Anchor-based Large Language Models (2024)

Abstract: Large language models (LLMs) predominantly employ decoder-only transformer architectures, necessitating the retention of keys/values information for historical tokens to provide contextual information and avoid redundant computation. However, the substantial size and parameter volume of these LLMs require massive GPU memory. This memory demand increases with the length of the input text, leading to an urgent need for more efficient methods of information storage and processing. This study introduces Anchor-based LLMs (AnLLMs), which utilize an innovative anchor-based self-attention network (AnSAN) and also an anchor-based inference strategy. This approach enables LLMs to compress sequence information into an anchor token, reducing the keys/values cache and enhancing inference efficiency. Experiments on question-answering benchmarks reveal that AnLLMs maintain similar accuracy levels while achieving up to 99% keys/values cache reduction and up to 3.5 times faster inference. Despite a minor compromise in accuracy, the substantial enhancements of AnLLMs employing the AnSAN technique in resource utilization and computational efficiency underscore their potential for practical LLM applications.1

heading: 1 Introduction
text: Large language models (LLMs) primarily utilize decoder-only transformer architectures, which necessitate caching keys/values information for historical tokens during the auto-regressive inference to supply contextual information and avoid redundant computation (Wei et al., 2022; Touvron et al., 2023a; OpenAI, 2023; Touvron et al., 2023b). However, due to their immense size and high parameter count, a considerable amount of GPU memory is
*Work was done when Jianhui Pang and Fanghua Ye were interning at Tencent AI Lab.
†Corresponding Authors. 1Our code and models are publicly available at: https:
//github.com/pangjh3/AnLLM.
required for loading. Furthermore, as the length of input text grows, storing keys/values caches requires more and more GPU memory, as evidenced in in-context learning, complex instructions, and extended conversations (Dong et al., 2022; Jiang et al., 2023; Wang et al., 2023a), which is not conducive to scenarios with limited computational resources. An alternative approach entails recalculating these extensive inputs, which, however, results in increased time overhead. Therefore, this study aims to reduce the storage demand for keys/values caches during the inference phase of LLMs, improving the memory efficiency and, consequently, accelerating the inference speed as well.
In a recent study, Wang et al. (2023a) demonstrate that label words in prefix demonstrations can act as anchors during inference, providing an effective context compression approach for improving inference efficiency in in-context learning. However, in practical applications, not all prefix inputs or demonstrations contain label words suitable for
4958
compressing information, making the reliance on label words a less universal approach for text information compression. Additionally, Pang et al. (2024b) observe that LLMs tend to attend to only a few, yet consistent, prefix tokens during inference. However, the specific tokens utilized are often unpredictable and uncontrollable. These observations raise an intriguing question: do natural language texts contain anchor points that compress the overall semantic information of sequences? In this context, previous studies on sequence embeddings have shown that the hidden state of a special token in neural network models can encapsulate semantic information (Baudiš et al., 2016; Devlin et al., 2018). Furthermore, contemporary LLMs typically utilize the causal self-attention mechanism during both training and inference phases (Touvron et al., 2023a,b), attending on each prior token. This suggests that the final token in a sequence may be better suited to serve as a natural information compression point compared to other tokens, as they cannot observe future tokens. Therefore, a methodical approach that identifies and exploits sequence anchor tokens in a dependable and controllable manner is essential for compressing sequence information, effectively reducing keys/values caches, and improving inference efficiency for LLMs.
To this end, we propose novel Anchor-based Large Language Models (AnLLMs), equipped with an innovative anchor-based self-attention network (AnSAN) and an anchor-based inference strategy. The AnSAN is designed to compel the models to compress sequence information into the anchor token (the last token in our implementation) during the training process, with the aid of anchor-based attention masks. During inference, the anchor-based inference strategy retains the keys/values caches of anchor tokens, which have aggregated the entire sequence information, and discards those of nonanchor tokens, thereby reducing memory demands. Specifically, the anchor-based attention masks for AnSAN serve two objectives: 1) to ensure anchor tokens attend exclusively to tokens within the same sequence, preventing attention to other sequences, and 2) to direct non-anchor tokens’ attention to previous sequence anchors, blocking the other nonanchor tokens from previous sequences. It is noteworthy that the technique of anchor-based attention bears similarities to the principles underlying sparse attention (Child et al., 2019). However, unlike the existing research that employs sparse attention to extend the context length of LLMs (Chen
et al., 2023; Ratner et al., 2023), our method focuses on continually pre-training the model to compress sequence information into the anchor token.
In our implementation, we utilize the publicly available RedPajama datasets (Computer, 2023) to continuously pre-train the open-source Llama2 models (Touvron et al., 2023b), resulting in AnLLMs that incorporate our proposed anchor-based attention mechanism. Experimental results on question answering benchmarks, as depicted in Figure 1, reveal that our method achieves up to a 99% reduction in keys/values caches and up to a 3.5-fold increase in inference acceleration ratios, while maintaining comparable accuracy to the original model. Despite a minor decrease in accuracy (within 1.5%), these findings underscore the significant improvements in computational efficiency and memory utilization offered by our method.

heading: 2 Related Work
text: Our research is inspired by the recent investigation into the understanding of in-context learning (ICL) within LLMs by Wang et al. (2023a). In their study, the authors delve into the underlying mechanisms of ICL, emphasizing the influence of label words in demonstration examples on information flow. They reveal that these label words serve as anchors, wherein semantic information converges into these anchors during inference, subsequently directing the LLMs’ final predictions. Motivated by their findings, our objective is to extend this feature to natural language modeling by guiding sequence information compression into manually designed anchor tokens, rather than solely relying on label words. This is crucial because natural language texts may not always contain an explicit label.
The most relevant method to our approach in the existing literature is the learning to compress prompts with gist tokens (Mu et al., 2023). Their approach centers around compressing task-specific prompts by fine-tuning the model using the proposed gist masking, thereby enforcing prompt compression. However, there are several crucial divergences between our study and theirs. Unlike their focus on compressing a task prompt, our objective lies in training the LLM to condense sequence information into the anchor tokens. Consequently, our approach can be universally applied to a range of tasks without requiring task-specific training, a feature not shared by gist tokens, as the anchor tokens are seamlessly incorporated into the model’s lan-
guage modeling. Furthermore, our anchor-based attention masks account for information compression within a sequence and information interaction between sequences, thus extending beyond the mere compression of task prompts.
On the other hand, FlashAttention (Dao et al., 2022) and PagedAttention (Kwon et al., 2023) both present memory-efficient attention mechanisms for LLMs. While they focus on optimizing attention computation and subdividing attention processing, our proposed method offers a distinct approach that specifically targets the compression of sequence information into anchor tokens, making it orthogonal to these existing works.

heading: 3 Anchor-based Large Language Models

heading: 3.1 Background
text: Transformers. LLMs are primarily realized as decoder-only transformers (Vaswani et al., 2017; Touvron et al., 2023a,b), incorporating an input embedding layer and multiple decoder layers. Each layer contains a self-attention network and a feedforward network with normalization modules. Crucially, causal attention masks are employed, allowing tokens to attend only to preceding ones.
Self-Attention Networks. Typically for decoderonly LLMs like Llama2 (Touvron et al., 2023b), self-attention networks (SANs) map queries Q, keys K, and values V into an output, as delineated in the following equations,
SAN(Q,K, V ) = Softmax(Q,K)V, (1)
Softmax(Q,K)i,j = Mi,jexp(QiK
T j )
ΣkMi,kexp(QiK T k )
, (2)
Mi,j = { 1, if i ≥ j 0, else , (3)
where M denotes an L × L masking matrix, facilitating the current i-th token to attend to only preceding tokens whilst disregarding subsequent tokens during the training and inference phases.
Keys/Values Caches. In the application of LLMs, the keys/values caches increase with lengthy prefix texts and continuously generated tokens during the inference phase, such as in question-answering (Saad-Falcon et al., 2023), text summarization (Basyal and Sanghvi, 2023), and machine translation (Pang et al., 2024b). The key and value matrices associated with tokens of prefix inputs are cached to avoid recomputation and expedite
subsequent token prediction (Radford et al., 2019). Additionally, the model generates the output tokenby-token in the real-time inference process, which requires more cache memory to store the newly generated sequence. Therefore, addressing the challenges arising from the ever-expanding texts is crucial for enhancing the efficiency of LLM inference.

heading: 3.2 Anchor-based Self-Attention Networks
text: Given an input text with n consecutive sequences, P = {S1, S2, ..., Sn}, their associated anchor tokens are the last tokens that represented as A = {a1, a2, ..., an}. The primary objective of AnSAN is to encapsulate the information of a sequence into its anchor token, with the anchor hidden states representing the comprehensive semantic information. In this manner, an AnLLM equipped with AnSAN generates subsequent tokens based on the keys/values caches of preceding tokens within the current sequence and the keys/values caches of anchor tokens from previous sequences.
Anchor-based Attention Masks. To accomplish this, we devise anchor-based attention masks, as illustrated in Figure 2. Assuming that the current token in the sequence is a non-anchor token, we allow attention towards previous non-anchor tokens within the same sequence and anchor tokens from preceding sequences, while blocking attention towards non-anchor tokens from previous sequences. This approach ensures that non-anchor tokens can only access information from anchor tokens in previous sequences and the current sequence’s information. Conversely, when the current token is an anchor token, which is the last token in the sequence, we exclusively permit its attention towards previous non-anchor tokens within the same sequence, blocking all other attention. This constraint forces the anchor token to aggregate information solely from its current sequence. Consequently, we replace Eq. (3) with anchor-based attention masks in Eq. (4) to determine the mask of the i-th token in the input text concerning the j-th token (assuming that the i-th token belongs to the k-th sequence).
Mi,j =    0, if ((wi, wj) /∈ A) ∧ (wj ∈ S≤k−1) 0, else if (wi ∈ A) ∧ (wj ∈ S≤k−1) 1, else if i ≥ j 0, else ,
(4)
where S≤k−1 represents previous k−1 sequences. The number 0 denotes blocking attention, whereas the number 1 indicates the opposite.
Anchor Token Selection. By implementing the AnSAN mechanism for training LLMs, we can compel the model to compress sequence information into the anchor token and generate new tokens based on the anchor token information from previous sequences and non-anchor token information from the current sequence.
The challenge now lies in selecting an appropriate anchor token. In our experiments, we propose two implementation methods: one using the endpoint as the anchor token, and the other appending a new token specifically as the anchor token.

heading: 3.3 Anchor-based Inference
text: By training the model to compress information into the anchor token of a natural language sequence, we can optimize the inference process by modifying the keys/values caching mechanism. Specifically, during inference, upon encountering an anchor token that condenses the comprehensive semantic information of preceding tokens in the current sequence, the model can reduce the keys/values caches by deleting the caches of nonanchor tokens within that sequence.
We introduce the inference method in Algorithm 1. The function “REDUCTION” in Line 1 is utilized to remove keys/values caches when the model processes prefix texts in Line 10 or generates an anchor token during the prediction of the next
Algorithm 1 Anchor-based Inference Require: Anchor-based LLM Θ, prefix text P with anchor
tokens, keys/values cache list C, predicted token wnew; Output: Generated text T ; 1: function REDUCTION(C) 2: j ← last anchor index in C; 3: C ← {c ∈ C | index(c) ≥ j or c is anchor}; 4: return C. 5: end function 6: Initialize T , C as empty lists; 7: M← GetMasks(P, C) using Eq. (4); 8: Update wnew, C using Forward(P ;M, C,Θ); 9: Append wnew to T ; 10: C ← Reduction(C); 11: while wnew is not [eos] do 12: M← GetMasks(wnew, C) using Eq. (4); 13: Update wnew, C using Forward(wnew;M, C,Θ); 14: Append wnew to T ; 15: if wnew is the anchor token then 16: C ← Reduction(C); 17: end if 18: end while 19: return T .
token in Line 16. This approach aims to reduce the keys/values caches for both prefix tokens and generated outputs during real-time inference.

heading: 4 Experimental Setups
text: In this section, we first detail AnLLM’s implementation, then outline the training procedure and model perplexity. Finally, we introduce the evaluation datasets and metrics.

heading: 4.1 Our Implementation
text: Llama2-7b (Touvron et al., 2023b) is adopted as the base model in our experiments, which is an open-source and English-centric LLM. In accordance with the principles outlined in Section 3, we present our implementations here. The crux is to identify which tokens in a sequence can be considered anchor tokens. In light of this, we describe two implementation strategies: one employs the endpoints directly, and the other involves appending a new token at the end of a sequence to serve as the anchor token. The details are as follows:
• AnLLM-EP. This approach uses punctuation marks within the sequence as anchor tokens. Punctuation marks, such as commas, periods, and question marks, are viewed as semantic boundaries within a sequence. As such, they can serve as anchor tokens in AnLLM. In our experiments of AnLLM-EP, we use the endpoint in English as the anchor tokens.
• AnLLM-AC. This strategy entails the introduction of a new token to act as the sequence anchor. In our implementation, we designate <AC> as the new token and initialize its embedding using the mean value of the embedding matrix. For training data, we use the sentence tokenizer from the NLTK package to split texts into sentences, appending <AC> at the end of each sentence as the anchor token.2 During inference, <AC> tokens can be flexibly added to the text based on user requirements, such as adding one anchor for each demonstration, allowing for flexible and controllable sequence compression.
2https://www.nltk.org/api/nltk.tokenize.punkt. html

heading: 4.2 Data and Training Procedure
text: Considering that AnLLMs are expected to predict subsequent tokens within the context of keys/values hidden states of anchor tokens, this presents a significant challenge for existing open-source LLMs. To this end, by substituting the self-attention networks with anchor-based self-attention networks as detailed in Section 3.2, we continually pre-train the Llama2 model using a publicly available corpus.
Data. We employ the RedPajama-Data-1TSample dataset (Computer, 2023) for the continuous pre-training purpose.3 This dataset comprises 850, 000 samples with approximately 1 billion tokens, which have been subjected to right truncation to fit the model context of 4, 096.
Training Procedure. We train each model via the next token prediction objective on the dataset for one epoch, with a batch size of 512. The learning rate is set to 0.00002 and constant after a linear warmup with 20 update steps. The AdamW (Loshchilov and Hutter, 2019) with β1 = 0.9 and β2 = 0.95 is adopted as the gradient backtrack propagation optimizer. All the training procedures are conducted with four 8× A100 GPU machines with 40GB GPU Memory.
Training Loss and Perplexity. The left-hand side of Figure 3 depicts the training loss associated with our models. The loss curves for AnLLM-EP and AnLLM-AC consistently decline to approximately 1.9, with AnLLM-AC achieving a lower loss. This observation suggests that continually pre-training an LLM using anchor-based attention
3https://huggingface.co/datasets/ togethercomputer/RedPajama-Data-1T-Sample
masks is indeed viable, enabling the LLM to effectively learn the process of compressing sequence information into anchor tokens.
The right-hand side of Figure 3 displays the perplexity evaluation of the models with varying context lengths. Full attention is utilized to assess the language modeling capabilities of all models. Following the settings of Chen et al. (2023), the perplexity is evaluated on the test samples of the Proof-Pile datasets (Rae et al., 2020). The results demonstrate that both AnLLM-EP and AnLLMAC models maintain a promising performance, exhibiting language modeling capacity comparable to the base model, Llama2-7B. Moreover, this finding suggests that AnLLMs are compatible with full attention, as indicated by minimal perplexity decline.

heading: 4.3 Evaluation
text: In our investigation, we employ a diverse collection of benchmarks with varying text lengths to evaluate our outcomes, including OpenBookQA (OBQA) (Mihaylov et al., 2018), WinoGrande (WG) (Sakaguchi et al., 2021), ARC-easy (ARC-e) and ARCchallenge (ARC-c) (Clark et al., 2018), PIQA (Bisk et al., 2020), HellaSwag (HS) (Zellers et al., 2019), SCIQ (Welbl et al., 2017), and BoolQ (Clark et al., 2019). These benchmarks provide a comprehensive evaluation of various aspects, including reasoning, comprehension, understanding of the physical world, and predicting future events. Importantly, they cover texts of varying lengths, facilitating a thorough assessment of our model’s performance across diverse tasks and text complexities, ranging from shorter input contexts in OBQA to longer texts in BoolQ. To measure the precision and efficiency of our models, we evaluate them across three dimensions using three distinct metrics for both zero-shot and five-shot settings. For AnLLMAC in the five-shot setting, we incorporate the anchor token <AC> at the end of each demonstration.
• Accuracy (Acc). This conventional metric is utilized to gauge the prediction accuracy of models. In accordance with previous studies (Gao et al., 2023), we choose the options with the highest probabilities as predictions and calculate accuracy using the gold-standard labels. • Keys/Values Caches Reduction (C⇓). In the context of the five-shot evaluation, the demonstrations can be cached in GPU memory for subsequent reuse. Nevertheless, extended demonstrations may require increased memory consump-
tion. This metric is designed to assess the memory efficiency of the AnSAN technique. • Inference Acceleration Ratio (T⇑). Similar to Wang et al. (2023a), capitalizing on the cached keys/values, we present the inference acceleration ratio, which serves as an indicator of the inference efficiency of the AnSAN technique.
Note that we first report full attention inference results for all models, then present results with the AnSAN method (+AnSAN) applied, compressing sequence information into anchor tokens.

heading: 5 Experimental Results
text: As evident from the results presented in Table 1, both the AnLLM-AC and AnLLM-EP models demonstrate promising accuracy, comparable to that of the base model, while simultaneously improving memory and inference efficiency.
Accuracy (Acc). The proposed AnLLM-EP and AnLLM-AC models exhibit commendable accuracy across various benchmarks.
In the zero-shot setting, with full attention, AnLLM-EP and AnLLM-AC achieve average accuracies of 64.6% and 65.1%, respectively, comparable to Llama2-7B’s 65.8% accuracy. This suggests that training with integrated anchor tokens barely affects the model capacity, emphasizing the robustness of LLMs. Furthermore, our models excel in OBQA, PIQA, and SCIQ tasks.
In the five-shot setting, with five prior examples, AnLLM-EP and AnLLM-AC maintain dependable performance using full attention. When implementing the AnSAN technique, a slight accuracy decline across all models is observed. This is expected, as AnSAN, designed for memory efficiency, necessitates token removal, potentially leading to information loss. The degradation in BoolQ is most pronounced, which contains the longest demonstration tasks, indicating that the longer the text, the greater the information loss after compression. However, the average accuracy reduction is minimal, approximately 1.5%, suggesting that AnSAN effectively balances memory-saving and model performance.
Keys/Values Cache Reduction (C⇓). The size of the keys/values cache is a critical factor in the practical implementation of LLMs, particularly concerning memory efficiency and computational resources. In this respect, the AnLLM-EP and AnLLM-AC models offer significant advantages.
OBQA WG ARC-e ARC-c PIQA HS SCIQ BoolQ AVG.
Llama2-7B 31.4 69.1 76.3 43.4 78.1 57.1 93.7 77.7 65.8 AnLLM-EP 33.2 68.0 73.4 40.8 77.8 55.0 94.4 74.4 64.6 AnLLM-AC 31.6 68.5 74.4 42.5 78.3 54.7 93.8 77.0 65.1
(a) The Zero-Shot Performance.
OBQA WG ARC-e ARC-c PIQA HS SCIQ BoolQ AVG.
Ld 89 133 145 209 262 426 603 804 334 Lx 18 26 36 42 42 90 130 169 69
By adopting the AnSAN, these models are designed to dramatically reduce the keys/values cache size during inference. As shown in Table 1, these models achieve remarkable reductions in cache size. Specifically, the average reduction percentages are around 90% for AnLLM-EP and an impressive 99% for AnLLM-AC. This is a substantial improvement compared to conventional approaches, which typically necessitate large cache sizes to store keys/values. These reductions in cache size translate to considerable savings in memory and computational resources, rendering these models highly efficient for practical applications.
Inference Acceleration Ratio (T⇑). The inference acceleration ratio serves as a crucial metric reflecting the model’s efficiency during the testing phase. By incorporating anchor tokens into natural language texts, we can repurpose the hidden states of anchor tokens as keys/values caches in the demonstrations, and then adopt an inference strategy as suggested by Wang et al. (2023a). In this
scenario, both the AnLLM-EP and AnLLM-AC models demonstrate significant improvements.
Specifically, in the five-shot testing, both AnLLM-EP and AnLLM-AC models attain an average inference acceleration ratio of approximately 1.7 times. This represents a considerable advancement over the conventional non-caching method, which typically necessitates prolonged processing times due to the large number of tokens involved. As Ld increases, reaching up to 3.5 times in the BoolQ task, the acceleration ratios also escalate, corroborating the findings of Wang et al. (2023a). This enhancement in processing speed leads to increased efficiency, making these models particularly apt for scenarios with limited resources.
The AnLLM-EP and AnLLM-AC models exhibit remarkable performance in natural language understanding benchmarks, effectively balancing accuracy, memory efficiency, and time acceleration. The incorporation of anchor tokens into AnLLMs, along with the utilization
of the AnSAN technique for reducing keys/values cache size, allows these models to maintain performance on par while significantly improving memory efficiency and inference speed. The equilibrium achieved between model performance and computational efficiency is noteworthy and opens up new possibilities for the advancement of LLMs.

heading: 6 Analysis
text: To further elucidate our method’s insights, we conduct a natural language generation experiment with the German-to-English (De2En) translation task. We evaluate the models using COMET-DA (Rei et al., 2022), indicating translation quality, and the Keys/Values Cache Reduction C⇓ metric, denoting memory efficiency as previously described. In line with previous findings, AnLLMs accept a minor accuracy trade-off (about 3 COMET-DA points) for enhanced memory efficiency. All LLMs are finetuned on the Alpaca dataset, combined with the newstest2017-2020 datasets, following Jiao et al. (2023). Results are presented in Table 2.

heading: 6.1 Compatibility and Flexibility of Full Attention and Anchor-based Attention
text: The results offer significant insight into the interplay between anchor-based attention and full attention mechanisms in the De2En translation task. Since source sentences are vital in translation tasks, applying full attention to them is crucial for maintaining model performance. Thus, retaining the source sentence keys/values caches is expected to enhance AnLLM performance when implementing the AnSAN technique. Specifically, when combining full attention with the AnSAN method, both AnLLM-EP and AnLLM-AC achieve approximately 80.0 COMET-DAE scores, comparable to other models using full attention exclusively. This indicates that the AnSAN technique is compatible with the full attention mechanism. Consequently, our proposed models allow users to choose between full attention and anchor-based attention for input texts based on their needs, emphasizing the compatibility and flexibility of our models.

heading: 6.2 Effective Cache Reduction for Real-Time Inference with the AnSAN Technique
text: The results in Table 2 show that our reduction strategy effectively minimizes keys/values caches during real-time inference. Specifically, as indicated in Line 15 of Algorithm 1, when generating an anchor token (i.e., the endpoint or <AC> tokens),
our AnSAN-equipped models execute the reduction function to minimize the current keys/values caches. When discarding source sentence caches, we achieve approximately 77% and 84% reduction for the AnLLM-EP and AnLLM-AC models, respectively, albeit with a low COMET-DA score. However, when retaining source sentence caches, we still reduce around 44% of caches for both models, achieving a COMET-DA score of approximately 80.0. These results confirm the effectiveness of our anchor-based inference strategy for practical real-time inference applications.

heading: 7 Ablation Studies

heading: 7.1 Impact of Anchor Positions
text: An intriguing question arises regarding the impact of anchor positions on model performance. In this section, we investigate the effects of varying anchor positions using the AnLLM-AC model, which enables us to modify the anchor position. Specifically, we employ the data settings from Section 3.2 and examine three position settings: the first compresses every 10 tokens, the second applies random compression, and the third compresses each demonstration, consistent with the setting in Table 1. For the second setting, an anchor token is randomly inserted after each token with a probability of 0.1. The experimental results are presented in Table 3.
Accordingly, we observe that the choice of anchor positions significantly affects the model’s performance across various question-answering benchmarks. The "every-demonstration" setting consistently outperforms the other two settings, achieving the highest average accuracy of 67.5%. This suggests that strategically placing anchors at se-
mantically meaningful positions, such as after each demonstration, can effectively enhance the model’s ability to capture and utilize the information contained in the input texts.
In comparison, the “every-10-tokens” and “random-prob-0.1” settings yield lower average accuracies of 57.6% and 57.8%, respectively. These results indicate that compressing input texts at fixed intervals or randomly inserting anchor tokens may not be as effective in facilitating the model’s understanding and reasoning processes. The suboptimal performance of these settings could be attributed to the potential loss of semantic coherence and structural information as a result of arbitrary compression or random anchor placement.
Overall, our ablation study highlights the importance of carefully selecting anchor positions in the AnLLM-AC model to maximize its performance on question-answering tasks. The superior performance of the "every-demonstration" setting demonstrates the benefit of aligning anchor positions with semantically meaningful boundaries in the input texts. Future research could explore more sophisticated strategies for anchor placement, taking into account the linguistic and contextual properties of the input data to further improve the model’s performance on complex reasoning tasks.

heading: 7.2 Training from Scratch
text: To evaluate the language modeling capabilities of our anchor-based language model (AnLLM-EP), we perform a comparison with the standard Transformer model. This comparison involves the training of compact models from scratch, using the Wikitext-103 dataset.4 Each model is configured with 18 layers, 4096 hidden states, and 16 heads. As shown in Figure 4, the AnLLM-EP model notably outperforms the standard Transformer model, achieving a lower perplexity of 32.81, compared to 36.57. This notable outcome suggests that the anchor-based training approach may enhance the
4https://huggingface.co/datasets/iohadrubin/ wikitext-103-raw-v1
effectiveness of language modeling tasks. In future research, it would be intriguing to further investigate the potential advantages of the anchor-based training strategy for training LLMs from scratch.

heading: 8 Conclusion
text: LLMs have emerged as a significant research area in the field of artificial intelligence. However, despite their exceptional performance across various natural language tasks, the practical application of these models is limited by their significant memory overhead and time efficiency. Implementing LLMs on resource-constrained devices, such as smartphones, poses a unique challenge. To address this issue, we propose anchor-based LLMs with the AnSAN technique. Our experiments demonstrate that by sacrificing a marginal 1.5% in precision, our approach saves 99% of keys/values cache memory while simultaneously improving inference speed by up to 3.5 times. Our methods’ application in machine translation showcases their compatibility and flexibility, effectively enhancing memory efficiency for practical use. Our novel approach is practical, straightforward, flexible, and compatible with existing methods, paving the way for further adoption of LLMs in real-world applications.
Limitations
While our proposed AnLLMs demonstrate significant improvements in memory efficiency and inference acceleration, there are several limitations to consider:
1. Accuracy Trade-off: As observed in the experimental results, our method incurs a minor decrease in accuracy (within 1.5%) compared to the original model. This limitation stems from the information compression process, which may lead to information loss. Despite its minimal impact, this trade-off should be considered in practical applications. In future works, additional evaluation methods could further enrich our assessment (Ye et al., 2024; Wang et al., 2023c).
2. Applicability to Various Tasks: Our experiments primarily focus on question-answering benchmarks and machine translation tasks. The effectiveness of our method in other NLP tasks and domains remains to be thoroughly investigated. Future work will explore the applicability and performance of our method across a broader range of tasks and domains (Zhao et al., 2023; Wang et al., 2023b; Fang et al., 2023; Pang et al., 2024a; Wang et al., 2024; Lan et al., 2024; Zhan et al., 2024).
3. Optimal Anchor Token Selection: In our implementation, we chose the last token in a sequence as the anchor token. However, the optimal anchor token selection may vary across different tasks and domains. We encourage further analytical studies to explore the selection of anchor tokens.
4. Scalability to Other LLMs: We have applied our method to the open-source Llama2 models. It remains to be seen how our approach would perform when applied to other opensource LLMs, such as Falcon and Qwen (Almazrouei et al., 2023; Bai et al., 2023). Evaluating the effectiveness and scalability of our method on more extensive language models is an essential direction for future research.
Despite these limitations, our work presents a novel approach to enhancing memory efficiency and inference acceleration in LLMs. Future research efforts should address these limitations, refining our method and extending its applicability to a wider range of tasks and model architectures.
Ethics Statement
We place great importance on ethical considerations and rigorously adhere to the ACL Ethics Policy. In this paper, we propose an anchor-based large language model that reduces the Keys/Values cache size and enhances inference speed during the inference stage. The resources and methods employed in this paper are publicly accessible and have been extensively adopted by researchers in the field of large language models. We ensure that the findings and conclusions presented in this paper are reported accurately and objectively.

heading: Acknowledgments
text: This work was supported in part by the Science and Technology Development Fund, Macau SAR (Grant No. FDCT/0070/2022/AMJ, the mainland China collaboration project, China Strategic Scientific and Technological Innovation Cooperation Project Grant No. 2022YFE0204900), the Science and Technology Development Fund, Macau SAR (Grant No. FDCT/060/2022/AFJ, the mainland China collaboration project, National Natural Science Foundation of China Grant No. 62261160648), the Multi-year Research Grant from the University of Macau (Grant No. MYRGGRG2023-00006-FST-UMDF), and the Tencent AI Lab Rhino-Bird Gift Fund (Grant No. EF202300151-FST). This work was performed in part at SICC which is supported by SKL-IOTSC, and HPCC supported by ICTO of the University of Macau. We would like to thank the anonymous reviewers for their insightful comments.

heading: A More Experimental Results
text: A.1 Detailed Perpelixity Evaluation
In this section, we present a comprehensive analysis of the perplexity evaluation results, as illustrated in Table 4. The perplexity scores were calculated for various evaluation context lengths, ranging from 256 to 4096 tokens, utilizing the ProofPile datasets (Rae et al., 2020). The table compares the performance of Llama2-7B, AnLLM-AC, AnLLM-EP, and their corresponding variants incorporating the AnSAN technique. Our findings reveal that the AnSAN technique, on average, leads to a one-point increase in perplexity, which negatively impacts the modeling capabilities of the models to some extent. These outcomes resonate with the trade-off between accuracy and efficiency observed in Table 1.
A.2 Testing Acceleration Ratio to Full-Caching Method
In Section 5, we report the testing acceleration ratio following the setting of Wang et al. (2023a), comparing the time difference between caching and non-caching inference. Although our method reduces the keys/values caches, enabling smaller space for prefix information storage and improving testing time up to ×3.5, we are still curious about whether it would enhance inference efficiency if conventional methods use full caches that save all keys/values of prefix tokens. As a supplement to Table 1, we present the testing acceleration ratio between anchor-caching and full-caching inference in Table 5. The acceleration ratios for AnLLMEP-AnSAN and AnLLM-AC-AnSAN achieve the highest improvements observed in tasks such as HS, SCIQ, and BoolQ. The average acceleration ratios for AnLLM-EP-AnSAN and AnLLM-ACAnSAN are 1.03. These results demonstrate that our anchor-based caching method can enhance inference efficiency even when compared to conventional methods that save all keys/values of prefix tokens. These results suggest that our anchor-based caching approach, which saves only the keys/values caches of anchor tokens, can effectively accelerate the inference process for the lengthy prefix texts.
A.3 Model Scalability Assessment
To examine the scalability of our approach, we extend the AnLLM-AC model to 13B and assess its performance on eight question-answering benchmarks using the same evaluation strategy as previ-
ously mentioned. In comparison to the 7B AnLLM models in Table 1, Results in Table 6 indicate that as the model size expands, the AnLLM-AC model achieves accuracies of 67.5% and 70.0% for 0-shot and 5-shot testing, respectively, resulting in up to a 2.4% improvement. Moreover, by incorporating anchor-based attention, the AnLLM-AC-AnSAN model achieves an average accuracy of 69.5%, signifying a 2.0% increase. The performance enhancement underscores the effectiveness of our methods in accommodating larger model capacities. The consistent improvements observed in the AnLLMAC model across various scenarios highlight its robustness and adaptability. Furthermore, the increased performance of the AnLLM-AC-AnSAN model, facilitated by anchor-based attention, emphasizes the potential of our approaches in optimizing LLMs. Collectively, these findings point to promising avenues for future research aimed at maximizing the utility and efficiency of AnLLM.
A.4 Case Study in Real-Time Inference
To elaborate on the optimization of keys/values caches by AnLLM-EP and AnLLM-AC during real-time inference, we reference examples from the translation task in Section 6.2. As per Table 7, AnLLM-EP and AnLLM-AC use "endpoints" (".") and "<AC>" tokens as anchor tokens, respectively. During inference, both models employ auto-regressive generation, creating outputs token-by-token. Upon generating an anchor token (as per Line 16, Algorithm 1), the Reduction function (defined in Line 1) is activated, preserving relevant caches and eliminating others. As a result, the Keys/Values Cache lengths are reduced to roughly the sequence length, averaging around 50 for AnLLM-EP and 35 for AnLLM-AC, as shown in Table 2.
A.5 Attention Pattern in AnLLMs
Regarding the attention pattern, we have conducted a case study using the AnLLM-EP model. As shown in Figure 5, given the sentence “Apple is delicious. He goes to the market. He buys an apple.”, we detokenize and split it into two segments: “_Apple _is _del icious . _He _go _to _the _market . _He _b” and “ys _an _apple .”. By employing a heatmap to visualize the attention pattern between the latter segment and the former, we observe that the token “ys” attends more to the second endpoint, which compresses the information of “He goes to the market.”. This is a reasonable and interesting
finding, as “ys” is part of the word “buys”. Additionally, the token “_apple” attends more to the first endpoint, which compresses the information of “Apple is delicious.”. These attention patterns offer some interpretability for our method.

heading: B Data Settings
text: To provide a thorough insight into how we continually pre-train the model into AnLLM and carry out evaluations, we showcase some data examples in this section for both training and testing data.
B.1 Training Data Examples In this section, we provide examples to illustrate the specific data format used in training the AnLLM models. For the AnLLM-EP model, the endpoints act as anchor tokens, allowing us to directly utilize natural language texts. For the AnLLM-AC model, we append a new token <AC> at the end of each sequence in the input texts, which are initially split into sentences using the NLTK toolkits.5 Some examples are presented in Table 8. All the trainig data are downloaded from HuggingFace6, an opensource community.
B.2 Testing Data Examples For the testing outlined in the results section (Section 5), we employ the same evaluation method as in previous work (Gao et al., 2023), which treats each choice as text generation and computes the corresponding probabilities, respectively. Table 9 presents some evaluation examples.
5https://www.nltk.org/api/nltk.tokenize.punkt. html
6https://huggingface.co/datasets/ togethercomputer/RedPajama-Data-1T-Sample

heading: Future Work
text: Collectively, these findings point to promising avenues for future research aimed at maximizing the utility and efficiency of AnLLM. A. 4 Case Study in Real-Time Inference
To elaborate on the optimization of keys/values caches by AnLLM-EP and AnLLM-AC during real-time inference, we reference examples from the translation task in Section 6. 2. As per Table 7, AnLLM-EP and AnLLM-AC use "endpoints" (". ") and "<AC>" tokens as anchor tokens, respectively. During inference, both models employ auto-regressive generation, creating outputs token-by-token. Upon generating an anchor token (as per Line 16, Algorithm 1), the Reduction function (defined in Line 1) is activated, preserving relevant caches and eliminating others. As a result, the Keys/Values Cache lengths are reduced to roughly the sequence length, averaging around 50 for AnLLM-EP and 35 for AnLLM-AC, as shown in Table 2. A. 5 Attention Pattern in AnLLMs
Regarding the attention pattern, we have conducted a case study using the AnLLM-EP model. As shown in Figure 5, given the sentence “Apple is delicious. He goes to the market. He buys an apple. ”, we detokenize and split it into two segments: “_Apple _is _del icious. _He _go _to _the _market. _He _b” and “ys _an _apple. ”. By employing a heatmap to visualize the attention pattern between the latter segment and the former, we observe that the token “ys” attends more to the second endpoint, which compresses the information of “He goes to the market. ”. This is a reasonable and interesting
finding, as “ys” is part of the word “buys”. Additionally, the token “_apple” attends more to the first endpoint, which compresses the information of “Apple is delicious. ”. These attention patterns offer some interpretability for our method