Title: Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings
Venue: NeurIPS 2025
Accepted: yes
Keywords: Attention Sinks

Mechanistic Interpretability

Language Models
Abstract: Large language models (LLMs) often concentrate their attention on a few specific tokens referred to as *attention sinks*. Common examples include the first token, a prompt-independent sink, and punctuation tokens, which are prompt-dependent. While the tokens causing the sinks often lack direct semantic meaning, the presence of the sinks is critical for model performance, particularly under model compression and KV-caching. Despite their ubiquity, the function, semantic role, and origin of attention sinks—especially those beyond the first token—remain poorly understood. In this work, we conduct a comprehensive investigation demonstrating that attention sinks: *catch* a sequence of tokens, *tag* them using a common direction in embedding space, and *release* them back into the residual stream, where tokens are later retrieved based on the tags they have acquired. Probing experiments reveal these tags carry semantically meaningful information, such as the truth of a statement. These findings extend to reasoning models, where the mechanism spans more heads and explains greater variance in embeddings, or recent models with query-key normalization, where sinks remain just as prevalent. To encourage future theoretical analysis, we introduce a minimal problem which can be solved through the 'catch, tag, release' mechanism, and where it emerges through training.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper discusses a potential reason why \"attention sinks\" (i.e., attention patterns where all attention concentrates on the first token) are ubiquitous in large language models (LLMs). It introduces a linear-algebraic mechanism which attention sinks execute in LLMs, which is labeled as \"catch, tag, release\" --- each attention head experiencing an attention sink will endow the input sequence with a component in a distinctive direction in embedding space given by the span of the value states corresponding to the sink tokens. Empirical evidence on both completely synthetic data and more realistic data show that attention sinks introduce a tag, that these tags explain much of the empirical variance of the outputs of the heads, that they have interpretable semantic meaning in controlled synthetic setups, that they occur more in reasoning-distilled models than only-pretrained models, and that attention sinks occur even when QK normalization is applied.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The overall mechanism of each attention head to influence the output is clear and succinct.\n- The experimental setup using large models is reasonable for many of the tasks, and may be sufficient to demonstrate the claims (modulo some ambiguity, see below).\n- The synthetic experimental setup is unambiguous, the claim is cleanly proved, and the setting may be valuable for future work (potentially in a more complex form).\n\nWeaknesses:\n- The writing is not very clear; there is a large ambiguity in the text that seems to be glossed over. Namely: what is the role of multiple attention heads? In particular, for a given input sequence, many attention heads are sinks, not just one. How do the tags interact with each other? Is there constructive or destructive interference (or something more complicated), or are all the \"tags\" orthogonal? It is especially confusing since in Figure 2 the tokens _later in the network_ seem to adhere strongly to the tags produced by the single discussed head earlier in the network, whereas multiple heads surely were attention sinks. It would be great to clarify these as it would strengthen the core contribution of the work.\n- There is little-to-no discussion of applications to the practice of LLMs. The main contributions are in clarifying the different mechanisms proposed in prior work. Even though this work is obviously not applied in nature, it would be good to discuss potential applications of the discussion, e.g. for large-scale interpretability or model surgery. \n- The theoretical analysis of the completely synthetic setup is for a 2 layer iterated attention network with one attention head in each layer, not necessarily a transformer. This is understandable because analyzing transformer dynamics is very challenging. Still it would be better to discuss the role of the MLP for instance.\n\nQUESTIONS:\nSee above, but to summarize:\n- What is the role of multiple attention heads in practice and in the synthetic setups considered in the work?\n- What is the role of other non-attention components in the network (MLP, layer norms, etc) in this mechanism?\n- Are there any potential applications of this mechanism to the practice of LLMs (capabilities, interpretability, etc)?\n\nLIMITATIONS:\nYes (Appendix B and the checklist)", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the phenomenon of \"attention sinks\" in Large Language Models (LLMs), where attention is disproportionately focused on a few specific tokens (like the first token or punctuation). The authors propose and validate a novel functional mechanism they term \"catch, tag, release.\" This framework posits that attention sinks \"catch\" the attention of other tokens, \"tag\" them by adding a common directional vector to their embeddings, and then \"release\" these tagged embeddings back into the residual stream. This tag, as the authors demonstrate, carries semantically meaningful information that can be used by deeper layers of the model.\nThe paper provides extensive evidence for this mechanism through the following contributions:\n1.\tVisual (PCA) and quantitative (\"variance explained\") analyses across a wide range of model families (LLAMA, PHI, QWEN, etc.) confirm the existence and significance of this tagging behavior.\n2.\tClever probing experiments on a truth-falsity dataset show that these tags can encode high-level semantic information, such as the truth value of a statement, with very high accuracy (up to 99.5%).\n3.\tThe authors show that the mechanism is more pronounced in models fine-tuned for reasoning and is robust to architectural changes like Query-Key Normalization.\n4.\tA key contribution is a minimal, two-layer Transformer model designed for a sequence-averaging task. The authors prove theoretically (Theorem 7.1) that the 'catch, tag, release' mechanism is a valid solution and demonstrate that this exact mechanism emerges spontaneously through standard training.\nThe paper concludes that attention sinks are not an artifact or a bug, but a fundamental and functional mechanism for information propagation and labeling within LLMs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1.\tThe \"catch, tag, release\" analogy is intuitive, powerful, and provides a functional explanation for the widely observed attention sink phenomenon. It shifts the perspective from sinks being a \"bug\" to a \"feature.\"\n2.\tThe paper's claims are supported by comprehensive experiments that are both qualitatively illustrative and quantitatively robust.\n⦁\tThe PCA visualizations in Figure 2 provide a clear and convincing visual proof of the \"tagging\" process, showing token embeddings clustering after the attention layer output.\n⦁\tThe introduction of the \"variance explained by tags\" metric allows for a quantitative comparison across many different models, layers, and heads, firmly establishing the ubiquity of the mechanism.\n⦁\tThe probing experiment in Section 4 is strong. By showing that the \"tag\" component of an activation can classify a statement's truth value far better than the non-tag or even the full activation, it provides powerful evidence that these tags carry specific, high-level semantic content.\n3.\tA major strength is the theoretical analysis in Section 7. By constructing a minimal problem and proving that the 'catch, tag, release' mechanism is a valid solution (Theorem 7.1), the authors provide a solid theoretical footing for their empirical observations. The demonstration that this mechanism emerges naturally through optimization (Figure 7) is a crucial result, suggesting it is a plausible and efficient strategy learned by neural networks.\n4.\tThe paper is exceptionally well-written and organized. The introduction clearly lays out the existing theories and their shortcomings through four specific open questions (Q1-Q4). The contributions (A1-A4) directly address these questions, creating a tight and compelling narrative. The figures are clear, well-captioned, and effectively support the paper's arguments.\nWeaknesses\n1.\tLimited Scope of Semantic Investigation: The paper brilliantly demonstrates that tags can encode truth-values. However, it leaves open the question of the full range of information these tags can carry. It would be interesting to see if this mechanism is also used for lower-level syntactic roles, coreference resolution, or other types of semantic relations.\n2.\tSimplicity of the Theoretical Model: As the authors commendably acknowledge in their limitations (Appendix B), the theoretical model is a highly simplified two-layer Transformer operating on 2D embeddings. While perfect for an illustrative proof-of-concept, it is a significant simplification of modern, deep, high-dimensional LLMs. The connection between the clean, emergent mechanism in this toy model and the potentially messier version in practice could be explored further.\n\nQUESTIONS:\n⦁\tThe probing experiments in Section 4 are very convincing. Are there other kinds of semantic or syntactic information encoded by these tags? For instance, in narrative text, could tags encode sentiment, character viewpoints, or tense, thereby \"tagging\" clauses for downstream processing?\n⦁\tThe \"Release\" step is described as the tagged tokens propagating through the residual stream. Are there specific types of attention heads or MLP neurons in subsequent layers that specialize in processing tokens based on the direction of these tags?\n⦁\tIn the theoretical model (Section 7), the \"tag\" and \"number\" components of the embedding are cleanly separated into two dimensions. In real-world models, these are obviously entangled within a high-dimensional space. The probing results suggest they are still highly separable. How \"clean\" is this separation in practice? Does the tag component have minimal projection onto the token's original semantic direction?\n⦁\tIn Section 3.1, the threshold ϵ for identifying an attention sink is set to 0.2. How sensitive are the quantitative results, such as the average number of sinks (Figure 3a) and the variance explained (Figure 3b), to the choice of this hyperparameter?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies attention sinks in large language models, which are tokens that receive a lot of attention, like the first token or punctuation marks. The authors argue that these sinks catch attention, add shared information to other tokens (tag), and release that information into the model’s residual stream. They provide visual and quantitative evidence for this mechanism in several popular models, show that the tags carry semantic meaning (like true/false labels), and observe that the effect is stronger in models trained for reasoning. They also introduce a small theoretical model where this behavior emerges naturally.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- **(S1)** Novel unified concept: the “catch, tag, release” mechanism offers a clear and intuitive interpretation of attention sink behavior. This concept unifies several previously scattered observations in the literature.\n- **(S2)** Clarity and accessibility: the paper is well-written, with clear diagrams and good structure. The visualizations are intuitive and support the narrative effectively.\n- **(S3)** Broad empirical scope: the authors examine the phenomenon across multiple model families (QWEN, LLAMA, Mistral, PHI), architectures (with and without QK norm), and tasks (reasoning vs. non-reasoning). This improves the generality of the claims. The experiments showing that sink-derived tags encode the truth value of statements are also a strong contribution.\n\nWeaknesses:\n- **(W1)** Lack of analysis on which tokens become attention sinks: while the paper compares the number of sinks between models (e.g., reasoning vs. pretrained), it does not analyze the types of tokens that serve as sinks. The discussion and analysis of attention sinks types could be very beneficial and provide more intuitive insights\n- **(W2)** Arbitrary sink detection metric: the definition of an attention sink (Eq. 1) relies on a fixed attention threshold, but this choice is not validated. There’s no sensitivity analysis and comparison to alternative metrics (e.g., attention entropy?). This undermines the robustness of the conclusions.\n\nQUESTIONS:\nMain Questions:\n- **(Q1)** Could you examine which tokens become sinks (e.g., punctuation, logical connectors), whether they equally contribute to explained variance, and whether this changes after reasoning fine-tuning? Is it possible to create a taxonomy, possibly using bigger and different dataset? This would clarify whether the additional sinks in reasoning models are semantically meaningful.\n- **(Q2)** The threshold-based sink definition is arbitrary, based on previous studies. Could you provide some intuition why this metric is indeed select all attention sinks (20% of explained variance in some cases is still low)?  Maybe you could provide some sort of sensitivity analysis over threshold and compare to alternatives (e.g., entropy or KL divergence) to ensure robustness? \n\nMinor Questions and Comments:\n- I have some doubts about value for activation probe for QWEN 2.5 3B in Table 1. How could it be that accuracy is still 50% after adding tags? \n- Though one can understand it intuitively, $\\theta_{activation}$ is not defined. As I believe you perform computations similar to tag/no tag, but it would be better to state it in the text\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the phenomenon of \"attention sinks\" in large language models, proposing that these sinks implement a \"catch, tag, release\" mechanism. In this process, specific tokens act as sinks to \"catch\" the attention of other tokens, \"tag\" them by adding their value vectors to the embeddings, and \"release\" this new information into the model's residual stream for use in subsequent layers. \nThrough probing experiments, they demonstrate that the tags carry semantically meaningful information, such as the truth-value of a statement. The study finds that this mechanism is more prominent in models fine-tuned for reasoning than in their pretrained counterparts. It also shows that attention sinks persist even in models with query-key (QK) normalization, an architecture that normalizes the query and key activations and may potentially suppress attention sinks. To ground these empirical findings, the paper introduces a minimal theoretical problem and provides a constructive proof showing that a two-layer transformer can solve it by explicitly implementing the \"catch, tag, release\" mechanism.\n\nSTRENGTHS AND WEAKNESSES:\nStrenghts\n- The paper introduces a novel \"catch, tag, release\" mechanism to explain the function of attention sinks in LLMs. This work contributes to a deeper understanding of LLMs' behavior, enhancing the interpretability.\n\n- The proposed mechanism is supported by a comprehensive empirical investigation. The authors validate the \"catch, tag, release\" mechanism across various LLM families and sizes (including Qwen, Phi, and Llama), demonstrating the generality of their findings. The claims are complemented by probing experiments that convincingly show that the tags generated by attention sinks encode semantically meaningful information.\n\nWeaknesses\n- While the core \"catch, tag, release\" mechanism is well-supported, the analysis for two of the claims (A3 and A4) seems preliminary and less developed. \n   - Claim A3 (Section 5): The paper shows that attention sinks and the proposed mechanism are more prevalent in reasoning models than in their pretrained counterparts. However, the authors do not go further to explain how this increased prevalence contributes to the models' reasoning abilities. Without this deeper analysis, the claim remains an observation.\n\n  - Claim A4 (Section 6): The analysis of models with QK normalization is limited to showing that the number of attention sinks does not decrease without further discussion. Moreover, the paper does not verify whether the \"catch, tag, release\" mechanism is still present in these models. This omission makes the conclusion in this section less impactful.\n\nOverall, while the core contributions of the paper are strong, the analysis in Section 5 and 6 is insufficient. These sections would need more evidence and discussion to be considered on par with the paper's primary findings.\n\nQUESTIONS:\n- The paper states in Section 7.1 that the theoretical model uses causal attention. However, the proof in Appendix C.1 appears to contradict this. Specifically, in line 580, the equation for the attention weight $A_{i,t}^1$. (for query $i$ attending to key $t$) defines the denominator as a sum over all keys from $k = 1$ to $T$. For causal attention, this sum should only be over keys from $k = 1$ to $i$.\n\n- I notice there is a discrepancy between theoretical constructions and empirical training results. The theorem and its proof rely on the assumption that the tag value $s_{tag} \\to \\infty$. However, in my attempt to reproduce the results using the provided code, I observed that the learnable parameter $s_{tag}$, initialized at $10$ as per Appendix D, does not grow large and instead decreases slightly to around $9.3$ during training. Furthermore, I was unable to reproduce the clear averaging behavior shown in Figure 7(c).\n\n- Minor: there is a typo in line 622 as the initialization value should be $10$ instead of $-10$ according to the codes.\n\nLIMITATIONS:\nYes, the authors address some limitations in Appendix B regarding the theoretical results in Section 7, but the discussion could be more comprehensive by also including the limitations of the empirical analyses in Sections 5 and 6.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}