Title: What are you sinking? A geometric approach on attention sink
Venue: NeurIPS 2025
Accepted: yes
Keywords: Attention sink

Transformer geometry

Position encoding

Self-attention
Abstract: Attention sink (AS) is a consistent pattern in transformer attention maps where certain tokens (often special tokens or positional anchors) disproportionately attract attention from other tokens. We show that in transformers, AS is not an architectural artifact, but it is the manifestation of a fundamental geometric principle: the establishment of reference frames that anchor representational spaces. We analyze several architectures and identify three distinct reference frame types, centralized, distributed, and bidirectional, that correlate with the attention sink phenomenon. We show that they emerge during the earliest stages of training as optimal solutions to the problem of establishing stable coordinate systems in high-dimensional spaces. We show the influence of architecture components, particularly position encoding implementations, on the specific type of reference frame. This perspective transforms our understanding of transformer attention mechanisms and provides insights for both architecture design and the relationship with AS.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a framework for studying *attention sinks*, and often discussed but rarely\nformally defined and studied property in attention-based models in which tokens corresponding to\nseemingly neutral or irrelevant information attract disproportionate attention.\n\nIt studies reasons why sinks emerge, formally defines sink tokens, and classifies different\nbehaviors in \"reference frames\", which are claimed to occur as a result of learning\nhigh-dimensionality latent spaces. Choices in the architecture such as the use of positional\ninformation, and the specific type of positional info result in different reference frame behaviors\n(centralized, bi-directional, distributed).\n\nThis framework and these findings are a great first step towards understanding attention sinks, and\nto my knowledge have rarely been investigated in prior works. Sinks present a wide range of\nopportunities and challenges in foundational models, from potential pruning methods for efficiency,\nto efforts in robustness and more effective quantization.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. Formal definition of attention sinks, with reasonable arguments as to why they occur. This has\n  been a rarity in papers discussing sinks directly in my experience.\n\n2. Investigation of the effects of different types of positional information / embeddings on the\n  types of reference frames and therefore sink behavior.\n\n3. Well-written and clearly organized, and relatively easy to ready and understand.\n\n### Weaknesses\n1. The major limitation in my opinion is a lack of background references. It would certainly be\n  helpful to readers in other research areas (including myself) to read prior works.\n  Geometric properties such as reference frames could use more extensive definitions, or references\n  for those unfamiliar (again including myself).\n\n2. As pointed out in the paper as well, the study is limited to only certain checkpoints of the\nmodels being studied. It would be interesting to see this study conducted throughout training. It\nwould also be interesting to find out whether much smaller models trained with limited resources\nexhibit similar sink behavior, and that way the study could be done more efficiently.\n\n3. (Minor) Experiments are limited to various LLMs, and not other foundation models in the visual\n  or audio space. While I think having more breadth would certainly help, I don't think it's a\n  necessity. More specifically, I think the findings affect those other applications as well, and\n  not just LLMs, so if at all possible, I recommend looking into other foundation models beyond\n  language.\n\n\n### Minor Notes:\n\n1. Multiple citations appear to be out of date or referencing the preprint instead.\n  Moschella et al. \"Relative representations enable zero-shot latent space communication.\" was published in ICLR 2023.\n  Xiao et al. \"Efficient streaming language models with attention sinks.\" was published in ICLR 2024.\n\n2. Other works prior to Xiao et al. (2023) have also discussed the behavior of sinks without using\n  the term:\n  * Clark et al. \"What does BERT look at? An Analysis of BERT’s Attention.\": https://aclanthology.org/W19-4828.pdf.\n  * Kovaleva et al. \"Revealing the Dark Secrets of BERT.\": https://arxiv.org/abs/1908.08593.\n\n3. In Figure 1, could you change the color or just the visualization of connections for Centralized\n  and Distributed, since they are not layer-specialized? It can be confusing as the straight line is\n  labeled \"Early layers connection\" in the legend. Or is that intentional?\n\nQUESTIONS:\n1. Section 3.3 suggests that the emergence of sinks is directly related to positional embeddings /\ninformation. Does that mean that models without positional biases / embeddings do not form sinks?\nAppendix A suggests that the softmax probability distribution mapping is one of the factors behind\nforming of sinks, so perhaps positional information are what bias the emergence of sinks to earlier\ntokens?\nCould we say that sinks emerge due to softmax and positional information? This is a more interesting\nfinding (assuming it's true), and I think it belongs in the abstract, where it currently says sinks\nare not an architectural phenomena (more on this in the next question).\n\n2. There are many instances in the paper that claim sinks are not an architectural phenomena.\nHowever, given that it only occurs in attention, I could argue that only architectures with\nattention (primarily transformers) exhibit this behavior. This is also discussed in the paper. For\nexample lines 27-30:\n\n> While prior work [...] treated attention sinks as model-specific phenomena, our geometric\n> interpretation unifies these observations as alternative solutions to the same fundamental\n> challenge, advancing understanding of transformer geometry.\n\nIn the end, it's all about transformers / NNs with attention. Therefore I recommend changing the\nlanguage to clarify this.\n\n\n3. I may be misunderstanding this, so maybe you can point me in the right direction. Line 56 says\nthat the study by Moschella et al. shows that NNs with high dimensional latent spaces naturally\nestablish reference frames. However, after going over the paper, based on what I understand they're\nproposing establishing a data-dependent coordinate system as to make it easier to compare latent\nspaces, among other things.\nCan you clarify this? If possible point out where Moschella et al. discusses the emergence of\nreference frames?\n\n4. Line 17:\n> Removing this pattern degrades model performance\n\nCould you please add references? I'm personally not convinced that eliminating sinks is necessarily\nbad for performance, as there are many different approaches that eliminate (or mitigate) the effect.\nOf course it's obvious that eliminating the tokens entirely post-training will affect quality, but\nit's not clear to me why designing a reasonable architecture that is not prone to this would be\ninferior in performance.\n\n5. I have a few issues with the definition of sinks (Sec. 3, Eq. 1). I understand that this may be a\nwidely accepted definition, but I still take issue with it, because it is too dependent on those\nthresholds, specifically $\\tau$. Why is the value of the 90th percentile a good choice? Is there a\nstudy on false positives?\nIf sinks are considered \"context tokens that attract disproportionate attention from most or all\nquery tokens\", then would it not make more sense to rank the attention weights for a single query\ntoken, and instead look for context tokens that appear in the top P%?\n\nI don't think this paper needs to discuss this, but I think it might be useful to point out that\nthe definition of sinks in Eq. 1 may not be the only one or the most reliable one, or if you believe\nit is, it would help to cite prior works using this definition or briefly discuss why.\n\n6. Can you comment on approaches such as Softpick or other similar modifications to the softmax\nfunction and what could be their effect on the emergence of sinks?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

comments: SUMMARY:
This paper studies transformer "attention sinks" as geometric reference frames that establish coordinate systems in representation space. The authors identify three types: centralized (single dominant anchor), distributed (multiple weaker anchors), and bidirectional (dual anchors), each determined by position encoding choices. Using topological, spectral, and information-theoretic analysis across multiple transformer architectures, they show these reference frames emerge naturally during training as optimal solutions to geometric constraints, serving as essential infrastructure for stable token relationships rather than inefficiencies.

STRENGTHS AND WEAKNESSES:
Strength

I'm not an expert on attention sink and related topics, but I found this explanation very interesting and novel. Understanding the attention in LLM could be very important to the field. 

The paper is clear about the assumption, experiment and analysis, making it clear about the general idea without knowing the theoretical details. 

In the experiment, the authors run a comprehensive cross-architecture analysis. The breadth of architectures tested demonstrates generalizability across different model families and scales.

Weakness

While the paper shows strong correlations between position encoding types and reference frame emergence, the authors don't perform controlled experiments changing only position encoding while keeping other architectural elements constant to definitively prove causation. Also, it lacks a fundamental theoretical explanation for why these specific patterns emerge.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a geometric interpretation of the attention sink phenomenon in transformer models, arguing that attention sinks are not merely architectural artifacts but manifestations of fundamental geometric principles related to the establishment of reference frames in high-dimensional spaces. The authors identify three distinct types of reference frames—centralized, distributed, and bidirectional—that correlate with the attention sink phenomenon and emerge as optimal solutions for establishing stable coordinate systems during training. They analyze how architectural components, particularly position encoding implementations, influence the specific type of reference frame that emerges. The paper's contributions include unifying diverse attention patterns across architectures under a single geometric principle, providing insights into how architectural choices affect attention organization, and laying the foundation for deliberate reference frame engineering in transformer models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The submission is technically robust and thorough. It provides a well-founded theoretical explanation for attention sinks, rooted in geometry and information theory. \n2. The paper’s empirical analysis is impressively comprehensive: it spans multiple model families and sizes, and uses complementary analysis techniques to validate each claim. \n3. The submission is generally well-written and organized. \n4. This work advances the mechanistic understanding of transformers by reframing attention sinks as necessary geometric structures rather than model artifacts.\n\nWeaknesses:\n1. One minor concern is that some of the analysis, while enlightening, is quite complex (e.g. persistent homology on attention graphs, information geometry formulations). This raises the question of whether all these analyses were necessary to support the core claims or if a subset would suffice – however, the inclusion of multiple perspectives does strengthen confidence in the findings.\n2. Although the analysis covers various transformer models and scales (up to ~12B parameters), generalizability to significantly larger models (e.g., 70B+ parameters) remains unclear. Clarifying theoretical expectations or providing additional empirical insights regarding larger-scale models would further strengthen the paper’s impact.\n3. The contribution primarily centers on theoretical understanding and conceptual insights rather than immediate practical improvements to transformer architectures or their performance metrics.\n4. The paper’s density of content is quite high. Readers who are not familiar with certain fields (e.g., topological data analysis or information geometry) might find parts of the presentation difficult to follow. A bit more gentle explanation or an appendix tutorial could help a broader audience.\n\nQUESTIONS:\n1. You have provided extensive analysis on several model families (LLaMA, Qwen, BERT variants, etc.) at various parameter scales. However, most analyses focused on models up to ~12B parameters. Could you discuss whether and how your findings (e.g., centralized, distributed, bidirectional reference frames) scale to significantly larger models (70B+ parameters, or GPT-class models)?\n2. Your analysis provides strong theoretical and empirical insights into attention sinks. Can you elaborate more concretely on how these insights might directly inform architectural choices or training strategies in practice?\n3. Your paper utilizes advanced methodologies (persistent homology, spectral graph theory, information geometry) extensively. Could you clarify why each analysis was necessary to support your core conclusions? Could any of these analyses be considered redundant or confirmatory?\n4. You briefly suggest in your conclusion the intriguing idea of \"reference frame engineering\" using attention sinks. Can you expand more explicitly on potential methods or experiments for deliberately manipulating attention sinks to achieve desired model behaviors?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a geometric interpretation for attention sinks in transformers, proposing that these sinks emerge as optimal 'reference frames' under softmax constraints in an emergent coordinate system. Crucially, they emphasize that these are not accidental artifacts but fundamental outcomes of geometric constraints on the learning dynamics of transformers. The authors identify three reference-frame types—centralized, distributed, and bidirectional—linked to inductive biases from positional-encoding schemes, and validate these patterns across 10+ model checkpoints (e.g. LLaMA, Qwen, BERT) using topological, spectral, and information-theoretic analyses\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths \n\n- The geometric analysis of attention sinks in transformer models presents a novel framing of these sinks as sites for an emergent coordinate system. Providing insight into what is going on at these tokens and why they emerge.\n- The experimental setup used is thorough, while the domain of sentences used to is somewhat narrow (i.e. just *stem-oriented* sentences), multiple architectures and models are tested in a comprehensive manner.\n- The conceptual framing of the paper is well presented (Sections 1-3) and the optimality perspective is interesting.\n- The authors bring to bear a rich set of analytic tools such as persistent homology, spectral graph metrics, KL/Fisher information to investigate their hypotheses.  \n\n\n### Weaknesses\n- **Sections 4-5 seems to be overly terse and a reads a bit like an information dump.** The authors introduce numerous analytic tools (persistent homology, Betti numbers, Ripser algorithm, persistence values, spectral properties, algebraic connectivity, star likeness, Gini coefficient, degree centralization, information theoretic measures, Fisher information matrix etc.) in a short span without guiding the reader to what question each tool is geared towards answering. It might be more helpful if these were presented in sections answering particular research questions / showing evidence for particular hypotheses, or at least some schematic of tool to information inferred.\n- **Metric values are not contextualized for readers:** in the various result tables there are many different metrics being presented without much contextualization of why these particular numbers imply the conclusions drawn. For example, Table 1 reports Betti₁ values but does not explain what constitutes a “high” vs. “low” Betti₁ or why changes across layers are meaningful.\n- **Some simple baseline analyses are missing:** \n\t- While it is natural that some background in the subfields used for analysis would be needed to fully understand the results, it would be helpful to start with some simpler statistical analyses that demonstrate evidence for the reference types found and then extend to the more in-depth analytic methods (potentially leaving some entirely for the appendix given the space constraints).\n\t- For example an initial visualization/table of the distribution of attention with respect to the sinks for each reference frame type across all the sentences tested (e.g. percentage of attention on sinks in specialized and possibly non-specialized heads, number of sinks per sentence). This is summarized in the \"Token specialization\" line in Table 1. But I would have expected to have seen that broken out in a bit more detail as initial evidence for the attention patterns described in prior sections.\n\nQUESTIONS:\n### Questions or Suggestions\n- The authors state that reference frames (and the specific reference frame types they observe) are *optimal solutions* under the softmax constraint to \"establishing stable coordinate systems in high-dimensional spaces\". In what sense do they mean \"optimal\" here, optimal with respect to which exact objective? Could the authors elaborate a little bit on this.\n- Could you clarify the definition of a specialized head is in Table 1? I'm trying to relate the dataset size to these measures, e.g. does it have to meet the reference frame attention threshold for all samples in the dataset? \n- What precisely is \"Token Specialization\" in Table 1?\n\t- e.g. What does 100% on BOS token mean?  I assumed it meant that all specialized heads (ones that demonstrate attention sinks) only attend to the BOS. \n\t- What about the *65.4% on \",\"*, for the distributed frame? What do the other specialized heads attend to? Are these commas in the same position (absolute or relative--i.e. the first comma in the sentence) across all sentences, or different.\n- As mentioned in the weaknesses section, could you provide charts of the distribution of attention sink patterns for each reference frame type across  the sentences tested (e.g. what tokens end up as attention sinks across all 500 sentences and what percentage of specialized heads focus on them, how many attention sink tokens per sentence, how many specialized heads etc) . I'm particularly interested to see how distributed among different tokens the \"distributed reference frame\" really is.\n- How were the thresholds for what constitutes an attention sink (Section 3) set?\n\nLIMITATIONS:\nThe authors include a brief limitations section. An extra limitation that readers find helpful is that the KL analysis on ablated sink tokens may not translate to impact on downstream tasks.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}