Title: The emergence of sparse attention: impact of data distribution and benefits of repetition
Venue: NeurIPS 2025
Accepted: yes
Keywords: emergence

sparse attention

in-context learning

induction head
Abstract: Emergence is a fascinating property of large language models and neural networks more broadly: as models scale and train for longer, they sometimes develop new abilities in sudden ways. Despite initial studies, we still lack a comprehensive understanding of how and when these abilities emerge. To address this gap, we study the emergence over training of sparse attention, a critical and frequently observed attention pattern in Transformers. By combining theoretical analysis of a toy model with empirical observations on small Transformers trained on a linear regression variant, we uncover the mechanics driving sparse attention emergence and reveal that emergence timing follows power laws based on task structure, architecture, and optimizer choice. We additionally find that repetition can greatly speed up emergence. Finally, we confirm these results on a well-studied in-context associative recall task. Our findings provide a simple, theoretically grounded framework for understanding how data distributions and model design influence the learning dynamics behind one form of emergence.


Peer reviews:
comments: SUMMARY:
The paper investigates the emergence of sparse attention patterns in Transformer models during training, focusing on how data distribution and repetition influence this process. The authors combine theoretical analysis with empirical experiments to uncover the mechanisms behind sparse attention emergence and its impact on model performance.

STRENGTHS AND WEAKNESSES:
> **Strengths:**
>
> (1) Good visualizations.
>
> (2) They explore sparse attention as a lens to understand emergence during training.

> **Weaknesses:**
>
> (1) Theoretical analysis primarily focuses on the single-location linear regression task, which is insufficient for analyzing the training dynamics of models.
>
> (2) On line 87, it states 'the relevant token location is always the same.' This will lead to limited diversity in training tasks, creating a gap with large language models. Additionally, it will also restrict the scope of applicability for theoretical analysis.
>
> (3) Line 104 "The model’s parameters (a, W) are learned by minimizing the expected mean square error between predictions y and targets y*."  Lines 142-144 "attention layer allocates B times more weight to the relevant tokens compared to the case without any repetition".  So, in both scenarios, does the pair (w, a) end up learning the same thing in the end? Does "B times" imply that the value of 'a' can be different?
>
> (4) The authors need to more clearly present the corresponding evidence to support the claim made on Lines 185-187.
>
> (5) Lines 278-288, what are the differences and connections between "grokking" and "emergence"?

QUESTIONS:
Please see Weaknesses. My primary focus lies in points 1 through 5.

LIMITATIONS:
yes, some Limitations are discussed in Sec. Discussion.
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-24

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the emergence of sparse attention. They adopt three settings: single-location linear regression, in-context repetition, and cross-sample repetition. In the simplified model, they discover a power law of the plateau length corresponds to the sequence length T and hidden dimension d. They build a simplified theoretical framework to explain the dynamics of attention weight a and value matrix W, matching the observation. They further use the theory to explain the benefits of in-context repetition and cross-sample repetition. They also extend the results to realistic transformers and learning problems.\n\nSTRENGTHS AND WEAKNESSES:\nAlthough the plateau is repeatedly discovered in in-context learning literature, the plateau length is rarely discussed. The authors make very interesting findings on the power law of the plateau length. They also make novel simplification of the training dynamics and provide theoretical insights on the observed power law. The insights from analyzing in-context repetition and cross-sample repetition are also interesting.\n\nFor the weakness, my major concern is the limited implication of the benefits of in-context repetition and cross-sample repetition. I understand that these are interesting findings. But they are restricted in the given in-context learning setup. \n\n1. The more interesting direction of in-context repetition (bustiness) is its relation to in-weight learning. It is less surprising that more in-context repetition leads to shorter plateau length. \n2. I would imagine that the cross-sample repetition is only useful given the special design of the in-context learning task. The benefits of data repetition in training, or the longer training (grokking), seem to come from reasoning mechanism (for example, see Nanda 2023), but barely relates to in-context learning. The coding example given in the last paragraph of the paper seems to be more related to reasoning rather than in-context learning. I also find it hard to relate the discoveries in the paper to active learning. In general, I feel the discussion in section 4 should be more carefully phrased.\n\nNanda, Neel, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. 2023. “Progress Measures for Grokking via Mechanistic Interpretability.” ArXiv.org. 2023. https://arxiv.org/abs/2301.05217.\n\n‌\n\nQUESTIONS:\n1. What is the motivation of removing layer norm in the 2-layer 4-heads Transformer? \n\n2. Just curious, have you observed the linear growth of the norm of $W$ (the value matrix in attention layer) along the training dynamics in 2-layer 4-heads TF if having layer norm and using Adam?\n\nLIMITATIONS:\nSee my previous comment.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies training time emergence (sudden jump in performance during training after an extended plateau) for a simplified in-context learning (ICL) task involving single-location linear regression. The authors simplify the problem to coupled dynamics of 2 scalar variables corresponding to learning the attention map and linear projection. They further study the effect of repetition (both in-context and cross-sample) to analyze how this affects emergence during training. Results in the toy model setup are verified qualitatively in Transformer training on an in-context recall task.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- Paper studies relevant problem of training-time emergence connecting it to attention map sparsity, and concretely shows how the two are related.\n- Toy setup is well-formulated and somewhat represents realistic Transformer setups.\n- Paper is overall well written and easy to follow.\n\nWeaknesses\n\n- Not a major weakness, but the theory setup of single-location in-context regression might be a bit too simplistic for many tasks (even the experiments considered in the paper). While it does seem to predict some emergent phenomena quite well, overall I am not sure if just a linear layer with a softmax-vector as Attention layer would be able to capture a lot of nuances in realistic setups.\n\nQUESTIONS:\n1. I couldn't follow why “in-context repetition makes the attention pattern to be learned less sparse, thereby simplifying the task” (line 138) - could the authors clarify how making the pattern less sparse simplifies the task? (I understood how having in-context repetition \"allocates B times more weight to the relevant tokens\" in Line 143, but not how that simplifies the task itself)\n\n2. Just to clarify, is Figure 1 (right) and 2 numerical simulation for the toy model? If so, is the takeaway from this plot that linearizing early-time dynamics is OK for estimating plateau length?\n\n3. Line 117-118: “in these two equations, the first term is the relevant signal and the second term is the noise coming from non-relevant tokens.” - I couldn't follow this argument, could the authors clarify?\n\n4. Line 189-190: “Removing layer normalization, which made solving the task more challenging” - is there a reason / hypothesis why this is true for this specific setup? \n\n5. In-context associative recall task: Does induction head formulation map in any way to the toy model discussed in Eq (2)?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the emergence of “sparse attention” using a toy setup of training a small Transformer on a synthetic linear‐regression task. They show that the number of training steps to emergence scales with model size and sequence length, matching their theoretical derivation. They also find that repeating examples in the training data accelerates the emergence. Finally, they validate these findings on a more realistic associative‐recall task, confirming that sparse‐attention emergence governs performance there as well.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The paper provides a detailed theoretical analysis of sparse‐attention emergence in a toy linear‐regression setup and corresponding simplified Transformer model. It then extends this analysis to a more realistic associative‐recall task, showing that key observations generalize.\n\n2. The authors skillfully bridge mechanistic interpretability and theory, moving from gradient‐based proofs to empirical validation.\n\n3. This work offers a novel proof of concept for how mechanistic insights can inform realistic training‐data selection.\n\n4. The experimental setup is clear and reproducible: the authors thoroughly explore different model sizes, sequence lengths, and optimization methods.\n\nWeaknesses\n\n1. Although the paper shows that data repetition accelerates sparse‐attention emergence, it only briefly discusses implications for tasks requiring diverse data. A deeper analysis of which tasks benefit most from repetition (e.g., those relying heavily on in‐context information) would strengthen the discussion.\n\n2. Some observations—such as the proposed power‐law scaling—do not hold consistently across the linear‐regression and associative‐recall setups. Since both tasks remain somewhat artificial, it is unclear how these insights will generalize to real‐world applications.\n\nQUESTIONS:\nQ: Do you observe any difference based on the position of the relevant token in the linear‐regression tasks? Would choosing later tokens help induce sparse attention more quickly?\n\nQ: It seems intuitive that repeating information in the context aids retrieval. However, that doesn’t necessarily require sparse attention. I’m confused about how repetition causally drives attention to become sparse—how are these two phenomena related?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}