Dynamics of Spontaneous Topic Changes in Next Token Prediction with Self-Attention
Authors: Mumin Jia
Jairo Diaz-Rodriguez
Venue: NeurIPS 2025 poster ()
Keywords: Self-attention
spontaneous topic changes
spontaneous human thought
next token prediction
Primary area: deep_learning

Abstract: Human cognition is punctuated by abrupt, spontaneous shifts between topics—driven by emotional, contextual, or associative cues—a phenomenon known as spontaneous thought in neuroscience. In contrast, self-attention-based models rely on structured patterns over their inputs to predict each next token, lacking spontaneity. Motivated by this distinction, we characterize *spontaneous topic changes* in self-attention architectures and reveal divergences from *spontaneous human thought*. First, we establish theoretical results under a simplified, single-layer self-attention model with suitable conditions by defining a topic as a set of Token Priority Graphs (TPGs). Specifically, we demonstrate that (1) the model maintains the priority order of tokens related to the input topic, (2) a spontaneous topic change can occur only if lower-priority tokens outnumber all higher-priority tokens of the input topic, and (3) unlike human cognition, the longer context length or the more ambiguous input topic does not increase the likelihood of spontaneous change. Second, we empirically validate that the effect of input length or topic ambiguity persists in modern, state-of-the-art LLMs, underscoring a fundamental disparity between human cognition and AI behavior in the context of spontaneous topic changes. To the best of our knowledge, no prior work has explored these questions with a focus so closely aligned to human thought.

Dynamics of Spontaneous Topic Changes in Next Token Prediction with Self-Attention
Mumin Jia Department of Mathematics and Statistics York University
Jairo Diaz-Rodriguez" Department of Mathematics and Statistics York University Toronto, Ontario M3J 1P3
Toronto, Ontario M3J 1P3 anyjik@yo
jdiazrodByorku.co ca
Abstract
Human cognition is punctuated by abrupt, spontaneous shifts between top- ies-driven by emotional, contextual, associative cues-a phenomenon known as spontaneous thought neuroscience. In contrast, self-attention- models rely on structured patterns over their inputs to predict each next token, lacking spontaneity. Motivated by this distinction, we characterize spontaneous topic changes in self-attention architectures and reveal divergences from spontaneous human thought. First, we establish theoretical results under a simplified, single- layer self celf-attention model with suitable conditions by defining a topic as a Token Priority Graphs (TPGs). Specifically, we demonstrate that (1) the model maintains the priority order of tokens related to the input topic, (2) spontaneous topic change can occur only if lower-priority tokens outnumber all higher-priority tokens of the input topic, and (3) unlike human cognition, the longer context length or the more ambiguous input topic does not increase the likelihood of spontaneous change. Second, we empirically validate that the effect input length or topic ambiguity persists in modern, -fate-of-the-art LLMs, underscoring fundamental disparity between human cognition and AI behavior the context of spontaneous topic changes. To the best of our knowledge, prior work has explored these questions with a focus so closely aligned to human thought.
Introduction
Human cognition is punctuated by abrupt, apparently unstructured topic changes, the hallmark of spontaneous human thought, phenomenon that has become central topic in cognitive neuroscience For example, spontaneous shift in focus during a conversation, sudden leap between ideas when brainstorming, or unexpected redirection in storytelling. These abrupt changes may be due to an emotional connection, such as recalling reading book during family vacation, where sensory details like the scent of the ocean the warmth of the sun trigger vivid memory. However, LLMS shift topics in response to contextual cues in the input, rather than initiating spontaneous topic changes their own. They follow a structured, statistical approach, remaining on topic unless explicit cues signal a change. Figure Iillustrates this distinction using the first sentence of the book "One hundred years of solitude"
8 9 23
32 34
Our work takes initial steps toward formalizing the dynamics of spontaneous topic changes in LLMs and analyzing how they relate to or diverge from spontaneous human thought. To this end, we ground our theoretical analysis in a single-layer self-attention model and empirically extend it to modern I.LMs, laying the groundwork for drawing comparisons between Al models and human cognition.
"Euual contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Pragment from "One hundred years of Solitude" (Gareia Minques, 1967,
input sequence
"Many years betee noc faoed the firing squueel (taionel Aurelians Boirdll west 60 momammber that distang afternoom when his fatherr Sook him to discover ind
Topic
One set TPGs
Dotnset A
Self-attention model A
Predicted next tuleen
Mixod-topie
Kespa topica
Belf-atturtion model (minsd-ltppc))
complicttin::
Datasot A and Untaset st
Predicted tollen
Ambigunus sequence
Many years luser as he faced the firing squad, he found hinsolll cablly calns, the wosght oft past no longer a burden but a story that had finally reoched its concuutoon
Changes topicA
The other set TPGs
+
Dataset B
Self-attencion model B
Figure 2: Overview of our theoretical framework. We define a topic as a set of TPGs [g( (Def. (2) and generate a dataset for each topic. The combination of dataset A and dataset becomes the dataset for the mixed-topic model. We train self-attention models independently on each dataset. Then, generate new input sequence from topic A and predict the next token with two models, self-attention model A and self-attention model (mixed-topie). The next-token prediction with the mixed- topic model is categorized into three outcomes: keeps topic A (topic continuity from Def. 3] ambiguous sequence (from Def. 4 or changes topic A (change of topic from Def. 5). Further details for each category are shown in Figure
Figure I: Illustration of the difference be- tween human cognition and LLMs. The original fragment of "One hun- dred years of solitude" I (top) has clear spontancous thought, but the GPT-2's completion (bottom), demonstrates continuity
Recent advancements in the related field have substantially deepened understanding of self- attention architectures. Li et al. (25. Tarzanagh et al. 48 have linked the self-attention to support vector machines (SVMs), offering optimization strategies for next-token prediction. Li et al. [26] highlight that in mixed-topic inputs, transformers achieve higher pairwise attention between same-topic words compared to different-topic words. In parallel, prior studies have recognized the practical challenges of spontaneous topic changes in LLMs and proposed approaches to address them 19 27 36 (5). Notably, spontaneous topic changes must be differentiated from hallucinations, generating incorrect or fabricated information without a clear contextual basis [20] 31.
Despite these advancements, our understanding of the dynamics of spontaneous topic changes in LLMS remains limited. Investigating the relationship between spontaneous topic changes in self- attention models and spontaneous human thought can provide valuable insights into the cognitive discrepancies of current language models compared with humans. Since modern LLMs rely on self-attention architectures, we begin by theoretically characterizing spontaneous topic changes in simplified setting. We then extend these findings through experiments on more complex, state-of- the-art models. To the best of our knowledge, no prior studies have investigated these dynamics so closely in relation to human thought.
Figure 2 outlines our theoretical framework. To make the mathematical analysis tractable, follow the same single-layer self-attention framework with log-loss objective function governed by Assumptions
1 from Li al. 225 Inspired by token-priority graphs (TPGs) 255 and building on attribution graphs from Ameisen et al. ① for exposing an LLLM's internal computation, we define topic set of TPGs. This graph-based based formulation aligns naturally with recent advances in structured representations for LLMs [42] (2). Furthermore, this mirrors neuroscience models of spontaneous human thought, in which concepts serve nodes connected by associative edges (32). Despite relying on these specific settings, our experiments extend our findings to modern LLMs, empirically confirming that relaxing these assumptions does not seem undermine our core insights.
1.1 Summary of findings
Imagine an oracle that is an expert on Topic A, capable following any conversation within that topic while staying true to its context. Now, suppose the oracle gains knowledge of Topic B and is following conversation about Topic A. Will the oracle's 's responses remain within Topic A, or will
Just to illustrate, we use the prompt Please continue this short sentence, forgetting about "One hundred Years of Solitude". ", since real conversation the LLM would be blind to the final output.

---

the influence of the knowledge of Topic B cause the conversation to drift? This analogy encapsulates the problem we address: understanding when and why attention models might preserve a topic or change to another spontaneously. Specifically, we make the following contributions:
1. Preservation of input topic priorities. Using controlled sandbox, we demonstrate in Theorem that self-attention models trained on mixed-topic datasets maintain the priorities tokens associated with the original topic of an input sequence (Topic A in our analogy).
2. Changing topics triggered by token frequency. In Theorem we show that the oracle's responses may reflect change of topic only if lower-priority token appears more frequently than all sigherrpriorit tokens of Topic
3. Impact of input length and topic ambiguity. Theorem establishes that longer input sequences decrease the likelihood of changing topics. Furthermore, input topic ambiguity acts as a stabilizing factor, not increasing the frequency of spontaneous topic changes.
4. Difference between LLMs and human cognition. In Section 6 we empirically extend Theorem to modern, deeper LLMs. Unlike human cognition, where extended discussions often encourage spontaneous thoughts and topie ambiguity promotes cognitive connections, our results highlight the opposite behavior in LLMs: neither longer prompts nor greater topic ambiguity appreciably increases the likelihood of spontaneous topic change.
Overview of the paper structure. We begin with the problem setup in Sec 2 Sec 3 introduces the definition of topic, and See eexamines how self-attentie attention models allocate the token priorities within the mixed topics. In Sec we establish the conditions under which a self-attention model induces spontaneous topic changes and show the dynamics of topic changes with longer input sequences or the presence of topic ambiguity. We then extend our analysis to frontier LLMs in Sec Related work and discussion are provided in Secs and respectively. All proofs are provided in Appendix A
Problem setup
2.1 Next topic prediction with self-attention model
In line with the approach presented by Tarzanagh et al. 49 and Li et al. 25 we frame the next- token prediction task as a multi-class classification problem. Given a vocabulary of size with an embedding matrix E (er e2 eœ]" E Kx we aim to predict the next token ID y [K] Rxd with x, E for all i E [T]. The training
based on an input sequence X [x] X2 dataset, denoted as
xT]
DSET ((X,, V/) RTxxd x [K))
contains sequences of varying lengths T. In our notation x is the embedding vector corresponding to the token ID v, this eg- For prediction, we utilize single-layer self-attention model with a combined key-query weight matrix W E gddd and identity value matrix as in Tarzanagh et al. [49] The self-attention embedding output
/w = S(Wx)),
(output)
where S(-) is the softmax operation and = XT, serves as a weighted representation of the tokens, allowing for context-sensitive prediction of y based on the final input token. Let é IR > IR be a loss function. For the training dataset DSET, we consider the empirical risk minimization (ERM) with:
L(W)
f(c Xx S(X,xx
(ERM)
We assume a well pre-trained classification head matrix [c] C2 ... c E Rxdd Each classification head c₁ E Rd is fixed and bounded for all & E [K]. Starting from ((0) e gdxd with step size 77 > 0, for 7 2 0 we optimize W with gradient descent algorithm
((++ W ()())(((+)
(Algo-GD)
We keep the first two assumptions from Li 25 Assumption 1. Vy, A E [K], k + y, c, e, 1 and c, e 0.
3

---

Assumption& 2. For any (X. E DSET, the token ey is contained in the input sequence X.
Assumption preprssett variation of the weight-tying approach commonly used in language models [40 [50]. Once training is complete, for a new input sequence X, and model characterized by W, we predict the next token ID l/w based on greedy decoding the probabilities from the softmax of the classification output
Dw E arg max [S (X)w(X)))) KE(K
2.2 Token-priority graph and global convergence of the self-attention model
Li et al. [25] defined a token-priority graph (TPG) as a directed graph with nodes representing tokens in the vocabulary. (SET( is subset of sequences from DSET with the same last token is ex = x. They defined TPGs [
[== such that every ( is a directed graph where for every sequence (X. v) (SET(*) directed edge is added from Gy every token E X. TPGs further divided into cononeecconeeeted components (SCCs), which capture subsets of tokens with equal priority. For tokens within two different SCCs, strict priority orders emerge, helping the model differentiate between tokens when learning next-token predictions. We use the same notation as Li et al. [25] given a directed graph G. for j E [K] such that f j:
E G denotes that the node belongs to G. j) E G denotes that the directed path (i + j) presented in & but j i is not.
(i = j) E G means that both nodes i and j are in the same strongly connected component (SCC) of (there exists both a path - and i).
For any two distinct nodes j in the same TPG, they either satisfy (i= (j= i) (i= j). Nodes in each (+)) represent indices in [K] and SCC structure supports the self-attention mechanism's 's ability to assign priority within sequences based on the conditioning last token. Theorem 2 of Li et al. 25 proved that under Assumptions and the self-attention model learned through Algo-GD converges the solution of the following Support Vector Machine (SVM) defined by the TPGs of the underlying dataset DSET
W arg min |W|F W
(Graph-SVMM
(= 0,
V(i x j) E g(t)
s.t. (e. eÿ)
We
Vii 5) g()) AK E [K].
2 1,
Here is a condensed version of the theorem:
Theorem (Li et al. (25)). Consider dataset DSET and suppose Assumptions Mand [annnpttioooaa@@@ hold. Set loss
function as {(u) = - -log(u). Starting Boorting(AlgoGuul Algo-GD from any (()) with constant size if W.om + O,
W(T)
Wam
W = -im
(2)
7-000
[W(]]
|
This convergence implies that the model predicts the next token based on priorities obtained from the SCCs within the TPG relevant to the last token the input sequence. Unlike the work Li et al. 255 which considers both hard retrieval and soft composition components and examines multiple loss functions in subsequent results, we focus exclusively on a log-loss function in this work, leaving the cxploration of other loss functions for future research. Since the soft composition component is not required for our subsequent definitions and theoretical results, we concentrate solely on the hard retrieval component.
We add here another reasonable assumption that prevents the probabilities in Equation from being equal due to improbable numerical reasons, and we present our first lemma.
Assumption 3. For any (X. 3) E DSET, Ži,j E [T] and u, U E such that u [(XWx))] =
S ((XWx)) if and only if u and [s(Xw)) SXXWx)
Lemma 1. Suppose conditions from Theorem 7 and Assumption 3 hold. Consider imput se-
quence X. from (SET(R) and corresponding TPG g()), Vi,j E [
we have |S (Cf =
[S (CCw(X))), iff (z+ x 2z) (())

---

This means that the tokens that maximize the probability for weights W in Equation 1 are all within the same SCC leading to the following definition:
Definition I (highest probability SCC), Consider an input sequence X from DSET and corre-
sponding TPG g(k). We define ((+) (X) g(h) as the highest probability SCC for X in G()) such
that Vx E ( (X) have [S (C (X))l, ||S (Cfw (X)
3 Defining topics
In order to answer our research questions regarding the dynamics of topic changes we need to define the concept of a topic. In the previous settings, a dataset DSET generates TPGs (g(4)) } conversely, existing set of TPGs generate DSET. Therefore, inspired by Ameisen al. [ that introduces attribution graphs to reveal the LLMs' internal computational structure, we define topic set of TPGs:
k== but,
Definition 2 (topic). A topic T is a
ae TP (g(k))__ Given topic TI defined by TPGs (g(k))
]k=
inpul sequence X belongs if x g(#), A sequence (X,y) is within T if X belongs to T and Vx X, (u = x) g(a)
Our graph-based formulation aligns with recent advances in structured representations of LLMs Given the finite number of edges, DSET can be generated from T such that it can reconstruct the exact TPGs { that define T. following the construction method in Li et al. This leads to the following reasonable assumption:
42
52].
l==1
255.
Assumption 4. A DSET generated from any topic T defined by [ the TPGs {(()) JK=
jk=1 exactly reconstructs back
Detailed explanation is provided in Appendix B This assumption enables the application of the results from Li al. [25), with the concepts of topics and TPGs being used interchangeably.
Definition (topic continuity). Given an input sequence X that belongs to T, a weight matrix W is said to keep topic T. for the input sequence X if úw E (X).
Remark. Given two topics, Ta and To, with corresponding datasets DSET, and DSET,, the union of (g Jkmi and (g) (1) 1 forms the TPGs for the mixed topics Tab, denoted by (k) [K
( 1K
Jkmi
It is clear that W. trained only with DSET, will always keep topic T 3 But we could also obtain Was with a dataset combining DSET, and DSET, as training sets. The central question is whether W Was keeps topic Ta. given an input sequence X that belongs to T.. or if it instead predicts tokens that prompt a topic change.
Attention within mixed topics
Let's first understand how attention models assign priority to tokens within mixed-topic setting, For simplicity, we elaborate our results using a two-topic scenario, but it straightforward to extend the results on multiple topics. Notice the self-attention embedding outppt is linear combination of X given by S(XWS Wx). The embeddings in corresponding to the highest entries in S(X will receive higher priority to predict the next token, therefore can hypothesize that models in which S(XWx) are ordered in a similar way will predict similar next tokens. This idea leads to our first main result which considers this situation within mixed-topic setting:
Theorem 2. Consider datasets DSET, and DSET, from topics To and To, respectively. Let DSET ab be the union of DSET, and DSET, Suppose Assumptions D Z 3 and
Set loss function as f(u) log( u.. Starting Algo-GD GDorom any initial point with constant size n and if Wamm + 0 and Wamm # 0: for given sequence X that belongs to Tax have that W. ab preserves the atlention priority Ta on input X. This is Vi.j E [T):
NNotation The subscripts of weights and objects correspond to the associated topic. For instance W. denotes the weights defined in Equation obtained from DSETa, which pertains topic Te.
5

---

61 for
To
Imputt saqueecc X from
Me D58T,
5,1000
Prras (00
00
(Tople continatty
Input eequence X from
Mg DSRT,
Pla
$. (a)
Pet. 5. (1
Sua
Self Attention
00 ba
W.. Trained on DSET...
Paic
(Amiguuur
Input so requeeoo X from
Trained DSET.
Prv.
5,0000
Sw., 6 (00 $, (0) 2be (X) (Changle of topii)
Self Attantion
Trained DSET,>
Figure 3: Depiction of each scenario in next token prediction. Left: Taking the last token e4 as an example, Gufl for Tos is formed by the union of G! and g,(4) The direction of edge is from output to input and the dotted square denotes the strongly-connece tokens have equal priority. Right: For each input sequence belonging to Ta. model trained on DSET, and another model trained on the mixed-topic dataset DSET as to predict the
components (SCC) in which self-attention
next tokens, denoted as 9ww, and fwa.. respectively. G(4) and Go represent the highest probability SCCs (Definition] I in mixed-topic setting and in T.. respectively. There are three scenarios, topic continuity (Definition 33, ambiguous sequence (Definition 4. and chunge of topic (Definition 5). The
numeric details for each scenario are provided in Appendix C.5
if [S(W ( })1 then
[S(X x6] = (XXW x)]]
((XW.x)) SS x)] }-
then
[SX x)] [S(W xbx)]]
. if S(XW.))), < S X a x)] then
SXXW axxx - SSX W xt)
This implies that for an input sequence X, a model trained in a mixed-topic setting will maintain the priority of the topic to which X belongs. Consequently, the attention will be allocated in the same order as if the model had been trained exclusively on the original topic of X. For the first input sequence X = [e,, e1, e3; ex] from Ta, as shown in Figure 3 (right), the predicted next token (w... is C3 and the highest probability SCC in mixed topics is Pub (X) = {cs}. Since Dwas. belongs to ((A)) (X). Was for input sequence X considered as topic continuity, based on the Definition
The only assumption about X on Theorem is that it belongs to T.. However, if X belongs to To and Tr, the priority will be preserved within both topics. Additionally, strict equality in the attention priority holds, but strict inequalities may not, as the union of their TPGs can form new SCCs. As
illustrated the left of Figure Gl and (()) denote the TPGs corresponding the last input token e4 for Ta and To, respectively. In gltt) the token priority is e; > eg >> e₁ 02 >> e4. In contrast, c(4) for the mixed topics, the priority order is e; > e3 in ab
e2 e4. The equality el e2 from (()) is maintained in g(4) whereas the strict inequality C2 > C4 is relaxed to C2 = C₁ in mixed topics, forming the new SCC, {ei, e2, e.}. in g(4)
Explaining topic shifts
The formation of new SCCs when combining datasets suggests that the highest priority SCC for some input sequences may increase in size in this new setting. This also suggests that topic shifts may arise from ambiguity within an input sequence rather than a straightforward change in topic. In our oracle analogy, gaining knowledge of both Topic A and Topic B might cause a conversation to be naturally followed within Topic A or also outside Topic A. We introduce the following definition to characterize this phenomenon:
Definition 4 (ambiguous sequence). Given DSET, and DSET, generated from two different topics T, and The Denote Tab as the combined topic defined by a combination of DSET, and DSET, A sequence X that belongs to Tr is ambiguous in Tab with respect to T. if War ab does not keep topic
To for X. but ( (X)
((8)
Jab (X).

---

Definition 4 defines an ambiguous sequence as one where the gighest-probabiliyy next next-token pre- dictions include tokens from both within and outside the input topic, reflecting natural ambiguity e4 T in Figure 3 (right) as an example. g64h) (X) is }e as depicted in g6 from Figure (left) and g4 (X) is fer, C4 as
from overlapping topics. Take the second input sequence X el 64, e1.
shown in G from Figure (left). g% (X) is subset of ab (X), although Bwas & 9 (X). We
can argue that the next token predicted from an ambiguous sequence cannot be considered as topic change, it lacks the clear trigger phenomenon observed in human cognition. To address this, we propose a formal definition for topic change:
Definition (change of topic), Given DSET, and DSET, generated two topics Ta and Th, and a sequence X that belongs to Ta. The weight matrix W ab changes topic Ta for sequence X if W does not keep topic T. for X and X is not ambiguous in Tab with respect to Ta-
ab
In Figure 3 (right), Was changes topic for the last input sequence X = [es, eq, eq, es]" following the Definition D Building the formal definitions of topic continuity, ambiguous sequences, and topic changes, we now present a necessary condition for a sequence to induce topic change. This is achieved by introducing our final definition, grounded in the tighest-priority SCC as determined by the order in the attention layer.
Definition 6 (highest priority SCC), Consider sequence X that belongs to T. We define ( (X) S ((3) as the highest priority SCC for X in ((8), such that Vx, E ((8(X) and x E gl have
(x: = Iz)) g(a) or (xxxx,,) E g(#)
Theorem 3. Under the same settings and assumptions in Theorem 2 let X be sequence that
belongs to Tu. Was ab changes topic for X then Bay & g (X) such that Vz, E ().)) the number of times Xj appears in X is greater than the number of times x; appears in X.
Theorem 3 implies that, for given sequence X from Ta and its corresponding TPG, a necessary condition for a topic change is the presence of lower-priority token that appears more frequently than any of the higher-priority tokens. This can be intuitively understood through our analogy: if the oracle is following conversation on Topic A but the conversation contains repeated components with lower importance in Topic A, its knowledge of Topic may steer the response toward Topic B, thereby initiating a shift away from Topic A. A natural question arises: what do these findings imply in practice? Specifically, how does the probability of change of topic behave as the input sequence length the topic ambiguity increases? The following theorem sheds light these dynamics.
Theorem 4. Under same setlings and assumptions datasets and training in Theorem
2 let X be
sequence that belongs to Ta with no repeated tokens, and be the number of elements in ((a) (X). x be a random sequence of iid random tokens sampled from X such that IP (xx = x). We have:
Let X' [x] x2= for a fixed p. = min 2 (X)
1. 1f > max " (X) P ( x). then limit P(W ab changes topic Ta for X') O.
2. If 1. increases then the probability that Br's of (+) (X) such that Val. E gl (X), X² outnumbers X in X' does not increase.
There are two implications of this theorem. First, as the input sequence length increases sufficiently, the likelihood of topic changes vanishes. Second, increasing l raises the probability of overlap between topics and reduces the probability of satisfying the necessary conditions for a topic change, effectively creating bound the frequency of topic changes. In practice, consider the oracle analogy: if the oracle is following a sufficiently long conversation on a specific topic, it becomes exceedingly unlikely to shift topics. Similarly, topics A and B become more interconnected, this increased ambiguity does not lead to more topic changes; rather, it may reduce their occurrence. This contrasts with human cognition, where longer conversations and greater inter-connectivity of knowledge increase the likelihood of spontaneous topic changes.
To illustrate Theorem through simulations, we generate embeddings with K 10 and d = 16. We approximate W. and Wab as the results obtained after 8000 iterations of AD We quantify the proportion of test sequences in which Wah keeps T, (keep topic), proportion of ambiguous sequences in Tes (ambiguity) and proportion in which W... ab changes topic (change topic). First, we
7

---

1.00
1.00
0.98
Preportion
0.96
Proportion
0.96
0.94
0.94
Topic continulty
0.92
Ambiguous sequence
0.92
Change of toplc
8
16
64
128
256
512
0.90
6
10
12
16
18
(a) Length of sequences.
(b) Topic ambiguity.
Figure 4: The proportion of topic continuity, ambiguous sequence, and change of topic as (a) input length and (b) topie ambiguity increase.
explore the effect of longer sequences by varying the length T of the test sequences Z. We increase from to 512. Figure 4a illustrates how the proportion of change topic decreases as T increases. Second, investigate the effect of topic overlap with increasing number of edges L. Intuitively, a higher L results in an increase and a greater overlap between TPGs of different topics. We vary from 4 to 18. Figure 4b demonstrates that L increases, ambiguity increases, while the proportion of change topic doesn't t increase. These two findings contrast with expectations derived from human cognition but align with the result of Theorem 4 Lastly, among the 85,000 test sequences generated for these experiments, 99.98% satisfy Theorem (i.e., topic changes occur when a low-priority token appears more frequently than high-priority tokens). The remaining 0.02% mismatched cases solely due to minor approximation discrepancies in the attention softmax. These results validate Theorem (see simulation details in Appendix
Experiments in frontier LLMs
To prove Theorem we work within the simplified, single-layer self-attention model of Li et al. [25 Although this abstraction omits many hallmarks of contemporary LLMs (deep stacks of attention blocks, alternative cost functions, and other training heuristies), it offers mathematically tractable setting that lets us derive interesting mathematical results. These results, in turn, can be used to understand how cutting-edge LLMs behave in terms of spontaneous topic changes. We empirically investigate such behavior on four frontier models: GPT-4o, Llama-3.3, Claude-3.7, and DeepSeek- V3.
Real dataset. We randomly select 100 arXiv papers published in March 2025 since the publicly disclosed knowledge cutoff dates for our study LLMs fall at the end of 2024 or carlier. This ensures that these models have not been trained on these data. We consider each paper different "topic".
Experimental setup. For two distinct papers A and B, and an input prompt (X) from paper A, we consider a measure of topic continuity as the cosine similarity between the embeddings of the texts generated when the LLM has contextual knowledge solely from paper A (ýw.) and when the LLM has contextual knowledge from both paper A and B (iw... We treat this cosine similarity as an empirical proxy for our formal definition of topic continuity (Definition 33: therefore the larger similarity, the smaller the chance that the model has led to change of topic. This proxy suggests two testable consequences which become the empirical counterpart of our Theorem 4 (I) cosine similarity is expected to increase with the length of the input prompt, and (2) it is not expected to decrease with increasing ambiguity in paper and paper
To more closely align with our theoretical framework, where model gains knowledge of topic A and incrementally gains knowledge of topic B, we implement a Retrieval- Augmenttd Generation (RAG) approach, retrieving information exclusively from paper A or jointly from papers A and B [51]. Based on the input prompt, we retrieve the top 3 most relevant excerpts from paper or paper to form the contextual knowledge set A or set B. The combined contextual knowledge set is simply the union of sets and B. We add set to the input prompt obtain the generated text with sole knowledge of paper A (fw.). and we add the combined set the input prompt obtain the generated text with

---

0.98
1.00
GPT-40 Claude-3.7
0.95
Cosine Similarity
VeepSeekv
Llama-3.3
Cosine Similarity
0.90
0.94
0.85
0.92
0.75
0.88
0.70
CPT-4o
Llama-3.3
Claudo-3.7
DeepSeck-VV
0.65
0.86
[0.43 0.52)
34,0 0.40 0.43) (0.25,0.34 (0.16, (0.16.0.25)
34)
(0.07,0.16)
10
30
50
70
90
110
130
150
Input Length
Keywords Similarity
(a) Length of sequences.
(h) Topic ambiguity.
Figure 5: Similarity between continuations generated with single-topic and mixed-topic knowledge as (a) input length and (b) topic ambiguity increase.
combined knowledge of paper A and B (iwa)). To closely follow our greedy decoding approach in our theoretical framework, we set temperature parameter to for all LLMs.
We designate each paper as paper A and randomly select 5 different papers from the remaining 99 papers as distinct paper B. For each input segment, we calculate the average cosine similarity between Úwaa and DWab across these five pairs of paper A and paper B, using each LLM. The results for each LLM are averaged over all 100 papers. See additional experimental details Appendix D
Experiment 1: Impact of input length. We use the first 10, 19,30.... ..... 150 words from each paper
A's abstract the input prompt. Figure 55 shows, for each LLM, the average cosine similarity as a function of input length; shaded bands indicate 95% confidence intervals. Across all models, similarity tends to increase with input length, aligning with the behavior predicted by Theorem Appendix D.3.J presents an additional experiment in which we extend the input length to 1210 words extracted from each paper's introduction; the results further support conclusions.
Experiment 2: Impact of topic ambiguity. We fix the input prompt length to the first 80 words of
each paper A's abstract. We quantify topic ambiguity by the average similarity among each paper A's keywords: lower keyword similarity signifies higher probability of overlap between paper A and other papers, consistent with setup in Theorem 4 We partition the papers into five equal-width bins along this ambiguity spectrum. Figure [5]mmmmmeesess the results: each boxplot shows the distribution of cosine similarities within an ambiguity bin, with the x-axis ordered from least to most ambiguous. Across all LLMS the median similarity does not seem to decrease, in agreement with the prediction of Theorem In Appendix D.3.2 present an additional experiment using an alternative ambiguity measure based on pross-paper keyword similarity, yielding results consistent with our conclusico fonclusions.
Taken together, the two experiments provide preliminary empirical support for Theorem 4 showing that its prediction, derived from a single-layer self-attention toy model, can be extended to today's s deep, multi-layer LLMs. Crucially, important divergence between machine and human cognition persists these frontier models: neither longer prompts nor greater topic ambiguity appreciably increases the likelihood of spontaneous topic change.
7 Related work
Training and generalization of Transformer. (1) Properties of Softmax. The self-attention mechanism employs the softmax function to selectively emphasize different parts of the input. Gu
al. [16), Goodfellow al. (13]. and Deng et [TO] underscore the pivotal role of the softmax function in shaping attention distributions, influencing how models process and prioritize information within input sequences. Bombari and Mondelli (5) examined the word sensitivity attention layers, revealing that softmax-basee attention layers are adept at capturing the significance of individual words. However, recent work has also pointed out limitations of the softmax function [4] TO). (2) Optimization in attention based models. Additionally, recent research interprets Transformer

---

models as kemel machines, akin to support vector machines (SVMs), with self-attention layers
performing maximum margin separation the token space 48 49 25 21. (3) Chain-of-Thoowe Thought (Co'T) and In-Context Learning (ICL). Moreover, transformers exhibit remarkable abilities in generalization through ICI., where models effectively learn from contextual cues during inference enhances this by breaking down reasoning processes into intermediate steps, highlighting the emergent reasoning abilities of transformers. (4) Improvement efficiency of transformers. Recent advancements aim to improve the computational efficiency of transformers 122 7 47 [53], ensuring their viability for large-scale deployment while maintaining or enhancing their representational capabilities.
[6,56] 37]. CoT prompting [54] 57
44 24
Next token prediction in LLMs. (1) Theoretical and architectural innovations. Shannon foundational work laid the groundwork for estimating the predictability of natural language sequences, providing basis for subsequent advances language modeling. Recent studies have expanded our understanding of how LLMs anticipate future tokens from internal hidden states, offering valuable insights into the efficiency and effectiveness of Transformer-
43
architectures [17] 38] 45 Despite
their impressive predictive capabilities, these models face fundamental limitations. For instance, Bachmann and Nagarajan [3] highlights the shortcomings of teacher-forcer training, emphasizing how this approach can fail and suggesting strategies to improve model robustness. (2) Efficiency and Optimization. Goyal et al. [4] introduces a novel method that incorporates a deliberate computation step before output generation, enhancing reasoning capabilities. Additionally, Gloeckle et al. [12] advocates for multi-token prediction, which significantly improves both efficiency and speed.
Self- A Atention and topic dynamics. Advancements in self-attention research have deepened our understanding of how transformers handle evolving semantic contexts. Prior work has explored di- verse aspects of topic modeling, such dynamic topic structures [35] hierarchical relationships (2)]. topic-aware attention mechanisms [39], and the mechanistic underpinnings of topic representation (26). While these studies provide insights into managing static and hierarchical topic structures, our work focuses on the topic changes with the given input sequences from specific topic.
8 Discussion
Our theoretical analysis on self-attention models and empirical investigations on modern L.LMs reveal fundamental clues regarding the distinctions between model-based spontaneous topic changes and spontaneous human thought, a phenomenon that is critical for comparing conversational dynamics across humans and AI. In era of growing concern about Al's cognitive resemblance to humans, our framework provides preliminary results differentiating these phenomena, thereby opening pathways for future interdisciplinary research at the interface of artificial and human cognition.
Limitations. Our theoretical framework builds the same simplified single-layer self-attention model with a log-loss objective from Li al. (25 and defines topics as TPGs. These abstractions do not fully capture the complexities of contemporary LLMs, including deep attention architectures, alternative loss functions, and diverse training objectives. Despite loosening these assumptions, our experiments suggest that the essence of our theoretical conclusions holds across modern LLMs within our framework of study. Future work will investigate how broadly these theoretical insights generalize complex architectures, for example within longer context windows and/or LLM outputs.
Code. The source code can be found on GitHub: http:://gthhbboomm/mmmmmiyyyyyms Spontancous- Topic-Changes
Acknowledgments and Disclosure of Funding
This work was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) under grant DGECR-2022-00 reviewers, and the session chair for their valuable feedback and insightful suggestions, which greatly improved the quality of this work.
The authors thank Prof. Yingcong Li, the anonymous
10

---

References
[1] Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Mareus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi. T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. Circuit tracing: Revealing computational graphs in language models. https: htttps:/ttaasfoooeee attribution- grwphs/maeeeeeeee methods. html 2025. Anthropic.
[2] Anthropic. Claude 3.7 sonnet: Hybrid reasoning ai model.
news/ 3 2025. Accessed: 2025-05-09.
[3] Gregor Bachmann and Vaishnavh Nagarajan. The pitfalls of next-token prediction. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 2296-2318 18. PMLR, 21-27 Jul 2024. URL hhttps://ppte htttp::/prroceedings mlr. wwwpp/ v235 2t333//aacmman22...m html
[4] Buddhika Bellana, Abhijit Mahabal, and Christopher J. Honey. Narrative thinking lingers in spontaneous thought. Nature Communications, 13(1),4585, 2022. ISSN 2041-1723, doi: 001138//41446-022-2920 123-6.... URL http:tttt/o.ooor//// 1038/ s41467 -22-32113-
[5] Simone Bombari and Marco Mondelli. Towards understanding the word sensitivity of attention layers: A study via random features. In Forty-first International Conference Machine Learning, 2024. URL Bttpstts::// httptttttt///orreiiiiii
net/forummid (tt.ormmmmdddaaaPppaaaP...
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020. URL wtwpttts:://///tttt htl_tttps//prrcceennggg neurips. wwtp:eeww..r...pppeeeeeeeeeeeeee wwwpppape_fileeeeec http///ieee/aaper///////l/
pdf
[7] Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy Colwell, and Adrian Weller. Rethinking attention with performers. In International Conference on Learning Representations, 2021. URL https //openreview www.:e/ooooommaaaaauuuua..eo
[8] Kalina Christoff and Kieran C. R. Fox. The Oxford Handbook of Spontaneous Thought: Mind- Wond-Wandering, Creativity, and Dreaming. Oxford University Press, 05 2018. ISBN 9780190464745.
doi: 10. 10100000ooffoddhh077999444444444 org/10. 1093/ wwt.forrbbb/88011944444450.0..10
.0001. URL httpp:///oo
[9] Kalina Christoff, Amanda Gordon, and Rebecca Smith. The role of spontaneous thought in human cognition. Oshin Vartanian and David R. Mandel, editors, Neuroscience of Decision Making, pages 259-284. Psychology Press, 2011.
[10] Yichuan Deng, Zhao Song, and Tianyi Zhou. Superiority of softmax: Unveiling the performance edge over linear attention. arXiv preprint arXiv:2310) 1685, 2023.
[11] Gabriel Gareía Márquez. Cien años de soledad. Editorial Sudamericana, 06 1967.
[12] Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Syn- naeve. Better faster large language models via multi-token prediction. arXiv preprint 19777 2024.
11

---

[13] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
attp://www.deellaminnngooo. org
[14] Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens. In The Twelfth International Conference Learning Representations, 2024. URL https
/ //openreview net/ Forum?id-ph04<RRkPd
[15] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
[16] Jiuxiang Gu, Chenyang Li, Yingyu Liang, Zhenmei Shi, and Zhao Song. Exploring the frontiers of softmax: Provable optimization, applications diffusion model, and beyond. CoRR, abs/2405.03251 2024. URL hittps://dee wtw.doi..ogggig/ wro/o0....00......... 48550/ arxiv. 2405 .062510311
[17] Hangfeng He and Weijie Su. A law of next-token prediction large language models. arXiv preprint arXiv:2408. 13442, 2024.
[18] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
[19] Yerin Hwang, Yongil Kim, Yunah Jang, Jeesoo Bang, Hyunkyung Bae, and Kyomin Jung. MP2D: An automated topic shift dialogue generation framework leveraging knowledge graphs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16782-17702, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 18653/v1/ 2024. wew..emmll-mmii..99 979. URL https: http::///caaaatoooooceeo org/ 22244. emnlp- main..ain... ieain..e main.979/
[20] Ziwei Ji, Naycon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
[21] Aaron Alvarado Kristanto Julistiono, Davoud Ataee Tarzanagh, and Navid Azizan. Optimizing attention with mirror descent: Generalized max-margin token selection. In NeurIPS 2024 Workshop on Mathematics of Modern Machine Learning, 2024. ltttss:::: net/forum? wtw.ooruummd-tttT999ll..
[22] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations, 2020. URL URL htRl.https://///t htttps://openrevieer
[23] Aaron Kucyi, Julia W. Y. Kam, Jessica R. Andrews-Hanna, Kalina Christoff, and Su- san Chitfield-Gabride Recent advances in the neuroscience of spontaneous and off-task thought: implications for mental health. Nature Mental Health, 1(11):827-840, 2023. ISSN 2731-6076. doi: 10. 00010888444200003-000333 844220--23-001333-
URL https: //dtss:///o..... 2xg/10.1130
[24] Hongkang Li, Songtao Lu, Pin- Yu Chen, Xiaodong Cui, and Meng Wang. Training nonlinear transformers for chain-of-thought inference: A theoretical generalization analysis. In The Thirteenth International Conference Learning Representations, 2025. URL https://
openreview wtt.riieiinntttteeeeeeeeeeeeeeee
[25] Yingcong Li, Yixiao Huang, Muhammed E Ildiz, Ankit Singh Rawat, and Samet Oymak. Mechanics of next token prediction with self-attention. In International Conference on Artificial Intelligence and Statistics, pages 685 693.. PMLR, 2024.
[26] Yuchen Li, Yuanzhi Li, and Andrej Risteski. How do transformers learn topic structure: Towards mechanistic understanding. In International Conference on Machine Learning, pages 19689-19729. PMLR, 2023.
12

---

[27] Sungsoo Lim, Keunhyun Oh, and Sung-Bat Lrog-Bae Cho. A spontaneous topic change of dialogue for conversational agent based on human cognition and memory. International Conference on Agents and Artificial Intelligence, 2010. URL http::/pppp...eaatccccccooo.o.a org/ CorpusID:11468255
[28] Jiangyi Lin, Yaxin Fan, Xiaomin Chu, Peifeng Li, and Qiaoming Zhu. Multi-granularity prompts for topic shift detection in dialogue. In Advanced Intelligent Computing Technology and Applications: 19th International Conference, ICIC 2023, Zhengzhou, China, August 10-13, 2023, Proceedings, Part IV. page 511-522, Berlin, Heidelberg, 2023. Springer- Verlag. ISBN 978-981-99-47555
doi: 10. 110011997--81-999--88111 1007/978 98----9----999---222-
42. URL https://doi.orr::
[29] Zhicheng Lin, HcGang Chen, Yuyin Lu, Yanghui Rao, Hao Xu, and Hanjiang Lai. Hicrarchical topic modeling via contrastive learning and hyperbolic embedding. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, editors, Proceedings the 2024 Joint International Conference Computational Linguisties, Language Resources and Evaluation (LREC-COLING 2024), pages 8133 8143, Torino, Italia, May 2024. org 2024. Lree-main. 712
ELRA and ICCL. /ttts:tt:///atttttttit
[30] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412. 19437, 2024.
[31] Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1919, Online, July 2020. Association for Computational Linguistics. doi: 10. 12111663311/2200.oom acI- main. URL
org/ 2020. acl - maiin
173
[32] Judith N. Mildner and Diana I. Tamir. Spontaneous thought as an unconstrained memory process. Trends in Neurosciences, 42((1):763-77) 767777, 2019. ISSN 0166-2236 doi: https: att.ccle/pp///0111222311000
/utb..og//0.11116...is2200000009 0000. URL htttps/////ww sciencedirect. cww/ ste//scienco
[33] Judith N Mildner and Diana I Tamir. Why do we think? the dynamics of spontaneous thought reveal its functions. PNAS Nexus, pga)e2g 06 2024. ISSN 2752-6542, doi: 10. 1093/ pnasnexus/pgae230.. URL
[34] Caitlin Mills, Andre Zamani, Rebecca White, and Kalina Christoff. Out of the blue: under- standing abrupt and wayward transitions in thought using probability and predictive processing. Philosophical Trunsactions of the Royal Society B: Biological Sciences, 376:20190692, 12 2020. doi: 10.10998/stb.201 2019.0692
[35] Nozomu Miyamoto, Masaru Isonuma, Sho Takase, Junichiro Mori, and Ichiro Sakata. Dynamic structured neural topic model with self-attention mechanism. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: Toronto, Canada, July 2023. Association for Computational org/
ACL 2023, pages Linguistics. doi: 10.11863/v11222 1/2033//11/223../ findings-acle ac1.366. URL
2023 www.inindngsaccccc acl. 366
[36] Jinjie Ni, Tom Young, Vlad Pandelea, Fuzhao Xue, and Erik Cambria. Recent advances in deep learning based dialogue systems: systematic survey. Artif. Intell. Rev., 5644))335555531155 August 2022. ISSN 0269-2821. doi: 01007//104622-022-00244 URL https:sst:///o..oo 10. D0.1007/s10442--00 022- 10248-8
[37] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.1 118599 2022.
[38] Koyena Pal, Jiuding Sun, Andrew Yuan, Byron Wallace, and David Bau. Future lens: Anticipat- ing subsequent tokens from single hidden state. In Jing Jiang, David Reitter, and Shumin Deng,
13

---

editors, Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pages 540-560, Singapore, December 2023, Association for Computational Linguis- tics. doi: 10. 10.18655.1/2023.colll...... Utt.hhttps://aclanbbbooo
org/2023.ccn111111 conll-1
37/
[39] Madhur Panwar, Shashank Shailabh, Milan Aggarwal, and Balaji Krishnamurthy. TAN- NTM: Topic attention networks for neural topic modeling. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages
Online, August 2021. Association for Computational Linguistics. doi: 10. (0.18653/11202) acl-long.299) URL org/ wtgg/02211122 2021 acl long........ 299
[40]
Ofir Press and Lior Wolf. Using the output embedding to improve language models. In Mirella Lapata, Phil Blunsom, and Alexander Koller, editors, Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 157-163, Valencia, Spain, April 2017. Association for Computational Linguistics. URL https://aclanthology org/ E17 2025/
[41] Hemanth Saratchandran, Jianqiao Zheng, Yiping Ji, Wenbo Zhang, and Simon Lucey. Rethink- ing softmax: Self-attention with polynomial activations. arXiv preprint arXiv:2410.18611 2024.
[42] Priyanka Sen, Sandeep Mavadia, and Amir Saffari. Knowledge graph-augmented language models for complex question answering. In Bhavana Dalvi Mishra, Greg Durrett, Peter Jansen, Danilo Neves Ribeiro, and Jason Wei, editors, Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE), pages 1-8, Toronto, Canada, June 2023. Association for Computational Linguisties. doi: 10.18653/v1/20222 .lrre- 1.1. URL https://aclanthhologyy org org/ mlrse-1.1/
[43] C. E. Shannon. Prediction and entropy of printed english. The Bell System Technical Journal, 30(1):00-64, 1951. doi: (..1002/j.1538--11 1661160336.6.
[44] Zhihong Shao, Yeyun Gong. Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. Synthetic prompting: Generating ofain-of-thought demonstrations for large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 30706-30775. mlr wwppss/ev20000 shao23a.
PMLR, 23-29 Jul 2023. URL URL.ttps::
[45] Buck Shlegeris, Fabien Roger, Lawrence Chan, and Euan McLean. Language models are better than humans next-token prediction. Transactions Machine Learning Research, 2024.
ISSN 2835-8856. URL hwtp::ttt:///ott //tptpeve//penrevieee www.:nee/fffoiiiiommm tett:/forrmmddddNssssdmmmm
[46] Mayank Soni, Brendan Spillane, Leo Muckley, Orla Cooney, Emer Gilmartin, Christian Saam, Benjamin Cowan, and Vincent Wade. An empirical study of topic transition in dialogue. In Chloe Braud, Christian Hardmeier, Junyi Jessy Li, Sharid Loaiciga, Michael Strube, and Amir Zeldes, editors, Proceedings of the 3rd Workshop Computational Approaches to Discourse, pages 92-99, Gycongju, Republic of Korea and Online, October 2022. International Conference on Computational Linguisties. URL
wwwp://...o/o022eomm.. co..-1.2....111111
[47] Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. In Anna Korhonen, David Traum, and Lluís Mårquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Lingnistics, pages 331-335, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/ P19- 1032. URL
wrg. 1032
[48] Davoud Ataee Tarzanagh, Yingcong Li, Christos Thrampoulidis, and Samet Oymak. Trans- formers as support vector machines. NewrlPS 2023 Workshop on Mathematics of Modern Machine Learning, 2023, URL https: :tttps://opeee http:///ppeeevevvee fest/fooummiii idd id=gLwzzmh779%
14

---

[49] Davoud Ataee Tarzanagh, Yingcong Li, Xuechen Zhang, and Samet Oymak. Max- margin token selection in attention mechanism. Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL. htt.htttp://ppenreeieeee wtt.:///w....oooooroaadoooooooo.
[50] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
[51] Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference Machine Learning, pages 35151-35174. PMLR, 2023.
[52] Haoyu Wang, Shikun Liu, Rongzhe Wei, and Pan Li. Model generalization on text attribute graphs: Principles with large language models. arXiv preprint arXiv:2502.1/836, 2025.
[53] Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.0477 2020.
[54] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. of-thouhtt prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 248377 2022.
[55] Huiyuan Xie, Zhenghao Liu, Chenyan Xiong, Zhiyuan Liu, and Ann Copestake. TIAGE: A benchmark for topic-shift aware dialog modeling. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1684-1690, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 10. 10.188///v1/022 1.2di... Bendings-emnll.1eee 145.
URL wtkL.ttpp:///cclaaaolll...ooooo org/ findiignnnnnnnnenn emm1p 145/
[56] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations, 2022. URL
[57] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schu- urmans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed H. Chi. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Con- ference on Learning Representations, 2023. URL htttps://ppereeieer net/ wwt./eeuut/iiiiee forum?id= WZH7099tgIM
15

---

Supplementary Materials
Technical proofs
17
Proof of Lemmall
17
Proof of Lemma
17
Proof of Lemma
17
Proof of Theorem
18
A.5 Proof of Theorem
18
A.6 Proof of Theorem
18
Explanation of Assumption
19
Detailed simulation studies with single-layer self-attention
19
Simulation process
19
Additional experiments to support Theorem
20
C
Simulation in Section
21
C.4
Additional experiments for convergence in mixed topics
21
C.5
Numerical analysis for each scenario Figure
22
D Experimental details in Section
24
D.1
Impact of input length
24
D.2
Impact of topic ambiguity
24
D.3
Additional experiments
24
D.3.1
Extended input length
24
D.3.2
Alternative measure of ambiguity
25
Computational resources for experiments
25
Impac statement
26
16

---

A Technical proofs
A.1 Proof of Lemma
Let S(XWx).
C/w(X)= = C XT 8 (Ww))
(3)
=C(xaa a)
(4)
r a [c] xx)
Ei= as (c] xx))
(5)
Σ as (ck x,)
(6)
Let k; be the number of times token x; appears in X. Then,
(X)),,)), kja,
From Assumption we have that
(X) (C][(X)),)
0; = aj
(7)
(,,xx)) (())) or x4 xy
(8)
If 04 # Zÿ then z; and zy are in the same SCC.
A.2 Proof of Lemmal
Lemma 2. For an input sequence X that belongs T and i.j [T].
[SXWx))] [s(XWx)], = (x, ( 22) (()) or =. j.
[(XW))) < [S(Wx))] = (x, 21) E g(=)
((XWx)): S(XWx) xx)]; > (+= mソ) g(2)
Proof. Since X belongs to T, Vx X we have g(z), therefore from the construction of TPGs by Li et al. [25] for every X, E X we have one of the these relationships: (I; = 2z), (z, = 2;). (x: x 2,) or r; Ty. From the constraints in Algo GD AD
S(XW))) = (XXW))) = (x- x1) = = xx) E g or = j.
S(XWx)) > (S(WX)) }) = (x,-x)) Wi>1
(zy 21) E g(a)
S(WW)) < [S(XW))] = -x) Wi <1 = (z 22) E g(
A.3 Proof of Lemma
Lemma 3. For an input sequence that belongs to T,
x = az SXWx)) ||S(XXx)
Proof. Let G {- [XWW)) ||(XW)
From Lemma V2 ar G. xx)
g(2), Therefore all elements in G belong to the same SCC. Also from Lemma 2 Vz, E G, G we have (a; = 2,)= E g( This means that every element in G has the highest priority among tokens in X concluding our proof.
17

---

A.4
Proof of Theorem 2
From construction, vk E [K], g
_ c This means that Vx,, X2 E X, we have:
if (1,x),,)> _o then (z; x x,) 9al
if (aj + m.) _ then (c) + x;) E g or (z) x xz) E 9ah
if (z, x3) E G then (z, + zy) E g or xxxx E g
Combining with Lemma 2
[SXW x)] = [S X x3] = (x, E g or i = j. then (xxxx))
= E or
j=j (SW )}. = (XX
[(XW,x)) < [S (x(Wxx)) = (zj + xz)) E ( then (z) + x;) E g (()
or
(x xxxz)) 0/22
(XWWax)) S ( xb>
B(XW,))]
[S(XW 8)ه - (x. xzح E g then (a. + xz)) E (
or
ab
(z; x 2z) E ga
S XW xx)] 2 [S(W x]]
L
A.5 Proof of Theorem
Let S(XW,8) x and b = S(XW x). Without loss of generality, suppose a is in decreasing order a1 2 ... 2 ar, From Theorem 2 we also have d1 2 ... 2 by. Let k; be the number of times
token x; appears in X. Following analogous procedure in Lemma 1
we get
(f = kja;
x;
Cf (X) kgb;
(10)
We will proof the contrapositive: If 3r, E g(() no change of topic, so Was keeps topic T, for input sequence X, or X is ambiguous in Tas with
(X) such that K 2 ky for all j E [K], then there is
respect Ta.
From Lemma if x4 E g((() (X), we have a; 2 aj for all j E [K]. Suppose Er, E g(()) (X)
such that h; ky for all j E [K]. we have that kja; Kyaz for all j E [K] then 2; G (X). Analogously since b, 2 by. 24 E ((2) (X). If Ex, E g(()) (X) with 2 # x, then kia; 2 kyaj for
all E [K]. then knaz kjag, Therefore from Assumption and Lemma 3 (an x x:) E g!a))
Analogously (x) x xz)) E g(f)) This means that if Ex, E g(())
(X) such that K; 2 ky for all E [K].
then g6 (X) S g(a)
(X). Then War at keeps topic T. for input sequence X, or X ambiguous in
Tas with respect to T..
A.6 Proof of Theorem
1. This is direct consequence from the law of large numbers. IfT oo the proportion of each token will match the probability. Since > =axx
(x) IP (x] = x). then the
probability that 3zy & ((*²) (x) such that Va! E cx(²)(x (X), the number of times x/. appears in X' is greater than the number of times x appears in X' will go to zero, and therefore the
probability of change topics will do it also.
2. Without loss of generality suppose g(3) (X) = (21, ... ,.). Clearly if we prove the (())
result assuming Vo E
(X), p= P (x = x). we will also have it for the more general
case p- = min XE (x) P (x = x).
Let X = |x|, X2, x/,]" be a random sequences generated as described in the theorem,
where the size of ((2))
(X) is l. Let k1, be the number of times x, is selected in X/- Let
18

---

A maxicisi ii and B = maxH+ssiK K4 Let P(t) = P(B, > A1). We want to prove 1] - P(t). We construct coupling between X and X by performing T independent trials. For each trial we generate uniform random variable U, in [0. and we choose tokens in X/ and X in this way:
If U, S pl both the selected tokens si,, and x "1,41 are in { f21, ;.))
If < U, S pll + 1), we select 2 21+1 if U; S pl + q क 21+2 otherwise, and we select 2,41 x1+1 where y is the probability of choosing #1+ in X/. Since > there is an interval where 24, 21+2 but 21,+1 21+1-
Uf > p(l + 1), then both the selected tokens x/,, and -2,41 are in 02, ..1). Notice that the probability of choosing 2; in X for i 2 [+ decreases because p constant.
From the previous coupling we have that K = k4 for 1 < i L, k1+1,1 S k441/41 for 1+1 1, and k 2 64,41 for 2 2+ This means that Ar+1 max(Ar, k141/411 max(Ar,441,411) 2 A and B1 = maxx h4,41 S Bt. Therefore P(I + 1) = P( > A141) S PP > A1) P(I).
B Explanation of Assumption 4
As illustrated in Figure the dataset for To and the dataset for To demonstrate interchangeability with g. and g64)) respectively.
Go
(4
64
64
Figure 6: Illustration of Assumption 4 Here are two datasets related to the TPGs, gl and g. from Figure 3 (left). From the directed arrows in gb4), we can generate dataset with the last token 64 for T., which can reconstruct back the 4) similar process applies for the g()).
C Detailed simulation studies with single-layer self-attention
C.1 Simulation process
Theoretical TPGs generation. For each token ex. L edges are randomly selected to construct the theoretical TPG Oeheo for 6 ensuring that e is involved, either a source destination node. Based on these selected edges, we add additional edges from e, to all other tokens included in edges, thereby ensuring that all tokens in 9k Kheor
be reached by 6g. Thus, obtain the theoretical TPGs {g theor 1= for Topic A. This process is repeated to generate another group of theoretical
(k)
K
TPGs (g the 1==1
for the Topic B. Let ((k)) aathoor and gbthhoo combine for each k, we obtain the theoretical TPGs for topics combinations {g(k)
J
(A)
===
KK
theor the Ik=1 k==
19

---

Training Dataset Generation. Generate training datasets DSET, and DSET, based on (9 (9tthrr J--1 respectively. For each input sequence in DSET, the sequence length Ttrain is 4, which means X = [X] X2 ... XT.vain | E TR xd with x; from E = [ei, e ee,,..e eş randomly selected as the last token and other tokens (other input tokens and the next predicted token) are chosen based on G(k) Specifically, the next token +Irain++1 is determined by sampling
(G 1K and
((k) with the weighted probability in Itheor where the weight for each token corresponds to the number of outcoming edges. Given Assumption
2 we randomly choose the position of the next token in the input sequence. Then, the remaining input tokens are randomly selected from tokens connected by incoming edges from е₁ (i.e., eş + e;) and placed in the random position within the input sequence. This process is repeated times to generate training data for each topic respectively. Empirical TPGs [g ampir Ik= and (g( 1/K are derived from the training datasets DSET, and DSET,, K k== are expected to be identical to the
(
According to Assumption 4. the empirical TPGs {
theoretical TPGs [ 1 A== for each topic. The experiments are conducted with 5000 instances, with each parameter setting evaluated over 50 epochs, consisting of 100 sequences per epoch.
Trained attention weights. We employ a single-layer attention mechanism implemented in PyTorch. The model is trained using the SGD optimizer with learning rate v) 0.01 for 8000 iterations. The training of attention weights is divided into two stages for each instance: (1) computing Wsmm for (2) get (T) each iteration for each topic. In Stage (1), prior using the CVXPY package to get W.vm SCCs are identified for each TPG derived from the using Tarjan's 's algorithm. Afterward, Wsvm is normalized to ensure consistency in subsequence computations. In Stage (2), the M Layer Attn function encapsulates the architecture of single-layer attention- based model. The training function is then used to optimize the attention weights by minimizing the loss defined in ERM Finally, the correlation between Wwm and W is calculated using the dot product.
each topic (
Next token prediction. To differentiate the input sequence length of the testing data from that of the training data, we introduce Trest- TPGs based on the training dataset DSET,, are utilized to generate test datasets consisting of 100 sequences from T, per epoch. Specifically, the last token XTe... of the test input sequence is randomly selected from K tokens (i.e. XTr..t = ex) and the remaining input tokens are randomly chosen based on the SCCs of 9% where tokens with higher priority are assigned greater weights. For instance, in G tokens e1, e2 e1 priority order e = 22 > C.. The weights assigned to input tokens e,, 62, and e4 are 0.4, 0.4, and 0.2, respectively. It reflects that e and e in the same higher-priority SCC, thus having greater weights compared to e4- Intuitively, tokens within the same SCC are more likely to co-occur than those from different SCCs. This approach enables the generated test input sequences mimic real word relationships and reflect their contextual groupings. Following the generation of the test dataset from To, the next tokens iw, and Ww.. are predicted by Equation with W. and Was obtained from the last iteration. To reduce the potential numerical issues in the outputs, S(X Wx) is rounded to three decimals, ensuring that tokens within the same SCC yield consistent softmax outputs.
e2,e2,
captured with the
C.2 Additional experiments to support Theorem
To further illustrate Theorem 2 we define the attention priority similarity of weights W' relative to W for sequence X as: Rw, w (X)
1 T-1 (S ( - - (X
- 1=1
where in, "., it is a permutation of 1, .... T such that (S (XWx)) 2... S [XW and
if 0. O.
g(w)=
otherwise.
The attention priority similarity quantifies how well the weights W' preserve the attention priority of the weights W. A value of indicates that the priority is fully preserved. Using this metric, we
O means the number of SCCs is 1 for gh, vk [K]. During the simulation, we proceed to the next instance when wwmm until reaching total of 100 instances.
'Note: W =
20

---

1.0000
Attenton priority sinii arity
0.9995
0.9990
0.9985
0.9980
0. 0.9975
0.9970
Wun |PLIT
0.9965
1000
2000
3000 4000 5000 6000 7000 8000 Iterations (X)
Figure 7: Convergence of attention priority similarity for War(r) W.(r)
[ relative W.(r)
(blue) and
[Fa(l]
[Wa(F]ll (orange).
Table Proportion of keep topic, ambiguous, and change of topic with varying Trest. (4, 8, 16, 24,32 64, 128, 256, 512}.
Trest
KEEP(%)
AMBIGUOUS($))
CHANGE(S)
4
98.60 + 1.54
1,40 = 1.54
0.00 0.00
8
98.50 + 1.47
0.96 + 1.11
0.54 0.76
98.06 + 1.33
0.54 0.76
1.40 1.07
24
98.48 + 1.31
0.24
1.26 1.10
32
98.84 + 1.15
0.33
1.04
64
99.10 1.07
0.20
0.86
99.64 0.53
0.14
0.54
256
99.98 + 0.14
0.00
1.02
512
100.00 + 0.00
0.00 0.00
0.00 0.00
conduct experiments, with results in Figure 7 We generate embeddings with K 10 and d 16, and randomly construct TPGs for Ta and Ty Using these TPGs, we randomly generate DSET, Wafr)
and DSET, We compute w.fr) F [[,(F]]l and Wasfr)
[Was(F]]e using the same procedure as Li et al. 25 We generate test sequences Z within Ta. and we calculate the attention priority similarity of Was(T)
[Wa(F)]l relative to both (z(z)
[F][] and FF We repeat this process for multiple TPGs and clearly demonstrates that the similarity .w.rr)
W.(r)
input sequences (simulation details in Appendix C. Figure 7
converges to 1 after 7 8000 iterations when evaluated relative to [Wa(r]]l (blue line), but fails to
Wz(r) (orange line). These observations align with the results of Theorem 2
converge relative [[.(T)
C.3 Simulation in Section
In Figure (aa) we predict next tokens for 5000 test sequences from T, with Trest {4,8, 16, 24, 32, 64, 128, 256, 512), while fixing L 4, d 16, Ttrain proportion of each scenario with varying T is illustrated in Table For Figure ( we predict next tokens for 5000
and K 10. The
sequences (the sequence length is Trest 20) using models trained with L == {4. 6, 8, 10, 12, 14. 16, 18), d = 16, K = 10. and Tirain = 4. The proportion of each scenario with varying illustrated in Table
C.4 Additional experiments for convergence in mixed topics
Building upon the convergence experiments in Li al. 25 our work demonstrates that the correlation ab
IP) (green lines) in Figure 8 measured with varying K = (6. 10, 14) and L = {8, 12, 16), approach to 1. These results indicate that Theorem ] extends beyond individual topies to also capture the convergence in mixed-topie scenarios, albeit with relatively slower convergence. In these experiments, we fix Tirain = and d = 16. Each point represents the average over 5000 randomly generated instances, trained with 8000 iterations. The shaded area around each line represents the 95% confidence interval, computed over 50 epochs.
coefficients (Was (T), Wmvm) / as (T) smm
ab
21

---

Table Proportion of keep topic, ambignous, and change of topic with varying L {4,6,8 8, 10, 12, 14, 16, 18).
L
KEEP
AMBIGUOUS(%)
CHANGE %)
4
98.22 + 1.43
0.26 0.60
1.52 + 1.31
6
98.30
1.37
0.50 0.68
1.20 = 1.11
8
98.18
1.49
0.68
1.14 1.23
10
98.42
1.25
0.76
0.82 + 1.02
12
98.14 + 1.32
0.82 0.92
1.04 + 0.97
14
97.96
+ 1.44
1.24 + 1.06
0.80 + 0.86
16
98.28
1.33
0.98 0.91
0.74 + 0.85
18
98.02
1.58
1.26 1 1.14
0.72 0.86
1.0
Coree Codffcoot
FXTE
RXTT
KGl BOSTE
0.0
0.
1800 2000 3005 1000 5000 600D 7000 8000
1000 0
2002 3000 4000 5000 0920 7000 8000
1800 2000 3000 4200 3000 6000 7006 8000
Beraterejjr
(a) K 6, -=
(b) K 10
(c) K 14, I= =4
1.0
Corresttoon Cooff 0012
20-00
1000 2000 3000 4200 6000 1000 n00o
1000 2020 3000 5000 2000 T0oo 8000
1000 3000 4002 5200 noce 7000 8000 Itanal ensid)
Beratoonct)
(d) K -
(c) K = L 12
(f K = 10, 16
Figure 8: Convergence W.(r)
(r [r) (blue),
Wa(r) [W. (orange), and
W [ (green) for varying K
W Wab(r)
and I, with fixed T Train and 16.
C.5 Numerical analysis for each scenario in Figure
Figure provides a numerical breakdown for each scenario Figure In Figure each distinct color corresponds unique token within the input sequence X, which consists of 4 tokens. e4 is the last token across all three input sequences. For each input sequence X, we apply Wa(7) and W (Tb(T) with 8000 to predict the next token, yielding iw. and fw... respectively.
Let [S(X (IXWa(T)s (7)x)); = a, and [S(XW (T)))); = b, for i E [T]. Following Equation and
Equation 10 we compute (C))). Ir. and ((/))(()(()))). and predict the next token each input sequence.
get the highest probability SCC
Topic continuity. In Fig. 9a input sequence X consists of four unique tokens: e5,
and e Based g( in Figure (left), the priority order of these tokens is O₁ C3 e₁ C4. with corresponding values: 0.45 > 0.25 > 0.20 > 0.1. Since [C]le = x 0.45 is the
largest, () {es} and /w. = es- the mixed-topic scenario, Was preserves the attention priority but e₃ and e₁ have the same priority: es > C3 e₁ C₁, with corresponding b, values: 0.40 > 0.30 > 0.15 = 0.15. Token e3 is still with the highest probability to be chosen, as x 0.40. Following the Definition 3 Wab keeps topic for the the input sequence
(r)
X les, e e3, e.]T.
Ambiguous sequence. Input sequence X in Fig. 9b has two unique tokens: e₁ and C4. The priority order is e > e₁. following c( in Figure (left). The corresponding values aj ag 0.3
22

---

(a(r) Trained
( (
SCAMP
(C
5, 000
5. 000
sequuence
Ave.
sequuence Fy
Frc.
(138)
E
000
[ (0
War(s) Trained in U.T
War(() Trained Tr UT,
G (9())
De GG (A))
Ragee condmuity
(11)) (00 the (1()) Ambiquous asquence
Topic continuity.
(b) Ambiguous sequence.
BXT
(+)4))
TABTT
(euuce 00)
Imput aequence T.
Pkc,
Ray (ox (13))
fh 0011
Wa (t) Trained In
(900)
Foras
(")) and 5. (1ig) (0())
(c) Change topic.
Figure 9: Numeric details for each scenario: (a) topic continuity, (b) ambiguous sequence, and (c)
change of topic.
and 02 0.2. Then (Cfw.,()) (X)le; 2 0.30 and [Cfw..r)(X)ee
0.20. Thus,
((4) is {es} with the highest probability. Wab makes and e with the same priority, indicated by g!4) in Figure (left). Both e₁ and e₁ are within the highest probability SCC, g(d) due to [Cfw (r) (X)]e, [Cfw..()))X (X) = 2 0.25. Although ywas gl go ((4) Therefore, the sequence X (ei, e4, e1, e4] ambiguous, based the Definition
Change topic. For the input sequence in Fig. Dc the only two unique tokens, e; and e₁, are with the same priority order in both g( and g(4) from Figure 3 (left): > With W (r) trained T.. the token eş has d = 0.70 and the token e4 has 02
0.70 0gg = a4 0.10. Obviously, [C.w (x),()))))l, > [Cffw, (X))) = 3 00 Thus, (44) consists of es. However, (44) consists of e4 instead of e5, due to 1 x 0.40 = [C. (X)) < [C (X)]e, = 3 x 0.20. Since
fw.. & ((41 and ((4) z g(4) W.. Wa changes topic for the input sequence X. Moreover, we have (es c.) e for [4], shown in 3igure (left). Thus, the highest priority SCC (Definition) 1 in Tos is g( (X) {es}. In the input sequence X [es, eạ, e4, ei] the lower-priority token ea g()) (X) appears more frequently than the higher-priority token es E ((1)) (X), illustrating our Theorem
23

---

D Experimental details in Section 6
In this section, we provide the experimental details in four LLMs: GPT-4o, Llama-3.3, Claude-3.7, and DeepSeek-V3. Here, outline the general procedure used in each model, under identical parameter settings, to generate continuations for each segment of the abstract.
Extract the first T words from paper A's abstract as the input segment X from Topic A.
2. Randomly select 5 papers different from paper A, as papers {B,
3. For the input segment X, apply RAG extract top relevant excerpts (chunks) from paper
A as the the knowledge A, denoted as Refa. Each chunk has 800 tokens length.
4. Similarly, retrieve top 3 relevant excerpts from paper B, the knowledge B,, denoted as Refg.-
5. Combine the knowledge from Topic A and from Topic B, as the knowledge AB; for mixed Topics, denoted as RefAB,.
6. For the input segment X. promot each LLM with Prompt, and Prompt to generate the continuations as 9w, and .w respectively. Notably, the only difference between Prompt, and PromptAn; is the reference excerpts provided RefA or RefAB, All LLMs are set with a temperature of 0 to match the greedy decoding in our theoretical framework. The maximum completion length was set to 1000 tokens to ensure that the generated continuations could complete the abstract.
(a) Promptx:
Here are some relevant excerpts from research paper(s) reference:Ref, Below is the 1tt fragment of an abstract from arXiv paper A: X. Please continue the 2nd fragment of the abstract based on the relevant excerpts without including the given content in the output.
(b) PromptAn,
Here are some relevant excerpls from research paper(s) reference:RefAs: Below the 1st fragment of an abstruct from arXiv paper A: X. Please continue the 2nd fragment of the abstract based on the relevant excerpts without including the given content in the output.
7. Calculate the average cosine similarity between iwa and jWab across five pairs of paper and paper B,.
D.1 Impact of input length
To investigate the impact of the input length, we vary T = {10, 30, 50, 70, 90, 110, 130, 150] for every paper Topic A. increasing the length of the input segment X extracted from the abstract of paper A, as shown on the x-axis from Figure 5a
D.2 Impact of topic ambiguity
We quantify topic (paper) ambiguity by computing the average similarity among each paper's keywords. Since arXiv papers do not provide keywords, we use Llama-3.3 to generate four keywords for each paper prior to generating continuations with the LLMs.. To investigate the topic ambiguity, we fix the input length with T 80 for every paper as Topic A and order papers by the average keywords similarity, shown on the x-axis of Figure 5b A higher keywords similarity corresponds to lower topic ambiguity.
D.3
Additional experiments
D.3.1
Extended input length
To further examine the impact of input length, we randomly select 50 out of 100 arXiv papers
introduced in Sec. 6 and extend the input length with the first 310, 460, ...., 1210 words from the introduction of each paper A as the input prompt. We start from 310 because shorter introductions provide insufficient contextual information to reliably extract relevant excerpts from the
24

---

full paper when using RAG. As shown in Figure 10 the average cosine similarity generally increases with the more introduction content from paper with the exception of DeepSeek, which exhibits only marginal improvement.
0.950
0.925
Cosine Similarity
0.900
0.875
0.850
0.825
0.800
0.775
GPT-4o
Llama-3.3
0.750
Claude-3.7
DeepSeek
310
460
610
760
910
1060
1210
Number Input Words
Figure 10: Extended length input sequences.
D.3.2 Alternative measure of ambiguity
As alternative approach validate results, also measure the ambiguity based the similarity between the keywords of paper A and those of paper B. Following the same setup, we rank the cosine similarity for each paper A by the increasing similarity between its keywords and those of five papers B, where higher level corresponds to a greater ambiguity. As shown in Figure [1 our results still hold since the cosine similarity remains relatively stable as the ambiguity level increases.
E Computational resources for experiments
In our simulations based on the single-layer self-attention model, each group of parameter setting requires hours to train two models separately, one for single input topic and one for mixed topics, followed by additional hours for next-token prediction.
1.0
Cosine Similarity
0.9
0.8
0.7
GPT-40
1lama-3.3
Claude-3.7
DeepSeek-V3
0.6
Level 1 Level 2 Level Level 4 Level 5
Keywords Similarity (Between Different Papers)
Figure 11: Topic ambiguity measured by the similarity between the keywords of paper A and those
of paper B. Higher keywords similarity indicate greater topic ambiguity between the two papers.
25

---

In our experiments on LLMs, we query GPT-4o, Llama-3.3, Claude-3.7, and DeepSeek- V3 through API calls. All experiments were conducted on standard laptop without specialized hardware. For each I.M,, the full process, including selecting relevant excerpts using RAG and generating continuations, requires approximately hours of runtime, with total of 30 million input tokens and 5 million output tokens. The total API usage cost for the experiments is approximately 350 USD.
F
Impact statement
Our investigation highlights fundamental differences between spontaneous topic changes in LLMs and spontaneous human thought, informing the development more natural and flexible Al systems in domains such as customer service and mental health support. However, improving such capabilities can raise ethical considerations, including inadvertent manipulation user focus, especially in persuasive sensitive contexts. Our work, while largely theoretical, emphasizes the importance of fairness, privacy, and user autonomy as developers refine these systems to serve users' interests, respect contextual boundaries, and remain accountable. This research has the potential to advance both Machine Learning and Human- Computer Interaction by informing new architectures that mimic human-like topic shifts; nevertheless, any real-world application of these findings should be accompanied by vigilant oversight to mitigate risks of misuse such as deceptive or manipulative dialogue shifting. There are many other potential societal consequences of our work, none which feel must be specifically highlighted here.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The fifth paragraph and Subsection I.] Introduction accurately reflect the paper's contributions, scope, assumptions and limitations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We introduce assumptions in Introduction and we have a subsection of limita- tions in Section 8 including the lack of theoretical framework for complex architectures, alternative cost functions, and diverse training objectives.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
27

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Building on Assumptions and from Theorem in Li [25], and
incorporating our new Assumptions and we present Theorems and 4 All proofs are provided in Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide full disclosure of the implementation details to reproduce experi- mental results. Section [nd Appendix Ciprssent the simulation details and results based on the single-layer self-attention model, while the experiments modern LLMs are detailed in Section [ and Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
28

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All code and data for reproducibility is provided as supplementary material. Upon acceptance we will also provide public GitHub repository.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
sublic/guidess htaaai//guiiee//ddeeuubiissssss
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https Sonpddeee///oSSSuummsaaea
stt.suubbmiisssoooooooooo.. for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We fully disclose the experimental setup in Section and the corresponding details in Appendix C.I for the single-layer self-attention model. The experimental setup and details for modern LLMS are provided in Section 6 and Appendix respectively.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
29

---

Justification: Figure 5a shows the 95% intervals of the average cosine similarity as the imput length increases. Appendix C Table and 2 the 95% intervals for the percentage of each scenario. Additionally, Figure shows the 95% confidence interval for the attention priority similarity, and Figure g presents the convergence of W with corresponding 95% confidence intervals.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Appendix E provides details on the computational resources used in our experiments, including execution time, API usage costs, and the total number of input and output tokens.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Our work complies with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
30

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Appendix we discuss the social impact of our work from both potential positive and negative impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymenn of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-us should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
31

---

Justification: Our work is conducted in Python with several open-source Python libraries. For generating continuations in our experiments, we access the following publicly released LLMs via API: GPT-4o 18 Llama-3.3 15]. Claude-3.7DD and DeepSeek-V2. V3neek-33333 30
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, wwpprsai thcode. com /daaaset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide detailed description of the dataset used for the single-layer self- attention model in Appendix C.I and the dataset used for the modern LLMs Appendix D
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
32

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work doesn't 't involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: We use LLM APIs, including GPT-4o, Llama-3.3, Claude-3.7, and DeepSeek- V3, to generate the continuations of the input segment. Section 6 and Appendix provide all experimental results and detailed process related the frontier LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non standaad components.
Please refer to our LLM policy (https: http::://neurppi cc/Confeeeeces//22222 /LLM) for what should or should not be described.
33


Reviews:
[{"review_id": "LWEQbVgmt2", "reviewer": "Reviewer_L1rU", "summary": "This paper presents a theoretical and empirical study of spontaneous topic changes in self-attention models, contrasting their behavior with the dynamics of spontaneous human thought. The authors develop a formal framework by defining a \"topic\" as a set of Token-Priority Graphs (TPGs) within a simplified single-layer self-attention model. This approach leads to several key findings that distinguish the mechanisms of large language models (LLMs) from human cognition: 1) Preservation of topic priorities: The model preserves the priority order of tokens related to an input's original topic, even when trained on a mix of different topics. 2) Conditions for topic change: A topic shift is shown to occur only under a specific condition: a lower-priority token must appear more frequently in the input than all higher-priority tokens associated with the current topic. 3) Impact of context and topic ambiguity: Contrary to human thought, where longer discussions or ambiguity can trigger new associations, the paper establishes that longer input contexts decrease the likelihood of a topic change in LLMs. Similarly, greater ambiguity between topics acts as a stabilizing factor rather than a trigger for spontaneous shifts", "questions": "- Clarity of definitions: Some symbols are used without definition. For instance, $\\mathbb{Z}$ in Assumption 3.\n\n- Justification for the topic ambiguity metric: In Section 6, the authors mention that a higher keywords similarity between two papers corresponds to lower topic ambiguity. This is counterintuitive as a higher overlap in the keywords of two papers generally implies higher topic ambiguity.", "limitations": "Limitations and potential societal impacts are discussed in detail.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n- The paper is well-motivated by the goal of theoretically understanding the distinctions between LLMs and human cognition.\n\nWeaknesses: \n- Limited discussion of literature on topic shifts: While the authors mention several papers on spontaneous topic changes in LLMs in the introduction part, a more detailed discussion comparing the proposed theoretical framework against existing work is needed. \n\n- Flawed application of prior theoretical results. The paper's theoretical claims are heavily dependent on Theorem 1, which is adapted from Theorem 2 in Li et al. (2024). However, there appears to be a critical misunderstanding in applying the theorem, which invalidates the subsequent proofs. \n  - Informally speaking, the work by Li et al. establishes that the attention weight $W$ converges to a structure that can be decomposed into two components, i.e., $W \\approx C \\cdot W^{svm} + W^{fin}$ with $C \\to \\infty$. The two components have different functionality: 1) Hard retrieval ($W^{svm}$): This term selects the high-priority input tokens according to the token-priority graph. 2) Soft composition ($W^{fin}$): A finite component that governs how to assign probabilities among the high-priority tokens selected by $W^{svm}$. \n\n   - However, in this paper, the authors only discuss the impact of $W^{svm}$, ignoring the role of $W^{fin}$. This omission is critical as it leads to incorrect conclusions. E.g., Lemma 1 builds an equivalence, stating that tokens have the same prediction probability if and only if they are in the same strongly connected component (SCC) in the token-priority graph. This is not true without knowing $W^{fin}$ which can (and generally will) assign different probabilities to tokens even if they are in the same SCC.\n\n\n[1] Yingcong Li, Yixiao Huang, Muhammed E Ildiz, Ankit Singh Rawat, and Samet Oymak. Mechanics of next token prediction with self-attention. In International Conference on Artificial Intelligence and Statistics, pages 685–693. PMLR, 2024.", "was_revised": true, "final_justification": "Most of my concerns have been resolved so I'm happy to raise my score from 2 to 4 and recommend for acceptance.", "rebuttal": "Thank you for your comments.\n\n**Weaknesses**\n1. We would like to begin by clarifying that we are confident in the correctness of our theoretical results under the assumptions we state, (and those from Theorem 2 in Li et al). We believe the confusion may stem from the perception that our analysis should involve the behavior of  $W_{fin}$, but let us explain why this is not needed for our analysis.\n\nTo clarify:\n\nThe core idea behind Theorem 2 of Li et al. (and its informal version in their Theorem 1) is that self-attention can be decomposed into a *hard retrieval* component ($W_{hard}$ or $W_{svm}$) which selects the highest-priority tokens, and a *soft composition* component ($W_{soft}$ or $W_{fin}$) which determines the probabilities among the selected tokens. Our analysis focuses specifically on the hard retrieval mechanism  ($W_{hard}$ or $W_{svm}$), that is: which tokens are selected as high-priority. Our three key definitions (*keep topic*, *ambiguous sequence*, *change of topic*) are based entirely on this component, therefore the analysis on $W_{fin}$ is not needed. We emphasize $W_{svm}$ in particular to preserve the conceptual parallel with the graph-based prioritization of ideas observed in models of human cognition.\n\nIn particular:\n\n- For our Theorem 2, our results concern the preservation of the highest-priority tokens in a combined-topic setting. Since this theorem is on weights $\\tilde{W}$ which comes from $W_{svm}$ (*hard retrieval*, no need of $W_{fin}$), the behavior of $W_{fin}$ is irrelevant to the proof. Our conclusions remain valid.\n\n- For Theorems 3 and 4, the definitions (*keep topic*, *ambiguous sequence*, *change of topic*)  depend solely on the highest-probability SCC which is based on the *hard retrieval* component. Again, no dependence on $W_{fin}$  is required, and the results are unaffected.\n\n- Lemma 1 applies to $\\tilde{W}$ which comes from $W_{svm}$ (*hard retrieval*, no need of $W_{fin}$). We agree the conclusion of the lemma could be misleading and we will change line 144 to \"This means that the tokens that maximize the probability for weights $\\tilde{W}$ in Equation 1 are all within the same SCC\".\n\n- Our formal proofs are provided in detail in the appendix, and the accompanying numerical simulations illustrate and support their validity.\n\nWe appreciate this comment and will revise the paragraph on line 133 to clarify that our focus is on the *hard retrieval* component of Li et al., as the *soft composition* step is not required for our definitions or theoretical results. \nWe believe this may be the primary reason behind the reviewer's lowered rating and we trust this explanation resolves the misunderstanding and confirms the validity of our contributions.\n\n2. Regarding the comment on the limited discussion, we thank the reviewer for the suggestion. While numerous theoretical frameworks exist for analyzing topic shifts in text, to the best of our knowledge, this is the first work to formally connect spontaneous topic changes in large language models (LLMs) with analogous phenomena in human thought, which is why we focused the discussion on this aspect.\n\n**Questions**\n\n- *Justification for the topic ambiguity metric: In Section 6, the authors mention that a higher keywords similarity between two papers corresponds to lower topic ambiguity. This is counterintuitive as a higher overlap in the keywords of two papers generally implies higher topic ambiguity.*\n\nWe believe there may be a misunderstanding. In Section 6 (line 309), we quantify topic ambiguity by keyword similarity within a **single** paper, not keywords from two or more papers. For instance, a higher keywords similarity within Paper A indicates a more coherent topic in Paper A, leading to less probability of overlap between paper A and others and thus implying the lower topic ambiguity for Paper A. Put simply, if a paper’s keywords are very diverse, it suggests that the paper spans multiple topics, increasing the likelihood of overlap with other papers and thus leading to greater ambiguity. We will clarify this explanation in the revision to avoid potential confusion.\n\n- *Clarity of definitions: Some symbols are used without definition. For instance, $\\mathbb{Z}$ in Assumption 3.*\n\nThank you for pointing this out. We tried to use standard notation, here $\\mathbb{Z}$ is the standard symbol for a set of integers.\n_______________\n\nTo conclude, we believe we are pioneering an underexplored area by drawing parallels between spontaneous human thought and spontaneous topic changes in self-attention models. To the best of our knowledge, this is the first study to examine these questions by investigating key similarities and differences between human cognition and AI models in the area of spontaneous thought. We view this foundational research as highly relevant to NeurIPS' mission of advancing meaningful, thought-provoking exploration in machine learning."}, {"review_id": "h8qF5ECzas", "reviewer": "Reviewer_V5wh", "summary": "This paper investigates spontaneous topic changes, i.e., abrupt, unprompted shifts in conversational topic, within self-attention-based language models and reveals their differences from spontaneous thought patterns seen in human cognition. The authors define a formal framework using token-priority graphs (TPGs) to model topic representation and transition dynamics in simplified self-attention settings. Their results indicate that self-attention-based language models rarely change topics spontaneously unless specific statistical conditions are met, such as when low-priority tokens dominate. Contrary to human behavior, longer inputs or higher topic ambiguity reduce the likelihood of topic change. Experiments on modern LLMs (e.g., GPT-4o, Llama, Claude) empirically support these theoretical findings.", "questions": "Please see the weaknesses in the \"Strengths And Weaknesses\" section.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n1. The paper explores an interesting and novel research question: the spontaneous topic change within LLMs and how it differs from human spontaneous thought. \n2. A rigorous mathematical framework is developed using TPGs and SVM-based analysis to explore the conditions under which spontaneous topic changes can or cannot occur in self-attention models.\n3.  Experimental results on real-world LLMs (GPT-4o, Claude-3.7, LLaMA-3.3, and DeepSeek-V3 ) empirically confirm the theoretical predictions. \n\n**Weaknesses**:\n1. Despite the paper tackles an interesting phenomenon: the spontaneous topic change within LLMs, it lacks a clear justification for its practical significance, leaving readers uncertain about why understanding or modeling such behavior in LLMs is important for real-world applications or cognitive alignment. \n\n2. The condition under which a topic change occurs: when a lower-priority token appears more frequently than all higher-priority tokens, is theoretically well-defined but practically hard to verify, since the paper’s formalism relies on constructing token-priority graphs that are not directly accessible or interpretable in real-world settings. Moreover, the paper does not provide evidence or discussion on how frequently such conditions arise in realistic language model usage.\n\n3. The paper focuses on analyzing the conditions for spontaneous topic change but does not explore potential applications of its findings, such as methods to encourage or suppress such behavior in language models for specific tasks.", "was_revised": true, "final_justification": "Considering the limited practical utility of the proposed method as currently presented, I will maintain my current score.", "rebuttal": "Thank you for comments. See here our reply:\n\n- *Despite the paper tackles an interesting phenomenon: the spontaneous topic change within LLMs, it lacks a clear justification for its practical significance, leaving readers uncertain about why understanding or modeling such behavior in LLMs is important for real-world applications or cognitive alignment.*\n\nOur research seeks to understand both the similarities and the differences between AI and human cognition. \nAlthough its immediate value is not an engineering deliverable, the work gives preliminary steps towards answering a foundational question: what truly distinguishes human thought from AI? Cognitive science characterizes spontaneous thought as a distinctly human trait. Neuroscientists model spontaneous thoughts as traversals across conceptual graphs, where the mind jumps between related concepts or topics. Our framework mirrors this process, aiming to explore what drives spontaneous topic changes in LLMs. Our aim is to draw a parallel between spontaneous topic changes in LLMs and spontaneous human thought, examining the phenomenon from both theoretical and empirical perspectives.\n\nSpecifically, understanding that longer inputs reduce the probability of topic changes has practical significance. Especially for the long-form question answering (multi-paragraph in response) for scientific or technical tasks, providing longer input contexts helps reduce unintended topic changes, leading to more coherent outputs.\n\nTo the best of our knowledge, our work is among the first to address this question through a combination of theoretical analysis and experimental validation.\n\n- *The condition under which a topic change occurs: when a lower-priority token appears more frequently than all higher-priority tokens, is theoretically well-defined but practically hard to verify, since the paper’s formalism relies on constructing token-priority graphs that are not directly accessible or interpretable in real-world settings.*\n\nThank you for acknowledging our well-defined claim. Our framework is based on Token Priority Graphs (TPGs), where a topic is defined as a set of TPGs. While this doesn't map one-to-one with real-world topics, it offers valuable insights. For instance, one could interpret a topic as a graph of ideas, where certain ideas consistently hold higher priority (a perspective aligned with how some neuroscientists model human thought). Under this view, our condition that “a lower-priority token appears more frequently than all higher-priority tokens” can be seen as a lower-priority idea dominating discourse. This remains a hypothesis, but it opens several promising avenues for future research.\n\n- *The paper focuses on analyzing the conditions for spontaneous topic change but does not explore potential applications of its findings, such as methods to encourage or suppress such behavior in language models for specific tasks.*\n\nFollowing up with our previous comments. Our work focuses on the simplified single-layer self-attention architecture. This controlled setting allows us to formalize the dynamics of spontaneous topic changes and draw parallels between human cognition and LLMs. While we don’t explore potential applications and frequency in real-world LLMs, our findings lay an important groundwork for future directions.\n___________\n\nTo conclude, we believe the reason for not receiving a higher rating may reflect some hesitation to view our contribution beyond its practical engineering implications. Our work represents foundational research aimed at addressing non-traditional questions that extend beyond typical engineering concerns. We see this direction as highly aligned with NeurIPS’ mission of fostering meaningful, thought-provoking exploration in machine learning."}, {"review_id": "9SiYEGX0p6", "reviewer": "Reviewer_2tA4", "summary": "This paper aims to propose a theoretical model of ‘spontaneous topic changes in autoregressive self-attention models, motivated by observation that spontaneous topic change as a fundamental cognitive feature in humans while LLMs topic change is rather initiated by contextual cue. Using a concept of topic priority graphs (TPGs), they propose a model of such topic change in a simplified 1-layer self-attention language model and provide empirical results on the frontier LLMs that capture the prediction of their model about input length and topic ambiguity.", "questions": "Line 310: isn’t it the opposite; lower similarity of the topics implies less overlap between A and B?\n\nI thing Figure 5 a is supporting the authors claim rather weakly; have you increased the context length further to see the effect better? Does your theory provide any tendency of the topic change probability change (linear, or sudden change)?\n\nHave you considered other metric of topic similarity, which might be able to capture more intricate relationship? (eg. hierarchy)", "limitations": "The theory is limited to a simple setup but the authors already mention it. \n\nWhile authors motivate the work with the dissimilarity between human cognition and LLMs in topic change, I do not see enough evidence of it. \n\nThe definition of similarity seems to be limited - what about hierarchy within topics or high dimensional topics?", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths:\n\nI think it is valuable to make a quantitative approach based on a simplified model to capture and describe phenomena in many complex models. Also, the authors provide the experiments with the frontier models at scale which captures some aspect of the simplified model, which validates their theory.  \n\n## Weaknesses:\n\nThe biggest weakness of the paper is clarity. I found the use of notations hard to follow and it was hard to follow the path to the ultimate end prediction about input length and topic ambiguity. \nThe visual illustration of the model is unclear; for example, it is not clearly described what is the direction of the edges and what is the dotted square. I inferred that it’s a label to input and SCCs but it took a lot of effort for me. Having explicit description of the figure would improve readability. \n\nWhile authors motivate the study by claiming the difference between human cognition and LLM on spontaneous topic change, I think its basis is not very well supported.", "was_revised": true, "final_justification": "Authors addressed the questions well, which led to an increase in score from 3 to 4. I appreciated the additional experiments on longer context, and I trust that the authors will update the corresponding bits of the paper for clarity + discussion of future work.", "rebuttal": "Thank you for acknowledging the value of our quantitative approach and the inclusion of real LLMs experiments. \nLet us first explain the basis of our motivations:\n\nOur research seeks to understand both the similarities and the differences between AI and human cognition. In the Introduction (line 22), we reference prior work on the area of spontaneous thought, which implies that spontaneous thought is natural and frequent for humans. Neuroscientists model spontaneous thoughts as traversals across conceptual graphs, where the mind jumps between related concepts or topics. Our framework mirrors this process, aiming to explore what drives spontaneous topic changes in LLMs. Our aim is to draw a parallel between spontaneous topic changes in LLMs and spontaneous human thought, examining the phenomenon from both theoretical and empirical perspectives and giving preliminary steps towards answering a foundational question \"what truly distinguishes human thought from AI?. To the best of our knowledge, our work is among the first to address this question through a combination of theoretical analysis and experimental validation.\n\nNow let us clarify some of your questions that will help supporting our conclusions:\n\n- *Line 310: isn’t it the opposite; lower similarity of the topics implies less overlap between A and B?*\n\nWe believe there may be a misunderstanding. In Section 6 (line 309), we quantify topic ambiguity by keyword similarity within a single paper, not keywords from two or more papers. For instance, a higher keywords similarity within Paper A indicates a more coherent topic in Paper A, leading to less probability of overlap between paper A and others and thus implying the lower topic ambiguity for Paper A. Put simply, if a paper’s keywords are very diverse, it suggests that the paper spans multiple topics, increasing the likelihood of overlap with other papers and thus leading to greater ambiguity. We will clarify this explanation in the revision to avoid potential confusion.\n\n- *I thing Figure 5 a is supporting the authors claim rather weakly; have you increased the context length further to see the effect better?*\n\nWe appreciate this comment. We intentionally kept the context length small for simplicity, but our experimental setup can be readily extended to longer contexts, further supporting our claims. In fact, below we provide a summary of real experimental results using GPT-4o with input lengths ranging from 0 to 1000 tokens. While we are unable to include figures, we hope the upward trend remains evident from the data provided.\n\n| Input words | Cosine similarity |\n|---|---|\n| 10 |  0.877 |\n| 210 | 0.915 | \n| 410 | 0.919 |\n| 610 | 0.927 |\n| 810 | 0.926 |\n| 1010 | 0.935 | \n\nWe can easily extend Figure 5a to 1000+ words in the final manuscript with all LLMs.\n\n- *Does your theory provide any tendency of the topic change probability change (linear, or sudden change)?*\n\nOur theoretical results show that the probability of topic changes vanishes as the input sequence length increases sufficiently and does not increase as the topic ambiguity increases. Our goal in this theorem is to show the difference between LLMs and human in spontaneous topic changes. We didn’t provide the specific tendency of the probability, but we thank you for your thoughtful question. \n\n- *Have you considered other metric of topic similarity, which might be able to capture more intricate relationship? (eg. hierarchy)*\n\nAs an initial step, our current framework aims to address when and why self-attention models preserve the topic or spontaneously change to another. While our work doesn’t model topic hierarchy, your insightful question offers a promising direction for future work, bridging spontaneous topic changes with hierarchical semantic structures. \n\n- Comments on visual illustration not being clear\n\nAs correctly inferred, the direction of edge is from output to input (see line 114), and the dotted square is SCC. We acknowledge our unclear visual illustration and thank you for pointing this out. We’ll clarify these details in the caption to improve readability. \n___________________\nTo conclude, we believe the reason for not receiving a higher rating may reflect some hesitation to see how our experiments and conclusions support our core motivation. We hope that our clarification on the notion of ambiguity, along with the additional validation using longer input lengths, helps reinforce the validity of our main conclusion: *unlike human cognition, increasing context length or input ambiguity does not make spontaneous topic change more likely in LLMs*. This insight opens a path toward addressing the broader question: How does human cognition differ from AI? We see this direction as closely aligned with NeurIPS’ mission to promote meaningful, thought-provoking exploration in machine learning."}]

Author rebuttal:

Meta-review: This paper examines spontaneous topic changes in self-attention models and compares them to the dynamics of spontaneous human thought. The reviewers agree that the topic is interesting and that the results are solid. However, they also note that the theoretical analysis is conducted in a highly simplified setting, which may limit the practical impact of the findings. After discussion, the authors have satisfactorily addressed several previous concerns raised by the reviewers. All reviewers now support accepting the paper. I concur with the reviewers’ assessments and recommend acceptance.
Decision: Accept (poster)