StreamingThinker: Large Language Models Can Think While Reading
Authors: Junlong Tong
Yingqi Fan
Anhao Zhao
Yunpu Ma
Xiaoyu Shen
Venue: ICLR 2026 Poster (2026)
Keywords: LLMs
Reasoning
Streaming
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose StreamingThinker, a framework that enables LLMs to think while reading.

Abstract: Large language models (LLMs) have demonstrated remarkable capabilities in chain of thought (CoT) reasoning. However, the current LLM reasoning paradigm initiates thinking only after the entire input is available, which introduces unnecessary latency and weakens attention to earlier information in dynamic scenarios. Inspired by human cognition of thinking while reading, we first design a **streaming thinking** paradigm for LLMs, where reasoning unfolds in the order of input and further adjusts its depth once reading is complete. 
We instantiate this paradigm with *StreamingThinker*, a framework that enables LLMs to think while reading through the integration of streaming CoT generation, streaming-constraint training, and streaming parallel inference.
Specifically, StreamingThinker employs streaming reasoning units with quality control for CoT generation, enforces order-preserving reasoning through streaming attention masks and position encoding, and leverages parallel KV caches that decouple input encoding from reasoning generation, thereby ensuring alignment and enabling true concurrency. We evaluate StreamingThinker on the Qwen3 model family across math reasoning, logical reasoning, and context-based QA reasoning tasks. 
Experimental results show that the StreamingThinker preserves performance comparable to batch thinking, while yielding an 80\% reduction in token waiting before the onset of reasoning and a more than 60\% reduction in time-level latency for producing the final answer, demonstrating the effectiveness of the streaming paradigm for LLM reasoning.
Code is publicly available at [this repository](https://github.com/EIT-NLP/StreamingLLM/tree/main/StreamingThinker).

Published as a conference paper at ICLR 2026
STREAMINGTHINKER, LARGE LANGUAGE MODELS CAN THINK WHILE READING
Junlong Tong Yingqi Fan? Anhao Zhao Yunpu Ma
Xiaoyu Shen
Shanghai Jiao Tong University Eastern Institute of Technology, Ningbo
NNngbo Key Laboratory of Spatial Intelligence and Digital Derivative, Institute of Digital Twin 'Hong Kong Polytechnic University 'Munich Center for Machine Learning, LMU jl il-tongesjtu edw.u.u..uuuuuuu......mmo edu.cn xyshen@eitech. edu. cn
ABSTRACT
Large language models (LLMs) have demonstrated remarkable capabilities in chain of thought (CoTT) reasoning. However, the current LLM reasoning paradigm initiates thinking only after the entire input is available, which introduces unneces- sary latency and weakens attention to earlier information in dynamic scenarios. In- spired by human cognition of thinking while reading, we first design a streaming thinking paradigm for LLMs, where reasoning unfolds in the order of input and further adjusts its depth once reading is complete. We instantiate this paradigm with StreamingThinker, through the integration of streaming CoT generation, streaming-constranl ing, and streaming parallel inference. Specifically, StreamingTThine Thinker employs streaming reasoning units with quality control for CoT generation, enforces order- preserving reasoning through streaming attention masks and position encoding, and leverages parallel KV caches that decouple input encoding from reasoning generation, thereby ensuring alignment and enabling true concurvency. We evalu- ate Streaming Thinker on the Qwen3 model family across math reasoning, logical reasoning, and context-based QA reasoning tasks. Experimental results show that the StreamingTThinn Thinker preserves performance comparable to batch thinking, while yielding an 80% reduction in token waiting before the onset of reasoning and a more than 60% reduction in time-level latency for producing the final answer, demonstrating the effectiveness of the streaming paradigm for LLM reasoning. Code is publicly available at this repository.
framework that enables LLMs to think while reading train-
INTRODUCTION
Large language models (LLMs) have shown impressive reasoning capabilities, as exemplified by systems like OpenAl-ol (Jaech et al., 2024) and DeepSeek-RI (Guo et al., 2025). Yet, most current approaches follow batch thinking paradigm, in which reasoning begins only after the entire input context has been received. This paradigm is problematic in scenarios that demand timely responses or dynamic information processing. First, waiting for the full input before reasoning introduces unnecessary latency. Second, as the input increases, attention earlier information becomes diluted due to the growing disconnect between reasoning steps and their relevant context (Liu et al., 2024; Levy et al., 2024: Zhang et al., 2025b). This weakens coherence and raises the risk of hallucination. To compensate, LLMs often rely on longer chains of thought (CoT) (Wei et al., 2022; Yeo et al., 2025: Chen al., 2025b) or repeated self-refinement (Wang et al., 2023; Ling et al., 2023; Madaan et al., 2023) to re- rocus. which in turn raise computational costs and inflate token usage.
In contrast, human reasoming often unfolds in an immediate and streaming manner. Research in psychology and cognitive science shows that during reading, humans process incoming information instantaneously, including text decoding, meaning construction, background knowledge activation, integrative reasoning, and actively generating inferences to build coherent understanding (Kintsch, 1988; Graesser et al., 1994). This "thinking while reading" mechanism not only enhances process- ing efficiency, but also allows reasoning to closely alongside the relevant context, minimizing cognitive lag and mitigating the risk of coherence degradation.
"Corresponding author

---

Published as a conference paper at ICLR 2026
(a)
Source Input
Streaming Thinking
Lange Language Model
Source Input Butfer
Contest:
Read
Think
<think-
(1) Charlic has two-day day conference schedule.
get the background of Chartie's schodule.
Wait Think
Read
(2) El atlends 3 conference 45 minutes each
Rosal
Think (2) Charlie's ia begins with three 45 ss-minute conference on the
Read
on the first day.
Resd
Thōnk
first day (3-45 - 135 minutes total).
(3) On socond day, he joins sominars, each
he attends semiinans
30 minutes
Read
lnsting hour
minutes.
Read
Think
nest day (2x1
minutes hours
Information Flow
Question:
Now,
the question about calculatingg the total
(-9 How many local hours does Charlic spend??
Charlie spand during days.
Prefill
Delay
-Whink>
answer hours.
Batch Thinking
Large Langudge Mookl
Think
<thiak>>
total hours Charlie spends the two days.
Olay,, see.
figure out how coany
Different Think-depth
(b)
Strowwing thinking direct answer
So the answer is 5.25 hours.
Think
day: atrends conference each minutes long. So, 3-15 135 minunes, next atsend seminnrss
Delay
Stroewing thluking with global shinking
So need to sum the previous results. Convent to hours and sum them: 135/00- hours 2.25 hours 25 hours.
Think
minutes, sumiinuss
2:900
Strvaming thiaking with global thinking & seff-rftection
minutes. Now, need
minutes.
the sotal minutes 135- 315 minutes. question zaks hours, Su convert minutes hours. 31 315660 =5 2,25 houns.
So need to sum the provious results Convent to hours and
sum them: 135/60- 3 hours 2 2.25-3 hours - 5.25 hours.
Think
Walt, let me check again. First day: 3-45 - 135 minutes.
[335600 -225 hours. Second day:
day: 2-400 180 minutes.
[80:60 - 3 hours. So total hours: 2,25 (3 3- 5.25 hours.
Figure 1: (a) Standard LLM reasoming follows the batch thinking paradigm, where reasoning begins only after the entire input is received, lcading to high latency and imbalanced attention to the input. The proposed streaming thinking paradigm enables LLMs to think while reading during input reception, substantially reduc- ing latency and maintaining attention aligned with the order of input. (b) Streaming thinking paradigm supports multi-depth reasoning, balancing latency with performance.
To narrow the gap between LLM and human reasoning, we propose a streaming thinking paradigm for LLMs. Streaming thinking unfolds reasoning steps alongside the input stream, allowing the model to reason while receiving information. Once the full input is available, the model can further refine its reasoning and adjust the depth of its analysis to match task complexity. As illustrated in Figure 1, compared with batch thinking, streaming thinking enables much faster responses while preserving consistency with the order of incoming information.
Accordingly, we propose Thinker,gThinker, framework that instantiates the streaming thinking paradigm. The StreamingThinker integrates a streaming CoT generation pipeline with training and inference frameworks that adapt LLMS the streaming paradigm. The generation pipeline inserts boundary tokens to the inputs to define minimal reasoning units, prompting LLMS to gen- erate serialized reasoning segments for each unit that are reconstructed into incremental content, filtered by quality evaluation, with reasoning depth controlled through token intervention (Wu et al., 2025). To support streaming training, CrreamingThinnee introduces two modifications: a streaming attention mask that restricts each reasoning step to past and current input, and a streaming position encoding that independently indexes input and reasoning tokens from zero eliminate positional contention, ensuring alignment with associated inputs. For inference, StreamingThinker employs parallel KV caches that decouple input encoding from reasoning generation and merge only during cross-attention, enabling true thinking while reading. We conduct comprehensive evaluation on Qwen3 model family (Yang et al., 2025), covering diverse tasks such as math reasoning, logical reasoning, and context-based QA reasoning. Experimental results indicate that Streaming'Thiner achieves reasoning performance on par with batch thinking, yet reduces token-level waiting before reasoning by 80% and overall answer lateney by over 60%.
The contributions of this work are fourfold.
To the best of our knowledge, we are the first to introduce the streaming thinking paradigm for large language model reasoning. This paradigm mirrors human cognitive processes, enabling LLMs to engage in more timely and continuous thinking in dynamic scenarios.
We propose a streaming CoT generation pipeline for this paradigm. Drawing on the prin- ciples of human streaming thinking, it ensures that the reasoning process remains aligned with the sequential order of the input context.
We provide an adaptation training and inference framework that implements the streaming thinking paradigm, in which training ensures alignment with sequential inputs and infer- ence enables efficient concurrent reasoning.
Extensive experiments on diverse reasoning tasks show that our method achieves reasoning performance comparable to batch thinking, while markedly reducing waiting latency.
Appendix A discusses the value and potential applications of human-like streaming thinking in practice.
2

---

Published as a conference paper at ICLR 2026
2 STREAMING THINKING PARADIGM
Paradigm Design Human streaming cognition involves two complementary processes: rapidly generating and updating representations input arrives, and subsequently performing global inte- gration to transform local, shallow understanding into holistic, deep comprehension (Kintsch, 1988).
Inspired by this process, we design a streaming thinking paradigm for LLMs, with an example illus- trated in Figure 1. At each step. the model incrementally processes the incoming sentence, focusing on progressive comprehension such (1) understanding and summarizing key information, (2) ex- plaining ambiguities and reorganizing semantic relations, (3) extending logical implications, and (4) skipping thinking step when the content irrelevant the question. After completing this incremen- tally reading and reasoning, we define multiple reasoning depths post-reasoning step, as shown in Figure I (b). The model may (1) directly produce an answer, representing the shallowest depth; (2) further integrate global information to achieve deeper comprehension; reasoning on top of global integration to obtain the most reliable reasoning outcome. Within the paradigm, reasoning depth adapts to question complexity and is explicitly controlled by instruction signals that guide the model toward different strategies.
(3) perform reflective
Ordering of Context and Question in Streaming Thinking In the batch thinking paradigm, all input is available simultaneously, so the order of context and question is often overlooked. Yet prior work shows that their relative order can influence reasoning (Chen et al., 2024b; Wei et al., 2024; Xie, 2024), effect amplified in streaming scenarios. In human reading, two natural input orders commonly occur. In the first, the question is presented before the context, enabling the reader to establish targeted associations as subsequent information is processed. In the second, the context precedes the question, which case the question remains unavailable during streaming reasoning and the reader must rely solely the context itself to construct plausible inferences. To approximate these scenarios, our streaming thinking paradigm explicitly distinguishes between the two orders and examines their impact on model reasoning. Some examples are provided in Appendix B.
immediate reasoning process that unfolds alongside the input stream, with reasoning depth flexibly adapting to the complexity of the problem. Formally, let Q denote the input question and C, the t-th sentence in the input context. At each step, the LLM generates an intermediate reasoning state RS corresponding to Ct. and Rg corresponding to the question Q. The instruction sets the reasoning depth, directing how intermediate states integrated into the final reasoning output R. The streaming thinking process can be described as:
Formal Definition Streaming thinking is defined
I1 S Rg--)) P(RgQQ, CST, RST) P(RQQ, CST, RIT, context first,
P Pstreaming
P(R(Q)) IT, (IRolQ.C5, Rg--) P(RQQ, Cer, Rgr, streaming thinking
question first.
with controllable depth
3 STREAMINGTHIKKER
This section introduces the StreamingThinnk a supervised fine-tuning framework that integrates streaming CoT generation with streaming training and inference mechanisms to adapt batch-oriented LLMs the streaming thinking paradigm.
3.1 STREAMING COT GENERATION
Streaming TreekeengThinker Thinker first constructs a streaming-like CoT dataset, as existing batch-style reasoning traces lack human-like incremental thinking. This step produces treaminggccompattbee traces with controllable depth, providing the foundation for subsequent training and evaluation.
Generation Process The streaming reasoning dataset is constructed through a multi-stage pipeline, as shown Figure 2. We first insert sentence-level boundary tokens <BOS> for the input to define minimal reasoning units. Then the LLM is prompted to generate order-preserving rea- soning for the preceding sentence and terminate the step with <EOT>, when encountering <EOS>.
3

---

Published as a conference paper at ICLR 2026
2 To further enforce sequential alignment, higher- parameter teacher model reconstructs the generated reasoning. Once all sentence-level reasoning traces are generated, they are evaluated using granularity score and sequential consistency score. Passing samples are enhanced with token-level intervention to generate depth-controlled reasoning variants. Samples that fail the evaluation are regenerated, and those still failing under Pass@2 2 (Chen et al., 2021) metric are discarded.
Beundary Tuken Insertion
than brodher? for <E35>
the <E0S>
Rude guided
"Yo are straaiing rouniner perforta onder reasoning banal os Mesestricr ralet: aor (contoxt)"
Prosupting
Reconstract Order
tcacher muddel
grido
Reasoning
Sequestial Consiseacy
Ressoning Quality Assessment
reject
Quality Assurance and Evaluation We propose two evaluation metrics: the granularity score mea- sures the fine-grained alignment of the streaming COT, while the sequential consistency score assesses whether the reasoning proceeds in streaming, order- preserving way. The granularity score is defined as the ratio between the number of boundary tokens in the input and those in the output: gramularity = Npos Nior' of boundary tokens in the input and output, respectively. When the granularity score is equal to 1, the reasoning matches the input in boundary count, suggesting ideal alignment. The consis- tency score is defined as the similarity between the input sentence and reasoning sentences, i.e., consistency = sim RR, Ci) = Tee besl where UR and ve denote the embedding vectors of the reasoning sentences RG and the input sentence Cr, respectively. We use SentenceBERT (Reimers & Gurevych, 2019) for sequential consistency calculation."
Grunulrity
accept
Token-level Interveation
Depth- Coutolladd Thinking
Global
Self-reflective
Answer
Thinking
Thinking
Figure 2: Generation for streaming CoT.
where NEOS and NEOT denote the counts
3.2 STREAMING TRAINING FRAMEWORK
A naive approach is interleave input and reasoning sentences. While it appears to be streaming, this method is inconsistent with the pretraining format of LLMs and still enforces serial execution, where reasoning prevents the model from simultaneously consuming new inputs. Beyond interleav- ing, we design an actual streaming training framework for streaming thinking paradigm.
Streaming Attention Mask Matrix According to Equation 1, the core constraint of streaming thinking is that the reasoning step at current time must not access future inputs, In contrast, the standard attention mask used for batch thinking exposes all inputs to every reasoning step. To adapt LLMSs to the streaming paradigm, we inject a streaming constraint into the attention mask. As illustrated in Figure 3 (a), within the attention from the reasoning sentences to the input sentences, we apply a causal mask that blocks attention from step t to input positions > We refer to the masked region as the streaming mask region. Let the input sentence have length T and the reasoning segment length L. The streaming mask is then defined as
Mancaming((.) = M(i.)) (-o0- M(i.)) JT,, s
where M is the vanilla casual mask matrix of LLMs, and I is indicator function.
Streaming Position Encoding The RoPE (Su et al., 2024) of LLMs represents relative positions by rotating queries and keys, with the attention between a reasoning token RG and an input token St expressed as Attn R,, S:) o R(T + t)ks, where T is the input length and t, t + T are their positional IDs, and R(T") is the rotary matrix. However, in streaming scenarios, the concurrent generation of output and reception of input induces positional contention in the encoding space (Guo et al., 2024; Tong et al., 2025). To address this issue, we assign independent position IDs to input and reasoning tokens, both starting from zero Formally, the positional IDs of reasoning token Ra and input token S, are both set to f, yielding Attn S.) = of R(t t)kg. This design removes positional contention in streaming parallel processing. Furthermore, identical position IDs ensure that, during streaming reasoning, a reasoning sentence is positioned nearest to its associated input sentence and distant from others, which conforms the essential principle of streaming alignment.
Boondary tokens mark minimal reasoning units and indicate the end of reasoning step during inference. In cases where single input sentence corresponds to multiple reasoning sentences, we regard them collec- tively as one segment. We provide the similarity map hetween the input and the reasoning the Appendix C.5.
4

---

Published as a conference paper at ICLR 2026
Input Sentence
Reasening Sentence
(b) Atrontonn Route During informnc Paraliel KV Caches for Streaming
Information Flow
Information Flow
Attend to Input
AAtcod to Reasoming
Gencration
Attemtion Mask Motrix During Training
Batch Thinking
Thinking
Streaming Thinking
[5y]
[8/][S>|[S (S 11 Rall
[82]
[Rg]
[8g1
[R,]
Interleaved Thinking
[Ry]
[Rg]
[521
Sinsening Mask Region
Large Language Model
[R2]
1821
18g1
Thinking
68
[5y]
[R/]
[R
[R2]]
Mal
(Rsl
Streaming Thinking
Source KV Cacho
Merge
[52]
[Ra]]
Tanget KV Cache
Spie
[R2]
2
Source Prefilt
Target Decoding
Figure 3: Training and inference framework of StreamingThinker (a) shows attention mask at training. (b) and (c) show attention routing and parallel KV caches for streaming thinking at inference.
3.3 STREAMING INFERENCE
Attention Route Figure 3(b) compares information flow across paradigms. In batch thinking, reasoning begins only after the full context is received, resulting in long attention routes and serial dependency. Interleaved thinking alternates reasoning with partial inputs but still updates a single cache sequentially, and its format diverges from the pretraining distribution. In contrast, stream- ing attention preserves consistency with batch-style pretraining while employing parallel caches, enabling concurrent processing with shorter routes and lower latency.
Parallel KV caches
To enable parallel processing in streaming reasoning, we design two KV caches during inference: source cache for input tokens and a target cache for reasoning tokens, as shown in Figure (c). As input arrives sentence by sentence, the LLM performs prefill in arrival order, storing hidden states in the source cache. Before decoding. the two caches merged that reasoning can attend to the inputs, and newly generated tokens are written into the merged cache. After finishing sentence, the caches split again. This design enables concurrency between source-side prefill and target-side decoding, whereas batch and interleaved paradigms rely on a single continuous cache, enforcing strictly serial execution.
EXPERIMENTS
4.1
EXPERIMENTAL SETTINGS
Datasets To evaluate StreamingThinker we conduct comprehensive assessment across three representative reasoning tasks: math reasoning. logical reasoning, and context- based QA reasoning. For math reasoning, we selected GSM-symbolic (Mirzadeh et al., 2025) and Metamath (Yu et al., 2024). For logical reasoning, we utilized LogicNLI (Tian al., 2021) and ProofWriter (Tafjord et al., 2020). Finally, for context-based QA reasoning, PubMedQA (Jin et al., 2019) and Hot- potQA (Yang et al., 2018) were employed. All datasets were partitioned into dedicated training and testing sets, with detailed specifications provided in the Appendix D.
Models and Baselines We implement our StreamingThine Thinker using models from the Qwen3 fam- ily. For streaming CoT generation, we utilize Qwen3-32B as the initial generation model to produce the preliminary streaming reasoning trace. We assign guidance model to reconstruct the preliminary trace. Then Qwen3-1.78 and Qwen3-4B as the backbone of Streaming'Thinke for evaluation. To provide comprehensive evaluation, we com- pare Streaming Thinker with three baselines representing alternative reasoning paradigms: (1) batch thinking (Batch, orignal), where the model generates reasoning after observing the entire context without additional supervision; (2) batch thinking with CoT distillation (Batch, SFT), where rea- soning traces distilled from stronger 32B teacher model to enhance reasoning ability; and (3) interleaved mode, a naive streaming variant that alternates between input segments and reasoning steps without parallel cache support.
22--2-235--A22--B1 Instruct as the teacher
5

---

Published as a conference paper at ICLR 2026
Table 1: Pass@1 1 accuracy (Acct) and token usage (Tokens.) results of the streaming thinking paradigm under the batch processing setting. The comparison includes: (1) the original batch think- ing baseline without distillation, (2) SFT models trained with RoPE or Streaming RoPE (SPE) dis- tilled from Qwen3-32B CoT data, and (3) the streaming thinking model executed in a batch process- ing mode SBatch-S) with RoPE or SPE. Reasoning depth is categorized into three levels, denoted as DI-D3. where DI = direct answer, D2 = with global reasoning, and D3 = with self-refiection.
Math reasoning
Logical reasoning
Context-based QA reasoning
Methods
GSM-Symbolic
MetaMathQA
ProofWriter
LogicNLI
HotPotQA
PubMedQA
Acct Tokens]
Acct
Takens
Acct
Tokens
Acct
Tokens|
Acct
Takens]
Acct
Tokens
Qwen3-1.78
Ratch, Original
0.645
1714.25
0.657
1751.40
0.488
1379.58
0.503
1384,47
0.484
638.91
0.616
619.92
Batch, SFT, RoPE
0.748
1104.05
0.755
1236.76
0.785
980.12
0.600
1120.38
0.530
512.42
0.642
602.38
Batch, SFT, SPE
0.740
1032.71
0.761
1195.68
0.790
970.44
0.604
1110.27
0.528
498.73
0.648
611.25
S,tch-S,Dee DI, RoPE
0.396
217.33
0.1795
253.26
0.470
360.18
0.515
872.33
0.456
324.71
0.598
452.15
Bateb-S, DI, SPE
0.401
212.42
0.183
250.26
0.476
352.07
0.519
865.27
0.463
318.64
0.601
448,92
Batch-S, D2, RoPE
0.688
396.34
0,703
611.61
0,540
555.21
0,542
1196,33
0.492
472.51
0,624
571.66
Batch-S, D2, SPB
SPE
0.692
391.25
0.706
618.53
0.544
545.37
0.546
1184,47
0.497
465.33
0.628
565.92
Batch-S, D3, RoPE
0.732
$62.88
0.736
829.58
0.804
720.46
0.560
1351,16
0.545
589.42
0.668
693.18
Bateb-S, D3, SPB
0.727
552.13
0.738
835.11
0.810
710.32
0.568
1362.82
0.552
583.61
0.663
688.74
Owen334BB
Batch, Original
0.855
1445.61
0.774
1630.67
0.620
1878.57
0.492
1421.92
0.575
617.40
0.609
463.37
Batch, SFT, RoPE
0.890
896.45
0.833
1029.66 66
0.863
935.36
0.647
1002.37
0.608
458.56
0,675
572.63
Batch, SFT, SPE
0,882
887.14
0.836
989.23
0.861
925.66
0.650
992.38
0.601
467.21
0.680
582.31
Bateh-S, D1, RoPE
0.433
203.23
0.662
201.60
0.592
386.25
0.627
683.17
0.552
269.45
0.598
358.14
Batch-S, SPE
0.437
199.64
0.668
200.73
0.596
376.13
0.623
672.61
0.561
257.11
0.592
351,47
Bateh-S, D2, RoPE
0.864
348.52
0.791
534.74
0.801
583.28
0.639
983.52
0.592
381.36
0.657
5U7.32
Batch-S, D2, SPE
0.871
35
0,806
528.33
0.795
596.61
0.642
978.52
0.601
388.93
0,651
498. 17
Ratch-5, D3, RoPF
0.867
499.81
0.821
661.96
0.856
721.89
0.645
1242.82
0.616
474.56
0.661
563.41
Bateb-S, D3, SPB
0.874
493.26
0.825
668.19
0.861
715.26
0.651
1176.65
0.621
466.14
0.669
571.22
Metric The evaluation of streaming scenarios can be viewed as a trade-off between performance and latency. For reasoning performance, we adopt Pass@1 score as the accuracy metric measure the model's ability to successfully solve problems. Latency is assessed at two levels: token latency and time latency. At the token level, we use to-es-to-f (TTFT) to measure how many input tokens must be observed before the model begins reasoning.* For time latency, we set the LLM's 's in- put speed as the average human speaking rate about 150 words per minute (Geva Yaghoub Zadeh, 2006; Jacewicz et al., 2010), and define the waiting time until the first answer token as the latency.
4.2 EFFECTIVENESS OF THE STREAMING THINKING PARADIGM FOR LLMS
We begin by validating the feasibility of the streaming thinking paradigm (sequential reasoning with depth adjustment) under the batch setting, which removes interference from streaming input and cache strategies. This controlled setup allows us assess the model's adherence to the paradigm and to further investigate two key aspects: (1) the impact on reasoning trajectories and reasoning depth, and (2) the role of streaming position encoding in maintaining alignment and stability.
Effect of Reasoning Depth in Streaming Thinking We first validate the streaming thinking framework on Qwen3-1.7B and Qwen3-4B. As shown in Table and Figure 4, the performance of LLMs improves consistently with increasing reasoning depth in the streaming paradigm. At shal- low depths, the model mainly performs local reasoning aligned with the sequential input, which provides fast but relatively coarse-grained understanding. When deeper reasoning stages are in- troduced- parttccuarryy with global reflection the performance approaches that of batch thinking, demonstrating that additional depth helps compensate for the information fragmentation inherent in streaming reasoning. Moreover, results in Table 1 show that under the batch processing setting, the
"This is similar with Detailed definitions and additional evaluation metrics are provided in Appendix F.
hut measured in terms of token count.

---

Published as a conference paper at ICLR 2026
(s)Streaming thinking depth
(b) RoPE SPE Performance
(c) Atzention (RoPE) (d) Attention (Streaming RoPE)
10h0
atrvaiing
glokal
reBection
Mulaamoth
Accurnry (Paeog) I)
6.8
5re Med
Acpuracy
Tokens
390
0.3
0.
02
Gptkybblic Orea34
420
Mataweac Swacc m
600
strosming
w/-global
w/-relection roslection
stre streeming
wi-glubal
s/-rddection
700
350
700 350
700
n.n
Tokens
Figure 4: Streaming thinking performance and attention patterns. Subplots (b) show accu- racy-token trade-offs for GSM-Symbolic and MetaMathQA under RoPE and streaming RoPF
(SPE); subplots (d) show average attention maps comparing RoPE and streaming RoPE.
streaming thinking paradigm achieves performance comparable to the original batch thinking, while demonstrating notable advantages in token efficiency.
The slope of the curves in Figure 4(a) highlights the marginal gains of each added depth. Notably, introducing the global reasoning stage leads to the most significant improvement, which aligns with the motivation of streaming thinking: since the streaming phase processes inputs in a lightweight, incremental fashion, global consolidation step is essential to fully integrate dispersed information and support complex inference.
Position Encoding in Streaming Thinking Streaming RoPE assigns consistent yet independent index groups to input and reasoning tokens, ensuring that each reasoning step is correctly aligned with its corresponding input. This design also prevents positional interference, thereby addressing the positional contention issue that arises with the original RoPE in streaming scenarios.
Our experiments further demonstrate that Streaming RoPE achieves performance comparable to the original RoPE under the same settings. As shown in Figure 4(b) and Table 1, in math reasoning tasks at the same depth, Streaming RoPE attains nearly identical accuracy and token consumption to the original RoPE. This indicates that adapting RoPE to the streaming paradigm preserves model capacity without incurring performance degradation.
The attention visualizations in Figures 4(c) and (d) provide additional insights. While the original RoPE exhibits no clear positional preference across the input context, Streaming RoPE shows a pronounced diagonal concentration, reflecting stronger focus the current context. This bias aligns well with the motivation of streaming thinking: reasoning should primarily rely on information already observed, thereby enabling the model to "think while reading."
4.3 LLMS THINKING WHILE READING UNDER THE STREAMING THINKING PARADIGM
extend evaluation to real streaming scenarios, where inputs arrive incrementally over time. In this setting, the model is required to perform reasoning online, relying solely on the partial context available each step.
After confirming the feasibility of streaming thinking in batch settings,
Latency of Streaming Thinker We examine the latency performance of Streaming Thinker us- ing the Qwen3-4B model. As shown in Table 2, streaming thinking achieves a markedly lower TTFT than batch reasoning, as reasoning can be initiated once the first input segment becomes available. The minimal latency observed at depth D1 further indicates that reasoning overlaps with input reading without incurring additional overhead. These results confirm that PrrcamingThinkee substantially reduces response delay, a property of particular importance for streaming applications.
Interleaved mode and Streaming mode The interleaved mode constitutes naive instantiation of streaming reasoning. Relative to batch reasoning, it exhibits lower latency- most prominently in terms of TTFT-as as reasoning can be initiated earlier, as reported in Table 2. Nevertheless, its accuracy is consistently lower and its overall delay higher than those achieved by streaming think- ing. This discrepancy can be attributed to the distributional mismatch between interleaved input
'It is important to note that the input rate is set to 150 words per minute to match the average speed of human speech in interactive scenarios. Given that L.M decoding operates at much faster rate, the effective bottleneck in streaming reasoning stems from input arrival, rather than output generation.
7

---

Published as a conference paper at ICLR 2026
Table 2: Results compare the original batch thinking (i.e., without distillation process) and streaming thinking paradigms. The streaming thinking results include both the naive interlcaved streaming mode (Interlcaved) and our proposed parallel streaming mode (Streaming), evaluated under three reasoning depths (D1 = direct answer. D2 = global thinking. D3 = self-reflection) Results are reported in terms of Pass @ I (Acc), token number first input token (TTFT), and time delay for generating first answer token (delay). All experiments are conducted on Qwen3-4B.
Method
GSM-Symbolic
MetaMathQA
ProofWriter
Acc?
TTFT
Delay4 (6)
Acct
TIFT
Delay|
Acct
TIFTL
Delay|
Batch, Original
0.855
94.74
47.70
0.774
100.51
53.81
0.620
232.11
61.99
Interleaved.
0.410
6.30
0.655
16.89
6.23
0.583
20.51
11.96
Interleaved,
0.829
16.55
20.51
18.07
Interleaved,
0.843
20.77
15.46
0.783
16.89
20.49
0.801
20.51
22.35
Streaming, DI
0.42T
20.77
0.66
0.657
16.89
0.68
0.588
20.51
0.71
Streaming, D2
0.842
20.77
5.973
0.752
16.89
10.98
0,761
20.51
6.50
Streaming, D3
0.856
20.77
9.768
0,780
16.89
15.18
0.813
20.51
11.05
Method
LogicNLI
HotPotQA
PubMedQA
Acc?
TTFT/
Delay (6)
Acct
TIFTJ
Delay| (8)
Acct
TTFT
Delay (s)
Batch, Original
0.492
46.92
0.575
1485.547
20.37
357.468
15.29
Interleaved,
0.591
42.06
21.17
0.537
24.32
8.33
0.571
21.74
11.09
Interleaved, D2
0.614
42.06
30.47
0.576
24.32
12.02
0.633
21.74
15.71
Interleaved, D3
0,627
42,06
38.50
0,598
24.32
14.45
0.646
21.74
17.45
Streaming, DI
0.603
42.06
0.72
0.544
24.32
0.69
0.379
21.74
0.74
Streaming, D2
0.629
42,06
9.9
0.581
24.32
3,92
0.641
21.74
4.92
Streaming, D3
0.634
42.06
18.45
0.603
24.32
6.50
0.653
21.74
6.76
sequences and the LLMs' pre-training corpus, which impairs reasoning fidelity. Moreover, the in- terleaved paradigm enforces sequential synchronization constraint, requiring the completion of ongoing reasoning before additional input tokens can be incorporated, thereby exacerbating latency. In contrast, StreamingThinke employs parallelized KV caches that disentangle input encoding from reasoning generation, enabling concurrent reading and reasoning. This architectural design effec- tively minimizes latency while preserving reasoning quality, thereby highlighting the necessity of streaming spreming--ppecffc mechanisms for efficient online reasoning.
4.4 ORDERING OF CONTEXT AND QUESTION FOR PRRFAMINGTHHIKKER
The ordering of context and question plays a critical role in streaming reasoning. Unlike batch settings, where the model has access to the entire input si- multaneously, the streaming paradigm requires rea- soning to unfold as inputs arrive, In real-world sce- narios, however, the order in which the question and context appear is often unknown. To account for this, we evaluate the proposed streaming thinking framework under both orderings to examine its ro- bustness across different streaming conditions.
Context first(CQ) vs question
1000
CD (Tokens)
-4- Co (Delay)
QC (Tokens)
OC
800
15
800
400
200
03
02
03
03.
100
09
0%
02
03
Math
Logic
Context-based QA
Figure 5: Comparison hetween context-first and question-first settings. Bars indicate model token consumption, while lines represent time latency.
Table 3 reports the performance of Streaming- Thinker under the context-firs input setting across different datasets. Overall, the results follow similar trend to the question-first setting, confirming the framework's ability to provide timely responses regardless of input order. As the depth of reason- ing increases, both accuracy and latency improve, albeit with gradual rise in token consumption.
Figure highlights the differences between the two settings.? When the question appears first, the model is aware of the reasoning target. This is particularly advantageous in Context-based QA tasks where critical information is sparse; the prior knowledge of the question allows the model to precisely capture key evidence, thereby avoiding the generation of reasoning for irrelevant context (in contrast to domains with denser information). However, for the identified relevant segments, the model tends to expand its logic immediately. This leads to higher token usage at depth D1,
For clarity, we report the average performance for tasks of the same type.
8

---

Published as a conference paper at ICLR 2026
Table 3: Results context-first setting, where the L.LM receives context before the question. (D1 direct answer, D2 = global thinking, D3 = self-reflection). Results are reported in terms of Pass @ (Acc), token numbers (Token), token number to first input token (TTFT), and time delay for gener- ating first answer token (delay). All experiments are conducted on Qwen3-4B.
Method
GSM-Symbolic
MetaMathQQ
ProofWriter
Acct
Token,
TTFT]
Delay
Acc
Token]
TTFTL
Delay
Acct
Token.
TIFT
Delay
Batch, Original
0.842
1483.65
94,74
48.20
0.774
1668.30
100.51
54.23
0.633
1786.42
232.11
58.05
Streaming. DI
0.476
165.42
18.68
0.77
0.608
114,50
20.40
0.94
0.528
457.91
8.12
1.06
Streaming, D2
0.837
451.81
18.68
9.23
0.760
463.59
20.40
11.59
0.740
697.20
8.12
7.33
Streaming, D3
0.844
797.63
18.68
20.45
0.788
617.36
20.40
16.71
0.798
875.19
8.12
13.11
Method
LogieNLI
HotPotQA
PubMedQA
Acct
Token,
TTFT]
Delay
Acc7
Token]
TTFTL
Delay]
Acct
Token.
TIFT
Delay
Batch, Original
0.483
1487.56
350.08
48.35
0.566
1492.41
1485.55
48.50
0.593
1320.36
357.47
42,90
Streaming, DI
0.476
604.84
5.60
0.89
0.532
1172.36
46.39
1.62
0.562
298.51
31.49
1.12
Streaming, D2
0.613
927.35
5.60
10.24
0.575
1266.41
46.39
5,90
0.628
475.63
31.49
5.54
Streaming, D3
0.625
122.33
5.60
16.55
0.591
1381.73
46.39
12.14
0.640
687.45
31.49
12.43
and since part of the reasoning is already completed during the streaming process, the incremental token growth at deeper levels (D2 and D3) becomes smaller compared the context-first setting. In contrast, when the context appears first, the model lacks knowledge of which information is salient. As result, its reasoning remains more conservative, proceeding sentence by sentence without extensive elaboration, which produces lower token usage at DJ when processing the same context sentence. However, this characteristic leads to inefficiency in sparse scenarios (such as the context-based QA tasks): due to the conservative strategy, the model fails to skip irrelevant information, resulting a significantly longer total generation length.
DISCUSSION
Efficiency Analysis We evaluate the model efficiency Qwen3-4B using 100 samples from GSM-Symbolic dataset. As shown in Table 4, the Streaming paradigm reduces the first-token latency (measured by time consumption) from 28.00s to 6.23s (~4.5x speedup). Crucially, the parallel KV cache operations introduce negligible temporal overhead, with splilıv and mergex. taking less than 5ms combined. Additionally, peak memory usage remains consistent with the baseline (~7..99 GB). While bandwidth cost increases, this is primarily due to the inherent multiple prefill phases in streaming scenarios rather than the parallel KV cache mechanism itself. Note that Table reports the latency for each stage separately; the actual end-to-end wall-clock time, which benefits from the concurrent execution of reading and reasoning, has been detailed in Section
Why Streaming Thinker Work? Prior studies (Fan et al., 2025; Laban et al., 2025) have high- lighted the risks reasoning over incomplete inputs. This phenomenon reflects the inadequacy of batch-processe procession LLMs when facing local contexts, their training objectives align with global visibility rather than incremental inference. However, Streaming Thinker circumvents these pitfalls through three distinct mechanisms. First, global information is deferred rather than lost. Unlike scenarios where critical conditions are permanently removed, our framework merely shifts the tim- ing of acquisition, ensuring the model incorporates the full global context after the streaming phase. Second, we employ a conservative reasoning strategy. Instead of attempting premature complex reflection, the model concentrates on shallow reasoning" (e.g., intermediate calculations and entity tracking) during the streaming phase. This functions as an incremental pre- pre-processing step that sim- plifies raw context. Third, this behavior is enforced via a specialized streaming training paradigm. Unlike batch models that rigidly apply full-context patterns to partial inputs- -ote-oofeenn falling into the over-thinking" trap- our model is explicitly adapted to local scopes, learning to process available information without jumping to erroneous conclusions.
'The model's conservative reasoning stems from our cautious data generation strategy- the question unknown, excessive logical expansion may cause overthinking.

---

Published as a conference paper at ICLR 2026
Table 4: Efficiency evaluation Qwen3-4B. We compare the efficiency of Batch and Streaming paradigms using 100 randomly selected samples from the GSM-Symbolic dataset. Metrics include exccution memory usage (Peak/A), bandwidth cost, and time consumption across different stages.
Batch Thinking Paradigm
Streaming Thinking Paradigm
Metric
Read
Prefill
Decoding
Read
Split KV
Prefill
Merge KV
Decoding
Count (times)
69.86
436.0
69.86
4.65
4.65
4.65
439.0
Peak Mem (MB)
7,991
7,998
7,957
7,993
7,940
7,999
Total Mem A (MB)
+222
+102
-289
+215
+217
+103
Bandwidth Cost (MB)
+29,173
+434
-3,310
Avg Time
0.4
0.052
0.039
0.4
0.001
0.041
0.001
0.037
Total Time (s)
27.945
0.052
17.178
27.945
0.005
0.190
0.004
16.392
First-token Latency (8)
28.003
6.231
RELATED WORK
Efficient Reasoning in LLMs Prior studies on efficient reasoning in LLMs have mainly focused on three directions: token compression (Aggarwal & Welleck, 2025; Xia et al., 2025; Zhang et al., 2025a; Zhao et al., 2026), structural quantization and pruning (Liu et al., 2025; Srivastava et al., 2025; Zhao et al., 2025), and efficient decoding (Pan et al., 2025; Liao et al., 2025). Token com- pression methods such as Tokenskip (Xia et al., 2025) condense CoT into fewer tokens to reduce the inference cost. Structural approaches leverage quantization or pruning to compress model param- eters for efficient reasoning such as (Srivastava et al., 2025). Efficient decoding has been explored through parallel sampling of generation paths (Pan et al., 2025) or by using smaller models for spec- ulative prediction (Liao et al., 2025) to reduce inference latency. In contrast, our work introduces streaming processing
complementary dimension of efficiency, allowing reasoning to proceed concurrently with input processing to reduce response latency.
Streaming LLMs Recent research streaming LLMs has focused three directions: archi- tecture adaptation (Raffel et al., 2024; Guo et al., 2024; Tong et al., 2025), latency control (Ma al., 2018; Ahmed al., 2025; Cheng et al., 2025), and modality adaptation (Chen et al., 2024a; 2025a; Xie & Wu, 2024; Lin et al., 2026). Architecture adaptation addresses the mismatch be- tween decoder-only transformers and streaming settings (Tong al., 2025), where issues such as positional interference and redundant re-encoding are mitigated through recurrent states, and mod- ified attention mechanisms. Latency control emphasizes fine- one-grained alignment between input and output, ranging from fixed policies (e.g., wait-k (Ma et al., 2018)) to adaptive scheduling that op- timize the accuracy-latency balance. Modality adaptation extends streaming L.Mss beyond text to speech recognition (Guo et al., 2024), speech translation (Cheng et al., 2025), and video understand- ing (Chen et al., 2024a) by integrating -ppeeity- encoders and synchronization mechanisms. From reasoning perspective, recent studies (Xie et al., 2025a; Chiang et al., 2025; Xie et al., 2025b) have explored alternating reasoning and generation to approximate streaming operation. Our work differs by explicitly modeling thinking while reading, enabling reasoning to evolve concurrently with incremental input.
7 CONCLUSION
In this work, we introduce the streaming thinking paradigm for large language models, inspired by the human ability to think while reading. Unlike conventional batch reasoning, this paradigm unfolds reasoning concurrently with input arrival and adapts its depth after reading is complete. To instantiate this paradigm, we developed StreamingThinker, which integrates a streaming CoT generation pipeline, streaming-constrainnn KV cache designs. Comprehensive experiments across math reasoning, logical reasoning, and long- context QA demonstrate that Streaming Thinker substantially reduces latency while preserving or improving reasoning quality. Our analysis further reveals the benefits of controllable reasoning depth, streaming-specific position encoding, and parallel inference for enabling true concurrency. These findings highlight streaming thinking as a promising new direction for efficient and coherent reasoning in LLMs, bridging the gap between artificial and human-like like cognition.
training, and parallel inference supported by specialized
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Pranjal Aggarwal and Sean Welleck. LI: Controlling how long reasoning model thinks with reinforcement learning. arXiv preprint arXiv:2503 04697, 2025.
Zeeshan Ahmed, Frank Seide, Zhe Liu, Rastislav Rabatin, Jachym Kolar, Niko Moritz, Ruiming Xie, Simone Merello, and Christian Fuegen. Non-monotonic attention-based read/write policy learning for simultaneous translation. arXiv preprint arXiv:2503.2203 2025.
Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao. Dongxing Mao, and Mike Zheng Shou. Videollm-online: Online video large lan- guage model for streaming video. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18407-18418, 2024a.
Joya Chen, Ziyun Zeng, Yiqi Lin, Wei Li, Zejun Ma, and Mike Zheng Shou. Livecc: Learning video IIm with streaming speech transcription at scale. Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 29083-29095, 2025a.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374 2021.
Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng. Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning A survey of long chain-of- thought for reasoning large language models. arXiv preprint arXiv:2503.0956/ 2025b.
Xinyun Chen, Ryan Andrew Chi, Xuezhi Wang, and Denny Zhou, Premise order matters in reason- ing with large language models. International Conference on Machine Learning (ICML 2024), pp. 6596-6620, 2024b.
Shanbo Cheng. Yu Bao, Zhichao Huang, Yu Lu, Ningxin Peng, Lu Xu, Runsheng Yu, Rong Cao, Yujiao Du, Ting Han, et al. Seed liveinterpret 2.0: End-to-end simultaneous speech-to-speech translation with your voice. arXiv preprint arXiv:2507.17527, 2025.
Cheng-Han Chiang, Xiaofei Wang, Linjie Li, Chung-Ching Lin, Kevin Lin, Shujie Liu, Zhendong Wang, Zhengyuan Yang, Hung-yi Lee, and Lijuan Wang. Stitch: Simultaneous thinking and talking with chunked reasoning for spoken language models. arXiv preprint arXiv:5507.15375, 2025.
Chenrui Fan, Ming Li, Lichao Sun, and Tianyi Zhou. Missing premise exacerbates overthinking: Are reasoning models losing critical thinking skill? arXiv preprint arXiv:2504.065544 2025.
Esther Geva and Zohreh Yaghoub Zadeh. Reading efficiency in native english-speaking and english- as-a-second-language children: The role of oral proficiency and underlying lonnitive-linguistic processes. Scientific Studies of Reading, 10(1):31-57, 2006.
Arthur C Graesser, Murray Singer, and Tom Trabasso. Constructing inferences during narrative text
comprehension. Psychological review, 101(3):371, 1994.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501.12944; 2025.
Shoutao Guo, Shaolei Zhang, and Yang Feng. Decoder-only streaming transformer for simultaneous translation. arXiv preprint arXiv:2406.00 03878, 2024.
Ewa Jacewicz, Robert Allen Fox, and Lai Wei. Between-speaker and within-speaker variation in speech tempo of american english. The Journal of the Acoustical Society of America, 128(2): 839-850, 2010.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.1672 2024.
11

---

Published as a conference paper at ICLR 2026
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.061460 2019.
Walter Kintsch. The role of knowledge in discourse comprchension: construction-intteraatin
model. Psychological review, 95(2):163, 1988.
Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville. Lims get lost in multi-turn conversation. arXiv preprint arXiv:2505.06/ 2025.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124 2024.
Mosh Levy, Alon Jacoby, and Yoav Goldberg. Same task, more tokens: the impact of input length on the reasoning performance of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), pp. 15339-1534 2024.
Baohao Liao, Yuhui Xu, Hanze Dong, Junnan Li, Christof Monz. Silvio Savarese, Doyen Sahoo, and Caiming Xiong. neward-guided speculative decoding for efficient I1m reasoning. arXiv preprint M.Xiv:2501.19324 2025.
Junyan Lin, Junlong Tong, Hao Wu, Jialiang Zhang, Jinming Liu, Xin Jin, and Xiaoyu Shen. Speak while watching: Unleashing true real-time video understanding capability of multimodal large 2026.
language models. arXiv preprint
Zhan Ling, Yunhao Fang, Xuanlin Li, Zhiao Huang, Mingu Lee, Roland Memisevic, and Hao Su. Deductive verification of ofainnoffthhuuhh reasoning. Advances in Neural Information Processing Systems (NeurIPS 2023), 36:36407-36453 2023.
Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models long contexts. Transactions of the Association for Computational Linguistics (TACI. 2024), 11:157-173, 2024.
Ruikang Liu, Yuxuan Sun, Manyi Zhang, Haoli Bai, Xianzhi Yu, Tiezheng Yu, Chun Yuan, and Lu Hou. Quantization hurts reasoning? an empirical study on quantized reasoning models. arXiv preprint arXiv:2504.04483 2025.
Mingbo Ma, Liang Huang. Hao Xiong. Renjie Zheng. Kaibo Liu, Baigong Zheng. Chuanqiang Zhang, Zhongjun He, Hairong Liu, Xing Li, et al. Stacl: Simultaneous translation with im- plicit anticipation and controllable latency using prefix-to-prefix framework. arXiv preprint 1rX1v:1810.08898, 2018
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems (NeurIPS 2023), 36: 46534-46594, 2023.
Seyed Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. The Thirteenth International Conference on Learning Representations (ICLR 2025), 2025.
Jiay Pan, Xiuyu Li, Long Lian, Charlie Snell, Yifei Zhou, Adam Yala, Trevor Darrell, Kurt Keutzer, and Alane Suhr. Learning adaptive parallel reasoning with language models. arXiv preprint a504v:2504. 15466, 2025.
Ruoyu Qin, Zheming Li. Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: A kveache-centric disaggregated architecture for Ilm serving. arXiv preprint arXiv:2407.00079, 2024.
Matthew Raffel, Victor Agostinelli, and Lizhong Chen. Simultaneous masking, not prompting op- timization: A paradigm shift in fine-tuning llms for simultancous translation. arXiv preprint arXiv:2405. 10443, 2024.
12

---

Published as a conference paper at ICLR 2026
Nils Reimers and Iryna Gurevych. Sentence-bert. Sentence embeddings using stamese bert- networks. arXiv preprint 19X1v:198..10066 2019.
Gaurav Srivastava, Shuxiang Cao, and Xuan Wang. Towards reasoning ability of small language models. arXiv preprint arXiv:2502. 2025.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568:127063. 2024.
Yang Sui, Yu-Neng Chuang, Guanchu Wang. Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, et al. Stop overthinking: A survey efficient reasoning for large language models. arXiv preprint arXiv:2503. 16419, 2025.
Oyvind Tafjord, Bhavana Dalvi Mishra, and Peter Clark. Proofwriter: Generating implications, proofs, and abductive statements over natural language. arXiv preprint arXiv:2012.13048 2020.
Jidong Tian, Yitian Li, Wenqing Chen, Liqiang Xiao, Hao He, and Yaohui Jin. Diagnosing the first- order logical reasoning ability through logicnli. Proceedings of the 2021 Conference on Empirical Methoos in Natural Language Processing (EMNLP 2021), 2021.
Junlong Tong, Jinlan Fu, Zixuan Lin, Yingqi Fan, Anhao Zhao, Hui Su, and Xiaoyu Shen. Llm ef- fective streaming processor: Bridging streaming- batch mismatches with group position encoding. arXiv preprint arXiv:2505. 16983, 2025.
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou Self-consistency improves chain of thought reasoning in language models. The Eleventh International Conference on Learning Representations (ICLR 2023), 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Sfain-of-thouuh prompting elicits reasoning in large language models. Advances in neural information processing systems (NeurIPS 2022), 35:24824-24837.222 2022.
Sheng-Lun Wei, Sheng-Kuang Wu, Hen-Hsen Huang, and Hsin-Hsi Chen. Unveiling selection biases: Exploring order and token sensitivity in large language models. In Findings of the Asso- ciation for Computational Linguistics ACL 2024, pp. 5598-56211 2024.
Tong Wu, Chong Xiang, Jiachen Wang, G Edward Suh, and Prateek Mittal. Effectively controlling reasoning models through thinking intervention. arXiv preprint arXiv:2503.243700 2025.
Heming Xia, Chak Tou Leong, Wenjie Wang, Yongqi Li, and Wenjie Li. Tokenskip: Controllable chain-of-thought compression in 1lms. arXiv preprint arXiv:2502. 12067, 2025.
Roy Xie, David Qiu, Deepak Gopinath, Dong Lin, Yanchao Sun, Chong Wang, Saloni Potdar, and Bhuwan Dhingra. Intericaved reasoning for large language models via reinforcement learning. arXiv preprint ar05v:2505. /9640, 2025a.
Zhifei Xie and Changqiao Wu. Mini-omni: Language models can hear, talk while thinking in streaming. arXiv preprint arXiv:2408. /6725, 2024.
Zhifei Xie, Ziyang Ma. Zihang Liu, Kaiyu Pang, Hongyu Li, Jialin Zhang, Yue Liao, Deheng Ye, Chunyan Miao, and Shuicheng Yan. mini-ommi-rrasoneer Token-level thinking-ing thinkingiin-spcakee insseakinng in large speech models. arXiv preprint arXiv:2508.15822 2025b.
Zikai Xie. Order matters in hallucination: Reasoning order as benchmark and reflexive prompting for largellagguage-moder arXiv preprint arXiv:2408.05: 050993 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv: / 1809.09600. 2018.
13

---

Published as a conference paper at ICLR 2026
Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain- of-thought reasoning in llms. arXiv preprint arXiv:2502.03377, 2025.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhen- guo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. The Twelfth International Conference on Learning Representations (ICLR 2024). 2024.
Jintian Zhang, Yuqi Zhu, Mengshu Sun, Yujie Luo, Shuofei Qiao, Lun Du, Da Zheng, Huajun Chen, and Ningyu Zhang. Lightthinker: Thinking step-by-step compression. arXiv preprint arXiv:2502. 15589, 2025a.
Yuwei Zhang, Jayanth Srinivasa, Gaowen Liu, and Jingbo Shang. Attention reveals more than tokens: Training-frre frree long-context reasoning with dhentiionguided retrieval. arXiv preprint arXiv:2503.098//9 2025b.
Anhao Zhao, Fanghua Ye, Yingqi Fan, Junlong Tong, Zhiwei Fei, Hui Su, and Xiaoyu Shen. Skipgpt: Dynamic layer pruning reinvented with token awareness and module decoupling. arXiv preprint wrxii:25506004799 2025.
Anhao Zhao, Ziyang Chen, Junlong Tong, Yingqi Fan, Fanghua Ye, Shuhao Li, Yunpu Ma. Wenjie Li, and Xiaoyu Shen. On-policy supervised fine-tuning for efficient reasoning. arXiv preprint arXiv:2602. 13407, 2026.
Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. [DistServe): Disaggregating prefill and decoding for goodput goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 193-210, 2024.
14

---

Published as a conference paper at ICLR 2026
A MOTIVATION AND PROSPECTIVE APPLICATIONS
A.1 MOTIVATION AND PROSPECTIVE APPLICATIONS
One motivation for introducing streaming thinking arises from the limitations of the conventional batch reasoning paradigm. In batch reasoning, model must wait until the entire input is observed before producing any reasoning, which introduces latency, prevents timely responses, and under- mines robustness in handling dynamic sequential information. In contrast, streaming thinking enables large language models to reason incrementally as input arrives, closely mirroring the hu- man cognitive process of thinking while reading. A second motivation lies in its broad practical value: this capability unlocks a wide range of applications where timely reasoning and continuous adaptation are essential. We provide some potential applications as follows.
Real-time Dialogue and Interactive Systems In advanced conversational agents or AI tutors, streaming thinking allows the model to perform continuous reasoning on a user's partial utterances. For example, it infer user's evolving intent, reason about the logical consistency of their arguments in real-time, and formulate clarifying questions or guidance without waiting for the user to pause. This enables fluid, collaborative dialogue rather than static, turn-based exchange.
Long- Con-eCn Analysis and Synthesis When processing lengthy documents, live transcripts, or codebases, the model can engage in incremental synthesis. continuously builds and refines men- tal model of the content, reasoning about causal links, logical dependencies, and thematic connec- tions across the information stream. This is crucial for tasks like real-time meeting summarization, where key decisions and action items must be identified and reasoned about as they emerge.
Auman-AI Collaborative Reasoning In creative and analytical workflows, streaming thinking facilitates true partnership. An AI can act as thought partner. reasoning alongside human analyst or writer. As the human proposes an idea, the Al can immediately reason about its implications, offer counter-arguments, or synthesize it with prior information, creating a dynamic and interactive brainstorming loop that accelerates discovery and innovation.
Dynamic Decision-Making and Planning In high-stakes environments such as autonomous nav- igation or real-time financial market analysis, streaming thinking is critical. An autonomous agent must reason about a constantly changing environment from a stream of sensory data to make timely decisions. This involves continuously updating its world model, predicting future states, and re- evaluating its action plans based on the most current information, process impossible under the latency of a batch thinking paradigm.
Embodied Intelligence and Robotic Control The streaming thinking paradigm is also particu- larly well-suited for the challenges presented by embodied Al. An embodied agent, like a robot, operates within dynamic physical environment, which distinguishes it from models that process static information. It continually receives multi-modal sensory data and is required to respond to this information in a timely manner. In this context, streaming thinking allows the agent to engage in continuous perceptual reasoning, interpreting incoming data to consistently update its internal model of the world. This capability supports dynamic instrumental reasoning. where the model can flexibly plan and re-plan its actions to navigate changing conditions and work towards goal. For example, a household robot would need to reason about multiple factors at once, such as the move- ment of a person, the delicacy of an object it plans to handle, and the best path through cluttered space, while adapting its motor commands accordingly. This close coupling of perception, reason- ing, and action in real-time loop is core aspect of embodied cognition, and achieving it presents significant challenges for a conventional batch thinking approach.
Streaming Multimodal Understanding Beyond text, streaming thinking is vital for interpreting continuous non-verbal data streams, such as live video feeds or audio environments. For instance, in video understanding, a model must reason about the temporal causality of events as they oc- cur-identifying that an action in frame f is consequence of an event in frame / This is essen- tial for applications like live sports commentary, real-time video surveillance for anomaly detection, or accessibility tools that provide live descriptions of the visual world for the visually impaired. By maintaining an evolving memory of the visual stream, the model can provide coherent, context-rich interpretations without the need process the entire video offline.
15

---

Published as a conference paper at ICLR 2026
A.2 RESEARCH SCOPE AND POSITIONING
This work positions itself as pioneering exploration into the paradigm of Streaming Think- ing-technically enabling Large Language Models (LLMs) perform concurrent reading and rea- soning. Our primary contribution lies in establishing the architectural and training methodologies required to adapt LLMs for efficient, continuous data processing.
Regarding our evaluation scope, we utilize mathematical reasoning, logic reasoning, and contextual QA tasks primarily as a controlled testbed to verify the logical consistency of our streaming frame- work. These deterministic domains serve as a rigorous testbed to verify that our model can maintain logical consistency and accuracy under the strict constraints of streaming inference. Our future work aims to extend this paradigm to complex streaming scenarios.
B STREAMING THINKING PARADIGM
B.1 COGNITIVE FOUNDATIONS OF STREAMING THINKING
Human reasoning during reading naturally unfolds in a streaming manner-people engage in com- prehension and inference as information arrives, without waiting for the complete context.
Our proposed StreamingThinee Thinker architecture draws structural inspiration from established models of human discourse comprehension, specifically the Construction-Integrann (CI) model proposed by Kintsch (Kintsch, 1988). The CI model posits that comprehension occurs in two alternating cycles:
The Construction Phase: A bottom-up up process where linguistic input triggers the activation of concepts and propositions based on local context. In this stage, the cognitive system acts as a high-bandwidth information buffer, prioritizing the establishment of local co- herence (e.g., resolving immediate pronouns or connecting adjacent clauses) over global consistency. Crucially, this phase is non-selective: it allows multiple, potentially conflict- ing interpretations to co-exist in temporary cognitive state. This ensures that no critical information is prematurely discarded before the full context is available
The Integration Phase: Once the initial propositional network is constructed, the cogni- tive system iteratively updates node activations based on their connectivity and connection strengths. Through this process, the system resolves local ambiguities (e.g., polysemy) and filters out contextually inappropriate inferences. The result is a refined macrostruc- ture where only the information strictly consistent with the global context remains active, ensuring a unified and logical understanding of the discourse.
Inspired by this cognitive process, proposed in the main text Streaming Thinking Paradigm that enables large language models to reason concurrently with incremental input. Our stream- ing phase corresponds to the construction phase, where the model performs "shallow reasoning" (e.g., entity tracking, intermediate calculation) to process incoming chunks and maintain local co- herence (Graesser al., 1994). Our final reasoning phase corresponds to the integration phase, where the model utilizes the fully accumulated context (KV cache) to synthesize globally consis- tent answer. This theoretical alignment explains why our model avoids the "hallucination" pitfalls of premature guessing: like human reader, it defers the final integration of complex causal chains until the necessary global information is available.
In this appendix, further elaborate this paradigm by providing detailed explanations and illus- trative examples that clarify its operational design and the ordering of context and question.
At each step, the model incrementally processes the incoming sentence, focusing on shallow and progressive comprehension, which aligns the construction phase in human comprehension. Specif- ically, we designed distinct categories of tasks for the model during local construction, serving as cognitive scaffolds for establishing local coherence. These tasks (e.g.. intermediate calculation, en- tity tracking) compel the model explicitly process and encode immediate details, transforming raw input into structured representations without prematurely committing to a global conclusion. (1) understanding and summarizing key information, (2) explaining ambiguities and reorganizing semantic relations, (3) extending logical implications, and (4) skipping thinking step when the con- tent is irrelevant to the question.
16

---

Published as a conference paper at ICLR 2026
After completing incrementally reading and reasoning, the model shifts its focus from local coher- ence to global interpretation. It leverages the full context now available its memory (KV cache) to perform the deep, unconstrained reasoning that deliberately deferred during the streaming phase. Specifically, the model may: (1) directly produce an answer, representing the shallowest depth; (2) further integrate global information to achieve deeper comprehension; (3) perform reflective reasoning on top of global integration to obtain the most reliable reasoning outcome.
B.2 EXAMPLES OF PARADIGM DESIGN
We instantiate Streaming Thinking on three representative tasks tasks-male math reasoning. logical reasoning, and long-context QA-to show how the paradigm incrementally processes input, manages interme- diate reasoning, and enables depth-wise answers (see Box B. la
Math reasoning (Box B.la). Given a multi-conditions problem, the model reads the context sen- tence by sentence, identifying rate/quantity information, skipping irrelevant text, and maintaining interpretable intermediate results (e.g.. partial counts). The pipeline supports three reasoning depths: DI direct computation from the current notes, D2 a globally consolidated solution that aggregates all intervals, and D3 a reflective pass that re-checks each arithmetic step to ensure reliability.
Paradigm design for Streaming Thinking in Math Reasoning
Input sentences
Streaming thinking
Question
Sthinks
popcoras does Mohamned brother?
Okay, need calculate
popcora
Motummed brother.
2. Mohammedd is popping popcorn for a snack)
3. As the skilllr of keemels hears the kemels sun popping faster,
<d pol
100 pop first 22 seconds of cooking, then times amount
Heating skillet accelerates kemel pooping.
popping terval, followed by a secood interval where the amount tiples. yiekiing 100x3=100 keendss)
process starts
5. The kernel inerease to 6 times the initial popping rate in the next 22
acconds:
In the nest scconds, the rite incresses limes the initial note. menning the number popped
based
seconds, popping slows
the
100, resulting
pesiod:
past 22 seconds.
For the final 22 seconds of cooking, the rite slows to half the proced-
7. After Mohammed takes the skiller off the heat, quarter of the muater
ing interval's rate, which gives 600s2=300 kemels.
kernels that popped the final 22 seconds cooking alsoo from the residual
After the skillet removed from the best, residuall heat pops ad final cooking period, adding
quaner amouni
8. Mohammed gives 2/5 the made popcorn his brother Yusuf.
300-1=75 kennels:
Finally, the woal amoour of popcorn the sum all inter
000-300-600-300-7555 Mobammed gives 2/5 of this to his brother Musaf, which
1375x225-550 kernets
Oppiooaal depth, streaming thinking with direct ansver)
So. the answer 275:
10ptional depth, streaming thinking with global thinking!
giving 550 pieces
1375-
550 The dHiference is 625-5501:255
[Opiional depth, streamingg thinking with global thinking & reflection) So, after giving 550 pizces to his brother, Motammed retains 1375: 550-315 The difference 520-550-275 Walt, check The 100, second
300, (not cumulative), 600 that's right. Fourth is half the previous in: serval's rate, so half 600 is 300, yes. Then residual best adds quarter that final 15 correct Total: (00-300--000300005---
mis
gives
1,3759845-550 Thar teaves Mohammed
Differnce: 878-550-275. steps check out yes, Mohammel his 275 more pinces than his brocher
<)<hink>
17

---

Published as a conference paper at ICLR 2026
Logical reasoning (Box B.1b). Premises are normalized as explicit statements; each incoming sentence is mapped to entities and relations, while unrelated sentences are skipped. Local impli- cations are updated online, and the final decision is produced immediate verdict when premises suffice; D2 integrates all premises to assess entailment vs. contra- diction; D3 performs reflective consisteney check, which may revise a premature "True/False" to "Unknown" when a critical premise (e.g., location) is not actually supported by the context.
different depths: DI yields an
Paradigm design Streaming Thinking in Logic Reasoning
Input sentences
Streaming thinking
Question
<t hiiikkk
I. the conclusion true given the premisest. ("Trae. "File", "Unknown")
1. Olay, the question is asking whether the conclosion thit William Dickinsoo went schools in the United Kingdom for both high school and university
Sonclasion: William Dickinsos went schools locased in the United King-
dom both school and
based given premises. passible answers Thee, False, Unksowa.
2.William Dickinson was a Brilish politician who sot in the House
William Dickinson atlended Westminster School for high school and then the University of Edinburghi
Commons William Dickinson atlended Wesuminster school for high school
and then the Universily of Edinburgh.
University Edinburgh univoritt loeated the United
University Edinbbugh located United Kingdom.
Kingdom
4. William Dickinson supported the Portland Whigs
5. People who supported the Portland Whigs did not get seal in the
<skipp
Parliament,
(Opiional depth, stroaming thinking direct answer)
Therefon: the inswer is True
[Opiional depth, streaming thinking with global thinking]
specifies
univerity
Kingdom.
explicitly states ottended
Wesminster School and the University of Edinbargh. Since the University of Edisbbrgh is contimedd to the UK the unives part sotisfted
However, the location of Westminster School directly stated
the context does not explicilly mention its location. Without explicit con- fimattoo that Westiinster School is in the LIK, the conclusion about bott instintions being in the UK remsins uncestiin. Therefore, the answer is
Optioonal depth, streaming thinking global thinking reflection
The conclusion specities that both his high school and univeriity were Jocased in the Unitd Kingdom, The context explicitly states he uttended
Wesminster School and the University of Eilinburgh Since the University Edinburgh confimeed
university part satisfied
However, the location of Westminster School not directly stated in the cuntest. Westminster School is well well- known school in London, UK but the context does not explicitly meation isi location. Withour explisir con- fimaation Westiinster School instiutions being
canclusion both
ramains uncertain. Walt, let
again.
The context only mentions the University of Edinburgh's is location in the UK. Wessminster School is not explicilly stated to be in the UK, Even
common knowledge might suggest the premises provided this
fully verifled the given premises. Therefore, the answer Unknown:
conclusion
<ichnnk>
Context-based QA (Box B.lc). For scientific document-level passages, the model streams through long context, filtering sentences that are irrelevant, extracting mechanistic cues (e.g., causal roles, experimental outcomes), and merging compatible evidence across distant spans. Dl gives concise answer when a single decisive statement appears; D2 synthesizes multi-sentence evidence for a stronger justification; D3 runs a reflective verification that cross-checks all cited evidence for agreement before committing to the final answer.
18

---

Published as a conference paper at ICLR 2026
Paradigm Design for Streaming Thinking in Context-based QA Reasoning
Input sentences
Streaming thinking
Question
1. mitechondria play a remodelling laoe plant leives during pro-
Okay, lets see The question asks whether mitochondrin contrilute the remoddlngg of las plant leaves during programmel cell death. I neod focus oa "awtcchondia"" "remedeling of loee plant leaves', and "peo- graaused cell death")
grmmmed cell death?
Context
Programmed cell death (PCI) the regulated death within
sspatisan
introduce Programmed cell (PCD)
3. The lace plant tApenageton malagisemennis)) produces perferations its leaves through PCD)
regulated death of cells within an organism
& la lace planss, leaf perforations are created through programmed well
4. The leaves of the plant consist latticework loegituiial and
Souh
enclosing arcales
pa
cells the center
arcoles and progresses
kekkpp
aucands, stopping apposimaaely five cells from the vasculattre
This study investigrnes mitochondiial invelvement in developmentll
6. The role mitochondria during PCD has been recognizeed in animals;
cell death lace plant leaves.
however.
studied
planss.
Researchers divided arcole three PCD cells, PCD cells, and PCD cells, reflecting the progressaon
7. following paper elacidates role
dynamics
during developmentally regalted PCD vivo A. madagnscriennss single arosle within a window stage keaf (PCD is ocvuriing) was divided
They stained mitechondris window-stage leaves MinTTraker Red to observe their properties.
into three based progression PCD: cells that will not (BPCD), cells
Minecheddri clussifee into four dynamic states (MI-M4) based
undergo PCD (NPCD), cells
distihuttoon mocilty membrane posential:
steges (LPCD))
10. DNA frameestation assoy (TUNEL) auclear degradacion conrspending with mitechendrial changes, linking mitechondiial dynam- ics to PCD progression.
8. Window stage leaves were stained with the mitochondiaal dye
MiloTrackeer Red CMXRos and examined
9. Mitochondrill dynamics delineated fouz (MI-M5)
11 <Skips
characirristis including distibution, motility membrane
12. The potencial involvement mitschondrial permeshilitt transition
pecenial (A x
pore was pharmacokogically with eyclosporine Al
10. A TUNEL assa showed fragmoned aDNA in gaidient over these
13. Treament reducad the mimher of leat perforations and prrserved mi- Rochindiia a state similar healthy cells, showing that Nocking milo- chondrial poce activity suppressed PCD.
milochondrial staees
Chioroplasts unssvscuolar stmads also examined
imaging.
12. The possibbe impoctanee of mitechendral permeabllity trassiiio
OOpiioaal depth, streaming thinking with direct ausverl Therefore, the answer is No.
pore (PTP) formation during PCD was indineclly examined via in viva syclosporine (CsA) treatment
OOiisnaal depth, stroaming thinking with global thinking)
significanily perforations compored commols, displayed mitnchondrial dynamies similar to that nos-CDD
treiment resulted
plant
study demonstratis
change
lly, and membrnne potential during the slapes of programmed sell death lake plaar leaves. DNA tragmentition cocresanns with these mitochon- drall staes, and inhibitic of the mitechondiall per permeahility transtrion
perforation
Tagether evidenez shows
tochondhia actively involved regulating and executing cell that sculpts lace plant leaves. Therefore, the answer Yes
Optiional desth, streaming thinking with global thinking & reflection)
The suuly demoestrnte that milochondias chinge disiiittion, modil fly
potential daring stages programmed cell hos plant leaves, DNA fragmentation corresponds with these milo- chondtial states, and inhibition of the miscchndrial permesbillly tran- sition reduces pesforation formarion. Together, this evidence shows mitechondria actively involved regelating exceating cell that sculpts plant Walt, me again. locs plant forms leat perforation through PCD, so milochundraa alfect PCD, they alfect remodeling. The study showed minchendia shift in disuibu: motiliy, membrune posential during PCD
that
align DNA fragmentatioe,
mitechondial peimaailliy
tnnsition pere reduced perforstions and preserved healthy- din This, mitcchnnria acrively regulate and evecute PCD in shoping lace plant leaves. Therefore, the answer Yes. <)chink>
19

---

Published as a conference paper at ICLR 2026
B.3 EXAMPLES OF THE ORDERING OF CONTEXT AND QUESTION
Math reasoning (Box B.2a). Question- first (left panel) anchors the target quantity upfront, each incoming sentence is parsed for rates/amounts relative to the goal, with irrelevant lines skipped and partial counts maintained online. Context-first (right panel) encourages incremental accumulation of numerical evidence without a declared target; once the question appears, the model consolidates cached quantities to compute the final difference. Both settings yield interpretable traces but em- phasize goal-driven vs. evidence-driven processing, respectively.
Example: Order of Context and Question for Streaming Thinking in Math Reasoning
Question first
Context first
Input sentences
Input sentences
Question
Context
1. How many more popsoens does Mohammed eat than his brother?
1. Mohammed is popping popeoen for a snack.
Context
2. As the skillet of kernels heats up. the kernels start popping faster.
2. Muhammed is popping popsoen for snack.
3. 100 the fins. 22 secomis cooking, then times that amount in
skillet of karnels heats up, the kernels start popping
the nest 12 seconds.
4. 100 pop in the finst 22 seconds of cooking, then 3 times that amount
The kernels increase to times the initial pupping rate in the next 22 s.
the next 22 secands.
5. In the final 22 seconds, the popping slews down half the rate as the
5. The kernels increase 6 times the inital popping rate in the nost 22
past 22 sccands.
6. the final 22 seconds, the popping slows down half
the
After Mohammed takes the skiller off the quaster the number that popped in tinal 22 seconds of cooking also pop from residual heat.
past 22 spconds.
7. After Mehamesed takes the skilles off the beat. a quaster of the mumber
of kernels that poyped in the tinal 22 seconds of cooking also pop from the residiall heat.
7. Mohammed gives 2/5 of the made popcorn to his bruther Yisuf.
Question
8. Molammad gives 2/5 the made popcoen his benther Yesuf.
many more popcoens Mohammeed thin his brother?
Streaming thinking
Streaming thinking
<t.inkk>
<think>
I. Okay, need calculate many more pieces of popcorn Mohummed
Dkay, the input sentences are start with that Mohammed pups popcorn
than brother.
snack:
2. <5k 1p
2. Then we know that popping speeds up as the skillet heats.
3. Heating the skilles accelerates kemel popping-
3. Then the process starts with 100 kernels popping in the first 22-scoand
The process starts
kernels pupping the first second in-
interval, followed by a second interval where the amount txplles yieking
terval, followed by second interval where the amount triples, yielding
100x3-300 kemels.
100x3-300 kemels.
4. the next 22 seconds, the rate increases to cimes the inital ratc, mean- ing the namber of popped kernels is based on the initial amount of 100, resulling 100w6=600 kemels for this thind period. seconds cuoking,
5. In the next 22 seconds, the rate increases to times the initial rate, moon-
the number of popped kernels is based the initial amount 100, resulting 100v6=6000 kermels this thind period.
For the ing ineevva'ss i's rate, which gives 600)-23000 kemels.
to half the preced-
6. For the final scconds of cooking, the rate slows
the presed-
ing interval's is rate, which gives us 6000:2-300 kensels.
6. After the skillet is removed from the hear, residual heat paps an additional quarter of the amount from the tinal cooking pariod, adiing 300+1=75 kemels.
7. After the skiller removed from the hear. residual heat pops additional quarter of the from the final cooking period, adding 300-1=75 kemels.
amount of popsoen sum intervals: 100+ 100+300+600+300 300-00003000-5--1111110 300-75-3355 Mohammed gives 2/5 of this to his brother Yesuf, which amounes to 1375x245-550 kemels.
Finally,
8. Finally, the lotal amount poprorn is the sum of all intervals:
100. (00.3005000 30000000-0511100 75-355, Mohammed gives 2/5 of this his brother Yosuf, which amounte to 137500-5-500 kemels.
8. Now have real the question and nesd to caleulate how many mooe pieces poproen Mehammed his beother.
[Optional deptk. strenming thinking direct auswer)
So, the answer is 275.
(pptional depti, streaming thinking with direct auswer)
[Optional depti, streaming thinking with global shinking
So, the answer is 275.
So, after giving 550 pieces to his brother. Mohammed retains 1375- 550=825, The difference 825-550=275.
[Optional depth, streaming thinking with global thinking]
So, after giving 550 pieces beuther, Mohammed retains 1375- 550=825, The difference is 825 $50=255
[Optional depth, strenming thinking with global thinking & reflection]
So, after giving 550 pieces to his boother, Mohammed retains. 1375. 550-825 The difference is 825-150-275. Walt, let me check again. The first interval is 100, second 3x that 300, thidd 6x the initial that's right. Fourth is the previous interval's rate, so half of 600 is 300, yes. Then residual hest adds a quatter of that final 300, so 75 correct. Total: 300+3006000 15077551,375 He gives 2/5 to Yosuf: 1, 1,77522-55500 Thit leaves Mohammed with 825, Diffee- 825. 250-550-275 All stepe check out, yes. Mohammod
[Optional deptlt, streaming thinking with global thinking & reflection)
So, after giving 550 pieces to his beother, Mohammed retains. 1375-
550-825. The difference 825-550-275. Walt, let check agsin. The times - 300, six times
cumulative),
first inerval was rate (not cumalative) -t 600, After that, half of the previous 600 - 300.
and tinally the residual quater of that 300 - 75, Adding them all, 100 300 600 + 300 + 75 1375, consistenr. Giving 2/5 to Yusuf: 1375 difference indoed
$50 Subemeting leaves 825,
picces thin brother.
275. So yes, after rechocking, the conclusion Mooil--e ands
<lthinkk>
with 275 more picces thas his brother.
<), clthiink>
20

---

Published as a conference paper at ICLR 2026
Logical reasoning (Box B.2b). Under question-first ordering, the model enters hypothesis- testing mode: each premise is checked against the claimed conclusion, distractors are skipped, and if a critical premise is missing, depth D3 returns UNKNOWN. With context-firs first ordering, the model first normalizes premises into entities and relations; after the question is posed, it adjudicates entail- ment/contradiicionn/kknooo
using the cached structure. Answers remain consistent across orders, while the traces reveal different timing of inference.
Example: Order of Context and Question for Streaming Thinking in Logical Reasoning
Question first
Context first
Input sentences
Input sentences
Question
Context
1. Is the conclusion trae given the premises", ['True', "Falle, "Unknown"]. 1.
1. William Dickinsn was a Briissh politicion who sot in the House of Com-
conclusion William Dickinson went schools located in the United King- dom both high school and univensity.
mons William Dickinson attended Westminster school for high school and
Univensity Edinburgh.
Context
The University Edinburgh is university lacated the UK.
2. William Dickinson was a British politician who sot the House of Com-
3. William Dickinson supported the Poctiand Whigs.
mons William Dickinson attended Wesmilnster school for high school and
4. People who supported the Portland Whigs did not sest in the Par-
then the University Edinburgh.
liament.
The University Edinburgh university located UK.
Question
4. William Dickinson supported the Portland Whigs.
S. the conclusion true given the premises", I I'True', 'aale", "Unknown") [ conclusion William Dickinsna went to schools located in the United King- dom for both high school and university.
5. People who supportad the Portaand Whigs did not get seat in the Par-
limentt
Streaming thinking
Streaming thinking
<thinks
<thinks
1. Okay, the question is asking whether the conclusion that William Dickinson schools United Kingdoos both high school and
Okay,
known that William Dickinson sllended Westminster
School high school and thes the University of Edinburgh
university true, based the gives premises. The possible answers ue
And, the University of Edinbargh a univeritty situsted in United Kingdom.
Trme, Fales, or Unkeown.
2. William Dickinson ansended Westminster School for high school and then
3. William Dickinson supports the Portland Whigs.
the University Edinburgh.
Then, according the rule, anyone who supported the Porllnd Whigs
3. The University Edinburgh located the United Kingdom.
not obsain seat Parliament. William Dickinson
scnt in Parliament.
5. <g
5. Now, the question is aoking whether the conclusion - that William Dick: inson went schools the United Kinglom for both high school and uni- versity crue.
(Optional depth, streaming thinking with direct auswer! answer is 275.
(Optional depth, streaming thinking with global chinking! The conclusion specifies that both his high school and university were located the UK, The cuntext explicilly states he assended Westminster School Univenstt Edinburgh. Since the University of Edinburgh is contimmed to in the UK, the university part is satistied, Howeves, the Incation of Westminstes School is not directly stated in the context, West minster School is well-known school in London, UK, but the contest does explicitly mention location. Without explicit confirmation that West minster School is in the conclusion about both instinations being the UK remains uncertain. Therefore, the answer is Unkaown.
(Optional depttt, streaming thinking with direct auswer)
Therefore, the unswer True.
[Optional depth, streaming fhinking with global thinking]
The context explicilly states be altended Westminster School and versity of Edinburgh, gh. Since the University Edinharg contirmed in the UK, the university pat is satisfidd, However, the location of Wost- minater School is not directly stated in the context. Westminster School a well known school London, UK but the does not explicidly tion its location. Without explicit confimation that Westminster School in the UK, the conclusion about both institutions being in the UK remzins uncertain. Therefore, the answer is Unknown.
[Cpptiona degtt,, steaming thinking with global thinking & reflection] The conclusion spacifies that hoth his high school and university were lo- coted in United Kingsom, The Westminster School and the University of Eddnbbugh. Since Universtty of Edinburgh is confirmed to be in the UK, the university part is satisfied. However, the location of Westminster School not directly stated the context. Westminster School well- known school London UK but context does explicilly mention location. Without explicit contir- mation that Westminster School is in the UK the conclusion about both in- stnutions heing in the UK remains uncestain. Wait. lot me chock again. The contest only mentions the Universtyy of Edinharhh's location the UK. Westminster School explicilly stated in UK Even though common knowledge might suggest the premises penvaded not con- firm this, Therefore, the conclusion cannot be fully verified from the given premises. Therefore, the answer Unknowa.
[Optional depel, streaming thinking with glabal shinhing & reflection] The context explicilly versity of Eddinburgo. Since the University of Edinhurgh confimmed in the UK, the university part is satistied, However, the location of West-
explicilly states he sttended
he sttended Westminster School and Uni-
minster School is not directly stated in the contest. Westminster School a well known school London, UK. but the context does not expliciily tion its location. Without explicit confimmation that Westminster School in the UK, the conclusion about both institutions being in the UK remains mncettnnn Wait, let me check again. The coneext only memtons the Univer- sity of Edinburhh's 's location in the UK, Westminater School is not explicifly suaed in the premises provided not confirm Therefore, the conclasion cannot be fully verified from the given premises. Therefore, the answer is Unknown.
Even though common knowlodge might suggest it
<)think>
<)thinn>>
21

---

Published as a conference paper at ICLR 2026
C STREAMING COT GENERATION
C.I DOES INCREMENTAL SENTENCE INPUT WORK?
A straightforward idea is to feed sentences into LLMs sequentially and expect sentence-by sereence-by-sentenee reasoning based on the accumulated input. However, our experiments show that when given only a single sentence without streaming fine-tuning. the original reasoning model tends to overthink (Sui al., 2025). other words, although this sentence-level data generation method appears to align better with the streaming property, the vanilla model lacks such generative capability and exhibits reduced controllability in CoT generation.
C.2 DOES DIRECT PROMPTING OF LLMs WORK?
Another intuitive approach is to directly prompt LLMs perform streaming-style reasoning with carefully crafted instructions. In this setup, the model explicitly asked to reason sentence by sentence. Nevertheless, our experiments show that, lacking fine-grained control, prompt engineering alone is insufficient: although the model may superficially follow the instructions, it often deviates from the intended trajectory. resulting in inconsistent overly verbose reasoning. Therefore, design pipeline for CoT trajectory control based on the complete input, which enforces sequential reasoning in LLMs through explicit boundary constraints and teacher guidance.
C.3 CONTROL COT PROCESS
In this work, we design multi-stage pipeline for streaming CoT generation and control. The pipeline consists four key components: (1) boundary-toker insertion finely define reasoning units and ensure sequential alignment; (2) instruction- prompting to guide the LLM toward the desired streaming paradigm; (3) teacher-model guidance to reconstruct reasoning chains with improved structural consistency and semantic fidelity; and (4) token-level intervention to control the trajectory of CoT, enabling adjustable reasoning depth and style.
Boundary Token Insert In our proposed streaming thinking paradigm, a complete sentence serves as a atomic unit of cognition. This design achieves a critical balance between the fine- ane-ggained na- ture of streaming processing and the preservation of semantic integrity in the information being processed. To implement this design, we introduce a structured protocol boundary tokens to ex- plicitly demarcate the model's 's thinking boundaries. Specifically, this protocol distinguishes between input signals and the model's generated reasoning states:
Input Demarcation: We employ two distinct tokens to structure the input stream. The <EOS> (End-of-SSenennce) token follows each contextual sentence, triggering an incremen- tal thinking step for context assimilation. In contrast, the <EOQ> -ot---Q-uuuttin token marks the end of the problem description, signaling shift from context processing to swer formulation.
Thinking State Demarcation: The model's thought process is, in turn, segmented by three corresponding output tokens. The <EOT> (End-of-Thought) token concludes each incre- mental reasoning segment associated with context sentence. The <BOQ> token concludes the reasoning phase triggered by the question. Finally, the <EOR> of-oofsoRnnnn to- ken signifies the completion of the entire streaming thinking chain with controllable depth, preceding the final answer generation.
important to note that the boundary tokens are introduced primarily as a data generation and evaluation methodology. Their purpose provide structured prompt for the LLMs generate the desired streaming chain-of-thought behavior. Moreover, in real streaming reasoning scenarios, once the LLM outputs a boundary token, the current reasoning unit can be regarded as completed. Thus, during training, boundary tokens serve as supervisory signals that guide the model to recog- nize when a reasoning unit should be terminated.
There is an example for boundary tokens insert.
22

---

Published as a conference paper at ICLR 2026
Boundary Tokens Insert for Generation
Input sentences
Streaming thinking
Question
<think>
Okiy. calcalate many pieces popsoen
1. How many more popcorms does Mohummmed cul than EECO
hammed has than his beother <E00>
Context
<5<ipa<BBT>>
2. Mohammed is popping popsoen for smok
<Bk bekkip <BCT>>>>>
the skiller kemells hears up, kenels start popping fister, <ECso
process with 100 kernel pooping fisst 22-second terval, followed by a interval where the amoust triple, yiclding
100 pop the first 22 seconds cooking, times aasount the next 22 sccands, xBC00
100c==000 kemells -EOT>
5. In the next 22 seconds, the rase increases to 6 times the initial rate. meaning the number pupped kernels based the initial amouns of resuling 100x6-6600 kemel findl seconds cooking, coding interval's mee, which gives us 600-2-100 kernels cnoms
5. The kernels incrense times the initial popping in next 22 scconds.EE88>
thind period-cratt
6. In the final 22 seconds, the popping slows down to half the rate the
slows
the pre-
past 22 seconds. <EOS>
7. Afier the skillet removed from the heat, residual hear paps
7. After Mohammed takes the skillet off quaster of mumber
additional quaster the from the final cooking period, adding
kernels that popped in
seconds cooking also pop the
residual heat. <E05>
300s4-75
Finally, the lotal amount pepcorn is the sum of intervals:
8. Mohammed gives 2/5 of the made popcorn to his bruther Yo5=
000-300-0000300-55::111 Mohummed gives 2/5 of this to his brother
Yosuf, which amounts (375x2+5==55 kernels <EOT>
So, after goving 550 pizces to brohher Mohammed retains 550-825, The difference is 825-550=275 cEoto <\chink>
Table 1: Streaming reasoning instruct template.
Input:
question, context, example.
You are a streaming reasoner to perform an order reasoning based on the following Question and Context:
Input Question: ( (questioon ion]
Input Context: (contexx)
Follow these striet instructions:
1. Must process the Context in order, as if the input arrive in a streaming way.
(1) Must perform reasoning one sentence at time, i.c. When you encounter an <EOS> in the Input Context, perform reasoning about the sentence that before this <EOS>.
(2) Upon encountering <EOS>, immediately reason about the preceding sentence and end with an <EOT>,
(3) Not skip any <ROS> and Do Not forget any <EOT>, i.e. each <BOS> in the Input must match <EOT> in the reasoning.
2. If you read any Question end with <BOQ>.
(1) Explain the Question that before the <BOQ> in the Input Question. (2) End your interpretation with <EOQ>, (3) Do not add assumptions beyond the explicit mcaning of the Question.
For each input sentence, smoothly integrate the following into coherent paragraph-styl output: (1) Understanding and summarizing key information: (1-1) Ignore redundant, irrelevant, ambiguous information. (1-2)Focus on extracting core facts. quantities, conditions, roles, time, location, and other essential elements.
(2) Explaining ambiguities and reorganizing semantic relation: (2-1) Clarify vague expressions, ambi- guities, or implied information. (2-2) Rephrase the original meaning using clearer, more straightforward, and structurally explicit language. without changing the factual content.
(3) Lightly extending logical implications: (3-1) Allowed examples include: Simple calculations (e.g..
totals, differences, ratios). Natural inferences about time, sequence, or spatial order. Direct factual inference that logically follows from explicit information. (3-2) Prohibited actions: assumptions, predictions, subjective evaluations.
(4) Skipping irrelevant input: If sentence not related the Question, output <Skip><<OT> imme- diately after reading <EOS>.
4. After reading and reasoning all sentencess: DO NOT summarize or infer any judgment.
There is an in-context example:(exxe) (example).
23

---

Published as a conference paper at ICLR 2026
Prompt Template The prompt presented in Table casts the model streaming reasoner that consumes the context seweenee-by-senteeee keyed by <EOS>, and emits an immediate reasoning step ending with <EOT>. It first interprets any question ending with <EOQ> and closes that inter- pretation with <EOQ>. Each step must extract core facts, clarify ambiguities, and permit only light, local inferences; irrelevant sentences output as <5k <Skip><ECT>> No global summary is allowed, and an in-context example anchors the desired behavior.
Teacher Guidance Template The prompt shown in Table directs teacher LLM to realign an initial streaming trace that every input sentence (terminated by <EOS>) maps exactly to one rea- soning step (terminated by <EOT>). If present, the question is explained once up to <EOQ> before processing the context in strict order. The teacher may only simplify, clarify, and streamline exist- ing content-no no new reasoning, no cross-sentence references, no merges, skips, or enforcing strict locality and one-to-one pairing.
Table 2: Instruction template for teacher LLM under both question-first and context-first settings.
Input: question, context, example, reasoning.
You have to reconstruct the following reasoning process based on the Original Input and the Initial Streaming Reasoning Process, ensuring full compliance with the requirements of a streaming reasoner. The Original Input consists of multiple sentences, each ending with <EOS>, Every sentence represents an independent unit of reasoning. Your objective is to revise the initial reasoning such that each reasoning step corresponds exactly.
Follow these strict instructions:
1. If any Question before reading the Input context.
(1) Explain the Question that before the <ROQ> the Input Question at first.
(2) End your interpretation with <BOQ>.
(3) Do add assumptions beyond the explicit meaning of the Question.
Must process the Context in order, as if input arrive in streaming way.
(1) Must perform reasoning one sentence at time, i.e. When you encounter <EOS> in Input Context, perform reasoning about sentence that before this <ROS>.
(2) Upon encountering <BCS>, immediately reason about the preceding sentence and output result ending with an <EOT>.
(3) Not skip any <EOS> and Do Not forget any <EOT>, i.e. each <EOS> in the Input must match a <EOT> in the reasoning.
3. For each input sentence, smoothly integrate the following into coherent paragraph-style reasoning
output:
(1) Understanding and summarizing key information.
(2) Explaining ambiguities and reorganizing semantic relation.
(3) Lightly extending logical implications.
(4) Skipping irrelevant input.
4. For each sentence:
(1) Do not merge, skip, or delay reasoning for any sentence. Each input sentence must be immediately followed by its reasoning step.
(2) Do not quote repeat the input sentence in the reasoning output.
(3) Do not introduce new reasoning not found in the original streaming process. You may only simplify, clarify, streamline what is already there.
(4) Your reasoning must be strictly local to each sentence. Do not refer to previous or later sentences. Avoid speculation, assumptions, or global summaries when streaming reasoning.
There is an in-context example: (allee:(xxxx)) (example).
Now, begin your streaming reasoning reconstruction:
Input question: (question}
Input context: (context)
Reasoning content: {reasoning)
24

---

Published as a conference paper at ICLR 2026
Table 3: Instruct template of streaming thinking intervention.
Streaming thinking with direct answer
< iimsttrr >user(n (Instruction Entire Input]< imm iimmiddddd an__nd] >/n
<| im start >ass prassisttan wwthhttttttinnn (tttatttttiinntttainnnnnnn Thinking Content). Now, let me direct
output the answer.
Streaming thinking with global thinking
< imiatttart >user(n [Instruction Entire Input) < immeend >)n
< im.start Dassistants ansiisttattnnttin anthinnk (Streaming Thinking Content]. Now, let me start
the global thinking, focus on high-level reasoning trace that leads the final answer.
Streaming thinking with global thinking & reflection
im_start >user(n ( (Instruction Entire Input)<< i >
(eeettiinn<<Sttammiiii (Streaming Thinking Content Now, let me start the global thinking. {Global Thinking Content). Wait, let me check again.
< iimstarr | bassistant saassiistannnsshinkkt \nsthin>>thhhhhtto
Thinking Intervention Template These prompts demonstrated in Table 3 modulate how the sistant transitions from streaming thoughts (<think>) to an answer: (i) direct answer ends the streaming trace and outputs the result immediately; (ii) global thinking triggers brief high-level consolidation before answering; (ii) global thinking & reflection adds a final self-check pass. All variants preserve the streaming trace while explicitly steering when and how the final answer is produced.
C.4 QUALITY EVALUATION OF STREAMING COT
As described in the main text, we evaluate the quality of data generation using two metrics: granu- larity score and sequential consistency score. The granularity score ensures that each input sentence corresponds to a distinct and non-overlapping reasoning segment, while the sequential consistency score verifies that the reasoning process unfolds in the same order the input sequence. In this appendix, we provide a detailed explanation the two evaluation metrics.
The granularity context and that of the reasoning process. Formally, it is defined as the ratio between the number of boundary tokens in the input and those in the output:
measures the fine- fine-grained alignment between the segmentation of the input
Ngos
granularity
NEOT
where NEoS and NEOT denote the counts of boundary tokens in the input and output, respectively. score indicates that the reasoning preserves the segmentation of the input exactly, suggesting ideal boundary alignment.
The sequential consistency score evaluates whether reasoning follows the input order in streaming, segment-by eegment--y-segmeet by-gmmennt manner. We compute the semantic similarity between each input sentence and its corresponding reasoning segment:
consistency sim R2, C.)
|R |vell
where en and to denote the embedding vectors of reasoning sentence R, and input sentence Ct, respectively. We employ SentenceBERT (Reimers & Gurevych, 2019) to obtain embeddings. A higher score reflects that the reasoning faithfully preserves both the order and semantic content of the input stream. In cases where multiple reasoning sentences correspond to one input sentence, we aggregate them as a single segment.
C.5 SIMILARITY MAP OF STREAMING COT
For further qualitative evidence, provide sentence-level similarity map in Figure 1, which visualizes how reasoning aligns with the input across time. This map highlights diagonal patterns when the reasoning process maintains strong sequential consistency.
25

---

Published as a conference paper at ICLR 2026
Sentonce Simiorfty Map
Sentence Siniarrty Hap
0.80
0.24
0.21
0.25
0.20
0.31
0.51
0.63
0:85
0.33
0.24
0.45
0.35
0.25
-.02
0.43
0.34
0.64
0.59
0:21
0.65
0.20
0.20
0.16
04
0.16
0.44
0.66
0.40
0.83
0.53
0.23
0.15
2
0.28
0.52
0.68
0.64
0.47
0.11
0.09
0.42
0:38
0:40
0:46
0.47
0.90
0.45
0:18
0:14
0.31
0.32
0.99
0:33
0.09
-0.02
2
0.37
0:61
0.35
0.64
0.06
0.59
0.06
0.57
0:30
0.62
0.10
0.10
0.36
0.20
0.32
0.80
0.66
0.04
0.18
0.11
0.30
0.09
0.17
0.79
0.13
-3
0.54
82
0:36
0:50
0.34
021
09
04
ad3
D43
D44
06
021
0d1
06
949
64
Figure/ I: Examples of sentence-level similarity map between the input sentences and the reason-
ing content (from math reasoning task). High similarity along the diagonal demonstrates strong alignment, enabling evaluation through sequential consistency.
D DATASET DETAILS
Math Reasoning We construct math reasoning data from GSM-Symbolic (Mirzadeh et al., 2025), MetaMathQA (Yu et al., 2024), and tulau-peroona--math-gana (Lambert et al., 2024).
GSM-Symbolic is a symbolic variant of GSM8K that reformulates arithmetic word problems into semi-structured symbolic expressions, explicitly preserving quantities, operators, and logical rela- tions. It consists of two subsets, GSM-Symbolic-P) (5K samples) and GSM-Symbolic-P2 (2.5K samples), where P2 includes longer and more complex problems. We randomly sample 4K and 2K instances from P1 and P2 for training, respectively, while the remaining data are used for evaluation.
MetaMathQA is an augmented dataset derived from the GSM8K and MATH training sets, containing 40K samples. It integrates symbolic reasoning with natural language explanation, covering multi- step arithmetic and algebraic reasoning tasks. We randomly select 1K samples for testing and use the remaining data for training instance generation.
comprises 50K math-related examples generated by GPT-4o under di- verse instructional and persona conditions. The dataset emphasizes reasoning diversity and linguistic variation, providing rich supervision signals for reasoning style adaptation. All samples are used for generating training data.
To align with the streaming objective of "thinking while reading," we retain inputs with at least three sentences and exclude tasks with very short inputs (e.g., proofss) which emphasize deep reasoning rather than streaming reasoning. We mix the generated data from all sources, and after filtering for correctness and evaluating the quality of streaming CoT, retain 7.9K samples for training.
Logical Reasoning For logical reasoning, evaluate ProofWriter (Tafjord al., 2020) and LogieNLI (Tian et al., 2021). ProofWriter is a benchmark designed for evaluating multi-step logical reasoning and theorem proving capabilities in natural language. Bank corpus, where each example consists of hypothesis and a set of supporting facts expressed in natural language. The task requires the model to infer whether the hypothesis entailed, con- tradicted, or neutral given the supporting facts, while optionally generating explicit reasoning chain that connects the premises to the conclusion. LogieNLI benchmark designed to evaluate the logical reasoning ability of language models beyond surface- level semantics. Each example in LogicNLI consists of a premise and a hypothesis pair, annotated with of three logical relations: entailment, contradiction, neutral. Unlike conventional NLI datasets that focus primarily on lexical or syntactic cues, LogicNLI emphasizes reasoning over for- mal logic structures such conjunction, disjunction, negation, quantifiers, and implication.
derived from the Entailment-
natural language inference (NLI)
26

---

Published as a conference paper at ICLR 2026
The ProofWriter dataset contains approximately 40K samples. We randomly sample 1K instances for testing. while the remaining data are used for generating training instances. For LogicNLI, we use its original Ik test set, while the remaining 16K training data is used for generating streaming COT data. After applying the streaming CoT generation process, the resulting dataset contains approximately 20k instances for training.
Context-based QA Reasoning For context-based QA reasoning task, we evaluate Streaming Thinker on PubMedQA (Jin al., 2019) and HotpotQA (Yang et al., 2018) datasets.
PubMedQA is biomedical question answering benchmark designed to evaluate factual reasoning and evidence-based inference scientific texts. Each instance consists of biomedical research question, context paragraph extracted from PubMed abstracts, and short-form answer annotated as yes, no, or maybe. The dataset emphasizes reasoning over factual statements, experimental find- ings, and logical connections.
HotpotQA is large-scale question answering dataset designed to evaluate multi-hop reasoning across multiple supporting documents. Each example consists of question, set of supporting paragraphs from Wikipedia, and a corresponding answer. Unlike single-hop QA datasets that require reasoning within single sentence passage, HotpotQA explicitly requires models to integrate information from multiple contexts to derive the correct answer.
The PubMedQA dataset contains 1K samples for testing and 61K samples for training, where all of training samples are used for streaming CoT generation. The HotpotQA dataset includes 90K samples. We randomly sample 1K examples for testing and 60K for streaming CoT generation. After correctness filtering and streaming CoT quality evaluation, we retain total of 16K samples for training.
E MODEL. DETAILS
E. TRAINING DETAILS
During training, we adopt a streaming mask region on top of the standard decoder-only LLM causal mask to enforce strict streaming constraints. Specifically, each training instance is composed of grouped imput and reasoning segments, where the streaming mask (Eq. 2) ensures that reasoning time cannot attend to future inputs. Additionally, the position encoding of both the input and CoT modified to group-wise streaming encoding, preventing positional conflicts.
For training, we set the batch size to 16 and employ the Adam optimizer with a learning rate of le- 5. We additionally enable activation checkpointing mitigate memory overhead from maintaining parallel caches during training. These choices align the training setup with prior LLM pretraining while introducing only the modifications necessary for streaming compatibility.
E.2 DECODING STRATEGY
During inference, decoding must also respect the streaming paradigm. Unlike standard autoregres- sive generation where all inputs are visible, our decoding operates in streaming- manner. Input tokens are continuously appended to dedicated input cache, while reasoning tokens are gen- erated parallel using a separate reasoning cache. At each step, the streaming mask ensures that reasoning about the current sentence depends only on past inputs and past reasoning, never un- seen future inputs.
For generation, we follow the sampling parameters of Qwen3. This balances fluency and diver- sity while maintaining consistency across sequentially produced reasoning sentences. Importantly, because input and reasoning caches are maintained independently, decoding proceeds with lower latency: new inputs can be consumed without flushing or rewriting the reasoning cache, and rea- soning updates can be emitted soon the corresponding input sentence completes. This strategy preserves alignment with streaming training while ensuring responsiveness real-time scenarios. The streaming thinking decoding process is shown in Algorithm 1.
27

---

Published as a conference paper at ICLR 2026
Algorithm 1 Streaming thinking with parallel KV caches
Input: Source length list S, target length list T'.
1: Initialize input KV cache Lacher output KV cache Ocaches and merged KV cache Meache.
2: Read the first input sentence, and save hidden state input KV cache Lnches
3: [Parallel] For Input. KV Cache (prefill):
4: while Input sentence is arrival do:
5: Separate Macke Teache and cache.
6: Read the input sentence, and save hidden state input KV cache packe-
7: end while
8: [Parallel] For Output KV Cache (decoding)
9: while Input sentence is arrival do:
if NEos 2 NEOT + Sentence arrival time less than LLM decoding time.
11:
Select a slice of the input KV cache Teache to keep Ngos == Ngor + 1.
12: elif NEOs < Neor 1: Sentence arrival time exceeds LLM decoding time.
13:
Wait for the input KV cache prefilling and keep NEOs NgOr + 1.
Select slice of the input KV cache 7' T'ache'
15: Merge l'acche and Ocache to Meache
16: Decode the streaming thinking tokens and save to M Meache
17: end while
[End Parallel]
19: Generate controllable deep thinking with thinking intervention.
20: Return: The streaming COT.
E.3 RELATION BETWEEN PARALLEL KV CACHES AND PREFILL-DCCOD SEPARATION
While our parallel KV caches mechanism may formally resemble techniques for Prefill-Decode separation (Zhong et al., 2024; Qin al., 2024), the foundational motivations and operational goals behind the two approaches are fundamentally distinct.
Prefill-Deoode separation is primarily a system-level optimization designed enhance throughput in batched inference. It bifurcates the generation process into two discrete phases: a computationally intensive prefill stage that processes the input prompt in parallel batch, and memory- memory-band bandwihthh- bound decoding stage for auto-regressive token generation. The core objective is to maximize hard- ware utilization by applying specialized computational kernels each distinct phase, thereby im- proving efficiency for fundamentally sequential task.
In contrast, the design our parallel KV caches is driven by the goal of enabling the model to generate output concurrently while still consuming input. In other words, our approach is designed to overlap the processing of incoming data with the generation the output, breaking the strict sequential dependency where generation only begin after the entire input has been processed.
F EVALUATION METRIC
F.1 REASONING ACCURACY
Pass@1 Accuracy This metric measures the proportion of test instances which the model's top-1 generated output exactly matches the ground-truth answer. Unlike relaxed metrics that con- sider multiple sampled generations (e.g.. pass@k) k), pass@ provides a strict estimate of the model's reliability under single-shot decoding, which is the default usage setting for most real-world applica- tions. A higher @ I score thus indicates stronger reasoning consistency and correctness without relying on sampling diversity.
28

---

Published as a conference paper at ICLR 2026
F.2 REASONING LATENCY
Token-length Delay In streaming tasks, token-length delay measures how many tokens must be consumed before the model begins its reasoning process. Formally, it is defined the gap between the start of the input stream and the position of the first reasoning token. A smaller token delay indicates that the model can initiate reasoning earlier the stream, leading faster overlap between input consumption and output generation. This metric thus reflects the responsiveness of reasoning onset in the streaming paradigm.
Token-Delay Position of first reasoning token
Time Delay Time delay measures the real-time latency between the arrival of the last input to- ken and the emission of the first answer token. This metric captures system-level responsiveness, incorporating factors such as decoding speed, cache management, and hardware throughput. While token delay evaluates the generation behavior in discrete units, time delay provides a wall-clock perspective that better reflects the user's perceived waiting time. The Figure illustrates the latency comparison between streaming thinking and batch thinking.
Timestamp
Input
Streaming Thinking
Interleaved Thinking
Batch Thinking
Behaviors
Time
Behavioms
Time
Behaviors
Time
monvby
<Redd>
Ez
fins
<Read while i Thinks
<Thike
<Read>
(3) the sccond day, be jeits somiees cach lestng! hour and N0
<Reed while il hinkk
<Reud>
<Rexd>
E3
minuas.
E4
(G) How mony total hours Chaifle Speade?
<Read while Thinks
<hinko
<Read>
<Think>
Read>
<Think>
<Answart>
<Thik>
Delby
<Read>
<Think>>
<Thnks
to
<Answer
<Answer>
Figure 2: Comparison of reasoning paradigms. Streaming thinking enables concurrent reading and reasoning with minimal delay, while interleaved thinking alternates between the but still accu- mulates overhead, and batch thinking postpones reasoning until all inputs are read, leading to the largest delay.
29


Reviews:
[{"review_id": "VmuOnHiZb7", "reviewer": "Reviewer_g9aY", "summary": "The paper “StreamingThinker: Large Language Models Can Think While Reading” proposes a new streaming thinking paradigm that enables large language models (LLMs) to perform reasoning concurrently with input processing, mimicking how humans “think while reading.” The authors instantiate this idea through the StreamingThinker framework, which integrates (1) streaming chain-of-thought (CoT) generation, (2) streaming-aware training via attention and position encoding constraints, and (3) parallel inference using dual KV caches. Experiments on multiple reasoning benchmarks (math, logic, and QA) demonstrate comparable reasoning performance to conventional batch reasoning while reducing token waiting time by 80% and overall latency by more than 60%. Overall, the paper addresses an important efficiency bottleneck in LLM reasoning and proposes a novel direction toward real-time reasoning.", "questions": "See weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper introduces an interesting idea of streaming thinking, where LLMs reason concurrently with incoming input rather than after receiving the full context. This paradigm is conceptually appealing and well-motivated by the analogy to human cognition, offering a fresh perspective on reducing reasoning latency while maintaining coherence.\n2. The proposed framework is technically complete, integrating data construction, streaming-aware training, and parallel inference into a coherent system. The design shows good engineering consistency across stages.\n3. The empirical evaluation is broad and covers multiple reasoning domains. The depth-controlled reasoning design effectively demonstrates the trade-off between accuracy and latency.", "weaknesses": "1. The authors should discuss more about why the method works. Thinking while reading is interesting, but it is obvious that the question is not complete while reading. There have been several papers [1,2,3] that mention that the incomplete questions will actually affect the model performance negatively. So, why think based on the incomplete questions during reading can improve performance?\n\n[1] Laban, Philippe, et al. \"Llms get lost in multi-turn conversation.\" arXiv preprint arXiv:2505.06120 (2025).\n\n[2] Fan, Chenrui, et al. \"Missing Premise exacerbates Overthinking: Are Reasoning Models losing Critical Thinking Skill?.\" arXiv preprint arXiv:2504.06514 (2025).\n\n[3] Li, Jinzhe, et al. \"Don't Take the Premise for Granted: Evaluating the Premise Critique Ability of Large Language Models.\" arXiv preprint arXiv:2505.23715 (2025).\n\n2. The paper mentions that “we set the LLM’s input speed as the average human speaking rate about 150 words/s.” This number appears inconsistent with the typical rate of around 150 words per minute. It would be helpful for the authors to clarify whether this is a typographical error or an intentional experimental setting. If the latter, providing a short justification or citation would make the latency-related results more transparent and credible.\n3. Tables 2 and 3 present comparisons between the question-first and context-first input orders, but both experiments are conducted using offline datasets with simulated streaming inputs. Since the differences are described mainly at an intuitive level, it might strengthen the analysis to include a short diagnostic discussion in the appendix, for example, outlining typical error types or explaining how sequence order affects reasoning behavior.\n4. The motivation drawn from human cognition is interesting, but could be supported a bit more clearly. A concise paragraph in the appendix linking the proposed method to established cognitive theories of reading or incremental comprehension would help reinforce the conceptual foundation.\n5. The dual KV-cache design is highlighted as an important part of the system, yet the paper does not include quantitative information about memory usage, bandwidth cost, or first-token latency. Including even a brief table or figure reporting these numbers would make it easier to assess the trade-offs and scalability of the proposed approach for larger or real-time systems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I41WtwrdBg", "reviewer": "Reviewer_PpX3", "summary": "StreamingThinker proposes a thinking while reading paradigm: the model generates step-by-step thoughts as the input streams in, rather than waiting for the full prompt. It implements this with by first generating streaming data to use for SFT, using a modified streaming attention mask, and parallel KV caches to decouple input prefill from reasoning. Across math/logic/QA tasks on Qwen3-1.7B/4B, it matches batch-style accuracy at deeper reasoning depths while cutting time-to-first-token and overall latency substantially. The method works for both question-first and context-first orders and allows controllable reasoning depth to trade off speed and accuracy.", "questions": "* How does the use of two KV caches compare in memory overhead and performance to the use of one KV cache?\n* How should the reasoning depth be decided on? It seems like D1 has a significant impact on performance and is not desirable.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* The paper is well written and figures are very clear.\n* The method and paradigm are novel since I am not aware of any other work which performs reasoning concurrently with input prefill. This idea is also well motivated and could be highly impactful.\n* Evaluation is performed over a diverse set of datasets and show a substantial reduction in number of reasoning tokens as well as latency with little reduction in accuracy.\n* The distinction and evaluation of the question-first and context-first settings in the streaming setting is interesting and provides valuable insights into how streamingthinker works.", "weaknesses": "* Tables lack variances.\n* The parallelization still occurs at sentence level chunks, and finer/larger granularities were not investigated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h1Mo7j4UIv", "reviewer": "Reviewer_VM9v", "summary": "This work introduces a paradigm that allows the model to stream while reasoning before outputting the results. Their experiments show that this streaming paradigm performs on par with batch thinking models but reduce the latency in getting the outputs. Both training and evaluation frameworks are provided for adpting LLMs to this new inference paradigm.", "questions": "- Presentation is mostly clear\n- Figure 4 has really small font making it hard to read", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Introduce paradigm that allows model to stream while reasoning\n- Intuitive, Well-engineered and clearly described training and evaluation frameworks\n- Demonstrate that using streaming performs on par with batch reasoning but with less latency\n- Use of multiple domains to demonstrate that this paradigm does well on different types of problems", "weaknesses": "- My primary reservation is the motivation behind the need for streaming thinking. What kind of application requires steaming reasoning? In what scenarios is batch thinking insufficient, especially when the batches are small? In the example in Figure 1, it makes much more intuitive sense to perform batch thinking to avoid \"overly eager\", not-so-helpful thinking tokens like \"Okay, now we get the background of Charlie's schedule.\" Intuitively, I would argue that math questions evaluated in this paper are more suitable with batch thinking or even thinking after the full input,especially in high-stakes applications. A clear vision of downstream applications would also improve the realism of the evaluation (for example, what is an interval that a user expect to see output).\n- It would be interesting to see if this paradigm can be used for other problems including planning domains to demonstrate generality", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)