Can Speech LLMs Think while Listening?
Authors: Yi-Jen Shih
Desh Raj
Chunyang Wu
Wei Zhou
SK Bong
Yashesh Gaur
Jay Mahadeokar
Ozlem Kalinli
Mike Seltzer
Venue: ICLR 2026 Poster (2026)
Keywords: SpeechLLM
Chain-of-Thought
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We enhance SpeechLLMs reasoning ability and proposed novel methods to allow concurrent thinking and listening.

Abstract: Recent advances in speech large language models (speech LLMs) have enabled seamless spoken interactions, but these systems still struggle with complex reasoning tasks. Previously, chain-of-thought (CoT) prompting or fine-tuning has been shown to significantly improve the reasoning abilities of text-based LLMs. In this work, we investigate the effect of CoT fine-tuning for multi-stream speech LLMs, demonstrating that reasoning in text space improves the accuracy of speech LLMs by 2.4x, on average, over a suite of spoken reasoning tasks. Beyond accuracy, the latency of the spoken response is a crucial factor for interacting with voice-based agents. Inspired by the human behavior of "thinking while listening," we propose methods to reduce the additional latency from reasoning by allowing the model to start reasoning before the user query has ended. To achieve this, we introduce an entropy-based metric, "question completeness," which acts as an indicator to guide the model on the optimal time to start reasoning. This method provides greater control over the accuracy-latency trade-off compared with heuristic-based approaches and, under equivalent latency conditions, yields a 4% accuracy gain on ARC-Easy. Finally, we use Direct Preference Optimization (DPO) on preference data created using rejection sampling to push the accuracy-latency pareto frontier further, resulting in a 70% reduction in latency without loss in accuracy.

Published as a conference paper at ICLR 2026
CAN SPEECH LLMS THINK WHILE LISTENING?
Yi-Jen Shih!?; Desh Raj²* Chunyang Wu²*, Wei Z .KK Bong', Yashesh Gaur", Jay Mahadeokar", Ozlem Kalinli, Michael L. Seltzer"
'The University of Texas Austin
Meta Superintelligence Labs
yjshiheutexas. edu, rdesh2 60gmai1 com, cwu5640gmaill com, showei@google.
ABSTRACT
Recent advances in speech large language models (speech LLMs) have enabled seamless spoken interactions, but these systems still struggle with complex rea- soning tasks. Previously, chain-of-thought (CoT) prompting or fine-tuning has been to shown to significantly improve the reasoning abilities of text-based LLMs. In this work, we investigate the effect of CoT fine-tuning for multi-stream speech LLMs, demonstrating that reasoning in text space improves the accuracy of speech LLMs by 2.4x, average, yond accuracy, the latency of the spoken response is a crucial factor for interacting with voice-based agents. Inspired by the human behavior of "thinking while lis- tening," we propose methods to reduce the additional latency from reasoning by allowing the model to start reasoning before the user query has ended. To achieve this, we introduce an entropy-based metric, "question completeness." which acts as an indicator to guide the model on the optimal time to start reasoning. This method provides greater control over the accuracy-latency trade-off compared with heuristic-based approaches and, under equivalent latency conditions, yields a 4% accuracy gain ARC-Easy, Finally, we use Direct Preference Optimization (DPO) on preference data created using rejection sampling to push the accuracy- latency pareto frontier further, resulting in a 70% reduction in latency without loss in accuracy.
suite of spoken reasoning tasks. Be-
INTRODUCTION
The traditional approach for building voice agents to cascade several components: an automatic speech recognition (ASR) model, text-based large language model (LLM), and text-to-speech
(TTS) model model|Huang Huang et al.] (2023); et al. (2024); Likhomanenko et al. (2025; Chen et al. (2025). The recent emergence of speech large language models (Speech LLMs) (Cui et al. (0244) offers a promising alternative to this cascaded pipeline. These models are designed to directly process speech input generate speech output, thereby eliminating the need for separate ASR or TTS mod- ules. This integrated approach can scamlessly process both the semantic content and paralinguistic features of speech, and also reduces lateney due cascaded components.
addition being used for specialized tasks such as speech understanding (Tang al. 2024 Hu et al.) 2024 Lu et al. 2024) and speech generation Ye et al.] (2025] [Du al. a 2024). these models have shown promise in end-to- end spoken dialog Défossez et al. (2024); Huang et al. (2025). Nevertheless, while they work well for casual conversations, speech LLMs often fall behind their text-based counterparts on complex reasoning tasks (Peng et al.] 2025 Wei et al. 2025 Lin et al. 2025). Consequently, the develop- ment of methods to enhance the reasoning capabilities of speech LLMs remains an underexplored and critical research problem to further their application smart companions.
Several approaches have been explored to enhance reasoning capabilities in text-based LLMs, with the most popular being chain-of-thought (CoT) (Wei et al. 2022). In this method, the model gen- erates a series of intermediate reasoning steps before producing its final answer. An LLM can be coerced elicit reasoning traces either through zero-shot prompting (Kojima al. 2022) or via pervised fine-tuning on a dataset of reasoning examples (Zelikman et al. 2022). Despite their strong
Work done Meta

---

Published as a conference paper at ICLR 2026
What the copital France? Barcctono, London, Paris?
User Audio
<Swlth ceo
< (00
<awR-h cbb>
<and, Loc>
sfissh STSP
What
copp
Franno
ceptaal
Beroe
of
Painc
Perai
<awtcct
16
1x
Parts
Text Monologue
System Audior
Paris.
Figure 1:
Training token sequence arrangement. We train the model to interleave reasoning tokens RT with streaming ASR| tokens QT on the text monologue channel, with special swwtch tokens for mode switching. After the CoT ends, the model generates text tokens which align with the spoken response R For simplicity, [PAD] and [EPAD] tokens
shown here.
performance, recent research highlights critical trade-off with CoT: the length of the reasoning trace is a crucial factor in accuracy. While longer CoT sequences generally yield better perfor- mance, this comes at the cost of increased decoding latency and computational overhead (Jin et al.) 2024). To tackle this problem, there are explorations about when and how long an LLM should reason Sprague et al. 2025), resulting in growing interest in "hybrid" reasoning models.
Although some recent work has adopted CoT in the speech domain, they focus primarily on ap- plications such speech translation Hu et al. 2 Du et al. (2255 Gállego et al. (2025), dia- logue Arora et al. (2025), or other detection tasks Mai et al. (2025); Park et al. (2025). The integra- tion of CoT in speech LLMs requires answering two research questions: (i) should models reason using text or speech, and (ii) how do we maintain the responsiveness required for spoken interac- tions? To answer the first question, we investigate both alternatives, showing that text-based CoT is as performant as speech-based CoT for improving reasoning in speech LLMs, while being 2x more token-efficient.
The sequential process of listening, reasoning, and responding introduces considerable latency; con- sequently, previous research has proposed methods to overlap CoT tokens with speech improve real-time conversational AI. Building upon the anthropomorphism of speech LLMs, concurrent works such as STITCH (Chiang et al. 2025) and Mini-Omni Mini-Ommn-RRasonerr (Xie et al. 2025) have pro- posed "thinking while speaking," i.e., the model begins its spoken response while its reasoning is still ongoing. This achieved by interleaving chunks of reasoning tokens with spoken response tokens, and subsequent CoT chunks are generated in the time it takes for the audio decoder to synthesize the preceding response. Despite showing reasonable improvements, this approach has notable limita- tions. For instance, the optimal chunk size for interleaving requires careful tuning and is dependent on hardware limitations. Moreover, despite a reduction in the time to first word, the model may inadvertently vocalize too much of its reasoning, leading to longer overall response time to fi- nal, conclusive answer. this paper, we draw inspiration from neuroscience (Donhauser & Baillet 2019) to propose novel "thinking while listening" paradigm, by enabling concurrent processing of text-based CoT and user speech.
Current speech LLM architectures may be broadly categorized into types: single-stream and multi-stream. Single-stream architectures merge user/system speech and text into unified token sequence (Kim al., 2024) Veluri al.] (024),, while multi-stream architectures simultaneously model distinct streams for each token sequence (Défossez. et al. 2024). In this work, we build upon multi-stream architecture due to superior capacity for the concurrent processing of user audio and reasoning tokens. This design provides significant flexibility by allowing the system's 's text stream to be revised independently. key advantage over single-stream models that lack this decoupling. Specifically, we fine-tune the publicly available Moshi model (Défossez et al. 2024) to generate CoT within its text monologue stream to improve its reasoning capabilities (Section 2}. To enable the model to think while listening, we propose two methods: (i) novel metric that estimates the completeness of the user's 's question at each timestep. and (ii) a preference tuning scheme to update the model's 's reasoning dynamically with new input (Section 3)
Since there are no existing standard reasoning evaluations for speech LLMs, we curated a suite of single-turn spoken reasoning tasks from well-known text-based reasoning benchmarks compris-
2

---

Published as a conference paper at ICLR 2026
ing mathematical reasoning, social/physical interactions, and other general reasoning tasks (Sec- tion 4.2). Overall, our contributions are summarized below.
1. Text-based CoT improves reasoning in speech LLMs. To the best of our knowledge, the first to explore text-based CoT fine-tuning on multi-stream speech LLMs. Our method obtains 2.4x improvement in accuracy, average, over the Moshi baseline across the SRQA tasks.
Thinking while listening reduces reasoning lateney. We demonstrate that auto-regressive mod- els that can generate tokens in syne with streaming user input can be taught to "think early" using entropy-based selection of trigger points. We achieve this using a novel Question Completeness metric that results in more controllable accuracy-latency trade-offs.
Preference tuning enables adaptive reasoning for early-CoT models. We use rejection sam- pling to curate correctness-baae based and length-based preference data and use them for DPO training, pushing the
pareto further and reducing ~07% latency without loss in accuracy.
MULTI-STREAM SPEECH LLMs WITH CHAIN-OF-THOUGHT
2.1 BACKGROUND: MOSHI
Moshi (Défossez et al. 2024) is full-duplex multi-stream model that simultaneously processes three distinct token streams at each timestep: user audio, system audio, and system text (referred to as the "text monologue"). For the audio streams, a separate codec model, Mimi, is used to encode audio waveforms into discrete tokens and back, operating at a frame rate of 12.5 Hz with 8 code- books. Eventually, all three streams of inputs are represented as tokens:
User Audio A E (I,....,NN
NA) LX8 N
(1)
System Audio A (I,...,,
N LX8
(2)
System Text: T E (1,....,Ty))
(3)
where NA = 2048 is the size of each Mimi code-book and Nr. 32000 is the text vocabulary size. All streams have time-aligned tokens; text tokens are interleaved with padding tokens ([PAD] and [EPAD]]] to align with the corresponding audio tokens. Notably, since such aligned text token sequences are significantly shorter than the corresponding speech, the majority of text tokens in Moshi simply padding tokens. The model architecture consists of temporal transformer and depth transformer, trained jointly using Negative Log Likelihood(NLL)) loss. At each timestep t, the temporal transformer consumes A and A and predicts T P This token is fed into the depth transformer, which generates A 4 The model is trained to estimate the following probability:
T S (A t+1 6+1
TS St. A.).
St.
Moshi was trained in multiple stages: (i) pre-training a text backbone LLM (Helium) using next token prediction, followed by (ii) post-training and fine-tuning with audio token sequences well as multi-stream data, and (iii) fine-tuning with user-system dialogue data. Subsequent studies also showed the effectiveness of alignment training with direct preference optimization (DPO) to improve aspects such as factuality and safety (Wu et al.) 2025). During inference, Moshi consumes user tokens and generates system text and audio token per timestep. Due to the nature of the training sequences, we can force Moshi to generate a response by inserting an [EPAD] token on the text monologue stream. Conversely, inserting a [PAD] token forces it to remain silent.
2.2 FINETUNING WITH CoT
Given user's 's spoken question, @A, our model predicts a spoken answer, A guided by reasoning trace, R. In the base Moshi model, all spoken responses AA are preceded by aligned text AAT on the text monologue stream. To integrate CoT in this framework, we allow the model to additionally generate text-only reasoning tokens RD without corresponding audio. Since both RD and A" are
1 [ EP AD] is used for indicating end of consecutive pad tokens. For simplicity, we neglect the delay pattern of first audio codebook and system text in notation.
3

---

Published as a conference paper at ICLR 2026
Table 1: Examples of questions where it is feasible to start reasoning early without impacting the correctness of the answer.
Question
Reasoning
Answer
What the capital of France it New York The capital of France is Paris. It's Paris. or Paris?
2 If you flip a fair coin three times and get It's a fair coin, so probubility heads each time... what is the probability of heads/tails is always 0.5. the fourth flip is heads?
The probability is 0.5.
generated on the text monologue channel, we demarcate them using special <start.cot> and <end_cot> tokens, as shown in Fig.
To help the model learn the relationship between the user's spoken question and the CoT, we also introduce streaming ASR component into the text monologue, with the corresponding tokens de- noted by QT (red tokens in Fig. D. Previously, AAoor et al. (2025) and Yuen et al. (2024) have suggested using the user's audio transcript LLMs, but they focused on offline ASR. In contrast, our model naturally learns streaming ASR through word-aligned user transcripts right-shifted by k: tokens for look-ahead. Based our pre- liminary experiments, we used & = 6 (equivalent to a 480 ms look-ahead), which was found to provide good balance between latency and word error rate (WER). Finally, the text monologue contains the user transcript Q the reasoning R." and the response text A²" To ensure all three streams-user audio (A ), system audio (A$), and text monologue (A U) have the same length, we insert silent audio tokens as needed.
intermediate step in the CoT process for speech
We fine-tune Moshi using the same NLL loss for next token prediction during the SFT stage, and use DPO loss for off-policy preference tuning. During inference, we apply force-decoding to our fine- tuned model and the baselines in two ways: (i) at the start of the user's question, we force-decode k [PAD] tokens to accommodate the streaming ASR, and (ii) we force-decode the <start_cot> token after the user's question ends the model has not generated it already.
THINKING WHILE LISTENING
Since our text-based CoT does not generate any aligned audio, naively inserting it between the query and the response can increase the perceived latency, thus impacting the naturalness of the human-system interaction. To alleviate this issue, our objective is to reduce this additional latency by mimicking the common cognitive trait wherein humans begin processing and reasoning before a question is fully articulated.
There are two scenarios where a model can begin reasoning early and yet provide the correct answer, as illustrated in Table The first scenario includes questions which can be considered "complete" before reaching the end. In such cases, the model can start reasoning early and simply ignore the remaining question. In the second scenario, sufficient information may be available start reasoning before the question ends, but the model still needs the remaining information to provide correct response. We propose two different methods enable early thinking. To endow the model with the ability for early reasoning, we created training examples by using our proposed Question Completeness metric. This metric is designed identify the optimal time for the model to begin generating its COT. Subsequently, we fine-tuned the model on this dataset to teach it to follow distribution of these carly-reasoning examples. Finally, we apply preference tuning to further enhance the performance of the model under early thinking scenario.
3.1 MEASURING THE QUESTION COMPLETENESS
Let us define the inflection point question as the timestep where sufficient information is avail- able to begin reasoning. Our objective is to teach the model to identify such points in order to start its reasoning trace. A naive approach to identify the inflection point may be to shift the reasoning trace by a fixed number of frames words, based on the heuristic that sufficient information is typically available a few words before the question concludes. However, this method fundamen- tally limited by its lack of semantic awareness. For instance, in the query "What is the capital of
4

---

Published as a conference paper at ICLR 2026
1.00
1.00
0.95
=
0.95
0.75
0.75
-= 19
((p)
0.50
((p)
0.50
Backcountry P-13
0.25
0.25
Mononyy 10k
0.00
Can
you think
of
broader
category that
gun would
fail under?
In
words what's a
thypar
the
word gund
Do
you know what kind
of
bear
featured was
the
movic
Figure 2: Examples of the Question Completeness curve (p). In the first example, reaches high value at the end of the main question, at which point it is feasible to begin reasoning. In the second example, the word "Backcountry?" is critical answer the question, and this is reflected in the corresponding Ç curve. More examples of the curve are provided in Appendix A.1 A.1
France?". model camnot reasonably begin its reasoning process until the final word, "France," has been received. Consequently, it is necessary to develop a method that instructs the model to initiate reasoning at the appropriate moment, informed by the semanties of the question. We do this through novel metric, which we call Question Completeness, denoted Ç.
Given training sample that contains the question Q1.N, the reasoning R, and the answer A, where N denotes the number of words in the question. Our goal is to find the index that splits Q into two halves: Qup and Qp+1:N such that
Pr R. A A|Q Pr R. A Q1:N].
(5)
Let X, denote the joint probability of R and A given partial question until the p-th word, i.e., X, Pr RR A A|Qo.p]- In practice, Xp can be estimated using an external language model. We define Question Completeness, as:
<(p) =1 DRL (XN I2)
(6)
DKI. (XN IX)
where DKL denotes the Kullback-Leibh (KL) divergence. Here, X and X, represent the extreme cases where the full question and no question are given, respectively. By definition, ((0) = 0 and <(N) 1, so we can regard < as semantic completeness progress bar Figure shows illus- trative examples of the Ç curve, indicating that can be good proxy for the progressive semantic completeness of question.
The inflection point for a training sample can be approximated using Ç by estimating P s.t.
ò -= min(p <(p) 2 0}.
where is hyperparameter. Further discussion and illustrative examples can be found in Ap- pendix A.I
3.2 SET TUNING FOR EARLY REASONING
To train the model to perform early reasoning, we construct the training token sequences by shifting the reasoning text tokens the inflection point (as shown in Fig [ Since the first <switch_cot> token corresponds to this inflection point, the model learns to identify the optimal timing by max- imizing the probability of predicting this switch token. During inference, the inflection point not provided. To initiate the reasoning, the model samples the first sswitch_cot> token condi- tioned on all past generated tokens and the currently available partial user question. Consequently, the model must internally predict the inflection point by assessing the evolving semantic content of the partial user query. The entire training methodology designed to increase the probability of sampling the the first <switch_cot token at the moment the model determines the input is sufficiently complete.
3C is not guaranteed to be non-decreasing, and in practice, there are small local fluctuations in probability X, due incomplete syntax. Nonetheless, the general trend of is still increasing from to 1.
5

---

Published as a conference paper at ICLR 2026
Correctness- basod preference
Longth based proference
eapinal France located on which eiver?
The capiol Fonce is Parie located central
The capital of Frarax Paris. located on
SFT Model
Divect Preference Optimizttion (DPO)
capital is Paris. is the
Paris is locoted on Seine.
ma(y)) =6
<atazt DOC>
Figure 3: The framework for curating preference data for DPO. We generate outputs from the SFT model (Tzet) by force-decoding <start_cctf early (e.g., before "on which river" is spoken). The preferred response (yw) is the one where the model is able to adaptively generate a correct and shorter reasoning trace.
3.3 PREFERENCE TUNING
While our question completeness metric allows for the creation of training samples that enable early reasoning, we observed that the model struggles to learn the distribution effectively via SFT and is often unable to update its CoT in response to new information in the user channel. Additionally, the CoT in our training data may be excessively long for simple questions, indicating a considerable opportunity to shorten the reasoning trace. To solve these issues, we created contrastive reasoning
pairs, D {((()) you (2)
using rejection sampling and preference-tuned the SFT model
using direct preference optimization (DPO) Rafailov et al. (2023). Fig. 3 illustrates framework for preparing the preference dataset.
For subset of prompts in the SFT training data, generate K responses using SFT model (fine- tuned with early CoT) where we force-decoded <start_cot at ((p) = 0 completeness. From these generations, we select a preferred output You and a rejected output y/- To improve adaptive reasoning, the preference is based on the correctness of the spoken response; for latency reduction. it is based on the both reasoning length and correctness. Kang al. (2025) and Hao al. (2024) have explored other techniques to reduce the the length of reasoning trace in CoT-based models.
Once we have the preference pairs, training is then performed using the DPO objective:
me (ywlz)
B log me (11(zz)
]
LDPO (Te; Tref) = -E log
[Blog Jree (Yon
Fret (y](z))
where тhe and Ref are the policy and reference model respectively, indicates sigmoid function and 3 is a hyperparameter. Practically, we initialize the policy model Te and the reference model oef with the same weights from SFT model n, but freeze the reference model during DPO training.
Following Wu et al. 2025), we calculate the token sequence prohabilities exclusively using the text monologue stream T for more stable training process, rather than using the full policy probability from eq. Additionally, we exclude user streaming ASR tokens Q" from this calculation to better differentiate between the probabilities of (yw|xx (x) and (y (z). We also adopt Sength-normalle DPO Mron et al. (2024) and add the NLL loss to further stabilize the training Xu et al. 2004).. The overall loss is given as:
Spref LDOO XB To (,wl=)),,
where À is hyperparameter that balances the two objectives.
3.4 INTERLEAVED REASONING WITH STREAMING ASR
In Section 2 we proposed that training the model to generate streaming user text tokens improves its textual reasoning capability. Predicting user text poses challenge when left-shifting the reasoning trace to occur before the user's question is finished, since the CoT tokens may overlap with the exist- ing streaming user ASR token sequence. To address this issue, we introduce two special switching tokens, sewitch_col and sswitch.ase two generation modes on the text monologue stream.
which enable the model to alternate between the
To prepare the interleaving pattern for training, first insert the user's streaming ASR tokens the text channel as usual. Then, we identify available blank spaces ([PAD] and [EPAD] tokens) and insert the CoT tokens into these spaces. Whenever mode switch occurs, we prepend the

---

Published as a conference paper at ICLR 2026
corresponding switching token. This approach preserves the time alignment between the user's streaming ASR tokens and the audio input. A detailed illustration of the token arrangement can be found in Figure
4 EXPERIMENTAL SETUP
4.1 TRAINING
Supervised fine-tuning for CoT based on the proposed modeling scheme requires training samples (2, A), where Q and A are in spoken formats, while R is in text format. Since there are no suit- able large-scale public spoken reasoning datasets available, we used text-based reasoning datasets for training by converting them into spoken format. Specifically, we used the CoT-Collection Kim jt al. (203)) dataset it contains samples from diverse sources along with reasoning traces, amount- ing to a total of 1.8M examples. The reasoning traces in this dataset are augmented by OpenAl Codex followed by some filtering to ensure quality.
Since the CoT-Collection was created for text LLM training, it is not readily applicable to voice- based models. For instance, several samples are instances of summarization problems containing long-form text, which may not be applicable to natural conversations. We performed careful curation to obtain a spoken-friendly training dataset from this source:
1. Remove all samples where Q contains more than 60 words, resulting in ~690K samples. Use an LLM to perform spoken-friendll rewriting for all questions, reasoning, and answers. The system prompt used for this rewriting is provided in Appendix[A.6)
3. Convert the rewritten questions and answers into audio waveforms using an internal TTS engine that generates 24Khz mono audios.
We have provided fine-tuning pppee--rrametess and other details in Appendix A.2
4.2 EVALUATION
Spoken reasoning question-answering (SRQA) benchmark. We prepared suite of spoken rea-
soning tasks from multiple domains, derived from popular text benchmarks: (i) A12 Reasoning Challenge (ARC) (Clark et al. 2018), (ii) Physical Interaction QA (PIQA) (Bisk et al.) 2019). (iii) Social Interaction QA (SIQA) (Sap et al., 2019), and (iv) Grade School Math (GSM8K) (Cobbe et al. 2021). For ARC, we prepared easy (ARC-H) and challenging (ARC-C) subsets, similar to previous work. Since these evaluation tasks are derived from text sources, we used the same method of LLM-rewriting and TTS as used for the Spoken CoT-Collection, forms. We designed customized rewriting prompts for each eval set to ensure that the rewritten questions and answers are reasonable. Since several of the tasks contain multiple-choice questions, these were rewritten such that the choices are listed in the spoken question. Additionally, we also tracked the accuracy mance for cases where reasoning may not be useful. The statistics and illustrative examples for all evaluation datasets can be found in Appendix A.3
convert them into spoken
ALaMA--Qustionn
(Nachmani al. 2024) to measure the model's perfor-
Accuracy Calculation. Throughout this work, we used LLaMA-3.1 405B Grattafiori et al. (2024) as text-based judge assess the correctness of the response. Since the model generates spoken response A, we used Pyannote VAD (Bredin & Laurent, 2021 Bredin et al. 2020) to first detect speech presence, followed by Whisper (Radford 20.3) to transcribe the response. The LLM- judge is provided the question and ground-truth answer, along with the model's transcribed response. first determines whether the model provided an answer and then evaluates its correctness. The system prompt for the judge can be found in Appendix A.6
Latency Calculation. In our experiments, latency is defined as the time interval from the end of the user's question to the initiation of the model's spoken response. To measure such latency, we first feed the evaluation user audio (question) into our model and save the entire response as an audio file. Then we apply Pyannote VAD (mentioned in the prior Accuracy Calculation paragraph) on the resulting audio file. This tool directly measures the end-to-end latency in seconds. However, instead of reporting in seconds directly, chose report latency in terms of number of tokens
7

---

Published as a conference paper at ICLR 2026
Table 2: Performance of text and speech LLMs on the SRQA benchmark. All models are roughly 7B but vary in sizes of pretraining data. Our proposed method significantly enhanced the reasoning abilities of Moshi baseline and got competitive results against other speech LLMs pre- trained with much more pretraining data.
of pretraining
Reasoning
Factuality
Model
text tokens
ARC-E ARC-C SIQA PIQA GSM8K LLaMA-QS
Text LLMs
Helium!
2.IT
79.6
55.9
51.0
79.4
LLaMA2-7b-Chat
2T
63.7
47.1
13.4
25.8
29.4
70.6
Bemma-7B-Instronn
6T
82.5
66.2
18.3
45.0
43.1
69.7
Speech LLMs
1oen2-Auiio-HH-maaee
2.4T
59.1
42.4
21.9
24.5
18.1
64.7
Kimi-Aui -B-Insruttt
18T
83.0
71.5
32.9
34.4
15.7
Moshi (buseline)
2.IT
30.2
21.5
22.8
23.8
8.7
42.8
Moshi CoT (ours)*
2.IT
77.7
59.8
56,1
56.9
16.1
57.8
w/o Streaming User ASR
2.1T
55.8
44.0
50.1
46.3
12.2
59.9
buckhone Moshi, results are not directly compurable. not publicly accessible, we report the metrics from Défossez et al (2024). which messured accuracy by ovaluating the log- likelihoooodd over the set given options.
Helium
model is
Contains streaming user ASR with delay of tokens.
Spri me Asi
betting ASR
Orttina
Accurasy 0S)
User RES
ASR
Delay (# tokens)
Deay (# tokens)
Delay 1# tokens!
Deby (P tokens)
Delyy 1# takens)
(a) ARC-E
(b) ARC-C
(c) SIQA
(d) PIQA
(e) GSM8K
Figure 4: Effect of streaming user ASR on accuracy for SRQA tasks. As we increase look-ahead,
the accuracy improves and approaches the "offline ASR" topline.
since the 80m multiplier is configurable based on the speech tokenizer used, but the number of tokens actually represents how much the model's 's reasoning affects time to first packet.
Additionally, it is possible to further speed up the decoding of CoT text tokens depending on dif- ferent implementations of model serving and hardware constraints. For example, we found that one forward pass for Moshi takes roughly 26ms on an A100 GPU. Hence, it is possible to decode (80ms/26ms) text tokens along with audio token to speed up the waiting time for CoT generation. Reporting latency token length provides a hardware and implementation agnostic metric that en- sures fair and consistent comparison between our proposed model and the original Moshi baseline. We leave the implementation of this specific speedup strategy for future work.
RESULTS
5.1
CHAIN-OF-THOUCOE FINE-TUNING IMPROVES ACCURACY
Comparison with baselines. Table 2hooess the accuracy of our CoT fine-tuned model on the SRQA tasks, compared to the Moshi baseline. We also include several publicly available text and speech LLMs of similar size for reference (Touvron et al. 2023 Mesnard al. [222]] Chu et al.) 2024 Ding et al. (2025). On average, our proposed method provided an absolute accuracy improvement of 29.1%, with most eval tasks improving by 2-3x, showing the effectiveness of our approach. Among the speech LLMs, our fine-tuned model showed competitive performance across the board, placing t on all reasoning tasks though the other speech LLMs pretrained much more pretraining data. The performance on LLaMA-QS also improved, but the gains were smaller.
Effect of streaming user ASR. In Section 2 we conjectured that training the model transcribe the user's audio through aligned text tokens on the monologue stream would improve reasoning. To justify this choice, we designed an ablation experiment by removing these streaming user ASR to- kens in training, but otherwise retaining the same training sequences, We also measured the topline accuracy using "offline" ASR, by training the model to transcribe the user audio after the end of
For Mimi Codec, the duration of cach audio token 80ms.
8

---

Published as a conference paper at ICLR 2026
ARC-E
ARC-C
GSM8K
60
Table 4: Effect of DPO train- ing with length-based prefer- ences on accuracy (%) and la- tency (# tokens). The base SFT model is trained with 0 0.75. With DPO training, we further reduced latency by 70% without compromising the accuracy.
75
Accuracy (98)
Accuracy
Accuracy
20
70
800
1.5
50
60
70
80
60
80
80
90
11c
Latency tokens)
Latency tokens)
Latency tokens)
SIQA
PIQA
Moshi + CoT WordShift
Arcuracy
-0-
Accuracy
Accuracy
Latency
25
Correct-DPO
Eval Set
SFT DPO
SFT
DPO
50
60
70
80
50
60
Latency (# tokens)
Latency tokens)
LLaMA-QS
56.2
56,9
35.6
20,9
words
we nos
words
words
words
words
ARC-E
62.8
65.4
49.2
12.0
95%
8-85%
7-%
05%
ARC-C
43.2
46.0
49.9
13.2
SIQA
45.1
45.3
50.0
12.9
Figure 5: Accuracy-latency curves for the proposed methods on SRQA reasoning tasks. QC exhibits better controllability in trade-offs. DPO training with correctness-bant -rasedd preference further improves the accuracy of the QC models.
PIQA
40.7
46.0
46.6
18.2
GSM8K
13.8
14.7
76.0
48.6
the question. Finally, trained several models with streaming ASR where the user text tokens delayed by different numbers of tokens (between and 6). In Table removing ASR significantly degrades accuracy all reasoning tasks while remaining almost the same on factuality, corroborat- ing that ASR helps reasoning. In Fig. 4 we observe that both user WER and accuracy consistently improved for SRQA tasks with an increasing number of delay tokens. The accuracy gain saturated after 4 delay tokens, though GSM8K showed continued improvement. At delay of 6 tokens, the streaming ASR showed comparable accuracy to offline ASR for the majority of tasks.
Reasoning in text v/s speech. A design choice in our CoT fine-tuming is to perform reasoning in text, since text is much more information-dense than speech tokens. To investigate the effect of this decision, we conducted an ablation by performing CoTT' fine-tuning in speech. For this, we used our internal TTS engine to synthesize the CoT into speech and prefixed it to the spoken response.
For these ablation studies, we used the train/test subsets of GSM8K, and the results are shown in Table 3 As expected, the Moshi base- line performed poorly GSM8K since it was trained primarily for casual dialog. CoT fine-tuning improved the overall accuracy from 8. 8.7% to 17.5% and 17.2% for bext-based and speech-basse reasoning, respectively. While Speech COT' has zero latency but it comes with the cost of 3x token length for the entire response. Direct fine-tuning on question-answer pairs ("No CoT"") degraded model performance, indicating that the improvement for CoT models cannot be attributed to our training data alone. Qualitative analysis revealed that in sev- eral cases where the Moshi baseline provided the correct answer, actually performed some reasoning first. By directly fine-tuning it without CoT, we forced Moshi "not to think" and therefore reduced its accuracy. More results are shown in Appendix A.4
Table 3: Comparison of text-based and speech- based CoT on GSM8K, in terms of accuracy (%).
Model
Accuracy
LLaMA2-7b Chat
29.4
Moshi (baseline)
8.7
Text CoT
17.5
Speech CoT
17.2
No CoT
3.5
5.2 EFFECT OF EARLY REASONING
In Section 3 we proposed two methods to teach the model to think while listening: first, based on Question Completeness (QC), (. and second, using DPO on reasoning traces generated with rejection sampling. For the QC method, we can control the onset of CoT during training based on 0. As simple baseline, we trained the model by left-shifting the CoT by fixed number of words of the user question. We refer to this as WS-N, to denote shift by N words. Fig, 5hohoss the accuracy v/s latency curves for our proposed methods as well as the baseline. The latency metric is reported in terms of the number of tokens between the end of the user question and start of system response.
bC-ba shifting outperforms word-count heuristic. First, it is evident that all latency improve- ments resulted in accuracy degradation, and different methods can only be compared based on their pareto-frontiers on the accuracy-latenc curve. The results for the WS baselines were mixed: while
9

---

Published as a conference paper at ICLR 2026
they showed gradual latency reduction ARC, the performance other tasks was haphazard. On PIQA and GSM8K, for instance, increasing N training did not result in expected reduction in la- tency, indicating that the model QC method, on the other hand, provided better control over the trade-off. On all eval sets, latency improved reduced the 0 for selecting inflection point (see equation 7 from 0.95 to 0.65.
unable learn any patterns for early reasoning. The proposed
Correctness-ased preference improves accuracy. On further DPO training with correctness-based preference data, we achieved consistent improvements on all evaluation sets, as shown by the pur- ple curve in Fig. 5 The marginal increase latency stems from the better alignment between our model's behavior and the ground truth. We report the average gap between start CoT position of pre- diction and ground truth in Table Negative sign indicates that the model generated CoT starts earlier than ground truth. Before ap- plying Correct-DPO, SFT models tends to start CoT earlier than the ground truth.
Table 5: Start CoT Gap (# tokens) on the valida- tion set. The gap is calcu- lated by subtracting the po- sition of <start eetaat_ccos ccttttt>>>>>co be- tween model generation and ground truth.
Gap (pred gt)
Length-based preference improves latency. Next, we trained the model using the length-basee preference data to further shorten the CoT length. For this experiment, chose the SFT model trained with 0 = 0.75 as our base model. From Table 4 we found that our method successfully reduced latency across all tasks by 30 tokens on average, while maintaining or improving accuracy.
SFT
Correct-DPO
0.95
-1.62
-0.60
0,85
-3,68
-0.76
0.75 -5.77
-1.56
0.65 -5.17
-0.32
Illustrative examples showing the improvements of our methods are shown in Appendix A.5
CONCLUSION
In this work, we integrate CoT into speech LLMs and proposed novel "thinking while listening" paradigm. We showed that text-based CoT can dramatically improve the model's performance, lead- ing 2-3x accuracy improvements over a suite of spoken reasoning tasks. significant fraction of this improvement came from explicit user understanding by teaching the model to transcribe the user audio in a streaming manner. To avoid the latency cost of reasoning, we trained the multi-stream model to reason concurrently with incoming audio guided by our proposed "Question Complete- ness" metric. This metric offers superior controllability over accuracy latency trade-off. We then performed preference tuning of this model with datasets created using rejection sampling. By curat- ing correctness-baset and length-base based preference data, we were able to (i) improve the accuracy of the carly-CoT model, and (ii) shorten the length of the reasoning trace, respectively. The resulting model demonstrated 70% lower latency without significant loss in accuracy. Our work opens up new direction for Speech LLMs: the ability to think while listening, capability that brings them closer to the natural, responsive flow of human conversation.
7 ETHICS STATEMENT
We acknowledge that we have read and adhered to the ICLR Code of Ethics. Our research on Speech LLM reasoning is intended to advance the communication between human and machine. The data source used is publicly available and has no personally identifiable information.
8 REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our experiments, we provide comprehensive implementation details in out paper. The overall pipeline for dataset preparation and evaluation is described in Section 4. Additionally, all LLM prompt templates and evaluation benchmark statistics are included in Ap- pendices A.6 and A.3 respectively. The detail of our model training parameters is provided in Appendix A.2
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Siddhant Arora et al. Chain-of-thought training for open e2e spoken dialogue systems. www.nttccccoooorooooooooooo.
ArXiv,
abs/2506.007 00722, 2025. 279075666
URL
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piga: Reasoning about physical commonsense in natural language. In AAAI Conference on Artificial Intelligence, 2019.
Hervé Bredin and Antoine Laurent. End-to-end speaker segmentation for overlap-aware resegmen- tation. In Proc. Interspeech 202/, Brno, Czech Republic, August 2021.
Hervé Bredin et al. pyannote.audio: neural building blocks for speaker diarization. In ICASSP 2020, IEEE International Conference on Acoustics, Speech, and Signal Processing, Barcelona, Spain, May 2020.
Junjie Chen al. Fireredchat: pluggable, full-duplex voice interaction system with cascaded and 06502
semi-cascaded implementations, 2025. URL. UK..ntttp.//aaxxx...r//////////
Cheng-Han Chiang et al. Stitch: Simultaneous thinking and talking with chunked reasoning for spoken language models, 2025.
baapgg/abs//50/ abs/2507 15375
Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen2-audio technical report, 2024.
URL httpp:://arxiv..rt
abs /207 1077..1100 10599
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try the ai2 reasoning challenge. arX1v:1880.004557v1, 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint 1xXv:2210014418, 2021.
Wengian Cui, Dianzhi Yu, Xiaoqi Jiao, Ziqiao Meng, Guangyan Zhang, Qichao Wang, Yiwen Guo, and Irwin King. Recent advances in speech language models: survey, ArXiv, abs/2410.03751, 2024. URL.tttss: api semant.
Ding Ding et al. Kimi-audio technical report, 2025. URL ULLatttps//arrrvvooo//bbss 18425
Peter Donhauser and Sylvain Baillet. Two distinct neural timescales for predictive speech process- ing. Neuron, 105, 12 2019. doi: 10. htt.://neeroo.019....
Yexing Du et al. Making llms better many-to-many speech-to-text translators with curriculum learn- ing. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, doi: 10. 10.18653/11220255.ll
pp. 12478, 01 2025.
Zhihao Du et al. Cosyvoice models. ArXiv, Abs/2412.10111 2024. URL https: pettps://api... www.nticscccoollaa........ /rg/
Scalable streaming speech synthesis with large language
CoxpusID:2747623::
Alexandre Défossez et al. Moshi: speech-text foundation model for real-time dialogue, 2024.
/ttp::::::/ /wwp///aavvvv///a org/ abtp///...22.../////. 00037
Aaron Grattafiori et al. The llama herd of models, 2024. URL https::/acxxv..rr/////ooo
2407.22 21733
Gerard Gállego, Oriol Pareras, Martí Cortada Garcia, Lucas Takanori, and Javier Hernando. Speech-to-Texe Translation with Choneme-Augmennt CoT: Enhancing Cross-Lingual Transfer in -ew-Ressuree Scenarios. In Interspeech 2025, pp. 31-35, 2025. doi: 10.21437/nterspeecchh 2025- 1954.
11

---

Published as a conference paper at ICLR 2026
Shibo Hao et al. Training large language models to reason in continuous latent space. arXiv preprint arXiv:2412.06769, 2024.
Ke Hu et al. Chain-of-thought prompting for speech translation. In ICASSP 2025 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, 2025.
doi: 10.1 00.1090CASSP4496022220 108500000..
Shujie Hu et al. Wavllm: Towards robust and adaptive speech large language model. Con- ference Empirical Methods in Natural Language Processing, 2024. URL https::///// //api.
semant cschollcasssc..ccc ar. org Corpus :
Ailin Huang et al. Step-audio: Unified understanding and generation in intelligent speech in- teraction. ArXiv, abs/2502. 11946, 2025. URL https://api www.essescaccooooo.e..........
CorpusTD: 276421777
Rongjie Huang et Audiogpt: Understanding and generating speech, music, sound, and talk- ing head. ArXiv, abs/2304. 12995, 2023. URL https::/ www.stetsmchotleeee..ee org/
Corpus 2583094330
Mingyu Jin et al. The impact of reasoning step length on large language models. In Findings of the Association for Computational Linguistics ACL 2024, pp. 1830-1842, 2024.
Yu Kang, Xianghui Sun, Liangyu Chen, and Wei Zou. C3ot: generating shorter chain-of- thought without compromising effectiveness. In Proceedings of the Thirty-Ninth AAAI Con- ference on Artificial Intelligence and Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence and Fifteenth Symposium on Educational Advances in Artificial In- telligence, AAAI"5/IAAA225/EAAA 10. 1609/a9/a.ii v39.23..934444 33668. URL https: org/ 1et.11169/aaaaa v33.23.
AAAI Press, 2025. ISBN 978-1-57735-897
doi:
Heeseung Kim et al. Paralinguistics-wware speech-empowered large language models for natural conversation. In The Thirry- Annual Conference on Neural Information Processing Systems, 2024. https: /ttptt::://ppeeevviieee wtw.://wwweee/oourtemmm
net. for.m?/rmmmmmmmm wtt.://oooo..aeeeeee/oeooeaoaaae
Seungone Kim, Se June Joo, Doyoung Kim, Joel Jang. Seonghyeon Ye, Jamin Shin, and Minjoon Seo. The cot collection: Improving zero-shot and few-shot learning of language models via chain- of-thought fine-tuning. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. TRLhttps: http://pw.neeeeeeeeeeeeee / www.rreviie.nett/ouummic.cc.cm. iinndidddDd o00.7=0070xxF
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 0781713871089
Tom Labiausse, Laurent Mazaré, Edouard Grave, Alexandre Défossez, and Neil Zeghidour. High- fidelity simultaneous speech-to-specd translation. In Forty-second International Conference on Machine Learning. 2025. URL /ttp:pp::/ppeeeeeveeeeeette/eeee 2tt
Tatiana Likhomanenko et al. Chipchat: Low-latency cascaded conversational agent in mlx. In ASRU, 2025.
Guan Guan-Ting Lin, Cheng-Han Chiang, and Hung yi Lee. Advancing large language models to capture varied speaking styles and respond properly in spoken conversations. In ACL, 2024.
Guan-Ting Lin et al. Align-SLM: Textless spoken language models with reinforcement learning from Al feedback. In Proceedings of the 63rd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pp. 20395 20411,, Vienna, Austria, July 2025. Association for Computational Linguisties. ISBN 979-8-89176-2511 acl-long.997. htttt:/ts://acc wwwaacaaanntooooooo anthol
doi: 10. 10.18653/v1/2022 997/
Ke-Han Lu et al. Desta: Enhancing speech language models through descriptive speech-text alignment. ArXiv, abs/2406.18871, 2024. URL https::/ /ats:///pp.... sewant org/ CorpusTD:2707643
12

---

Published as a conference paper at ICLR 2026
Jialong Mai, Xiaofen Xing, Yangbiao Li, and Xiangmin Xu. Slain-of-Thoughh Distillation with Fine-Grained Acoustic Cues for Speech Emotion Recognition. In Interspeech 2025, pp. 5438- 5442, 2025. doi: 10.21457/mmexxpecch.2222 1979.
Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple preference optimization with reference-free reward. In The Thirtt- Annual Conference on Neural Information Processing Systems, 2024. URL
bettps::/openreveee
net
Thomas Mesmard et al. Gemma: Open models based on gemini research and technology, 2024. 08295
URL http:////xxxivvorr//oo//
Eliya Nachmani, Alon Levkovitch, Roy Hirsch, Julian Salazar, Chulayuth Asawaroengchai Soroosh Mariooryad, Ehud Rivlin, RJ Skerry-Ryan, and Michelle Tadmor Ramanovich. Spoken question answering and speech continuation using cettrogram-powerree LLM. In The Tweifth Interna- tional Conference Learning Representations, 2024. URL https::/
Forum? witimmiddiiiiz Seiiro0IJovvy
Chanwoo Park, Anna Seo Gyeong Choi, Sunghye Cho, and Chanwoo Kim. Reasoning- Ap- proach with Chain-of-Thouuht for Alzheimer's Detection Using Speech and Large Language Models. In Interspeech 2025, pp. 2185-2189, 2025. doi: 10.21455/lmeexpeechh225-5 202-p/eccchh.2-1222222 1226.
Jing Peng, Yucheng Wang. Bohan Li, Yiwei Guo, Hankun Wang, Yangui Fang. Yu Xi, Haoyu Li, Xu Li, Ke Zhang, Shuai Wang, and Kai Yu. A survey speech large language models for understanding, 2025. URL.
abs////////// 18908
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and llya Sutskever. Robust speech recognition via large-scale weak supervision. In Proceedings of the 40th Interna- tional Conference on Machine Learning, ICML'23, 23. JMLR.org, 2023.
Rafael Rafailov et al. Direct preference optimization: Your language model is secretly a reward model. In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023. URL.
https: //ppnrevpeeeeeieww net/ forum? Fid- Hid-HHPSSXJaaas.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Com- monsense reasoning about social interactions. In Kentaro Inui, Jing Jiang. Vincent Ng, and Xi- aojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4463-4473, Hong Kong, China, November 2019. Association for Com- putational Linguistics. doi: 1..18853/v1/DD99 1454. URL
oorgg
D19-14547
Zayne Rea Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. To cot or not to cot? chain-of-thought helps mainly on math and symbolic reasoning. In The Thirteenth International Conference on Learning Representations, 2025.
net forum?
1d-w6n1cS5kknn
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https: 7 //openreview net wotp://foommmmm11nnnaaaa-
Hugo Touvron et al. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https:
//arxiv. org abs/ 2307.09288
Bandhav Veluri et al. Beyond turn-based interfaces: Synchronous llms as full-duplex dialogue agents. In EMNLP, 2024.
Chengwei Wei, Bin Wang. Jung-Jae Kim, and Nancy F. Chen. Towards spoken mathemati- cal reasoning: Benchmarking speech-based models over multi-faceted math problems. ArXiv, 1tt.22555.15000 2025.
URL hhtpps: :/ap:::///p.... www.ntisscoooeeeeeee.. o wttp://aaa..rr/ooouuoouueo..eeee
278782499
13

---

Published as a conference paper at ICLR 2026
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Ad- vances in Neural Information Processing Systems, 2022. URL Forum? d=nenidd=__a Ses___iQMMeS00
net.
Anne Wu, Laurent Mazaré, Neil Zeghidour. and Alexandre Défossez.
Aligning spoken dia- logue models from user interactions. ArXiv, abs/2506.21463. 2025. URL https://api.
semant 1ts.cholll..or///rruustt 2800012214
Zhifei Xie et al. Mini-omniireassoner: Token-level thinking-in-speaking large speech models,
2025. URL BLLLL//tss///waw..cr////////222.
Haoran Xu et al. Contrastive preference optimization: Pushing the boundaries of Ilm performance in machine translation. In ICML, 2024. URL httpss://ppennereeeeentttt www.ooruuntteeemmmm/ i
5liwkio8ph
Zhen Ye al. Llasa: Scaling train-time and cnference-time compute for llama-based speech synthesis. ArXiv, abs/2502.04128, 2025. URL htllhhtt/:ppppp httppppp...mmmtttcccccoooo....r.
CorpusID:276161207
Robin Shing-Hei Yuen, Timothy Tin-Long Tse, and Jian Zhu. Internalizing asr with implicit chain
of thought for efficient speech-to-speech conversational Ilm. ArXiv, abs/2409. 173533, 2024. URL
https::/api. http:ppp..smmmnniscch........... wwwpolaarrrrggoooouuuooo ID: :2:29111262
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. STar: Bootstrapping reasoning with
reasoning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Ad- vances in Neural Information Processing Systems, 2022. URLhttps: _d____E_Rd3g2qq11
nett/
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 THE "QUESTION COMPLETENESS" METRIC
In our preliminary experiments, we considered using entropy or log-probability as metries for mea- suring completeness. However, both were found to be less robust, as they were more susceptible to noise from the incomplete syntax of partial questions. Consequently, we adopted Kullback-Leiblee (KL) divergence for this purpose.
For determining the inflection point, we set a specific percentage-baset threshold rather than captur- ing the largest jump in the curve, suggested in prior work [Laiiauss al. (2025). A large jump may occur early in a user's question when key term is mentioned, but it does not mean that this partial information is sufficient to answer the question correctly. We conjecture that metric based on a completeness percentage is more semantically reasonable. Furthermore, through manual exam- training data, 95% threshold was empirically found align well
ination of the QC curves with human perception of question completeness, serving as conservative and effective criterion. We put more QC curves in the Appendix. Table shows more examples of QC curves.
Table 6: Examples of the Question Completeness curve (p). The word at inflection point p is shown as red and underlined. Each point on the horizontal axis corresponds to the cumulative sequence of words in the partial question up to and including the current word.
/=
b000
10007
0.8
Bumination monay:
0.6
produce??
0.4
rickal
cadmmium
0.2
magiins
What
0
8
12
13
14
16
p
1
0 0.95
Stndban
1ccned
(())
0.6
Wastington
Redskkin
0.4
Uhe
0.2
0-
Hey
you
know
what
0 - 0.96
byperomym Rortovie
0.8
trombone
words, NEWIES
Belongs
category
that
Uhink
0.8
0.95
keyboard?
0.6
Typroslly
has
something
0.4
0.2
Can
You
2
3
P
15

---

Published as a conference paper at ICLR 2026
0.8
0 0.96
Glessic
Ibarary
tary
15627
Sodwman and
boin
who
know
= 0.95
branch
Doverneent
dons
the
Unitod
States
0.2
Whisk
0
0
5
10
P
0.95
125
107
0.8
0.6
maing-
gold
into
tie
veniabbl
0.4
0.2
the
0at3
0
4
8
10
11
12
13
16
P
16

---

Published as a conference paper at ICLR 2026
A.2 TRAINING AND FINE-TUNING DETAILS
We fine-tuned the entire model with a learning rate (LR) of 4e-6 and batch size 128 using fully- sharded data-parallel (FSDP) on 8 A100 GPUs. All models were trained for 8K steps with warmup of 400 steps followed by LR annealing. We used Llama3-8B--han (Grattafiori et al. 2024) to es- timate X, which is required for estimating the inflection point á ($ 3.1p. For preference tuning experiments, we selected models that are fine-tuned with different 0 as the base models. We set learning rate 5e-7,
0.1, À 0.1, and trained with batch size for 1200 steps. Final checkpoint was selected based on saturation of reward accuracy. To get better monitor our model training, we curated the validation set with a more strict filtering process. Specifically, we only keep examples with question length less than 80 words and the question shouldn't 't include key- words such as "paragraph", "article", etc. and no special character allowed. The rest of dataset preparation procedure as same the training set.
17

---

Published as a conference paper at ICLR 2026
A.3 SPOKEN REASONING BENCHMARK
7 shows illustrative examples for each of the tasks in our SRQA benchmark. Since the source prompts for ARC-E, ARC-C, PIQA, and SIQA are choice-based tasks, LLM rewriting includes the vocalized options with the questions to make them suitable for spoken tasks.
Statistics and illustrative examples for each task in the Spoken Reasoning Question An- swering (SRQA) benchmark
Table 7:
Multiple choice
Q statistics
Task
Size
Example
Dur. (s)
#words
ARC-E
2376
14.5
40.5
Q: Plants use sunlight to make something, but what is it? Is it soil, minerals, food, or water? Food
ARC-C
/
1172
16.9
18.6
Q: What is the mass of carbon that has protons, neutrons, and electrons? it 6, 7, 13, or 19? A: 13
PIQA
1838
12.4+4.7
43.9=15.3
Q: want to install some cabinet pulls and I'm considering two options: either gluing old jewelry under the cabinet knob gluing it on top the cabinet knob. Which do you think would be the better idea? A: I think gluing the old jewelry on top of the cab- inet knob would be the way to go.
STQA
/
1954
15.1
49.8
Q: Hey, I was just watching this game and Ash had a pretty rough moment. He tried to redeem himself after missing an open shot. How do you think he's feeling right now? Would he be feeling disappointed, frugal. or maybe trying hard to shake it off? A: I think he 'd be feeling disappointed.
GSM8K
X
1319
15.3
46.3
Q: A candle melts by 2 centimeters every hour that burns. How many centimeters shorter will candle be after burning from 1:00 PM to 5:00 PM? A: 8
LLaMA-QS
X
300
3.0+08
8.5+21
Q: Which river is the longest in South Amer- ica? A: Amazon
18

---

Published as a conference paper at ICLR 2026
A.4 COMPARISON OF TEXT-BASED AND -ASECH-BASED COT
In 3 we showed comparison between text v/s speech modes of CoT, when trained and tested on GSMKK. While the accuracies than speech, with the token sequence lengths being 119.0 + 54.4 and 368.8 +: 190.3, respectively. Speech-based CoT may still be useful introduces no additional latency compared the baseline Moshi model, while still yielding substantial reasoning gains. This characteristic is similar to recent work such as STTTCH (Chiang et al., 2025) that does "thinking while speaking". Still, text-based CoT results in shorter latency to the model's final answer since model generates the reasoning trace, without having to vocalize
similar, reasoning in text was much more token-efficient
shows the variation in accuracy for the different methods by the number of ground-truth Figure 6
reasoning steps required for the solution. For models, accuracy degraded with an increase in # of steps, which indicates more complex questions.
35
Moshi (baseline).
30
Text CoT
Accuracy (%)
25
Speech CoT
No CoT
20
15
10
2
4
6
# reasoning steps
Figure 6: Accuracy variation by number of reasoning steps for text-based and speech-based CoT, measured on GSM8K.
19

---

Published as a conference paper at ICLR 2026
A.5 QUALITATIVE EXAMPLES
Table 8: Examples generated by models. We show some examples of the model output. Words in blue shaded are the spoken from the model. Words in red shaded are the user audio ASR. Words in green shaded are the CoT generated by the system.
Example I: User Question:
Steve put together puzzle that took 10 bours hard work to complete. Anna put together the same puzzle hours more than half Steve's time. How long did it take Anna to finish the diflicult puzzle?
Ground Truth Moshi :ashhiBascllie))
Based on the information from the previous paragraph. can infer that Anna took around six hours to complete the puzzle,
Moshi+CoT:
Steve put together puzzle that look hours of hard wark complete. Anna put together the puzzle in hours more than half Steve's ' time. How long did take Anna to finish the difficult puzzle? If Steve's 's time to complete hours, then half of this time hours. Anna finished the puzzle in 7 hours. Anna took seven hours to linish the puzzle, took
puzzic
0 hours. hours more than half Steve' time
2=
QC(0=0.95)
Steve put lugether a puzzle bours more than half Steve's 's time. First find half of How long did tike Anna finish the ditticult the puzzle? time it took Steve to put logether the puzzl: : 1 hours / 2 = hours. Then add hours to lind Anm''s 's time : hours + hours 7 hours. It took Anna hours finish the puzzle.
hours of
work complete. Anna put logether same puzzle
Example 2:
User Question:
What would he hest seen through telescope? Would he a moon, a cell, light ray, or molecule? moon
Ground Truth:
QC(8=0.75)
What would be best seen through telescope? Would it be moon. The telescope cell. device that uses a light ray, lenses or a molecule? or mirror s to mag ily images of distant objects. The most common use for telewcope is in astrooomy, where they used to observe celestial hodies such planets and stars. A telescupe the moon because it's an object located in space. However, a cell connot be seen through a telescope hecause cells are small and therefore not visible without some sort of magnification device like microscope. Light rays also do not need to he viewed through telescope s; instead they are just part of the process by which we see things with our eyes. Mol ce ules don' t exist outside of chemical reactions so socing them would roquire more than just looking at them through a lens. Therefore, "a moon the correct answer here you' be able to see the moon through telescupe.
+CorretDPOO
be used to
@C(5=0075) +LengthDPO
What would he hest through telescope? Would it he moon, The light see my, or the a molecule? moon through a telescope, You' d be abli to see a moon through a telescope,
answer a cell, is hecause you
Example 3: User Question:
Hey, I've got a friend named Jesse who just finished college after taking a few years to get degree. Now that he's finally done, what do you think be'll want to do next? Will he want to earell in college again. get job next, pay for college!
Ground Truth:
think he'l II want get joh next.
QC(# = 0.75):
Hey, I' got a friend named Jesse who just finished college after taking few years to get his degree, New that he's finally done, what do you think he" want river flow tough get a joh valley next, s" or pay for college? Actually, that's hat's an unexpected answer, although sooms unrelated. However, according to the answer rivers flow through valley s.
next? Here's the rationale he want enroll college again.
OC(0=0.75) +CorretDPO
friend named Jessie who just linished college after taking few years to get his degree. Now that be's finally done, what do you think he'll want to do next? Here's 's the rationale Will he want to enroll ; in college again, " has finally get job next, finished callege, what or pay for college? will he want do next? he will probably want to get a, job I think he" Il probably want to got job next.
Hey.
We present several qualitative examples generated by our models in Table 8 In Example 1, after fine-tuning with CoT, our model correctly answers the question, whereas the Moshi baseline fails. With our proposed QC-based early thinking, the model begins generating its CoT trace immediately after all information are provided. Therefore it reduces the latency.
In Example 2, we show example requires minimal reasoning but the model generate long COT, which increase the latency a lot. By applying our Length-DPO fine-tuning, we were able to significantly reduce the CoT length while still maintaining the correct answer
Example illustrates a limitation of the ba-dd early thinking. If the model initiates reasoning too carly-in in this case, before the answer "get job" is spoken-it is prone to generating incorrect reasoning trace and, consequently, an incorrect final answer. With Correct-DPO tuning, the model overcome this failure. Even when the CoT trace starts at the same early point, the model correctly considers subsequent incoming information from the user question, leading to a correct answer.
20

---

convontedraccinaie should be Published as a conference paper at ICLR 2026
A.6 SYSTEM PROMPTS
System prompt for LLM-judge scoring
You aew provided with queal iun, ground Leuil. an.sver and mudel Eeagonse, Your Look Lo detecmion Only whothez the modal response is correct.
nwer judge aun cactual compasenca in the ewaponae, such as opiniion, grvetingss bellefa, aud jeu statenents, follow-up questiens.
resgonse wish information provided.
Now Given
quest lonz [iuuatioon]]
ground
ligt. snewer]]
nodel__esponae [inodel_output]]
Output should
with dictionery containing keys (nodell_finelaannn julge_
De not include any other text.
Yor fmodel Finel answer', please extract the firal ansver From the model podel_rregee response.
If the mndel mutel__regume dowan') output final model resgonee charactersl.
Fina, annveeeeeeee
conclusior, ousput
Ido octpet any special
For "judge_ result', plesse nutput "ico_fiiaa annall output "<carrects" it the tinal ansver is equivsient to the tounnd_rrutharre
tolloving three options:
reagonse
final
3, output "sincorreats" it the model I tinal anaver is not equivalent to the ground truch ansver, Orly autput of the sbove three lons for "jadge reggllr judge
on the only given grouns wewth_
question nocol_ tinal_ttaeeeaeeeet
Example:
[Response]
answer": r20*, "judge resule"; "ccorrent>"|>
[Response]
System prompt for LLM rewrite on CoT-Collection
You helsful conversational
into natural, spoker ccaversation. not throw away information required ansvering the quastion. The question iteait should ze selz-cortaines for pecpue to answer it,
question
You are given question anawar and rationale. Ploase convert then into natural spoken converaction. multipic choice quastion, given queation is too lona, please sumnrrize and Includa informaiion sequired anavering. If the givan quesiton refers to an article, passage, paragraph, please include the essencisi information in the converted questicn.
neation. cholces
spoker
converted_quest.
conversed_ansver should written formas ecacise possiblel.
spoken
Do not use any special cherocters in the converted converted_meerton and converted_onsve Make suze the ranaartad_attional. is cobereat with the convrraa__uusttion and converted The shoud
follovine.
"..." "cameetues_mwweec "..." "esmerteddsatioonle: ?...")
Sxample:
[What waa roaction whan Choose the nost suitakle option to answez the above quastion. Options:
Quaations
weze given
axcsame
joyod
play chess
skip
sadnass]]
Anavers [B]
Nationalw: [The cliildewn were given Low cream, Lhey wn: juyed. So anaves B]
Octputs
Peunveeted "Ghaa swaution when childden
iow Did Lhey Seel noze suitable", "coavercadaansver Thaaa nost
envode, play chess, skip or aadneas? likely anjoyed. "converted tonel="; *The obi dreed were given fow crean, they enjoged. So the ansver joyad.n)
Now given
Ouest fons [gg
Anavers [[anawzz]
Netionele: [1ng))
Outputs
21

---

Published as a conference paper at ICLR 2026
System prompt for LLM rewrite SRQA benchmark
ARC-E
You are helpful assistant. Your taak to corvert kritten guestion on into a natural, spoken converaation
not throw awsy intormation required for ansvering the question. The quastion itselt should be selt- ansver it.
contained for
given question,, several oge eonioon.. mention the options in the coaverted spoken questicn.
then
spoken
The outpot should be in d53N format as the fol lowirg. "eorvected quextiin "...*)
Exanplel:
Questions IAn astrenomez observes rotacion?) likely
rotates
meteorite inpact. thich
Options: (I, Planetary deneity sill decrease.
Planetary will becone longer, Planstacy days
sharter.
Planetary gravity will become stongge:.
("corvetted ("convettel_ppetiiin""
observes planet the nost Likely effect of this increass in rotaticn? Mi11 planctary consity decrezse or planetary years become longer planetary days become shorter or planetary gravity bocome stronger?")
impact.
given
Ouestion: [iquestion]]
Options: [lopticae_sttr)
Octput:
ARC-C
You are helsful assistant. Your Lask corvert kritien questiin iate natural, spoken conversation. not throw away information required for ansvering the quastion. The quastion itself should be self- contained for people ansver it. quastion, naation the options The ouigul should be in JEON furmet as the following. quaveeted_ "...")
then nstural apoken conversation, Make coaverted spaken questicn.
Examplel:
Quastion: (An 2strenoner observes that planet zotazes faster after a neteorita inpact. thich is the mest Likely
this
la cotalian?]
Options: onst [1. Planetary denaity will decresze.
2, Planelacy will become longer.
Planstary
sharter.
Planetary gravity will becone at ronger.]
Octput:
Conveettdd_gueet ion':
metworite ingast. moat likely effect of increase sotation? Mill planelary densily decsease glanetasy years become longer or pl anetary days becore shorter or planetary gravity become stronger?")
Now given
Quest ionz [gue
Opt (loptione stri]
Output:
PIQA
helpful
goal solution
spoken
ccaversation. Do not throw eway information required for answering the questicn. The questicn itself should be selt-contained for people to anexer it,
You
solation suro to nention the eptions
Please convert then into natural spoker conversation. Make converted speken geal.
The output should in JSW tormat as the colloving. "oorvestod_aazii *...", ansxxer" "...*)
given
Goal: [1gea1H]
Solution 1: [lopticni]]
Solution 2: [1opticn21]
Ansvers [[answer]]
Octput:
SIQA
22

---

Published as a conference paper at ICLR 2026
are helsful assistant. Your task corvert kritien questien inte natural, spoken conversation. not throw away informacion required for answering the quastion. The quastion itselr should be selz- You contained ror people ensvEE it.
sucs to quastion, options the apcions The ourgut should be in JS2N tormet as the tollowing. p"ooverttedeoai" *..."y "conversed_nssee"" "...?)
convarced speken geal. into conversation.
Now given
Goal: [Icontent] (question]]
Option [ostionl): I
Option Hegtion2) |
Option 3: 3opt)on3) ]
Anaver: [[nnauz)]
Outputz
23


Reviews:
[{"review_id": "yOtWhUjud6", "reviewer": "Reviewer_8SXS", "summary": "This study presents the following key contributions:\n\n1. It extends the existing full-duplex SLM, *Moshi*, by integrating a streaming ASR component into its text channel.\n2. It incorporates the reasoning process while introducing a dedicated mechanism to determine the appropriate timing for reasoning during a user’s utterance, thereby aiming to reduce latency.\n3. Furthermore, it employs Direct Preference Optimization (DPO) to guide both response length and answer correctness.\n\nThrough these methods, the authors improve the overall performance of the conversational model and demonstrate that by adjusting the threshold related to reasoning timing, one can effectively balance and trade off between latency and performance.", "questions": "The only question I raised is addressed in the *Weaknesses* section, and there is a citation typo on line 348 (a missing parenthesis).", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written overall and effectively uses figures to support understanding.\n2. The authors successfully demonstrate the effectiveness of the introduced ASR modeling component and the use of DPO.\n3. They also show that latency can be reduced through appropriate adjustment of the threshold.", "weaknesses": "I have some fundamental questions regarding the underlying assumption of the scenario proposed in this paper. In typical real-world interactions with voice assistants, I am not sure how frequently situations arise in which it is actually reasonable or necessary for the model to perform reasoning *during* the user’s utterance. \n\nMore specifically, while it seems plausible for the model to reason incrementally based on partial chunks of the user’s speech, I wonder how common it really is for a partial question and the entire question to yield essentially the same reasoning trajectory. In addition, since a 95% threshold would likely result in only a very small difference in word count compared to the full question, I remain somewhat skeptical about whether it is actually possible to reduce latency meaningfully without degrading performance.\n\nAs the authors also mention, lowering the threshold to around 0.65 in order to achieve noticeable latency improvement leads to significant performance degradation. This makes me wonder whether, in most cases, the positions where the model determines that it can safely perform reasoning, say, around the 95% mark of the utterance, are in fact not meaningfully different from having access to the full question.\n\nEven in the figures provided by the authors, except for a few special cases, the reasoning appears to begin near the final 3~5 tokens of the input. Since this portion is part of the prefill stage, I am uncertain whether this actually produces a substantial latency advantage. Consequently, I find myself questioning the necessity and justification of this specific module and setup.\n\nPerhaps I overlooked it, but I would like to know, based on standard speech-related benchmarks (e.g., *VoiceBench*), approximately how many of the final words are typically excluded on average when using a 95% threshold.\n\nTo be clear, my comments are not meant to question the overall necessity of the task itself, but rather to express skepticism about whether the proposed setup reflects a truly *practical* real-world scenario. I hope this perspective is taken in that spirit.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gwFvX48I34", "reviewer": "Reviewer_DB5G", "summary": "This paper presents a Speech Language Model (SLM) that enables \"thinking while listening.\" The model is finetuned on Moshi and introduces a question completeness metric to support early reasoning when only partial user queries have been received. Additionally, the model leverages Direct Preference Optimization (DPO) to further enhance its performance.", "questions": "- Does the latency discussed in this paper refer to reasoning token length or answer token length? This differs from the \"first package latency\" commonly referenced in SLM systems. The distinction should be clearly explained in the abstract and introduction to avoid confusion.\n- Is the streaming ASR naturally supported by the Moshi model, or is it trained by the authors?\n- How much latency is added compared to the original Moshi model? Is it just the 480ms? Can you count all the latencies, e.g., Codec encoding, 480ms, LLM inference, and Codec decoding?\n- In the speech-based CoT setting, the authors appear to prefix the speech CoT sequence to the spoken response. Why not allow the model to generate the speech CoT reasoning sequence itself? A fairer comparison between Text CoT and Speech CoT would involve the model autonomously generating the Speech CoT.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of enabling \"thinking while listening\" in SLMs is novel.\n- Experimental results demonstrate strong performance for the proposed model.\n- Comprehensive ablation studies convincingly show the effectiveness of each proposed module.", "weaknesses": "- The paper does not evaluate end-to-end latency in terms of seconds (since the streaming ASR and the output of reasoning tokens may introduce further latencies). Latency is a critical factor for achieving natural conversational experiences, especially for full-duplex SLMs such as Moshi.\n- Training relies solely on synthetic audio, limiting the model's ability to handle diverse input voices. Although not a critical weakness, this raises concerns about potential overfitting to the voice characteristics of the training speech. It would be valuable to assess the model's robustness on varied input voices. The authors could consider benchmarks like [VoxEval, 2025] (support both mathematical reasoning and diverse input audio conditions), [VoiceBench, 2024] (support diverse input audio conditions), or similar alternatives.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "61kIrak5R8", "reviewer": "Reviewer_d8Do", "summary": "This paper presents a method to interleave chain of thought (or any reasoning approaches) into speech language models. The naive way is to just perform CoT in text, before projecting it back to speech. This is a naive baseline that works, but increases latency.\n\nThe paper proposes a method to efficiently reason while the model is still listening to audio tokens. The method first uses something called a question-completeness metric that indicates when there is sufficient information in the question stream for the model to begin answering/reasoning. This metric is calculated using a model's output logits. The authors claim this allows us to generate a training dataset that we can perform SFT on the model.\n\nThen, the authors introduced a preference optimization to further improves the model training.", "questions": "1. Please elaborate how the training dataset can nudge the language model to start reasoning at certain inflection point. if the author can elaborate on this clearly in the rebuttal, I will consider increasing the score.\n2. The authors also claimed that their method is Moshi + CoT (ours)♣ and because of this, they outperformed Moshi by a lot (2-3x). This is kind of misleading - I think simply injecting CoT into speech modeling is not that novel and should only serve as a baselines. The authors should focus on the improvement in latency in the approach (by interleaving reasoning while the inputs are still streaming in).\n3. Third, I think time is something that is indicative of a good speech model. The authors mostly use number of tokens as an indication of latency but what kind of time are we talking about here? Are we able to reply in a small amount of time?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 4, "strengths": "I think the paper's overall idea is very relevant and practical for speech-based language models. In its introduction, it gives a clear outline about what the approach is about. I quickly could understand it is trying to interleave reasoning early on while the language model is still listening to the input streams. \n\nThe question-completeness metric also makes perfect sense, allowing us to identify when the LLM has enough information to start thinking about the answer (instead of listening till the end). Hence, I mark the paper's contribution as \"excellent\"\n\nHowever, the paper leaves out the details on how exactly we can use their findings to fine-tune the LLM (e.g., how to use the offline training data created to actually perform SFT, and how that actually helps the LLM to determine the inflection point during runtime). I have some unanswered questions regarding this part, which is arguable the most important part of the paper.", "weaknesses": "While the paper does well in explaining the big idea, it has a significant weakness in explaining the details of its approach: it leaves out important details on the actual implementation that makes use of the question-completeness metric to fine-tune it to begin reasoning early. \n\nLet me give an example. In section 3, the authors mentioned that:\n> The first scenario includes questions which can be considered “complete\" before reaching the end. In such cases, the model can start reasoning early and simply ignore the remaining question. In the second scenario, sufficient information may be available to start reasoning before the question ends, but the model still needs the remaining information to provide a correct response. We propose two different methods to enable early thinking.\n\nThis is a great introduction to the motivation behind their approach, and it is very clear. But then the authors proceed by saying:\n> To endow the\nmodel with the ability for early reasoning, we created training examples by using our proposed\nQuestion Completeness metric. This metric is designed to identify the optimal time for the model to\nbegin generating its CoT. Subsequently, we fine-tuned the model on this dataset to teach it to follow\nthe distribution of these early-reasoning examples.\n\nFirst, I understand this metric can identify the optimal time to begin generating its CoT and we can create some kind of training dataset from it. But how does this lead to \"Subsequently, we fine-tuned the model on this dataset to teach it to follow\nthe distribution of these early-reasoning examples.\" I can understand this sentence on its own but I cannot see how the previous metric allows us to \"follow the distribution\" (this phrase is vague on its own). Even when I read on to section 2.2, I cannot clearly see how we can use the created training examples to \"force\" the model to start thinking earlier. My very vague guess is that we need to insert the \"start_COT\" token to the end of the inflection point, which forces the model to start thinking and fine-tune the model from this? In addition to this, we need to append some reasoning traces to the training dataset as well? I might be wrong (and I'm not surprised I am), but that's because the papers leave us to piece together the explanation.\n\nI think there is some logic gap here (arguable at the most important part of the paper) and the authors should elaborate the details better. I'm sure the authors know how the details work but we as readers need to make educated guesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)