Speech World Model: Causal State–Action Planning with Explicit Reasoning for Speech
Authors: Xuanru Zhou
Jiachen Lian
Henry Hong
Xinyi Yang
Gopala Anumanchipalli
Venue: ICLR 2026 Poster (2026)
Keywords: speech
spoken language understanding
state-action
causal reasoning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Current speech-language models (SLMs) typically use a cascade of speech encoder and large language model, treating speech understanding as a single black box. They analyze the content of speech well but reason weakly about other aspects, especially under sparse supervision. Thus, we argue for explicit reasoning over speech states and actions with modular and transparent decisions. Inspired by cognitive science we adopt a modular perspective and a world model view in which the system learns forward dynamics over latent states. We factorize speech understanding into four modules that communicate through a causal graph, establishing a cognitive state search space. Guided by posterior traces from this space, an instruction-tuned language model produces a concise causal analysis and a user-facing response, enabling counterfactual interventions and interpretability under partial supervision. We present the first graph based modular speech model for explicit reasoning and we will open source the model and data to promote the development of advanced speech understanding.

Published as a conference paper at ICLR 2026
SPEECH WORLD MODEL: CAUSAL STATE-ACTION PLANNING WITH EXPLICIT REASONING FOR SPEECH
Xuanru Zhou', Jiachen Lian Henry Hong", Xinyi Yang' & Gopala Anumanchipalli
"Zeejinng University, PCC Berkeley
xuanruzhoul gmai com, jiarchen
(
Please analyze speech provide stap-by-step breakdown of your reasoning.
Instruction:
The man the audio spesking English and seying. "Lknow! Don't switch hands, obav?" is difficltt to determine the exact context the conversation without further information.
Qwen2-Audio
The audio clip seems to be a humorous exchange between two people, possibly teachor and student, as indicaled by the phrase "don' saltch handss ohay? This could playful instruction classroom setting. the teacher trying the sludent
Voxtral
sounds sameone giving instruction, possibly to someone else performing a task that requires focus, like a sport or a game. The speaker seems he emphaszing not to switch hands, which might suggest they are guiding or coaching somcone.
ChatGPT
The audio in a communication context that likely involves physical activity or interaction, such passing objects, the speaker's fearful tone conveys serse urgency sbay?" simply reessurance and clear commimmntt from other party. This implies need consistency and trust the arrangement, as the speaker is looking for definiive answer to resolve their fear and feel more secure the situation.
Ours
Speech World Model
surface underfying
switch
Figure 1: Speech World Model. Demo audio link: http:///it.11/4pB...
ABSTRACT
Current language models (SLMs) typically use a cascade of speech encoder and large language model, treating speech understanding as a single black box. They analyze the content speech well but reason weakly about other aspects, especially under sparse supervision. Thus, we argue for explicit reasoning over speech states and actions with modular and transparent decisions. Inspired by cognitive science we adopt modular perspective and world model view in which the system learns forward dynamics over latent states. We factorize speech understanding into four modules that communicate through causal graph, es- tablishing a cognitive state search space. Guided by posterior traces from this space, an instruction-tunel language model produces concise causal analysis and a user-facing response, enabling counterfactual interventions and interpretability under partial supervision. We present the first graph based modular speech model for explicit reasoning and we will open source the model and data to promote the development of advanced speech understanding.
INTRODUCTION
Speech language models (SLMs) (Cui et al. 2024) provide unified framework for general-purpose speech understanding, enabling multitask perception, reasoning, and multi-turn interaction. At the same time, ongoing debate persists regarding whether such models truly exhibit genuine reasoning capabilities or merely perform sophisticated pattern matching and statistical abstraction over tokens (Shojaee al.) 2025). One direct observation is that current SLMs appear to combine outputs from isolated tasks such as automatic speech recognition, emotion recognition, communicative
"Equal contribution.

---

Published as a conference paper at ICLR 2026
Causel Graph Training
Test lust took cart couit you help
Instruction Tuning
Taet: lost bank card, coud you (anceliaan) pico aoemoont
Causal Graph
*
LLaMA LLM
WMA
Adapter
Speech Encoder
Text Encoder
Ressoning
Jaferency
Sai "Aoquestdeettr Ping: "Yoen Comphertrr
Response
Prop
Chven2-Audio
Fusion
Adaster A
Figure 2:
The Speech World Model pipeline with a running example, illustrating the "Causal Graph- Guided Explicit Reasoning" process. (1) Causal Graph Training: multimodal inputs (text æ, acoustic a. prosody 2) are encoded and fused to
o(h ha, h2). Each node state S, is inferred from its parents Pa(v) and fused feature g via S. = softmax (o (fo
yielding structured these states are used as explicit guidance for the large (speech) language models to generate response y by maximizing f = - Σ log Po Instr, x, S).
{Su (oeeee))))) xieeddng)
reasoning. (2) Instruction Tuning:
function detection, intent classification, and pragmatic inference and then present this aggregation as if it reflects reasoning. However, this overlooks the intrinsic dependencies among these speech components and provides only partial supervision, which we argue fundamental to enabling genuine reasoning in speech understanding. From human-centered cognitive perspective, speech perception is thought to be hierarchically organized into partially specialized modules-such as acoustic, linguistic, and paralinguistic processing- that interact through causal and reciprocal relations (Hickok & Poeppel| 2007). However, this does not entail designing strictly brain-like modular systems (e.g. verified AI (Seshia et al., 2022)). as human perception itself operates through overlapping and partially opaque processes that remain incompletely understood (Angelaki et al. 2025).
An alternative line of research seeks intermediate approach that models latent internal states, thereby enabling more interpretable predictions of system dynamics. A representative example is World Models (Ha & Schmidhuber 2018 LeCun 2022), which posit that the subsequent state is conditionally predictable from any given current state. In the domain of speech, human perceptual systems also anticipate interdependent dynamics across states such as emotion, intent, and linguistic content, such that changes in one dimension systematically constrain others (Hickok & Poeppel (007). These observations point to an underlying structure of causal dependencies. is worth noting that Chain-of-TT Thoo-- (CoT) prompting (Wei et al.) 2022c) represents a parallel approach, which expands the search space to improve goal-directed reasoning at the expense of increased computational cost. However, the search process is not grounded in human speech perception principles.
In this work, we propose Speech World Models (SWM) to advance speech understanding and reasoning by modeling the dynamics of internal speech states. SWM comprises four explicit wodlles-World Model Action (WMA) (Mesaros et al., 2018). Theory of Mind (ToM) (Sclar et al., 2023). Speech Act (SA) (Bothe et al., 2020), and Pragmatic Intent (Prag) (Stolcke et al. 2000 subspace of the cognitive perception system. A causal-graph- -raph--bee- based perception module is introduced to bridge these components and capture the flow of causal information. In the first stage, this graph is trained in both supervised and semi-supervised settings to establish a cognitive state search space. Notably, leveraging causal dependencies among modules yields faster convergence than training the modules independently. Moreover, because the graph approximates cognitive state structure, it can reliably infer unlabeled modules from partially labeled data, suggesting that causal graphs function as natural generative structures for latent variables. Once the search space is established, in the second stage we incorporate the outputs of the causal graph-samples from the search space, state nodes, and information flow-into prompts for instruction tuning. This provides an explicit reasoning advantage: by enforcing structured reasoning chains, SL.S are guided to reduce hallucinations and achieve greater overall reasoning capacity.
an approximated
Experiments demonstrate the superiority of SWM over traditional SLMs across a range of reasoning metrics. First, when comparing against randomly initialized graph without cognitive grounding, we find that our causal graph converges faster during training and achieves substantially higher reasoning scores (c.g., ACE, ICS). Second, postruction-tuning results show that SWM outperforms open-source SLMs such as Qwenl (Chu et al., 2023). Qwen2-Audio (Chu et al., (024). and Voxtral (Liu et al.) 2025) in reasoning ability, with particularly strong performance in emotion recognition even
2

---

Published as a conference paper at ICLR 2026
surpassing commercial hoaddls-highlihhhinnn the benefits of explicit reasoning. Third, while the overall speech understanding performance of SWM (as measured by our proposed metrics) remains slightly below that of Gemini 2.5 Pro (Gemini Team 2025), our training cost is dramatically lower (only 20 GPU hours), validating the efficiency of the causal gaassal-gaah-bbasee explicit reasoning paradigm.
2
RELATED STUDY
2.1 COGNITIVE FOUNDATIONS OF SPEECH PROCESSING
Cognitive neuroscience shows that speech perception and comprehension emerge from modular computations over latent states, rather than a single monolithic process. Core frameworks include the
model linking ventral (semantic) and dorsal (sensorimotor) pathways (Hickok & Poeppel (007), staged accounts of syntactic and semantic processing (Friederici (011). and analyses of the superior temporal gyrus that reveal intermediate phonological and prosodic representations (Lonnar& & Chang 2022). Complementary perspectives highlight cortical oscillations aligned with hierarchical linguistic units (Giraud & Poeppel| 2012) and predictive coding views of the brain as a generative model updating latent states via prediction errors (Friston et al. 202T). Together, these perspectives converge on four principles: (i) modular and hierarchical decomposition of speech, (ii) forward predictive dynamics, (iii) integration of perception and action, and (iv) generative modeling of latent auditory states. Our Speech World Model operationalizes these principles computationally.
2.2 SPEECH LANGUAGE MODELS AND REASONING
Early work on speech language models (Gong al. 2024 2023 Tang al. 2024 Kong al. 2024) pioneered the use of instruction tuning pipelines for speech understanding and preliminary reasoning tasks. Subsequent studies adopted Chain-of- of-Thuugh (CoT) prompting (Wei et al. 2022c) to construct reasoning-cle
-aaninnggghhiinnn chain search spaces, which advanced reasoning capabilities in speech-related
tasks (Ma et al. 2025 Xie et al. 2222a b Kong al. 2025); see also (Cui et al.) 2024) for broader review. Nevertheless, these CoT-based approaches remain largely heuristic and are not grounded in
principles of human speech cognition (Hickok & Poeppel 2007).
2.3 WORLD MODELS
The concept of world models has long-standing roots in cognitive science and neuroscience, where internal models are thought to support prediction. planning, and reasoning (Craik) 1967, Friston 2005; Hickok & Pocppel 2007). While recent computational advancements have successfully
operationalized this concept via latent dynamics for planning and reinforcement learning (Ha & Schmidhuber 2018 Hafner et al. 2019 Schrittwieser et al. 2020 LeCun 2022). a parallel line of research emphasizes the structured nature of these models. Ullman & Tenenbaum (2020) and Lake et al. (2017) argue that robust learning resembles theory building, where world models internalize intuitive theories of physics and psychology. Instead of merely predicting sensory streams, such cognitive world models explicitly reason about causal mechanisms and agents' mental states (Baker (et al., (017).. In the context of generative AI, this perspective inspires systems that not only model data distributions but also capture the underlying hierarchical structure of the domain, such as vision (Garrido et al. 2024) and language (Lin et al. 2024).
3 SPEECH WORLD MODEL SYSTEMS
In this section, we present the methodology behind our proposed Speech World Model (SWM). At its core, the framework relies on causal graph that factorizes speech understanding into modular components, enabling explicit reasoning over speech states and actions. These states then condition (spoken) language models, which are fine-tuned to generate both transparent reasoning trace and a potential response to given utterance. This design allows us to validate the hypothesis that explicit state representation improves Speech Language Model's reasoning capacity and reduces hallucinations (Atwany et al. 2025).
3.1 WORLD MODEL CAUSAL GRAPH
We proposed a probabilistic causal model, represented by a Directed Acyelic Graph & = (V,E), where 1 is set of nodes representing different modules and E is set of directed edges representing
3

---

Published as a conference paper at ICLR 2026
Cousall Graph
Explicit Remsoning Chain
8141 fl
reprace (staten)
Statsmenk Complaint
trening
prompi
"Input"
WMA
SA
AState y Action
World Mocel
Pertegrion
ToM
aZEON
Resoning Chiin
perOUsT
Noutral - Angor
Prag
Resporos
Pregram
"Ouiput"
Generative
Proposed Causal Graph
Language World Model
Figure 3: A unified perspective on world models. Both the Generative (left (Garrido et al. 2024)) and Language (right) world models are forward dynamic models. Our Causal Graph (center) provides a structured, explicit formulation of these dynamics. See Appendix A.1 for details.
causal relationships between modules. This structure allows to explicitly model the dependencies between different aspects of speech and perform causal reasoning.
3.1.1 MODULES AND STATES
To construct holistic casual graph for speech understanding, select a set of latent variables that are both necessary and sufficient to describe the communicative process. Drawing from the standard model of communication (Shannon 1948) and cognitive pragmatics (Bara 2011), we define four key modules. These modules are not arbitrary selections but represent a hierarchical cognitive chain, progressing from situational grounding (Context) to agent modeling (Mental State), and finally to communicative action (Act) and goal (Intent). This combination ensures comprehensive coverage of essential speech properties, spanning the "where, who, what, and why" to form a complete cognitive perceptual subspace. The complete label spaces are provided in Appendix A.4.2
World Model Activation (WMA) represents the situational grounding of the interaction. Speech does not occur in a vacuum but is situated a specific functional context. Drawing on the concept of Situation Models (Zwaan & Radvansky 1998), WMA captures the active domain scenario (e.g., SmartHome, Finance, Alarm) (Bastianelli et al. 2020). This serves as the global precondition, "activating" the relevant knowledge partition required to interpret subsequent events.
Theory of Mind (ToM) models the agent's internal state. Originating from cognitive science, ToM refers the ability to attribute mental states (e-g. intents, emotions) (Premack Woodruff 1978 Baron-Cohen et al. 1985). While recent research has extensively explored ToM capabilities
Large Language Models, ranging from emergent behaviors (Kosinski) 2023) to robust reasoning evaluation (Sap et al. 2019 Sclar et al. 2025). explicitly operationalize this concept for speech understanding. In our framework, this module captures the speaker's ' affective state and personality traits, acting the internal driver for the observed speech behavior.
Speech Act (SA) identifies the communicative function of the utterance. Based on the foundational Speech Act Theory by Austin (Austin) 1962) and Searle (Searle) 1969), this module categorizes the utterance into illocutionary acts (e.g.. Questioning, Commanding, Asserting). It represents the surface-level "action" performed by the speaker (Bothe et al. 2020).
Pragmatic Intent (Prag) represents the underlying goal or the perlocutionary effect the speaker aims to achieve. Pragmatics deals with the "unspoke" logic and implicature (Levinson) 1983). It captures why the speaker performed a specific act (e.g., using a question to sarcasm or solicit help), grounded in the theory of Indirect Speech Acts (Ruytenbeek) 2021).
Graph Perspective. Each module corresponds node, instantiated as a neural network perform- ing its own computation. Formally, we model the state of each module as a discrete categorical variable. The outputs of these modules are categorical latent states, and collectively they form the structured representation of speech, presented in Fig.
World Model Perspective. Inspired by the essence of image world models (Garrido et al., 2024). view the entire system forward dynamics model: the current latent state, together with an
4

---

Published as a conference paper at ICLR 2026
SA (Latent)
SA
SA Forward Computation
@ WMA
WMA
WMA
Gradient Flow
Supervise Loss
ToM
8 Prag
ToM
Prag
ToM
Prag
Semi-supe sumi-iupervised Causal Graph
Fully-supervised Causal Graph
(C) Fully-supervised Random Graph
Figure 4: Comparison of gradient flow under different training scenarios. (A) Semi-supervised causal graph: when module lacks direct supervision, it act latent variable generator, and gradients propagate backward through causal edges from downstream modules. (B) Fully-supervised causal graph: all modules have labels, losses are applied locally, while causal dependencies still guide gradient flow. (C) Fully-supervised random graph: supervision is present at every node but edges are unstructured, leading redundant and less efficient gradient propagation.
action, leads to the next latent state. Within this view, the notion of action extends beyond external commands. Instead, an action can understood as the causal influence exerted by one state on another. For example, when the Theory of Mind module shifts from neutral to anger, the downstream Speech Act may change from Statement-opinion to Complaint/Escalate as shown in Fig. 3
Thus, actions in framework best understood the state transitions induced by causal relationships within the graph. Formally, for a node with parents Pa(e):
S. fo ( Pa(u)), Au-ro),
(1)
where S. is the latent state node v, and Au denotes the action, interpreted the causal influence (action) of parent node on
3.1.2 GRAPII CALCULATION
Given an input speech signal X, the casual graph infers a joint posterior distribution over latent states = { (AW ZTM As 3Prag]. y. By the DAG factorization:
X = p(awMalA X) p(ZroM|(X)
p(zsAl ZWMA, STMM. D(ZPraglZSAr 2S.. ZTMM 2 2W,
Each conditional p(z,|Pa(u)) (P) X) is parameterized by neural module fu(;; modeled as multi- class classifier. Our proposed casual graph is indicated in Fig. (B). The detailed calculation and model architecture are detailed in Appendix A.2.I
3.1.3 MULTI-TASK TRAINING WITH CASUAL GRAPH
Causal graph is trained in multi-task manner to form reasoning-chain search space. The supervised loss is shown Eq where mi,r indicates whether the label for node is available sample
Suuu
Σ mi,r CE(yi,; St.),
VEV
Teacher forcing (Bengio et al. 2015) is applied edge-wise: each child node receives either the ground-truth parent state the predicted distribution. For edge u u-
5,a
Tiu+v onchot + (1 +e) stopgrad(S..).
Tu. Maa--o-eernlllll Bernouli(ppu)))))) (4)
The mixed signal Si, is used as the parent input when computing the state of child
3.1.4 SEMI-SUPERVISED LEARNING
Motivation. Annotations across modules are costly and often incomplete (e.g., WMA labels are missing more frequently). If parameter updates restricted only to nodes with available labels, large fraction of training data would be wasted. Semi-supervised training allows unlabeled parent nodes to be optimized through the supervision signals of their labeled children.
5

---

Published as a conference paper at ICLR 2026
Gradient Flow. Consider parent node j without labels in sample i (m,. = 0). where of its child node k has label (ma, = 1). To ensure a differentiable path to the unlabeled parent node j. teacher forcing disabled (pz->==0), forcing the child & to rely on the parent's continuous prediction Siz. Thus, the loss Li,A CE Si,K) propagates to 0g through the chain rule:
OC; 30g
iCi, Ōmik OS..j Omi.i Omck aS..j Ōm.. 30,
Σ
+ 0.
(5)
l:: 1=
jEPa(k)
where the gradient also flows into the parent distributions p(*" (a). Thus unlabeled parents act as latent variable generators that receive updates from their supervised children as shown in Fig. (A).
Multi-task supervision ensures "learn whenever label is available" (Eq. 3 while semi-supervised training ensures "learn even when labels are missing" via chain gradients (Eq. (5. Together they allow effective learning from incompletely annotated data under the causal graph structure.
3.2 RANDOM GRAPH
To validate the benefits our proposed causal structure, grounded by cognitive speech perception, we introduce the Random Graph strong, unstructured and data-driven baseline. This model replaces the predefined DAG with a fully connected graph where every module can, in principle, influence every other module. Consequently, the prediction for each module is conditioned on both the speech input and the initial, teacher-forcel estimates from all other modules in the graph, allowing the model learn arbitrary dependencies directly from the data.
x(
= II p(z 2 X) DEV
By comparing this model with our structured causal graph, we can quantify the contribution of imposing theoreticallyggoondde causal priors speech understanding. The detailed, step-by-step computation for this iterative process is provided in Appendix A.2.2
3.3 SPEECH UNDERSTANDING AND REASONING
To bridge the gap between the structured latent states inferred by the causal graph and human- interpretable analysis, we employ an instruction tuning (Wei et al. 2022a) phase. This final stage designed to validate our core hypothesis: Conditioning on these explicit graph states tends to guide the reasoning-chain search toward human-aligned spaces.. To this end, the model is trained to produce a transparent reasoning process and contextually appropriate response based on the graph's outputs. We explore two distinct settings for this purpose: language-only approach and multi-modal approach.
3.3.1 LANGUAGE-ONIA SETTING
In this setting, we fine-tune standard LLM to reason over the symbolic outputs of the pre-trained causal graph G. The input is prompted with general instruction (Instr) and textual serialization of the graph's 'ss output, I (Gxx), which encodes the inferred states and their causal relationships. The prompts used for training
detailed in Appendix, A.5.2
The model is trained to produce a target sequence y that contains both step-by-step reasoning process and potential dialogue response. The training objective is minimize the standard cross- entropy loss over the target sequence, where 0 represents the parameters of the LLM,, and the target the ground-truth reasoning and response pair.
(i)
Σ (a.))D
log
Instr, 1(G(x))
3.3.2 MULTI-MODAL SETTING
This approach leverages a speech language model to directly ground the symbolic states in the acoustic properties of the raw speech signal. The input the SLM consists of the instruction (Instr),

---

Published as a conference paper at ICLR 2026
the raw speech input æ, and the set of inferred states {So hev. The model's task is to generate the same target sequence containing a reasoning process and a response.
(0) = Σ
log Po Instr, æ, SWMA. SToM, SSa, Spras)
(8)
(x.y)ED
This objective forces the SLM to build joint understanding of the audio signal and the structured states, effectively learning to associate "what was said" and "how it was said" with the "why" provided by our causal graph's 's analysis.
EXPERIMENTS
4.1
REAL-WORLD DATASET
The following four datasets serve as the primary speech sources for our experiments. Please refer to Appendix A.4.1 for detailed statistics regarding these datasets.
MELD (oria et al. 2019) is conversational dataset for emotion recognition. It consists of over 13,000 utterances from the TV series Friends, annotated with emotion and sentiment labels.
IEMOCAP (Busso et al. 2008) an audiovisual and multi-speaker database of dyadic interac- tions between actors, containing approximately 12 hours of data annotated with categorical and dimensional emotion labels.
SLURPP (Bastianelli et al. 2020) is challenging spoken language understanding (SLU) dataset featuring single-turn user interactions with a voice assistant. Utterances are annotated with a hierarchical schema, providing both user intent and corresponding action.
VoxCeleb (Nagrani et al., 2017) is a large-scale rext-independent speaker identification dataset. comprises hundreds of thousands of utterances from celebrities, extracted from YouTube videos.
4.2 LABEL-GENERATIIN
A central challenge for scaling up is that real-word speech data usually comes with only partial supervision: some modules may be annotated while others are not. To construct complete instances for sulyy-supervised graph training and instruction tuning, implemented robust two-stage label generation pipeline leveraging Vicuna-13b-v1.5 (Chiang et al. (023) as the teacher model.
Stage 1: Constrained Label Imputation. Given the speech transcription and any available module
labels, the goal is to infer missing labels. We employ one-shot prompting strategy (Brown et al. 2020), providing the model with the complete label space definition and single demonstrative example to enforce the desired output format. To ensure the generated labels map strictly to valid categories, we apply a constrained decoding mechanism that maps the generated tokens to the nearest canonical label by minimizing the Levenshtein edit distance. As quality control measure, instances where the model fails to generate parsable or valid labels after matching are discarded.
Stage 2: Reasoning and Response Synthesis. By leveraging the robust causal priors inherent in LLMs (Kiciman et al.) 2024; Binz & Schulz 2023). we explicitly aim to simulate the latent data- generating process of speech. Conditioned on the complete labels (from stage 1) and transcription, we query the LLM again to synthesize the downstream instruction data: (i) a comprehensive reasoning analysis which explains how the inferred states interact through the causal graph to shape the utterance, and (ii) a contextually appropriate response aligned with these states.
Full details of prompts and other implementation specifics are provided in the Appendix A.4.2
4.3
METRICS
Node quality. We evaluate the performance of individual modules (nodes) their respective sub-tasks using standard classification Accuracy and weighted FI-Score.
Edge causal efffect. To validate each learned edge X + Y, measure its causal strength using the Average Causal Effect (ACE) (Pearl] 2010) and its directional consistency with the Intervention Consistency Score (ICS). ACE indicates the magnitude of post- iost-intervention effects, while ICS indicates effects align with a predefined hypothesis. See Appendix A.8. for computational details.
7

---

Published as a conference paper at ICLR 2026
Inturvention Constatency Scoregcs
16
Average Caesal 194
26-
80
25
23-
30
22
25
20
20
Fully (Buseline)
(loM) sup (SA)
sup (Baselino)
sup
19
Semi-aup (WMA)
Scmi--up (WMA)
Scral (SA)
WMA
ToM
Prog
Prog
5A Prag
WOAA
ToM
WMA .Prag ToM Prag
Prag
Causal Edge
Causal Edge
Figure 5: ACE and ICS of each casual edge under both fully-supervised and semi supervise training.
Instruction tuning. To provide a scalable and consistent evaluation of our instruction-tuned models, we employ the Model-as-Judee (M..) methodology (Zheng et al. 2023). where GPT-4o (OpenAI 2024) acts as an impartial judge to score both reasoning and response quality. The overall M.J. score a weighted sum of reasoning and response scores. EM and EA refer to emotion mention rate and emotion classification aceuracy, respectively, while R-Len represents the length of the reasoning process. The detailed prompting strategy and scoring rubric are indicated in Appendix A.9
4.4 EXPERIMENTS SETUP
Our method employs a two-stage training pipeline: causal graph training followed by instruction tuning. The causal graph consists of MLP-based classifiers with temporal attention, trained under fully- and superviie settings. To validate training efficiency, we report wall-clock times averaged over seeds (42, 123, 2025) single NVIDIA A6000 GPÚ. Our Causal Graph converges in 2.07h (# 0.04), achieving ~5x speedup compared to the Random Graph baseline (10.39h + 0.15) under identical settings. For instruction tuning, we apply LoRA (Hu et al.] 2022 to both Llama3. 1-BB Meta (024) (language-only) and Qwen2-Audio (Chu et al.) (2024) (multi-modal). Training on 4 NVIDIA A6000 GPUs required 19h for the language-only model and 24.6h for the multi-modal model, respectively. We benchmark against a Owen2-Audhe baseline fine-tuned with CoT prompts. Full architectures, configurations, and ablations are detailed in Appendices A.5 A.7 and A.6
4.5
GRAPH EVALUATION
The results of our graph evaluation are presented in Table. and Table. 2 with detailed breakdown of edge causal effects shown in Fig. [5] These results first demonstrate the efficiency of our Causal Graph, which achieves node accuracy comparable
Random Graph baseline at a significantly lower 2 exposes fundamental flaw in the Random Graph: structural instability. The
training cost. Table dominant information flow in the Random Graph fluctuates chaotically with training hyperparameters (teacher-fae (eeachr-forcing ratio), suggesting it functions as a black-box that exploits spurious correlations rather than learning invariant mechanisms, contrast, our Causal Graph maintains consistent ACE and ICS scores, proving it captures stable, interpretable causal dependencies. Fig. JJoffrrs granular view of model's robustness under partial supervision. Crucially, when a specific module (c.g.. ToM) unsupervised, the drop in ACE is logically localized the edges connected to that node (e.g., ToM-SA), while other pathways (e.g., WMA -SA) remain unaffected. This orthogonality confirms that our modules have learned disentangled representations, allowing the system to effectively infer missing states via the causal structure rather than collapsing.
4.6 SPEECH UNDERSTANDING AND REASONING EVALUATION
Our speech understanding and reasoning results (Table. 3 reveal the distinct con- tributions of our reasoning framework. To isolate these effects, we created a tuned baseline by fine-tuning Qwen2- Audio with our CoT-style instruction data. This baseline alone surpasses other open-source models, confirming the high quality of our instruction tuning data, and our full SWM models, which leverage the causal
Table 2: Node quality and information flow analysis for random graph baseline. (Detailed in Appendix A.8.2)
Node (Accuracy %
Information flww
plotscher_foree)
ToM
Strungod
Weakes
Ni1
75.2
66:1
82.5
SA WMA
WMA Prng
69.7
76.0
67.5
83.6
ToM SA
ToM Prag
73.1
ToM
74.4
TrM
SA
TeM
8

---

Published as a conference paper at ICLR 2026
Table 1: Performance evaluation of the causal graph on node accuracy and edge validity under
different supervision settings. The gray background in the semi-supervise rows highlights the accuracy of modules that were left unlabeled during training, demonstrating the model's 's ability to infer latent states via causal structure. The rightmost columns evaluate the strength of learned causal dependencies using verage Causal Effect (ACE) and Intervention Consistency Score (ICS).
Method
Setting
Node Quality (Accuracy %. f)
Edge Casual Effect
WMA
ToM
SA
Prag
Ave. ACE (%. 1)
Ave. (%. f)
Fully-supervised
69.4
73.5
65.3
81.4
23.57
43.29
Semi-supervised
Latent module: WMA
34.8
75.0
70.7
83.2
21.71
26.9
Latent module: ToM
69.1
43.3
69.6
83.5
21.98
28.9
Latent module; SA
69.3
77.0
34.4
82.5
21.65
29.3
Random Graph
69.7
74.0
67.5
83.6
graph for explicit reasoning, significantly outperform this tuned baseline. This demonstrates that while good data is important, the causal graph's 's explicit guidance is the critical factor driving superior reasoning ability. This benefit is particularly stark in emotion recognition, where our model's accuracy exceeds even top proprictary models. We posit that the graph serves as a structural anchor, facilitating the explicit disentanglement of affective states (ToM), which effectively mitigates the common text-dominance" bias and thereby reducing hallucination. Finally, while model like Gemini 2.5 Pro (Gemini Team 2025) holds an edge in the overall score, our approach achieves this competitive performance at dramatically lower training cost. This demonstrates that integrating structured cognitive priors allows smaller models to punch above their weight, offering cessurce-efficieet alternative to the prevailing reliance massive parameter scaling.
Table 3: Performance comparison against open-source and proprietary baselines. We report results under both Direct and CoT prompting styles. The Overall M.J. Score is the primary metric, calculated weighted aggregate (0.6 R2 0.4 R2)) to balance the Reasoning Score (R,) and the final Response Score (Rp)). The Reasoning Breakdown columns provide granular metrics, where EM and EA denote emotion mention rate and emotion classification accuracy, respectively. R-Len indicates the average length of the generated response in words.
Prompi Style (Interence)
Reasoning
Response Rp
Ressoning Breakdown
0.0
x
(nord))
Our Modelr
SWM (Lluma3.1-8h)
CoT CoT
7,81
7.84
1.76
97.80
105.70
SWM (Quen--Aulio)
104.64
Tuned Baseline
Owen2-Andi-Corf
CoT
5.18
4.76
5.82
192.11
34,72
102.44
Baselines
Qwen Audio Chreet ml.
0023
Direct
2.70
2.20
3,46
14:20
8.00
28.99
Audio-AAddo
Direct
2.63
2,088
3.47
5:34
15.38
15.60
Quen2-Andi
C.T
2.39
1.96
3.04
6.11
17.50
21.19
Vastral
Lin
Direct
2.89
2,46
3.54
10.28
5.88
69.64
Voxtral Cir 2015
CoT
2.92
2.52
3.52
10.39
5,56
71.42
Propriencyy Medei
Go Open
2023
CoT
7.41
6.98
8.06
68.
45.16
105.23
Geman 255 Pio (S3min Team 2015
Col
8.12
X.02
8.28
82.47
$1.29
112.22
5 LIMITATIONS AND CONCLUSIONS
In this work, we proposed Speech World Model (SWM), which surpasses the current open-sourced speech language models in understanding and reasoning ability by modeling the dynamics of internal speech states. However, there are still some limitations. First, the current model includes only four modules, while more diverse states could enhance speech understanding and better capture complex speech dynamics. Second, the causal graph in SWM is predefined, limiting the model's ability to adapt to unseen dependencies. Future work could focus on learning adaptive casual structures to increase ftexibility. Third, while instruction tuning effectively integrates reasoning traces and responses, it relies heavily on the label generation pipeline, where inaccuracies can propagate to both the reasoning and response stages. Therefore, achieving efficient data annotation and generation for speech remains a crucial and collaborative challenge within the speech community.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
We thank Anastasis Stathopoulos, Alexander H. Liu and Alexei Baevski for their valuable advice and support throughout this work.
ETHICS STATEMENT
Our work adheres to the ICLR Code of Ethics. All experiments were conducted on publicly available datasets, ensuring data privacy and consent. Our research is intended for positive applications, and we strongly discourage any misuse of this technology for surveillance or manipulative purposes.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our results, we will make our source code, including implementation. training. and evaluation scripts, publicly available at httptts::/// /ttp::///ithue com/ wwrreka335/eureaa222 github. All datasets used are public and cited accordingly. The main experimental setup described in Sec. 4 For more detailed breakdown of the model architecture, hyperparameters, and implementation specifics, please refer to Appendix. A.4 A.9
REFERENCES
Dora Angelaki, Brandon Benson, Julius Benson, Daniel Birman, Niccolò Bonacchi, Kcénia Bougrova, Sebastian A Bruijns, Matteo Carandini, Joana A Catarino, et al. A brain-wide map of neural activity during complex behaviour. Nature, 645(8079):177-19, 2025.
Hanin Atwany, Abdul Waheed, Rita Singh, Monojit Choudhury, and Bhiksha Raj. Lost in transcrip- tion, found in distribution shift: Demystifying hallucination in speech foundation models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: ACL 2025. pp. 23181 13320233,,, Vienna, Austria, July 2025, Association for Computational Linguisties. ISBN 979-8 8-86166-57766.. doi: 10.18653/ v12202 ftto22...idins--aa acl. 190. URL http:tttta/aaaattooooooo org 2625. (oni25.fiinings.. findings-acl. 1190/
John Langshaw Austin. How to do things with words. William James Lectures. Oxford Uni- versity Press, 1962. URL http:h::/c/ccooarggoogoooee descholar...eeeoaa.............. http:/owww...bbbooeooooooo//o//d scholar scholar.google com http://ooooooctccctcco..o.aaoaaa
Chris L. Baker, Julian Jara-Ettinger, Rebecca Saxe, and Joshua B. Tenenbaum. Rational quantitative attribution beliefs, desires and percepts in human mentalizing. Nature Human Behaviour, 1, 2017. URL htt.tt/s//ppp..cmmmiissssoo..rrr www.:eeeeaaannnoo.eeee http:roor//oooo::3333333
Bruno Bara. Cognitive pragmatics the mental processes of communication. Intercultural Pragmatics, 8, 09 2011. doi: 10. 1115/ppr..2011111 2..00000 1.020.
Simon Baron-Cohen, Alan M. Leslie, and Uta Frith. Does the autistic child have a "the- ory of mind" ? Cognition, 21(1):37-46, 1985. ISSN 0010-0277. doi: htps://doi.or// (0.100660010022770559902022 pi1/000160277859002222
httpsttss://wwe wttss///wwwiiieeeereeeeetttttt.. com/ httpe//cccccmmcccee///ttticcllc
Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser. SLURP: A spoken
language understanding resource package. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7252-7262, Online, November 2020, Association for Computational Linguistics. doi: 10.18653/v120020.mmmll maimmmpmmain88..... URL https .emnnlp--mpp main. 588/
//aclanthologyyorgg////
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL htwpt:ttpt:ttttttm http:/tttsss//occcdiiiiiii ccpppps.cppppp files/paper/201 fflee/99550000007000441171111100
10

---

Published as a conference paper at ICLR 2026
Marcel Binz and Eric Schulz. Using cognitive psychology to understand gpt-3. Proceedings of the National Academy of Sciences, (206)))22118231222 2023, doi: 10. 10.10733pnas.221882211 URL
htttps://wwe pnas. wwwps/orgg//oo//////ee/ doi/abs/ /10 1073/ wwa.////......./.. 2218523120
Chandrakant Bothe, Cornelius Weber, Sven Magg, and Stefan Wermter. EDA: Enriching emotional dialogue acts using ensemble of neural annotators. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 620-627, Marseille, France, May 2020. European Language Resources Association. ISBN 2020 1rec-1.78/
URL httttts///c::ttttiii org/
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Hlerbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL htt.https:// httpps://procceeiinggs neur htt.://e666000111111111112222262
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jean- nette N. Chang, Sungbok Lee, and Shrikanth S. Narayanan. IEMOCAP: interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42(4))335-359, December 2008.
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, and Furu Wei. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE JSTSP, 2022.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023, https
//lmsys. org.blog.rg/blo//////o 200e:/22233000 1000003-0-3--ii--- 30-vicuna/
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919 2023.
Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo. Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen2-audio technical report. arXiv preprint 2rXiv:2407.1077 10759, 2024.
Kenneth James Williams Craik. The nature of explanation, volume 445. CUP Archive, 1967.
Wenqian Cui, Dianzhi Yu. Xiaoqi Jiao, Ziqiao Meng, Guangyan Zhang, Qichao Wang, Yiwen Guo, and Irwin King. Recent advances in speech language models: A survey, arXiv preprint arXiv:2410.03751, 2024.
Florian Eyben, Martin Wöllmer, and Bjöm Schuller. opensmile The munich versatile and fast open-source audio feature extractor. Proceedings of the ACM Multimedia Conference (MM), pp. 1459-1462, Florence, Italy, October 2010. ACM. ISBN 978-1 1-----1--59--6666
Angela Friederici. The brain basis of language processing: from structure to function. Physiological Reviews, 914)):357-1392, 2011 1. doi: 10.1152200 520pphyseee.0000062000
Karl Friston. A theory of cortical responses. Philosophical transactions of the Royal Society B: Biological sciences. 360( (80(115)):1158832) 2005.
Karl Friston, Noor Sajid, Daniel Quiroga-Martinee Martinez, Thomas Parr. Cathy Price. and Emily Holmes. Active inference: a process theory of language. Neuroscience Biobehavioral Reviews, 134: 1063-1067, 2021.
doi: 10.101 m00.moeee...2221.001111

---

Published as a conference paper at ICLR 2026
Quentin Garrido, Mahmoud Assran, Nicolas Ballas, Adrien Bardes, Laurent Najman, and Yann LeCun. Learning and leveraging world models in visual representation learning, 2024. URL
www./b/////eeo..o 2403. 00504
Google Gemini Team. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality. long context, and next generation agentic capabilities, 2025. URL https://arxiv org/abs/ 2507.06261
Anne-Lise Giraud and David Poeppel. Cortical oscillations and speech processing: emerging computational principles and operations. Nature Neuroscience, 15(4):511-517, 2012. doi: 10. 1038/nn.3063,
Yuan Gong, Alexander H. Liu, Hongyin Luo, Leonid Karlinsky, and James Glass. Joint audio and speech understanding. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 1-8, 2023. doi: LLL109//ARRUS7794222330339999
Yuan Gong, Hongyin Luo, Alexander H. Liu, Leonid Karlinsky, and James R. Glass. Listen, think, and understand. In The Twelfth International Conference on Learning Representations, 2024. URL
https://openrevieer http://pprrrrreeeee.eeeerrrrrrrp wtw.ne//foorrmmddddn/exxxx
David Ha and Jürgen Schmidhuber. World models. 2018. doi: 10.5281/ZENODO. URL
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference machine learning, pp. 2555-2565. PMLR, 2019.
Gregory Hickok and David Poeppel. The cortical organization of speech processing, Nature Reviews Neuroscience, 8(5):393 402, 2007. doi: 10.1038/nm2213.
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Alunn-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL
net wwret/oooomiiii http:/orrmmmmiieeeeeeeeeeee
Emre Kiciman, Robert Ness, Amit Sharma, and Chenhao Tan. Causal reasoning and large language models: Opening a new frontier for causality. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL httt::///ooeeneeeeeeeeeeee//rddd Certification.
Featured
Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro. Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities. arXiv preprint 2-Xiv:2402.018// 2024.
Zhifeng Kong, Arushi Goel, Joao Felipe Santos, Sreyan Ghosh, Rafael Valle, Wei Ping, and Bryan Catanzaro. Audio flamingo sound-cot technical report: Improving tf-thugghh reasoning in sound understanding, 2025. URL httl:/ttss///axiioor////b//
11818
Michal Kosinski. Theory of mind may have spontaneously emerged large language models, ArXiv, abs/2302.02083, 2023. URL htt.http:://
Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, and Samuel J. Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, 40:e253, 2017. doi: (0.10017/00140555X6000000
Yann LeCun. A Path Towards Autonomous Machine Intelligence Version. 2022.
Matthew K. Leonard and Edward F. Chang. superior temporal gyrus. Annual Review of Psychology. 73(1):79-102, 2022. doi: 10. we/.////////neeepp// wwy.annrrrr--------eee ennurev-psyche mesych-022321-11 -222321-035255
Speech computations of the human
Saoych-022221-035222
URL https://dol.org/// 1146/
Stephen C. Levinson. Pragmatics. Cambridge Textbooks in Linguistics. Cambridge University Press, 1983.
12

---

Published as a conference paper at ICLR 2026
Jessy Lin, Yuqing Du, Olivia Watkins, Danijar Hafner, Pieter Abbeel, Dan Klein, and Anca Dragan. Learning to model the world with language. In Forsy-first International Conference on Machine Learning, 2024. URL
Alexander H. Liu, Andy Ehrenberg, Andy Lo, Clément Denoix, Corentin Barreau, Guillaume Lample, Jcan-Malo Delignon, Khyathi Raghavi Chandu, Patrick von Platen, Pavankumar Reddy Muddireddy, Sanchit Gandhi, Soham Ghosh, Srijan Mishra, Thomas Foubert, Abhinav Rastogi, Adam Yang, Albert Q. Jiang, Alexandre Sablayrolles, Amélie Héliou, Amélie Martin, Anmol Agarwal, Antoine Roux, Arthur Darcet, Arthur Mensch, Baptiste Bout, Baptiste Rozière, Bau- douin De Monicault, Chris Bamford, Christian Wallenwein, Christophe Renaudin, Clémence Lanfranchi, Darius Dabert, Devendra Singh Chaplot, Devon Mizelle, Diego de las Casas, Elliot Chane-Sane, Emilien Fugier, Emma Bou Hanna, Gabrielle Berrada, Gauthier Delerce, Gauthier Guinet, Georgüi Novikov, Guillaume Martin, Himanshu Jaju, Jan Ludziejewski, Jason Rute, Jean-Hadrien Chabran, Jessica Chudnovsky, Joachim Studnia, Joep Barmentlo, Jonas Amar, Jos- selin Somerville Roberts, Julien Denize, Karan Saxena, Karmesh Yadav, Kartik Khandelwal, Kush Jain, Lélio Renard Lavaud, Léonard Blier, Lingxiao Zhao, Louis Martin, Lucile Saulnier, Luyu Gao, Marie Pellat, Mathilde Guillaumin, Mathis Felardos, Matthieu Dinot, Maxime Darrin, Maximilian Augustin, Mickaël Seznec, Neha Gupta, Nikhil Raghuraman, Olivier Duchenne, Patri- cia Wang. Patryk Saffer, Paul Jacob, Paul Wambergue, Paula Kurylowicz, Philomène Chagniot, Pierre Stock, Pravesh Agrawal, Rémi Delacourt, Romain Sauvestre. Roman Soletskyi. Sagar Vaze, Sandeep Subramanian, Saurabh Garg. Shashwat Dalal, Siddharth Gandhi, Sumukh Aithal, Szymon Antoniak, Teven Le Scao, Thibault Schueller, Thibaut Lavril, Thomas Robert, Thomas Wang, Timothée Lacroix, Tom Bewley, Valeriia Nemychnikova, Victor Paltz, Virgile Richard, Wen-Ding Li, William Marshall, Xuanyu Zhang, Yihan Wan, and Yunhao Tang. Voxtral, 2025. URL htttss////aaiii org/ abs/ 250. 13264
Ziyang Ma, Zhuo Chen, Yuping Wang, Eng Siong Chng, and Xie Chen. Audio-cot: Exploring chain-of-thought reasoning in large audio language model, 2025. URL/httss://arxi..oo / 2501 07246
Annamaria Mesaros, Toni Heittola, and Tuomas Virtanen. A scene classification, 2018. URL httph:tts//// //arxi wwt.:o/gaab//..o0g0000
dataset for urban acoustic
Meta. mettalllamalllnn--3-------- -8b-issrrctt 2024. URL https::/hugginggacce.cm com llama/
Llama- 10-18-8-Instucc Accessed: 2025-02-21.
Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. Voxceleb: large-scale speaker identifica- tion dataset. In Interspeech 20/7, pp. 2616-2620, 2017. doi: 10.2 1tt22144///nerppecch21117/..
OpenAl. Gpt4-o.
http://ppw....oo////ddd//e////aa 2024.
J Pearl. Causality: Models, Reasoning, and Inference, Second Edition. Cambridge University Press, London, 2009.
Judea Pearl. Causal inference. Causality: objectives and assessment, pp. 39-58, 2010.
Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada
Mihalcea. MELD: multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 527-536, Florence, Italy, July 2019. Association for Computational Linguisties. doi: 10.18653/v1/ P19-1050. httphhttts//aaaattttttte/ http://-ooog/0/00/////01 1050/ 1050/
org
David Premack and Guy Woodruff. Does the chimpanzee have theory of mind? Behavioral and Brain Sciences, 1(4):515 526, 1978. doi: 10.0017/50144555500076666
Nicolas Ruytenbeek. Indirect Speech Acts. Key Topics in Semanties and Pragmatics. Cambridge University Press, 2021.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020. URL htttps:/aaaxiv.og//abb////////// 01100
13

---

Published as a conference paper at ICLR 2026
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Common- sense reasoning about social interactions. Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIPP) pp. 4463 4473, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v 1Ttp:////D19--1111 URL htttps://aclanthiittet www.oo/gg...o/19194444444
Julian Schrittwiseer, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604 609, 2020.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, (09(5):612-) (1-566611-3366 634, 2021. doi: 10. N0.1109/PROCC.2211335558
Melanie Sclar, Jane Dwivedi- Yu, Maryam Fazel-Zarandi, Yulia Tsvetkov, Yonatan Bisk, Yejin Choi, and Asli Celikyilmaz. Explore theory of mind: program-guided adversarial data generation for theory of mind reasoning. In The Thirteenth International Conference on Learning Representations, 2025. URL / http:///ppppaeeeeee.eee/////onnn www..ne//forruii/d226rrriiim
John R. Searle. Speech Acts: An Essay in the Philosophy of Language. Cambridge University Press, 1969.
Sanjit A Seshia, Dorsa Sadigh, and S Shankar Sastry. Toward verified artificial intelligence. Commu- nications of the ACM, 65(7):46 05,66555222 2022.
C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27(3): 379-423, 1948. doi: 10. 10.10022).15338-73000 153--7305. 19487.95...99111333 1948.1001338.x
Parshin Shojace, Iman Mirzadeh, Keivan Alizadeh, Maxwell Horton, Samy Bengio, and Mehrdad Farajtabar. The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity. arXiv preprint arXiv:2506.. 06941, 2025.
Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational Linguistics, 26(3):
339-33 339-374, 2000. URL httphtss//////Lnntooooo..ooorttt 1rtp://tthhoooooo//0000000000000 3003/3
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. SALMONN: Towards generic hearing abilities for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL TtLLhtttss///pneeeett met//orrmm2d==14n//ea
Tomer Ullman and Joshua Tenenbaum. Bayesian models of conceptual development: Learning as building models of the world. Annual Review of Developmental Psychology, 2:533-558, 12 2020. doi: 10.1 (e//aannree-devssyeee 121318-084833
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du. Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022a. URL http:://ppppnreeeeeeeeeee/ooeooe gEZrGCozdgR
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoe V Le. and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022b. URL httpss////ppeeeeeeeeeeeeeerrerre _JQLMMSBB_
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of thouttt prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2444 24-837, 2022c.
14

---

Published as a conference paper at ICLR 2026
Jingran Xie, Shun Lei, Yue Yu, Yang Xiang, Hui Wang, Xixin Wu, and Zhiyong Wu. Leveraging chain of thought towards empathetic spoken dialogue without corresponding question-answering data, 2025a. URL
www.aaawwi.oooggaaaa 2501. 10937
Zhifei Xie, Mingbao Lin, Zihang Liu, Pengcheng Wu, Shuicheng Yan, and Chunyan Miao. Audio- reasoner: Improving reasoning capability in large audio language models, 2025b. URL https
//arxiv.org/aal laarg/abb/22553.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas I.. Griffiths, Yuan Cao, and Karthik R Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In seirth- Conference on Neural Information Processing Systems, 2023. URL /ttpss//
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Joseph E Gonzalez, and Ion Stoica. Judging llm-as- a-judge with mt-bench and chatbot arena. In Thirty-seventh Conference Neural Information Processing Systems, 2023.
Rolf A. Zwaan and Gabriel A Radvansky, Situation models in language comprehension and mem- ory. Psychological bulletin, 123 2: 2:162-85, 1998. URL https: //api. semanticscholas org/ CorpusID: 1063416
A APPENDIX
A.1 DISCUSSION ON TWO TYPES OF WORLD MODEL
As stated in Fig. 3 we consider both generative world models, such as those conceptualized by LeCun (LeCun (2022) ) and large language models as fundamentally being forward dynamic models. This perspective helps unify their roles and clarify the contribution of our proposed Causal Graph.
Generative World Model. This model is often associated with agent-based learning, aims to learn a transition function of the environment. This function, typically formalized as 8t+1 = [(s,, at). allows agent to predict future sensory inputs (the next state 5(++) 1) based on the current state (s;) and a chosen action (a,). The learned representation 8; is often a low-dimensional, latent state that captures the underlying dynamics of the world, enabling the agent simulate outcomes and plan.
Language World Model. This model can be viewed forward model operating in a symbolic space. Its core function is to predict the next token (W++1) given a history of previous tokens (w1..t). This autoregressive process, P(w++1|w.... Here, the "state" the context provided by the preceding text, and the "action" is the generation of the next token. This process unfolds trajectory through the space of language. Techniques like Chain-of- Thought (CoT) make this explicit, where the model generates a sequence of intermediate thoughts, effectively simulating trajectory through conceptual state space reach conclusion.
is conceptually parallel to predicting the next state.
Causal Graph. While the generative model learns implicit, sub-symbolic dynamics of an environ- ment, the language model executes explicit, symbolic reasoning. Recognizing their shared foundation forward models, our proposed Causal Graph offers third, structured approach to represent these dynamics. It distills the world's complex dynamics into an explicit graph of causal relationships between key cognitive states (WMA, ToM, etc.). This structured knowledge then provides powerful prior for the language model. Instead of navigating the vast space of possible text sequences based only on statistical patterns, the language model's forward-reasoning process guided and constrained by the causal dynamics provided by our graph, leading to more grounded and coherent outputs.
A.2 DETAILED GRAPH CALCULATIONS
A.2.1 CAUSAL GRAPH
Let x denote the transcription of given speech, a the acoustic feature (WayLM (Chen et al. 2022) ).
and the prosodic feature (Eyben al. 2010). We define text feature with distil- BERT encoder (Sanh
et al. 2020):
h
Frese (1:)
15

---

Published as a conference paper at ICLR 2026
An optional pre-fusion operator (attentoo//aleeddrasffoeee)) combines these with prosodic features:
= Φ (fece 2, a).
(10)
We consider four categorical latent states:
SWMA E A Com-1 STOM E A Cun-1
SSA E E Ac-1,
Sprag E A Ama-!
(11)
each represented by probability simplex.
For any node e with parents Pa(v), compute its state by node-specific neural map fo from the parents' distributions and available features:
S. ((S. uu Pa(u)), Ec),
(12)
where Eu denotes the feature bundle used by node e:
ΕWMA
(Iuests a), {TOM - 9; ESA = (g or hea),
Eprg (g Meet))
Concretely, each node produces logits and then a softmax:
S. softmax Vla(L (Su)uePa(oi)))
(13)
where ()x(.) is nonlinear transformation and W,, is task-specific projection.
A.2.2 RANDOM GRAPII
This architecture serves as ablation, testing whether a model with the flexibility to learn arbitrary dependencies from data can outperform our proposed causal structure.
Instead of uni-directional information flow, the Random Graph employs iterative refinement process. The computation unfolds over two main iterations. Let S be the probability distribution (state) of module E V at iteration t, and let H represent the fused input features derived from the speech-text pairs, i.c., H = g(X).
Initialization (t=0). At the initial step, the states of all peer modules are unknown and repre- sented by zero vectors.
( = 0, Vu E
(14)
First Iteration (t=1). Each module v computes an initial state estimate in parallel, conditioned on the input features H and the arrtiiniializzd states of all other modules. This step allows each module form preliminary prediction based solely on the primary input modalities.
s fo Hin. (0 E V, u # v)
(15)
Here, fu is the neural network parameterizing module The output (!1) is the softmax-normalized probability distribution over the module's labels.
State Refinement with Teacher Forcing. Before the second iteration, generate refined input state ((1) for each module u. As in the causal graph training (Eq. 4). this is stochastic mixture of the predicted states s(!)) and the ground-truth label yi., via teacher forcing. This allows the model to correct initial errors and propagate more accurate information in the next step.
Final Iteration (t=2). In the final step, each module computes its output by conditioning on the input H as well as the refined states g(1) from all other modules. This allows for a form of message-
passing where modules can adjust their predictions based on the initial estimates of their peers.
(((ma)) = (His 201) V, (
(16)
The final logits from S (final) for all modules are then used to compute the multi-task loss defined in Eq. By comparing this model with our structured causal graph, we can quantify the contribution of imposing theoretically-gronnde causal priors on the task of speech understanding.
16

---

Published as a conference paper at ICLR 2026
A.3 THEORETICAL ANALYSIS OF THE SPEECH WORLD MODEL
In this section, we provide a rigorous analysis of how the causal structure of the Speech World Model enhances training efficiency compared to unstructured baselines. We analyze this improvement through two theoretical lenses: (1) the optimization of gradient flow via structural priors, and (2) the pruning of the search space for downstream instruction tuning.
A.3.1 STRUCTURED PRIORS AND OPTIMIZED GRADIENT FLOW
The core advantage of Speech World Model lies modeling the joint distribution over latent variables as factorized probability, where each module is conditioned only on its causal parents (Pearl 2009):
P(Y XN II P(Y|
(17)
To rigorously quantify the efficiency gains afforded by this factorization, we analyze the gradient propagation dynamics, contrasting our structured approach with standard monolithic architecture.
Gradient Flow and Redundancy in Monolithic Models. Consider a monolithic baseline where the latent states ,,,,..., } are fully interconnected. The state of any module S; potentially Si+1...., SN. X). Let the total loss be L = E (S,), where A E {0, }} indicates label availability. The gradient with respect to the parameters 9r of a single module l expands via the chain rule into dense sum over all paths:
function of all other states: S, = f.. S
OC
N
N
0ʃ as; + 05. aS
A
(18)
00
as, 00, OS₁ 00
US. is non-trivial for all pairs. The model must implicitly In this fully connected setting, the term 80,
learn to "prune" irrelevant connections by driving the corresponding Jacobians 0 as, zero, which is
computationally inefficient and data- iata-intensive. We formally define this wasted optimization effort as learning redundancy:
Redundancy x
Σ I1, 1 C(0,) mo0e:)mmnommmmmm con-causal
(19)
where II; serves importance indicator. High redundancy implies the model expends gradient updates discovering structural independencies that could have been provided as priors.
Sparse Gradients in the Causal Model. Our Speech World Model eliminates this redundancy by construction. Since the state S, depends strictly on its parents Sra(i). the system's 's Jacobian matrix becomes sparse, satisfying as, if j & Pa(i). Consequently, the gradient calculation for parameters 0g simplifies dramatically: O₂ only influences the loss module j if l: an ancestor of (denoted k An(j)). The total gradient becomes:
Øs,
ac
OLk d 0A
Oh,
Σ
Xj 00
(20)
08g
jEDesc(k)
Unlike the monolithic case, each term J9, By explicitly setting non-causal dependencies to zero, the model precludes the need to discover relationships from scratch. This factorization focuses updates strictly along causally relevant path- ways, directly minimizing the redundancy term and accelerating convergence, which aligns with the established advantages of causality-inspired explainability (Schölkopf al. 202
ac; here is computed along unique, valid causal path.
A.3.2 CONSTRAINED SEARCH SPACE FOR INSTRUCTION TUNING
The causal graph provides a critical advantage by constraining the combinatorially large search space of latent cognitive states, y 3i x x YN, to valid, low-entropy subspace Sg C Y. This
17

---

Published as a conference paper at ICLR 2026
reduction in complexity is quantifiable by information entropy:
H(YIX,9) Σ H(Y.) x,9)
log Dil
(21)
This simplification transforms the learning task for the language model. Existing reasoning frame- works, such as Chain-of- Chain-of-Thought (Wei et al. 2022b) and Tree-of-Thoughts (Yao et al. 2023), require the the model to navigate the vast state space y. either linearly via tree-search, to approximate the joint distribution Th (Y: , X). This implies a computationally expensive search for valid reasoning chains from scratch. Our approach simplifies this to learning conditional policy & R X Yg). By explicitly providing the causally pruned state Yg, we introduce a control variate that reduces gradient variance, thereby accelerating convergence and improving sample efficiency.
The improvement in sample efficiency discussed above can be rigorously derived by analyzing the variance of the gradient estimator used in fine-tuning.
Instruction Tuning as Policy Gradient. Instruction tuning can be framed as learning policy (A(C)) that produces a response given a context C. The learning objective to maximize the expected log-likelihood of a gold-standard response A":
(())= E(C,At)= log (A*(C)))
(22)
This objective is typically optimized via stochastic gradient ascent. The efficiency of this learning process is highly dependent on the variance of the gradient estimator, 9 V log T C).. A lower variance estimator implies that fewer samples are needed to approximate the true gradient direction effectively.
Variance in Standard Chain-of- The LLM must generate both the intermediate reasoning steps (the chain of thought, Y) and the final answer (/t), so the action is A = (Y, Rt.. The gradient estimator is thus fCST = V log T (Y",, R2" (X). The variance of this estimator is high because for any given query X, there can
Chain-of-Thought. Thought. In standard CoT, the context is the input query, C = X.
multitude of valid reasoning paths (Y*} leading to the correct answer R" The model receives noisy learning signal as may be penalized for producing a perfectly valid reasoning chain that does not exactly match the single ground truth example.
Variance Reduction via Causal Conditioning. Our method fundamentally modifies the con- text C. By first executing the causal graph. we generate a structured, low-entropy state vector Yg P(Y|X,)) G). The language model is then conditioned on this richer context, C" = (X, Yg), simplifying its task generating only the final response R. The policy becomes T (R|X, Yg) and the gradient estimator is Óswm V log a [ Yg).
We can formally attribute the efficiency gain the law of total variance:
Var(ğeur) Eyg [Var(gswM|)g) Yg) Varya [E (g))
(23)
Our Variance
Variance Reduced
The gradient estimator in our model, OsWM, corresponds to the inner conditional expectation term. Since we condition on the explicit information provided by Yg, our observed variance is limited the first term. The second term, Vary, [[(oo)] Y.)]. represents the variance arising from the ambiguity of latent cognitive states. Because Yg is highly predictive of the final answer R" conditioning on effectively "explains away" this large source of variance. As hypothesized, the provided state Yg acts as control variate, significantly reducing the gradient variance and directly leading the greater sample efficiency observed in our experiments.
18

---

Published as a conference paper at ICLR 2026
A.4 DATA
A.4.1 DATASET STATISTIC
We construct our training dataset by aggregating four publicly available real-world datasets, as detailed in Table. These datasets provide a diverse range of labels, which we leverage to train different modules our Causal Graph. Specifically, we use the "Emotion" labels from MELD and IEMOCAP as ground truth for our Theory of Mind (ToM) module, and the "Intention," "Action," and "Scene" labels from SLURP for the Pragmatics (Prag) and World Model Activation (WMA) modules. For any remaining labels within given dataset that are not directly used, we employ label generation to create pscudo-labels for the corresponding modules. For evaluation, we adhere to the official test set splits provided by the MELD and SLURP datasets. For IEMOCAP and the VoxCeleb subset, randomly sample 10% the data to create our test sets, ensuring no overlap with the training data.
Table 4: Statistics for Real-world Dataset.
Dataset
Samples
Source
MELD (Poria al.
2019
13,000
13
Emotion
series Friends
(Busso
Dyadic actor dialogs Voice assistant Youtube
SLURP Basiunelli et
2020
72.000
58
Intention. Action, Scene Speaker identity
VosCelch (subses) Nagrani
2017
30,000
30
Total
~ 125,000
~ 113
A.4.2 LABEL GENERATION
The complete label space for each of the four modules in Causal Graph: World Model Activation (WMA), Theory of Mind (ToM), Speech Act (SA), and Pragmatics (Prag) is detailed in the box below.
Label Space for Causal Graph Modules
WMA: Alarm, Calendar, Contacts, Communication, SocialMedia, Email, Music, Radio, Podcasts, Au- diobooks, Games, Wcather, News, Traffic, Transport, Shopping, FoodAndDrink, Cooking, SmartHome, IoTDevices, Cleaning, Finance, Stock, Conversion, Knowledge, Math, Definition, Chitchat, Recom- mendation, GeneralControl.
ToM: Neutral, Joy, Sadness, Anger, Surprise, Fear, Disgust.
SA: Statement-noon-oiinionn Statement-opinion, Yes-No- Yes-No--uestion, Wh-Question, Acknowledge (Backchannel), Action-directive, Conventionall-losinng Appreciation or Assessment, Agree or Ac- cept, No-Answer, Yes-Answer, giggaalloo--uddsstannndnnnn Quotation, Affirmative non-yes answers, -oarrial-Quesioon, Rhetorical Backchannel, Hedge, Open-question, Thanking, Declarative Yes-No- Question, Reformulate, Conventional-oppennng,. Apology, Other Forward Function.
Prag: Request-Action, Request-Info Command/Directive. Complaint/Escalate, Thanks/Appreciation,,
Apology, Acknowledgemmee//Ageeene Aggeemmtt,, Social/Chitchat, Affer- Confirm/Disconfrmm, General- Control, InformationQuerry EntertainmennRequess, Task-Completion.
19

---

Published as a conference paper at ICLR 2026
The prompt for Stage of our label generation process (Label Completion) is follows:
System Prompt:
You speech understanding model task-oriented speech.
MODULE DEFINITIONS (each must choose exactly one label from the provided label space):
MM (Norld Model Activation): What concrete world/context the utterance lives in.
ToM (Theory of Mind): The speaker's discrete nental-affective stance.
SA (Speech Act): The communicative function of the utterance.
Prag (Pragnatic Intent): The downstream task intent or plan implied by the utterance, grounded in an action ontology.
REQUIREMENTS
1. You will receive:
LABEL SPACE: ISON listing the allowed labels for (WMA, TaH, SA, Prag).
INPUT: ASELTRANSCRIPT, KNOWNLLABELS MISSING_KOOULE (list modules predict).
For module
SIISSING.HOOULESSS select EXACTLY ONE label from LABEL_SPACE(modulee: Decide choice based given ASRSTRANSCRIPT and KNONNLLABELS
Select EXACTLY ONE label from [mnc_uul for MISSING.MODULES
Label strings must match EXACTLY (case-sensitive, cheracter-perfect)
When choosing between
multiple plausible options, select the contextually specific
OUTPUT FORMAT for each module in MISSING.MOOULS:: <hoduLLekameecOOdd1////dddd.dd
EXAMPLE:
Input:
"ASR_ TRANSCRIPT": "Oh ny Cod, he's lost it. He's totally lost it.",
KNOWN_LABELS"
"ToM": "sadness",
"SA": "Statement-noon-ppiioon""
MMOSEENG ["WMА", "Prag"]
Output: <#WbbCalendrr//MM>>
sprapiRequst-Info//raaa"
20

---

Published as a conference paper at ICLR 2026
The prompt for Stage of our label generation process (Reasoning and response generation) is as follows:
System Prompt:
speech understanding and reasoning model. complete
nodules, generate
detailed analysis and appropriate response.
MODULE DEFINITIONS (each must choose exactly one label from the provided label space):
MMA (World Model Activation): What concrete world/context the utterance lives in.
ToM (Theory of Mind): The speaker's discrete nental-affective stance.
SA (Speech Act): The communicative function of the utterance.
Prag (Pragnatic Intent): The downstream task intent or plan implied by the utterance, grounded in an action ontology.
REQUIREMENTS
1. You will receive:
INPUT: with ASRTTRANSRIPPT and COMPLETE_STATE containing all four module values (MM, TOM, SA Prag)
2. Generate two components:
<Analysis> A detailed causal reasoning chain (4-8 sentences) explaining:
How the world
(MNA) shapes situation
How the speaker's mental state (ToM) influences their commnicattinn
How these factors lead to the specific speech act (SA)
How the pragnatic intent (Prag) emerges from this combination
What this suggests for the
interaction step
<Response> A natural, contextually appropriate user-facing reply that:
Aligns
speaker's emotional state (ToM)
Serves the comnunicative function (SA) Addresses the pragmatic intent (Prag)
Fits the world context (WMA)
3. OUTPUT FORMAT:
<Analysis=YOUR <Response>YOUR NATURAL USER-FACING REPLYc/Response>
REASONING CHATN</Analysis>>
EXAMPLE:
Input:
"ASR_TRANE TRANSCRIPT": "Thanks so nuch helping ne find that document!",
"COMPLETE_STAN "
"WMA": "office-work",
"ToH": "grateful",
"SA": "Thanks/Aprecciation,,
"Prag": "Task-completion"
Output:
whnilysis-Within office work context (MMA), the speaker has just received assistance with document- related task. Their grateful emotional state (ToH) reflects successful problen resolution and appreciation for support received. This gratitude naturally manifests as thanks/appreciation speech act (SA), explicitly acknowledging help provided. The pragnatic intent task completion (Prag) indicates reached successful conclusion, with the speaker si signaling that their innedi ate need
the interaction has been satisfied. This combination suggests the interaction is winding down and the speaker is ready to el ther nove to closure or potentially engage in follow-up activities. weteiiiiies...saaaaaaae </Analysis> <Response>Yuu very come! glad could that document you. Is there anything else you need help with tedaes</Respense>
A.5 EXPERIMENT SETUP
A.5.1 CAUSAL GRAPH
For our casual graph baseline, we trained the model for 30 epochs with batch size 32. utilizes a two-layer gated fusion mechanism and was optimized using AdamW with learning rate of le-3 for all randomly initialized modules. consistent teacher- teacher-forcing probability of 0.3 was applied to the ToM, WMA, and SA modules. The entire training conducted single NVIDIA RTX A6000 GPU and completed in 2.07 hours. Under identical training settings, the Random Graph baseline required 10.39 hours complete the same hardware.
21

---

Published as a conference paper at ICLR 2026
Table Ablation study the Causal Graph under the fully-supervised setting. We investigate the impact of different fusion mechanisms (Gated, attention, transformer), teacher-forcing probabilities (p), and the removal of specific causal edges. The baseline model from the main text is highlighted.
Meihod
Ablation Seciing
Node Quality (Accuracy
Weighted Fl St,
Edge Casual Bffect
Frag
Ave.
Pally aupervieee
Baseline (Gated, p
72.2
Gased. -0.0
68.7
68.4
73.5 72.5
66.5
82.1
81.6
21.71
40.34
Gated, 0.5
72.8
65.6
81.5
24.12
Gased,
73.4
81.3
24.91
Gased, 1.0
68.9 68.7
740 73.2
66.7 -6644644
82.5 82.0
24,90
45.60
attention, 0.3
66.7
25.72
imnoformer. 0.3
68.1 66.5
69.3 60.6
58.4 53.3
76.3 72.3
27.40
41,40
Fully supervised
ToM SA
68.7
68.5
73.9
73.0
61.9
60.
818
81.2
22.97
43.0
WMA
68.9
68.6
71.9
65.3
82.6 $1.9
23.04
42.9
Table 6: Ablation study on the Causal Graph under the semi-supervised setting, We compare our primary latent module approach (highlighted) against an ablation where the input to the specific module is changed from the fused multi-modal feature g to only the text feature.
Method
Ablation Setting
Node Quality (Accuracy
Weighted Fl %.
Edge Casual Effect
WMA
ToM
SA
Prag
Ave. ACE (%. f)
Ave. ICS (%. f
Semi-supervised
Lateat Module: WMA
34.8
75.0
70.7
83.2
21.71
26.9
{sa feegt
32.7
77.3
70.5
82.7
21.69
Latent Module: ToM
69.1
43.3
21.98
{sA Racset
48.1
Latent Module: SA
69.3
77.0
34.4
82.5
21.65
29.3
Epray 9 haces
69.4
75.4
35.3
83.6
21.48
A.5.2 INSTRUCTION TUNING
For the instruction tuning stage, fine-tuned two separate language models handle the multi- modal and language-only settings, respectively.
In the language-only setting, instraction-tuned Llama-3. 1-8B using LoRA (Hu al. 2022) We applied LoRA adapters with a rank (r) of 64, an alpha (ax) of 16, and a dropout of 0.1. We trained for 20 epochs with cosine learning rate scheduler (peak LR: 10-5 50-5) and an effective batch size of 128 (per device batch size=8, gradient accumulation steps=4). This was performed on 4 NVIDIA A6000 GPUs, using float16 precision and gradient checkpointing with a sequence length of 1024, completing in 19 hours.
In the multi-modal setting, we fine-tuned Mween-Audioo7B-INne efficient approach using LoRA with rank (r) of 16 (per device batch size=2, gradient accumulation steps=2), an alpha (a) 32, and dropout rate 0.05. The model was trained for epochs using cosine learning rate scheduler (peak LR: 2 x 10 conducted on NVIDIA A6000 GPUs using bfloat16 mixed seeedd-recision with a maximum sequence length of 4096, completing in 24.6 hours.
We employed parameter-
10-4) 4) with effective batch size of 16. Training was
System Prompt (Qwen2-Audio2):
assistant consumes
Task: Given the audio and provided states, first produce concise but justified reasoning, then ureenfriendl reply.
Output format MUST be exactly: KEEAs ressomiigg-REESSONSSS]ss reply>
module labels:
NMA: World Hodel ietivation-what concrete world/centext the utterance lives in.
ToM: Theory of Mind-the speaker's 's discrete mental-affectiv stance.
SA: Speech Act-the communicative function of the utterance.
Prag: Pragmatic Intent-the downstrean task intent plan implied by utterance, grounded action ontology.
The causal dependencies between modules are: MMA >> SA, ToM >> SA, WMA -> Prag, ToM -> Preg, SA >> Prag.
22

---

Published as a conference paper at ICLR 2026
Average Cansal Entect (ACT) 199]
032
Culsal DLS
Daseltio Cuted. DLO
Goted
Trundoctact 0.3
Alterion 0.3
Intorvention Constatoncy Scoredcs
88
36
44
12
26
24-
40
iga0od,
38
Galed, 0.5
Cabed, ILB
WMA-SA
Prog
Prog
5A Prag
WMA Prag ToM PPag
Prag
Causal Edge
Causal Edge
Figure ACE and ICS of each casual edge for ablation study (fusion mechanisms and Seucher-forcing probabilities) on casual graph under fully-supervised setting.
28
Reastro riated,
Intorveation Conalstames Scare(13)
Toretnn caaned,
27
T:M SA
47
TaKC
Average Causal Entect (ACE)
46
45
24
83
82
WMA-5A
Prog
WMA-
ToM
WMA ToM
Edge
Edge
Figure 7: ACE and ICS of each casual edge for ablation study (removal of specific causal edge) casual graph under fully-supervised setting.
System Prompt (Llama3.1-8b)
an expert speech analysis.
The four speech modules
WHA: World Model Activation-wie concrete world/context the utterance lives in.
ToM: Theory of Mind-the the speaker's discrete mental -antal-ffeective stance.
Speech Act-the commuiicative function of utterance.
Prag: Pragnatic Intent-the downstrean task intent plan implied by the utterance, grounded in action ontology.
Analyze the given utterance and provide both your reasoning process and an appropriate response. The causal dependencies between modules Prag.
SA, SA, NMA Prag, ToM -> Prag,
Output format MUST be exactly: [REASoNING]<your reasoni Bornnnngg<RRPPREE [RESPONSE]syour reply>
A.6 ABLATIONS FOR CASUAL GRAPH
We conduct a series of ablation studies to validate the key design choices for our Causal Graph, with results shown in Table. and Table. and with detailed breakdown of edge causal effects shown in Fig. g
In the fully-supervised setting, our ablations confirm that gated fusion provides strong, balanced performance. While transformer-based fusion improved the Average Causal Effect (ACE), it came the cost of lower node classification accuracy. The model is also highly robust to the teacher- forcing probability, maintaining stable performance for values between 0.3 and 1.0. To validate the learned structure, we ablated causal edges; removing the ToM + SA link significantly impairs the SA module's accuracy, confirming this connection's importance.
In the semi suneriuuer setting, we tested a more challenging condition by changing the input to the children of the latent module from the fused multi- mootl feature only the text feature heast- This effectively cuts off their direct access to acoustic information, thereby forcing them to infer the necessary context from the output of their latent parent module. As shown in Table 6, the model's
23

---

Published as a conference paper at ICLR 2026
Intervention Constatoncy ScoredCS)NN
80
Average Cansal Entect (ACTS 194
Phat (VMA
Laben Madulic SA
Lenent Nedbie: TiM
Latent Musaae: WoRk
Latent Modale TYM
low g gat 15A)
Medill
30
15
20
WAA-
Prog
Prag
SA Prag
WMA
WMA Prog ToM -Pag
Prag
Causal Edge
Causal Edge
Figure ACE and ICS of each casual edge for ablation study on casual graph under semi-supervised setting.
overall performance remains remarkably stable, with node quality and ACE scores comparable to the baseline. This demonstrates the robustness of our semi-supervised framework, as the graph effectively propagate sufficient information through the remaining supervised pathways even when module partially disconnected from the acoustic modality.
A.7 MODEL ARCHITECTURE
Our model processes three inputs: 1) disstlbbertbbss--ucaedd model with a 2-layer Transformer on top for text; 2) CNN-LSTM adapter for pretrained WavLM features, producing 64-dim vector; and 3) 88-dim prosody feature vector.
Encoders.
Fusion Mechanisms. We explored three mechanisms to fuse the encoder outputs into a 256-dim representation. (I) Gated Fusion. Computes weighted sum of modality features, where weights are dynamically generated by small gating network. (2) Attention Fusion. Applies a single multi- head self-attention layer across modality features, which are treated as tokens in sequence. (3) Transformer Fusion. A deeper version of Attention Fusion that adds positional embeddings for each modality and processes them through multi-layer Transformer Encoder. Our baseline model uses Gated Fusion.
WMA Module. Temporal Self-Attention + MLP (hidden: 256). Output: 30 context classes.
ToM Module. Temporal Self-Attention + MLP (hidden: 128). Output: 7 emotion classes.
SA Module. Residual MIP (hidden: 256). Ousput: 24 speech act classes.
Prag Module. Residual MLP (hidden: 128). Output: pragmatic intent classes.
A.8 EVALUATION
A.8.1 CASUAL GRAPH EVALUATION
Average Causal Effect (ACE).
DrV Drv DTy(Po()do(Xx= x),2), P(Y12))
ACE(X + +Y ENND
(24)
|Cx
xeCx
Intervention Consistency Score (ICS).
ICS(X + EEND [I (Y = x)).) Pmnxx (rlz) + +)]
(25)
24

---

Published as a conference paper at ICLR 2026
Algorithm 1 Average Causal Effect (ACE) Computation
Require: Model M, DataLoader L, source module name X, target module name Y Ensure: Average Causal Effect score ACE(X + Y)
1: Initialize total_effect <- 0
2: Initialize samples f- 0
3: for batch in C do
4: P(Y 22 +- (z)y
Get baseline predictions for the hatch
5: for each sample in batch do
6:
cample_effeett +- 0
7:
Let Cx be the set of all classes for module X
8:
for each class x E Cx do
9:
P(Y|do (X x) zi) M. Mnttervene X, effect + Drv (P do z), z;), P(Y())))
Intervene and predict
10:
sample effect sample effect effect 12: end for
13:
total_effect + total_effect + (samplc.effcct//G, num_samples +- num-samples + 15: end for
14:
16:
end for
17: return total eelllefffeetttte nnum samples
Algorithm 2 Intervention Consistency Score (ICS) Computation
Require: Model M, DataLoader L, source module X, target module Y
Require: Positive intervention class x Cx. threshold r
Ensure: Intervention Consistency Score ICS(X- + Y)
1: Initialize consistent_com count +- 0
2: Initialize total_count 0
3: for batch in L do
4: P(Y (z) M (z)y
Get baseline predictions for the batch
5: for each sample in batch do
6:
Phuse t- max P(Y z;)
7:
PdYldo(X xt Pintervened f- max P(Y do = x,,)),))))))))))))))))))))))xx
M Minteveenee X, x,
Intervene and predict
8:
9:
if Pintervened Peuse then
10:
consistent.count + consistent.come +
11:
end if
12: total count <- total count +
13: end for
14:
end for
return consisten_ccount/toaal.ommm
A.8.2 RANDOM GRAPH EVALUATION
To quantify the information flow between any two modules in the random graph, we measure the influence a source module (Ms) has target module (Mr). This is achieved by comparing the output of My under two conditions for each batch of data:
1. Baseline (Without Source): We compute the target module's 's output logits. denoted L wiithout by replacing the input from Mg with a zero vector. This represents the hehavior of My without direct information from Mg.
With Source Influence: We compute the target module's output logits, Liith by providing it with the actual probability output from Ms. This represents the behavior of My when influenced by Mg.
The influence is then calculated by comparing the probability distributions derived from these two sets of logits, pwithout softmax BaaiaaxxTTiiitaa /.whhhout) and pwith softmax (T,).")), using three metrics:
25

---

Published as a conference paper at ICLR 2026
KL Divergence (DKL): Measures how much the output probability distribution of the target module changes when the source module's s information is added. larger value indicates a greater change.
DKL DRz (Pwith ||
pwithout)
(26)
Prediction Change Rate (Aprea): The fraction of samples in batch for which the predicted class (the one with the highest probability) changes. I(-) is the indicator function.
A prod [arg max (L7 + arg max (iithou))
(27)
Confidence Change (Acont): The absolute difference in the average prediction confidence (the value of the max probability) between the two conditions.
A conf [[[max(Pw)) (Pwih)) - R[[mxx (Pwithnun)
(28)
Final Influence Score MT the simple average of these three normalized metrics, providing single value to represent the strength of the connection.
The final Information Flow Influence Score (IFS) for the connection Ms +
IFS - MT) (DKr. + Apred + Acont)
(29)
A.9 SPEECH UNDERSTANDING AND REASONING EVALUATION
The prompt used to have GPT-4o score the reasoning and responses generated by models is as follows:
System Prompt:
You are strict model-as-judge. You will assign two scores 6-10 scale for each sample: one for the model's reasoning ("analysis") and one for the model's 's response ("response"). transcript, sample's analysis and response.
You are given:
with
(WM, TaM, SA, Prag), plus
You must follow the rubric and output strict JSON only. No extra explanations, no code blocks, no markdown | only valid JSON.
[Scoring Rubric]
(1) reasoning score (e-10) | for analysis:
Completeness: does it cover and explain all four dimensions (MHA, TOM, SA, Prag) and their relations? Does reference key ASR info?
Accuracy:
align with STATE labels exactly? inventing changing labels.
Causal logic: does it present clear, coherent causal chain from ASR state judgnent pragnatic intent?
Guide:
causal and clear. 8: mostly correct, minor gaes generalization.
10: all four dimensions, fully
6: partially correct, superficial, 1-2 inaccuracies.
several inaccuracies or weak logic, clear mismatches.
2: mostly irrelevant confused reasoning.
0: missing, fabricated, totally wrong.
(2) response score (6-10)
respanse:
Logicality and appropriateness: does it respond reasonably to the ASR context (e.g. anger, wh wh-question)? Polite, safe wording?
Match with STATESTATE: does it reflect WMA, TOM, SA, Prag consistently? No contradictions.
Guide:
10: highly consistent, natural, well-suited strategy.
8: generally consistent, minor flaws or templated.
partially consistent, weak key info. multiple mismatches. awkward.
2: mostly irrelevant or inappropriate.
8: nonsensical, unsafe, or severely inconsistent.
EXAMPLE:
[Outout Format strict only]
[
"reasoning.scors" score": <float 0-10>,
"response_score" <float
26

---

Published as a conference paper at ICLR 2026
A.10 CASE STUDIES AND ANALYSIS
To provide concrete intuitions regarding the effectiveness of our approach, we present case studies comparing our Speech World Model against the Voxtral baseline (Liu et al. 2025), which achieved the best overall performance among open-source methods in our experiments, surpassing Qwen-Audio series. We focus on challenging scenarios, such sarcasm, indirect commands, and emergency detection, where acoustic cues contradict or enrich the surface text. To facilitate the analysis of the generated reasoning chain, we color-code the inferred cognitive states corresponding to the causal graph: WMA, ToM, SA, and Prag.
Case 1: Sarcasm Detection
Input: "Oh, brilliant. That is just what I needed right now."
Baseline (Voxtral):
The speaker starts with positive reaction to something, using the word "wonderful" and "just brilliant". The speaker seems be expressing satisfaction or appreciation for something they've just heard or experienced. The speaker then states that this is "just what needed right now". The speaker expressing gratitude and satisfaction, and they feel that this experience or information has been particularly helpful or timely for them. So, the speaker is likely expressing their appreciation for something they' 've just encountered. [Analysis: Failed to detect sarcasm, interprets text literally.] Speech World Model (Ours):
Predicted Graph:
[WMA: Traffic], [ToM: Anger], [SA: Statement-opinion [Prag: Complaint/EEsclatte]
Reasoning:
The audio is situated in a traffic or transit context, likely involving an unexpected delay or obstruction. The speaker's angry and frustrated tone, characterized by heavy emphasis and a sigh, contradicts the positive linguistic content. While on the surface the utterance "Oh, brilliant" positive statement-opinion, the causal conflict between the text and the affective state indicates the underlying intent is a complaint or escalation. This sarcastic interpretation implies the speaker is actually expressing strong dissatisfaction with the situation.
Case 2: Indirect Command
Input: "It's getting a bit dark in here."
Baseline (Voxtral):
The audio begins with slight pause, and then the speaker says "It is getting a bit dark in here." This statement suggests that the speaker is noticing a change The speaker could be in a room or an environment where the light is decreasing, possibly due the time of day or the presence a window that letting in less light. The speaker might be preparing turn [Analysis: Fails to resolve the indirect speech act, treating the command as a passive environmental observation.]
the lighting conditions.
light or asking someone else do
Speech World Model (Ours):
Predicted Graph:
[HMА: SmartHome], [ToM: Neutral], [SA: Statement-opinionn)..P Command/Directive]
Reasoning:
The audio belongs to a SmartHome context, specifically regarding environmental control. The speaker maintains neutral yet authoritative tone, suggesting a routine expectation of service rather than casual observation. While on the surface they offer a statement regarding the light level, the functional context transforms this observation into an underlying command or directive. This implies a tacit request for the system to take immediate action, specifically to turn on the lights to remedy the darkness.
27

---

Published as a conference paper at ICLR 2026
Case 3: Urgency//merrenncy
Input: "Where is the nearest exit?"
Baseline (Voxtral):
The speaker is in location where they need to find an exit quickly. The speaker is likely in a building, a mall, or another enclosed space where they need to find a way out. They are asking for the nearest exit to get out of the current location. The tone is urgent, indicating that the speaker might be in situation where they need to leave immediately. Given this context, the speaker likely asking for directions information about the nearest exit to ensure they can leave the area safely.
[Analysis: Correctly identifies urgency but limits intent "seeking directions", inrectiioss,falliinnnn failing to escalate to emergency assistance.]
Speech World Model (Ours):
Predicted Graph:
[WMA: GemeralControll, [ToM: Fearl, [SA: Wh-Question], [Prag: Request-Action]
Reasoning:
The audio occurs transport or general safety context, possibly during critical incident. The speaker's & fearful and panicked tone, marked by fast tempo and breathiness, conveys sense of immediate danger. While on the surface this a standard Wh-question asking for location information, the extreme emotional state elevates the underlying intent to request for urgent action or assistance. This implies the user requires immediate guidance to safety and reassurance, rather than generic informational response.
A.11 THE USE OF LARGE LANGUAGE MODELS
During the writing this paper, we leveraged GPT-5 to aid in polishing the writing. Its use was exclusively limited to improving grammar, phrasing, and overall readability. The core scientific contributions, experimental results, and intellectual content are entirely our own.
28


Reviews:
[{"review_id": "mVg0bOtqFX", "reviewer": "Reviewer_FHEH", "summary": "This paper introduces speech world model, a structured formalism that factorizes speech understanding into four modules that communicate through a causal graph. Instruction-tuned languages models can provide better reasoning and response guided by posterior traces from the proposed formalism. The authors show empirical validation for their approach on several speech datasets.", "questions": "Writing suggestions: \n- On Page 7, Sec. 5 and subsections 5.1 and 5.11 are next to each other without any text. And 5.11 begins with bullet points. These patterns are not the best practices for scholarly writing. Consider adding guiding text/sentences for clarity.\n- Also on Page 7, why citing \"Tree of Thoughts\" when you only talk about CoT (Line 348). I suggest that the authors carefully verify their citations in this work.\n- Currently, the paper leaves little room for discussion. I would recommend moving a bit of the technical details to the appendix and include more discussions/interpretations/limitations at the end. As of now it feels that the paper ends abruptly.\n- Some of the figures/tables could benefit from more detailed captions. For example, Table 3 is supposed to be a main result table, but it only has a 1-sentence caption. It would be better if it briefly explains/reminds the reader of what some of the important columns mean.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This paper has an interesting framing. It recognizes that speech understanding with language model still has large room for improvement, and it proposes to use a structured and theoretically grounded framework to improve speech understanding. The notion of world models makes sense.\n- The authors experimentally show that the causal graph matters (Table 1 and 2) and SMW significantly improves the performance of LLMs (Table 3). Experiments are done with several different datasets, providing evidence for generality of the approach.\n- This paper tackles a problem that has considerable real-world relevance.", "weaknesses": "- The literature review in this paper is highly unsatisfactory. Theory of Mind, Speech Act, and Pragmatic Intent all have long traditions in cognitive science and linguistics. Yet the authors only primarily cite one work for each, and the selection is a bit ad-hoc. Why not cite the classic papers? Also, for example, Theory of Mind with LLMs alone is a popular topic in the past few years, but that is not reflected in the paper. Overall, I think the authors should conduct a more thorough review of related work; otherwise I do not think this is at publication standard.\n- Similarly, for concepts around world models and cognitive science, I recommend checking out [1] and references therein. It offers a perspective on cognitive world models that supports what this work aims to do, so engaging with that line of research would support this work.\n- How did the authors decide on the combination of WMA, ToM, SA, Prag? Why not more or less? Why these four in particular? These choices need more arguments/justifications. I think there are ways to justify them---currently just saying \"We define four key modules...\" is not enough.\n- I would appreciate more concrete examples on why the proposed approach works (which can go into the appendix). Figure 1 and 2 are helpful, but I think a reader would like to get more intuitions and concrete case study/analysis.\n\n[1] Ullman, T. D., & Tenenbaum, J. B. (2020). Bayesian models of conceptual development: Learning as building models of the world. Annual Review of Developmental Psychology, 2(1), 533-558.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fv9PRTDpnz", "reviewer": "Reviewer_xGU2", "summary": "This paper proposes a Speech World Model (SWM) that explicitly models causal relationships among four key components of speech understanding: World Model Activation (WMA), Theory of Mind (ToM), Speech Act (SA), and Pragmatic Intent (Prag). The model builds a causal graph connecting these modules and uses the inferred latent cognitive states to guide a language world model for reasoning and response generation. To handle incomplete supervision, the framework employs semi-supervised learning, allowing unlabeled nodes to benefit from gradients propagated through causal dependencies. Experimental results show that the proposed causal structure improves node inference accuracy under sparse labels, enhances reasoning performance compared with open-source baselines, and achieves these results with significantly lower training cost.", "questions": "NA", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. While prior work has explored explicit modeling of context or theory of mind, this paper uniquely integrates them into a causally grounded framework that links perceptual and pragmatic dimensions of speech. The structured reasoning graph offers interpretability and aligns with cognitive principles of human speech understanding.\n\n2. The causal dependencies enable effective gradient flow even when some states are unlabeled, improving learning efficiency and robustness compared with fully independent module training.", "weaknesses": "1. It would strengthen the paper to include experiments that isolate the benefit of the causal graph itself—for example, by training all four modules independently or removing all edges—and then feeding these disconnected representations into the language model. This would help quantify how much causal reasoning contributes to downstream reasoning and response quality.\n\n2. The current graph structure is manually predefined based on cognitive intuition rather than learned from data. While this design is interpretable, it may limit adaptability to new domains or cultural variations in communicative patterns. Exploring learnable or adaptive causal graphs could make the model more generalizable.\n\n3. The label completion and reasoning–response generation pipeline relies heavily on pseudo-labels generated by large language models (Vicuna, GPT-4o). An analysis of label accuracy or a comparison with human annotations would clarify how this synthetic supervision affects model reliability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ur0jn8PyDG", "reviewer": "Reviewer_eLqM", "summary": "The paper introduces a Speech World Model that enables explicit reasoning over speech states. It defines a four-node  DAG (WMA, ToM, SA, Prag) whose topology is fixed from cognitive science; the mechanisms are learned via neural classifiers over speech/text features and parent states The graph’s inferred states are (i) serialized to instruction-tune a text LLM and (ii) fed directly to a multimodal SLM. The proposed learning method operates in both a semi-supervised and a fully supervised manner, improving data efficiency where annotations across modules are sparse. Evaluation is split into two axes: a) Quality of the estimated graph, and b) Speech understanding and reasoning.", "questions": "- Equation (4) seems to be doing what I would call “stochastic teacher forcing”. I am a bit lost by the motivation and the need for this. Can you elaborate?\n- Why only one judge for LLM-as-judge? Please add more judges and report win-rates and agreements, given known style biases of LLM judges [1].\n- Can you also report results on true labels only? The current setup may conflate the performance of the imputation and the overall performance as some labels are generated while others aren’t.\n- Please evaluate your imputation by masking gold labels at random, using your imputation procedure to recover them and evaluating accuracy/macro-F1.\n- Check multimodal causal-faithfulness: flip a single parent state in the graph serialization passed to the multimodal generator; extract the child state from the response and check movement in the predicted direction. Include negative controls by flipping a non-parent.\n- The label imputation paragraph should be explained in more detail in the main text.\n- Claims on learning speed should move to the main text with seeded means+std and wall-clock details.\n- Clarify whether causal claims target the data-generating process or the learned representation.\n\nMinor:\n- Figure 2: Instcution->Instruction\n- The type of the causal variables is not explicitly defined in the main text and has to be inferred from the rest of the text.\n- Make tables self-contained: state which datasets feed which rows/metrics.\n- L416-417: Can you elaborate on the instability of Random Graph?\n\n[1] Benjamin Feuer, Micah Goldblum, Teresa Datta, Sanjana Nambiar, Raz Besaleli, Samuel Dooley, Max Cembalest, & John P Dickerson (2025). Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking. In The Thirteenth International Conference on Learning Representations.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Strengths:\n- This paper proposes a causality-inspired method where the underlying causal graph is built from expert knowledge. This is an approach that more papers could follow.\n- The paper is terse but well-written.\n- Explicit, interpretable reasoning over speech states is novel and useful.\n- Solid conceptual grounding; edge-level analysis and ablations are appreciated.", "weaknesses": "- Label generation with no human oversight casts uncertainty on the quality of the labels.\n- The label-imputation pipeline is under-specified in the main text.\n- Section 3 is particularly hard to follow; a running example would help.\n- Section 4 feels disconnected: redundancy/last-paragraph argument is unclear and Eq. (9) (CMC) belongs earlier.\n- Evaluation relies on a single LLM judge.\n- Results often mix gold and imputed labels, conflating model learning with label quality.\n- Causal scope is unclear: interventions (and ACE/ICS) are performed on model-internal states.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)