From Text to Talk: Audio-Language Model Needs Non-Autoregressive Joint Training
Authors: Tianqiao Liu
Xueyi Li
Hao Wang
Haoxuan Li
Zhichao Chen
Weiqi Luo
Zitao Liu
Venue: ICLR 2026 Poster (2026)
Keywords: Large Multimodal Models
Multi-token Prediction
Non-Autoregressive Learning
Primary area: foundation or frontier models, including LLMs

Abstract: Recent advances in large language models (LLMs) have attracted significant interest in extending their capabilities to multimodal scenarios, particularly for speech-to-speech (S2S) conversational systems. However, existing multimodal models handling interleaved audio and text rely on autoregressive (AR) methods, overlooking that text depends on target-target relations whereas audio depends mainly on source-target relations. In this work, we propose Text-to-Talk (TtT), a unified audio-text framework that integrates AR text generation with non-autoregressive (NAR) audio diffusion in a single Transformer. By leveraging the any-order AR property of absorbing discrete diffusion, our approach provides a unified training objective for text and audio. To support this hybrid generation paradigm, we design a modality-aware attention mechanism that enforces causal decoding for text while allowing bidirectional modeling within audio spans, and further introduce three training strategies that reduce train-test discrepancies. During inference, TtT employs block-wise diffusion to synthesize audio in parallel while flexibly handling variable-length outputs. Comprehensive experiments on audio question answering (Audio-QA), automatic speech recognition (ASR), automated audio caption (AAC) and S2S benchmarks show that TtT consistently surpasses strong AR and NAR baselines, with additional ablation and training-strategy analyses confirming the contribution of each component.

Published as a conference paper at ICLR 2026
FROM TEXT TO TALK: AUDIO-LANGUAGE MODEL NEEDS NON-AUTOREGRESSIVE JOINT TRAINING
Tianqiao Liu Xueyi Li Hao Wang', Haoxuan Li', Zhichao Chen', Weiqi Luo', , ZZiao Liu
"Guangdong Institute of Smart Education, Jinan University
2ALL Education Group, Peking University
liutianqia aol @t @tal. com, wixuey watueeelestu22211 www.
ABSTRACT
Recent advances in large language models (LLMs) have attracted significant inter- est extending their capabilities to multimodal scenarios, particularly for speech- to-speech (S2S) conversational systems. However, existing multimodal models handling interleaved audio and text rely on autoregressive (AR) methods, over- looking that text depends on target-target relations whereas audio depends mainly on source-target relations. In this work, we propose Text-to-Talk (TTT), a unified audio-text framework that integrates AR text generation with non-autoregressive (NAR) audio diffusion single Transformer. By leveraging the any-order AR property of absorbing discrete diffusion, our approach provides unified training objective for text and audio. To support this hybrid generation paradigm, we de- sign a modality-aware attention mechanism that enforces causal decoding for text while allowing bidirectional modeling within audio spans, and further introduce three training strategies that reduce train-test discrepancies. During inference, TrT employs block-wise diffusion to synthesize audio in parallel while flexibly handling variable-length outputs. Comprehensive experiments on audio question answering (Audio- (AA, automatic speech recognition (ASR), automated audio caption (AAC) and S2S benchmarks show that TtT consistently surpasses strong AR and NAR baselines, with additional ablation and training- -trratggy analyses confirming the contribution of each component. Our code and data are publicly available htthtps:: / com/ aide//T44dd///
INTRODUCTION
The recent success of large language models (LLMs) has catalyzed a paradigm shift towards general- purpose multimodal large language models (MLLMs) capable of processing and generating infor- mation across diverse modalities (Xu et al. 2025 Team et al. 2023 Li et al. 2021). Among these, S2S conversational systems have emerged interaction. Conventional systems typically decompose this problem into cascaded pipeline of automatic speech recognition (ASR), LLM-driven response generation, and text-to-speech (TTS) synthesis. While effective to degree, this modular design introduces significant latency accumu- lation and error propagation between modules, hindering naturalness and real-world applicability. In response, recent end-to-end approaches like Moshi (Défossez et al. 2024)), GLM-4-Voice (Zeng) et al. 2024). and VITA-Audio (Long et al., (0255) have sought to unify speech understanding and generation within single model. These models are typically trained through multi-stage pipelines that involve text-to-audio tokenizer training, interleaved data construction, text-audio alignment and
pivotal component in facilitating natural human-AI
task-oriented supervised fine-tuning (Huang al. 2025 Li et al. 2025al Ding al. 2025 Chu
et al. 2024 Kang et al.] 2022). As shown in Figure 1 these methods aim to generate interleaved text and speech tokens in autoregressive (AR) manner, which are then decoded into continuous audio waveforms by separate neural codec or diffusion-based decoder (Mehta et al. 2025 Kong al. 2020). However, this emerging paradigm faces fundamental challenge. As illustrated in Fig-
ure we identify a fundamental mismatch in prevailing approaches that employ single language model autoregressively generate both text and audio tokens while using identical AR training ob-
jectives across both modalities (Zeng et al. 2024
Xie & Wu 2024 Borsos et al. 2023
Dang et al.)
"Corresponding author: Xueyi Li.

---

Published as a conference paper at ICLR 2026
Inconsistent size of last audio span provious ones
Text Token
Audio Taken
Target-target
Consistent audio
Special Token
dependency
span size
dependency,
End of Sequence
Autoregressive MLLM
System Prompt
Audio Query
Figure 1:
Distinct dependency structures for text and audio, and the resulting variable-length audio spans caused by disparate tokenization rates.
Rubenstein et al. 2023) Such uniform objective overlooks a critical distinction in their underlying generative processes. Text generation inherently follows a sequential causal structure characterized by strong target-target dependencies (Box et al. 2015), where each token explicitly conditions on previously generated tokens. Consequently, an incorrect token prediction can propa- gate and introduce subsequent errors due to the exposure bias inherent in AR models (Ranzato al., 2015). In contrast, audio token generation is predominantly driven by source-target dependencies (Ren et al. 2020), where audio output primarily conditions the source text rather than the pre- ceding audio tokens. Specifically, within the current non-autoregressive (NAR) span, audio token generation should remain faithful to the source text when previous audio tokens incor- rectly predicted. Applying a purely AR objective to audio generation thus introduces unnecessary sequential constraints, leading suboptimal training dynamics and magnifying error propagation.
2024
This problem can be substantially alleviated by adopting a NAR generation strategy, which aligns better with the source-dependent nature of audio modeling. Recently, discrete diffusion has emerged as a compelling alternative to AR for discrete sequence modeling (Yu et al. 2025 Gong et al., 2025 Austin et al., 2021 Sahoo et al.) 2024). Beyond empirical gains, recent theory shows that absorbing discrete diffusion can be interpreted as modeling the conditional distributions of clean tokens and admits a tight connection
any-order AR objectives (Ou et al., (024). This raises
research question: Can unified model combining AR and NAR generation miligate the mismatch introduced by applying a uniform AR objective to both text and audio?
In this paper, we introduce Text-to-Tall (TtT), unified audio-text MILM that integrates AR text generation with NAR audio diffusion within a single Transformer initialized from a pretrained LLM. Text segments are trained with a standard AR cross-entropy objective, while audio segments modeled via an NAR discrete diffusion process. During inference, the model dynamically switches between AR and NAR decoding strategies based special control tokens. In summary, work makes the following contributions:
We identify and formalize the fundamental asymmetry in dependency structures between text and audio modalities. Leveraging the any-order AR nature of absorbing discrete diffusion, we establish unified theoretical framework that proves our joint training objective provides an upper bound on the negative log-likehle likelihood of the desired joint distribution.
We propose a hybrid AR-NAR MLLM that seamlessly integrates AR text generation with discrete diffusion-bante based audio synthesis within single Transformer initialized from a pretrained LLM. Our design preserves the reasoning and instruction-follovinn capabilities of the base LLM while enabling efficient parallel audio generation.
We introduce three principled training strategies to address the inherent train-test discrepancies hybrid AR-NAR learning, enabling stable training and robust content-aware variable-length generation that bridges the gap between training and inference conditions.
Extensive experiments across Audio-QA: ASR, AAC and S2S benchmark demonstrate that TtT consistently outperforms strong AR and NAR baselines, highlighting the advantage of the hybrid AR-NAR framework.
2

---

Published as a conference paper at ICLR 2026
2 PRELIMINARY AND NOTATION
In this section, we establish the basic notation for interleaved audio-text sequences and provide brief overviews of the two core generative paradigms employed our framework: AR modeling and absorbing discrete diffusion. These form the theoretical foundation of our proposed method in Section
Tokens, Vocabulary, and Interleaved Layout We consider interleaved discrete text-audio se- quences of length L: = (a'..... ,..,,,,.. .²) with unified discrete vocabulary V Viest Vandio S, where x' E V denotes the I-th token in the interleaved sequence I, Vixt and Vandio are the dis- crete vocabularies for text tokens and audio tokens, respectively, and contains special tokens such as (SOA) (start of audio), (EOA) (end of audio), (EOS) (end of sequence) and the ab- sorbing mask token [M]. A sequence 2 is structured as series of alternating text and audio spans: = (Ti.A,,-... TM, (EOS)), where M denotes the number of text-audio span pairs, m {l...., M) indexes the mn-h pair, and the total length satisfies L =EM- (ITml [mmm] +1, with the spans defined as follows:
Tm (Im,1,--. :.m.1.....,m..TTm.l... mm.l).))) (Vuex U {(SOA))) are text tokens.
Am (am.l Am.)Am)) E (Vaudio U (((O))))) are quantized audio tokens.
Let fo : VL + Lxd be single Transformer. We use a shared output head W [gdx[M] (typically tied with input embeddings) to produce per-position logits over the entire vocabulary V.
AR Modeling AR models factorize the joint probability of sequence product of conditional probabilities, based on the chain rule: p(x) I (x=), where < (x: x' This imposes sequential, causal structure on the generation process. For detailed discussion, please refer to Appendix A
(x',..., x²) into
Absorbing Discrete Diffusion Absorbing discrete diffusion models are NAR paradigm for quence generation. They consist of a forward process that corrupts a clean sequence by gradually replacing tokens with special absorbing mask state [M]. and learned reverse process that aims to recover the original sequence from the corrupted input. A key insight from (Ou et al. 2022] is that the learning objective simplifies modeling time-independent conditional probability of the clean data. Specifically, the score for unmasking token at a corrupted position is given by:
Pel ...... Pel Pel.... [M.....
Po(u Iivis)
e a(t)
clean conditional probubility
concrete score
time scalar
where fI....,I) L} indexes the corrupted position with xf [M], and & denotes the same sequence except that the i-th position is set to (i.e., .² = v). Here, pr(+) denotes the marginal distribution of the forward corruption process at time I, po(-) the clean data distribution, Tvis denotes the visible (unmasked) tokens in the corrupted sequence (with their positions), and a(t) the cumulative corruption schedule. This denoising formulation is precisely the objective of an any-order AR model (AO-ARM), predicting a token given arbitrary context of unmasked tokens. As demonstrated by (Ou et al. 2024), the diffusion training objective mathematically equivalent to this A0-ARM objective, which averages the prediction loss all possible permutations of the sequence: (zo = ExsU. Σ log qe( ge(200 z =)(())) where To = (a)..... =b) is the clean token sequence. is drawn from Uz. the uniform distribution
all permutations of 7 (1-
{1...., I.). π (I) denotes the index at the 1-th position of the permutation, . 1) = 1)} denotes the set of indices revealed before step and qo I) is the AO-ARM parameterized by 0. Therefore, training an absorbing discrete diffusion model is equivalent to training a powerful ensemble of AR models that can operate in any order. More details are provided in Appendix A.3.2
JOINT TEXT-AR & AUDIO-NAR MODEL
In this section, we introduce our proposed model, which integrates AR generation for text and discrete diffusion for audio within single unified Transformer architecture.
3

---

Published as a conference paper at ICLR 2026
3.1 AR MODELING FOR TEXT
We model text generation using a fixed, canonical AR order. Let Trext denote the natural left- to-right permutation over all text token positions in the sequence, namely Texx < Texu( (2) < < [T<<((<<M )) where TeM = - Im collects all text token indices. At the span level, the probability of generating the m-th text span Tm (fm.1....,,m,, (T.IT...) conditioned on IT,=2 Tmal pe(tm.y/T<ms [T<m= Acms km, where tm.sj (mmj............- is the prefix of text tokens within the current text span. To express the joint probability of all text tokens in the sequence, we account for the conditioning on preceding therefore defined as the product of the probabili- ties of each text span, conditioned on all prior spans: (T(ee)))) = M Pe (Tm TTmm = I 1 po vo(mm.lT<mm Arms km, the model is trained by minimizing the standard causal cross-entropy loss over all text positions:
all prior context is given by: Pe TTm: Acm)
audio spans. The joint probability po (Xtert)
(xR(x)
Σ
logpo (bm.s Acms bm
3.2 ABSORBING DISCRETE DIFFUSION FOR AUDIO SPANS
Building on the theoretical foundation established in Section 2 we apply absorbing discrete diffu- sion to audio spans AgM - This design choice aligns with the fundamental difference dependency structures: audio tokens exhibit strong source-target dependencies (conditioning on source text), making them well-suited for the any-order AR nature of diffusion, while text tokens follow target-target causal dependencies, better handled by standard AR modeling.
Audio-specific Corruption and Denoising For each training sample, sample a masking level ~ U(O,, 1I) and independently mask each audio token with probability X, while preserving all text tokens. This creates corrupted sequences where audio spans contain mixture of original tokens and mask tokens [M], but text spans remain intact. To enable efficient parallel training across all audio spans simultancously, we apply masking operations to every audio span Am in the sequence, rather than processing them sequentially. This parallel masking strategy significantly improves training efficiency while leveraging the time-independent nature of the denoising objective (Eq. IT.
Training Objective for Audio Generation The model learns to predict the original audio tokens for masked positions by minimizing the X-denoising cross-entropy loss over all audio spans. As dis- cussed in (Ou et al.) 2024), this objective is mathematically equivalent to the any-order AR objective, and can be equivalently expressed in the AO-ARM form:
M
Lso(x)=
=
Σ m=
Ex m
log qo (Amm (j) (T) Army Om (<)))
j=l
where nm is a random permutation over the positions within audio span Am. and am (xj) denotes the audio tokens that appear before position j in the permuted order. This formulation makes explicit that the audio generation objective is learning to predict each audio token conditioned on an arbitrary subset of other tokens within the same span, plus the full cross-modal context from text. This any- order AR nature is what enables parallel generation during inference.
3.3 MULTIMODAL FACTORIZATION AND UNIFIED OBJECTIVE
Having established AR modeling for text in Section 3.1 and discrete diffusion for audio in Sec- tion 3.2 we now formalize how these two paradigms can be unified within single probabilistic framework. The key insight is to leverage the distinct dependency structures of each modality through partial-order factorization that respects the causal nature of text while allowing flexi- ble ordering within audio spans. Recall that text tokens exhibit strong target-target dependencies requiring causal ordering, while audio tokens primarily depend on source-tar ttrget relationships with their corresponding text. This suggests that within each audio span Am, the tokens can be generated
in any order as long they condition the appropriate cross-modal context Tsm Arm- We
4

---

Published as a conference paper at ICLR 2026
formalize this intuition using partial orders over token positions. A partial order on set V is binary relation 3 that is reflexive, antisymmetric, and transitive. set equipped with such a relation b a;
is called a partially ordered set (poset). Two elements a, b E V are comparable if otherwise, they are incomparable. An antichain is a subset of V in which every pair of distinct elements is incomparable, so no internal ordering constraints exist among them (Davey & Priestley 2002).
Partial-order Formulation Let (V, be a poset over all token indices in the sequence, where V represents all token positions and 3 encodes precedence relationships. For our interleaved text-audio setting, we define: (1) Each text token fm.j precedes fm.j+1 (maintaining left-to-right causality within text spans). (2) All tokens in span precede all tokens in span (maintaining cross-span dependencies). (3) Tokens within each audio span An form an antichain under I (no mandatory internal ordering), but the model is permitted condition the same span during training and inference under any linear extension. For any token i, let Pa(i) denote its of predecessors under this partial order. By construction, each audio token am. has predecessors Pa(am.)) = bm. Any linear extension of the partial order (V, 3) induces a valid chain-rule factorization: p(x) = II p(Ze()) #ra(e())) Since audio tokens within each span form an antichain, there are multiple valid linear extensions differing only in the within-span ordering of audio tokens. Rather than committing to single extension, we can marginalize over all possible orderings within audio spans.
previously generated tokens within
TT U A.m. while for text tokens Pa(tm.j) = Tem U Acm U
Order-marginalized Factorization for Audio Spans For an antichain S S V (such as tokens within an audio span), we define the order-marginalized conditional by averaging over all permuta- tions of po (x sy)s) = EeePems(S) [Les (0(#((s) IV Fn(4))), where (o represents the any-order AR learned through discrete diffusion. When applied to our audio spans, this gives:
[Am]
Po (a Tsm; Acm) Eum--am
II
go (o(m, Tsm Arm, Om (45))
(E1))
Intuitively, this averages the likelihood over all possible within-span orderings, reflecting the fact that audio tokens can be generated any order given the appropriate cross- moosl context. Note that while tokens within Am form an antichain under the partial order (i.e., mandatory sequential constraints). the arder-marginalizees conditional in Eq. I alloow the model to leverage local target- target dependencies that may arise under specific generation orders. This flexibility enables the model to capture useful intra-span structures when beneficial.
Hybrid AR-NAR Joint Distribution Combining fixed-order AR for text with order- onder-marginalized factorization for audio, our model induces the joint scoring function:
M
Tm
II m=
Po (tm.s Agms km,<s)
(AAm T Acm)
(5)
marginalized any AR audio
single-order AR for text
This formulation reveals that both modalities are fundamentally AR: text uses single linear exten- sion (left-to-right), while audio integrates over all linear extensions consistent with the partial order.
Training Objective and Upper Bound Analysis In practice, we cannot directly optimize (e because the garginazrgi conditional in Eq. requires computing expectations over all per- mutations. Instead, we use the training objectives CAR(x) (I)) and (ao(2) derived in Section 3.2 The key theoretical insight is that our combined training objective provides tight upper bound on the negative log-like ligllikei of the desired joint distribution. To see this, consider the audio term:
LAmi Σ
E ~
log qe (am, Tsmm (m,r...ss)))
(6)
[mml
2 -llo E Bu Unm
go (m
Ts A am.aa(s))
5

---

Published as a conference paper at ICLR 2026
The right-hand side is precisely log ( Tsm:A<mi from Eq. the audio loss term for span m in our practical training objective (x)
The left-hand side is exactly
To establish the unified upper bound, now sum the inequality in Eq. 6 over all audio spans m M:
M
IAm! Σ f=1
Σ E
-log qu (m),)m()) ()) Tsmm Acms (m
Σ
loggoo Acm))
The left-hand side is exactly Lso. For the text terms, the CAR loss is defined in Eq. Combining the text and audio terms according to the joint factorization Eq.
Lunified (2) ) A (IR(T) + (x) 2 - (x))
(8)
Detailed derivation of this inequality is provided in Appendix A.I.I where this final inequality fol- lows from combining the text equality with the audio inequality derived above. Thus, minimizing our practical training objective (z) corresponds to minimizing an upper bound on the nega- tive log-likelihool of the theoretically motivated joint distribution pe (z). This result is significant because: (1) It provides theoretical justification for our hybrid AR-NAR training approach. (2) It guarantees that optimizing the computationally tractable objective Lumiteed)) will not deviate arbi- trarily from the theoretically optimal objective - log (I)).
Training Pipeline and Loss Computation Our training pipeline starts from a pretrained text LLM and expands its vocabulary with discrete audio codebook tokens and control symbols ((SOA), (EOA)). Each training sequence is organized as interleaved text spans and audio spans. We provide an illustration of loss computation in Appendix(A... Despite its thcoretical and practical advantages, the hybrid AR-NAR paradigm introduces significant train-test discrepancies that can degrade gener- ation quality. During training, audio spans are partially masked according to the diffusion process, while during inference, the model must generate audio and text tokens conditioned complete text context and previously generated clean audio tokens. To bridge this gap, we propose three principled training strategies:
Batchwise AR & NAR Objective Mixing (BANOM): With probability Pmix. skip the diffu- sion noise addition process for certain samples and compute gradients only on text tokens using AR loss. This ensures that, during training, text tokens occasionally observe clean, unmasked audio spans, which is consistent with the inference setting where text generation conditions on previously generated complete audio content rather than partially masked spans.
Prefix Preservation Masking (PPM): For a fraction Ppretix of training samples, we randomly se- lect cutoff index m and ensure that all preceding audio spans A&m = Am).....Am1) remain unmasked, while applying NAR diffusion loss only to spans Azm {Am, Am+1. AM }. This strategy ensures that during training, when generating span Am. the model observes clean representations of all previous spans Açm, matching the inference scenario where audio spans are generated sequentially and each span A.,. conditions on fully generated, clean preceding spans Agm rather than their corrupted, partially masked versions.
Stochastic Span Truncation (SST): We address the positional bias in (EOA) prediction by ran- domly truncating audio span AM during training. Due to disparate tokenization rates between text and audio, audio tokens significantly outnumber text tokens, resulting in fixed-size spans A1....,44 AM-1 and a variable-Iength final span AM- Since all audio spans undergo simultaneous diffusion training, the model learns to predict (EOA) at fixed positions for early spans, creating strong positional bias that hinders content-aware termination learning for the final span. To mitigate this, we implement stochastic truncation: with probability Prrme we randomly select truncation length k < | and create truncated span Altune (0M,,)..... am.k) by removing the original (EOA) token and suffix tokens (aM, aM.IAs=)) This creates training sam- ples where span termination occurs arbitrary positions rather than fixed boundaries, forcing the model to predict (BOA) based on semantic content and contextual text rather than positional cues.

---

Published as a conference paper at ICLR 2026
Toxt Token
10|l||o|| Audio Decoder
Final NAR Output:
00
EDA ends NAR
Special Token
Start
Start
Prompt Token
a000
@
coA
0
Diffuzion Reverse Process
NAR
AR
Rernack
Remaat
xN
Unified MLLM
D0000
Unified MLLM as Mask Predictor
System
User
Audio
Prompn
Query
Encoder
D
DMomo Masked Block
Prompt
Prompt
Tokens
Masked Block
1w1
Speech
(a) Figure 2: Overview the proposed framework and its diffusion reverse process. (a) TrT framework. A unified MILM that interleaves AR text and NAR audio generation. The model alternates between AR text decoding and NAR audio synthesis based on control tokens. (b) Diffusion reverse process. NAR audio generation through iterative denoising.
(b)
3.4 MODALITY-AWARE ATTENTION MECHANISM
Our attention design enforces a step-wise pattern across three content types: (I) the input prompt uses standard causal attention; (2) text tokens Tmn apply striet causal attention to the prompt, all prior spans, and preceding tokens within their span; and (3) Audio tokens Am use hybrid attention, at- tending bidirectionally within the span and causally the prompt and all carlier spans. This enables parallel audio-span training in single forward pass while preventing cross-span interference. See Appendix A.5 for illustration.
3.5 INFERENCE PROCESS
Figure 2 shows the overview of TTT and its inference process. During inference, TtT alternates between AR text decoding and NAR audio synthesis within unified framework. Given input audio, the encoder produces discrete tokens, which are processed by AR generation until (SOA) is reached. The model then switches to NAR mode, where block-wise diffusion (see Algorithm in Appendix A.4 generates audio spans in parallel. Upon predicting (EOA), remaining tokens in the block are dropped and decoding returns AR mode, repeating the cycle until (EOS) is generated. Each completed audio span is immediately sent to the audio decoder, enabling parallel synthesis with low first-token latency and continuous streaming generation.
EXPERIMENTS
4.1 EXPERIMENTAL SETUPS
Datasets To effectively train and evaluate our proposed TYT framework, we follow prior work (Zeng et al.) 2025 Ding al. 2025 Long al. 2025) and adopt a diverse collection of multi-task datasets, including ASR, TTS. audio chat, text chat, AAC, speech emotion classification (SEC), acoustic scene classification (ASC), and interleaved text-audio data. total, the combined train- ing corpus contains approximately 6.3 million samples across these tasks. For evaluation, we focus on three representative capabilities: (I) conversational reasoning via Audio-QA, (2) cross-modal alignment via ASR, and (3) audio comprehension via AAC. To further validate end-to-end S2S performance in realistic conversational scenarios, we additionally evaluate on IRO-Bench (Yan et al. 2025), comprehensive benchmark that integrates reasoning, understanding, and oral con- versation tasks. Full details on training data composition and evaluation protocol are provided in Appendix A.6 Appendix A.7
7

---

Published as a conference paper at ICLR 2026
Table 1: Comprehensive evaluation of TtT framework. Higher (1) is better for Audio-QA, lower (U)
is better for ASR. Datasets abbreviations are available in Table
Audio-QA QA (1)
ASR (1)
Models
AE.
LQ.
TQA.
WQ.
Fzh.
A2.
A1.
WS m.
WS n.
Fen.
Main
Results
Qwen2.5-1.55 (AR)
10.85
1.00
0.00
0.10
103.18
81.84
95.96
103.15
95.54
105.62
Qwen2.5-1.5B (NAR)
10.70
0.00
0.40
0.20
86.97
224.37
191.11
123.96
143.76
108.25
TrT-1.5B (AR-NAR)
15.68
23.75
3.47
7.70
44.36
14.89
16.72
52.23
41.52
49.00
Qwen2.5-3B (AR)
14.42
10.00
0.60
0.70
90.32
54.94
72.01
80.01
73.64
74.47
Qwen2.5-3B (NAR)
11.31
0.67
1.21
0.70
68.94
212.27
160.58
89.22
111.29
83.51
TYT-3B (AR-NAR)
17.46
34.68
6.53
11.61
55.67
12.53
13.65
53.83
44.29
64.31
Ablation
Study
TYT-3B w/ BANOM
13.87
19.87
2.81
5.12
58.25
18.58
21.35
58.48
49.52
68.87
TrT-3B w/o PPM
14.27
22.79
2.71
5.54
58.86
15.63
18.83
57.76
47.92
67.37
TrT-3B w/o SST
14.12
10.20
1.30
3.72
56.39
25.43
31.03
64.41
56.70
62.60
TTT-3B (AR-NAR)
17.46
34.68
6.53
11.61
55.67
12.53
13.65
53.83
44.29
64.31
Training
Strategy Comparison
TvT-3B (AR-NAR)
17.46
34.68
6.53
11.61
55.67
12.53
13.65
53.83
44.29
64.31
Preirain+AAR
29.45
15.93
3.61
11.45
23.37
9.79
12.67
26.75
20.91
19.49
Pretrain+TYT
26.73
40.07
11.07
21.43
18.99
6.80
5.78
27.59
19.85
19.10
Evaluation To effectively evaluate our model on the above tasks, we carefully design the evalua- tion protocol and metrics: (1) For Audio-QA, we introduce an ASR-LLM pipeline that transcribes the model's spoken responses using language-specific ASR systems (Paraformer-zh for Chinese and Whisper-Large-v3 for English) and leverages powerful LLM-as--aJudee model (Qwen3-30B. D to assess semantic correctness against ground-trum truth answers; (2) For the ASR task, we di- rectly measure transcription accuracy using word error rate (WER): (3) For the AAC task, we adopt the evaluation prompt from CLAIR-A (Wu et al. 2025), using thinking model Qwen3-30B-A33 to judge the caption quality; (4) For URO-Bench, we directly use the official evaluation code and protocol to ensure fair comparison with existing systems. More details are in Appendix A.77. A.7.1
Baselines We compare TtT with state-of- the-----tttttttt audio-language models, including Moshi Défossez et al. 2024). SpeechGPT ZhacchGGPZZhnggggg
Zhang et al. (2023). Kimi-Audio Ding et al. (2025), VITA-Audio
Long et al. 2025). LLaMA-O Fang et al. (2025). GLM-4- Voice Zeng et al. (2024), Mini-Omni
Xie & Wu 2024) and SLAM-Ommi Chen et al. (2025a) (detailed descriptions in Appendix A.8).
Model Configuration We adopt the Qwen2.5-Ban model the backbone, experimenting with parameter scales of 1.5B and 3B, and fine-tune all parameters during training. For the audio compo- nents, we directly follow the audio tokenizer and decoder design introduced in GL-M-4 (Zeng eet al., (024).. These modules have been shown to provide efficient and high-quality speech tok- enization and synthesis, and they allow our framework to leverage strong audio modeling without requiring additional architectural modifications. The training details are provided in Appendix A.9
4.2 VALIDATING THE HYBRID AR-NAR ARCHITECTURE
To evaluate the effectiveness of our proposed TtT framework, we compare with two represen- tative variants, purely AR backhone and purely diffusion based NAR backbone. For fairness and scalability, all three frameworks are instantiated with backbones of 1.5B and 3B parameters.
Performance Analysis on Audio-Qu QA and ASR Tasks Table T (Main Results) provides the com- parative results for the Audio-QA and ASR tasks. Our proposed TtT framework consistently out- performs both pure AR and NAR variants across all metrics. Specifically, at the 3B scale. TIT-3B surpasses Qwen2.5-3B (AR) by +3.04, +24.68, +5.93, and +10.91 on Audio-QA task. For ASR tasks, TTT-3B yields improvements of 42.41 and 58.36 absolute WER points over Qwen2.5-3B (AR). These substantial gains stem from our hybrid AR-NAR design: the NAR diffusion compo- nent enables efficient parallel denoising for tighter audio-text alignment, capturing audio's 's inherent
8

---

Published as a conference paper at ICLR 2026
Table 2: Performance comparison on Audio-QA, ASR, and AAC tasks. Higher (1) is better for
Audio-Oa QA and AAC; lower (J) is better for ASR. Datasets abbreviations are available in Table 7
Audio-QA QA (1)
ASR (1)
AAC (1)
Models
Size
AE
L.Q. TQA. WQ
Fzh. A2.
AL.
WS.m. WS.n. Fen.
Clo. MACS
Large Models (> 7B)
Moshi
25.63
48.30
16.75
16.85
4.32
12.01
SpeechGPT
10.00
30.96 16.53
24.53
101.45
120.77
111.81
123.15
124.86
45.15
2.10
3.95
Kimi-Audio
7B
19.49
57.53 43.51
43.20
2.87
2.53
0.61
6.34
5,39
4.87
55.92
64.90
VITA-Audio
7B
40.20
54.30
18.59
30.75
6.35
5.56
4.58
20.38
15.88
9.58
6.18
7.94
a.A-Mmmmi
8B
39.59
48.46
21.80
30.28
2.53
4.56
GI-M-4-Vocce
9B
44.87
62.67
44.99
48.47
13.15
12.67
Efficient Models (S 3B)
Mini-Omni
0.5B
15.73
2.00
1.10
2.42
182.73
342.40 442.06
294,42
335.80
22.74
3.61
4.45
SLAM-Omni
0.5B
17.47
24.75
3,51
7.90
54.52
50.46
Qwen2.5-3B (AR)
3B
14.42
10.00
0.60
0.70
90.32
54.94
72.01
80.01
73.64
74.47
9.73
48.64
Qwen2.5-3B (NAR)
3B
11.31
0.67
1.21
0.70
68.94
212.27
160.58
89.22
111.29
83.51
9.54
27.40
TTT
3B
17.46
34.68
6.53
11.61
55.67
12.53
13.65
53.83
44.29
64.31
12.63
48.87
Pretrain+TYT
3B
26.73
40.07
11.07
21.43
18.99
6,80
5.78
27.59
19.85
19.10
11.55
42.86
source-target dependencies, while AR text generation maintains coherent cross-modal conditioning and respects target-target dependencies. Purely NAR models perform notably worse due to order confusion from applying order-agnostic objectives inherently sequential text-audio sequences.
Ablation Study To better understand the contribution of each training strategy in our hybrid AR- NAR framework, we perform an ablation study based on the full model TrT-3B (AR-NAR). The variant w/o BANOM corresponds to removing batchwise AR & NAR objective mixing from the full model, w/o PPM removes prefix preservation masking, and w/o SST removes stochastic span truncation. Table [ (ablation study) presents the detailed results of ablation experiments. From these results, we draw the following conclusions: All three training strategies have a positive impact on model performance, and removing any one of them leads clear degradation on most metrics. For instance, on the LQ. dataset, removing SST reduces the score from 34.68 to 10.20. This drop occurs because stochastic truncation mitigates positional bias in (EOA) prediction, forcing span termination by semantic content rather than position. Removing it weakens variable-length audio generation and reduces flexibility conversational outputs. Meanwhile, removing SST also sub- stantially degrades ASR on major benchmarks, e.g., increasing WER on AISHELL-2 from 12.53 to 25.43 and on AISHELL-1 from 13.65 to 31.03.
Effect of Multimodal Alignment Pretraining To further investigate the effectiveness of method on top of a multimodally aligned pretrained model, we perform large-scale multimodal pretraining based on the @Wenn..5-BB-ass model. Specifically, we construct corpus of approxi- mately 200B tokens covering ASR, TTS, text-only data, and interleaved text-audio data. The model trained with a standard AR objective using a global batch size of 256 for 140k steps. This pretrain- with strong cross-modal alignment ability before applying our hybrid AR-NAR learning framework. Table [ (training strategy comparison) compares the AR-only and AR-NAR frameworks under two different training strategies, specifically training directly from Qwen2.5-3BB 1--22--3---aass without multimodal pretraining (TYT-3B) and initialization from the multimodally aligned pretrained model (Pretrain+AR and Pretrain+T(T).. From the table, our TYT framework achieves comparable even superior performance to the AR-only baseline, indicating that the hybrid AR- NAR design is already competitive without pretraining: (2) when applied on top of the multimodally aligned pretrained model, Pretrain+TTT consistently matches or surpasses Pretrain+AR across both Audio-Op QA and ASR tasks. These results demonstrate that TtT not only performs strongly from scratch, but also benefits significantly when built upon large-scale multimodal alignment pretrain- ing.
ing stage equips the backbone model (Qwen2.5-3B-BB)
we observe that: (1) When trained directly from Qwen2.5-3B-Bass
4.3 BENCHMARKING AGAINST STATE-OF-THR-ART MODELS
Having validated the effectiveness of our hybrid architecture and the benefits of multimodal pre- training, now compare our best model (Pretrain+TTTT) against state-of-the-art audio-language

---

Published as conference paper at ICLR 2026
Table Evaluation results on URO-Bench. Higher (1) better for all tasks.
Basic Task (1)
Task (1)
Perceptual Quality
Models
Size
Under- standing soning Conversation standing soning Conversation NMOS
Rea-
Oral
Under-
Rea-
Oral
UTMOS
Large Models (> 7B)
Moshi
7B
18.23
24.21
36.65
26.38
21.06
33.93
3.10
3.05
SpeechGPT
7B
9.26
13.34
35,50
19.03
14.29
28.88
4.04
3.92
Kimi-Audio
7B
83.89
53.88
54.44
53.25
41.44
50.17
3.52
2.93
VITA-Audio
7B
52.08
51.45
54.97
32.36
54.77
45.81
3.95
4.24
LLaMA- A-miii
8B
$3.71
41.93
64.05
34,66
51.51
43.91
4.09
4.00
GLM-4- Voice
9B
85.82
61.63
69.90
55.47
51.89
61.30
3.86
4.15
Efficient Models (Σ 3B)
Mini-Omni
0.5B
15.01
14.80
29.71
23.51
33.09
33.46
4.42
SLAM-Omni
0.5B
26.45
42.20
34.49
27.39
40.23
4.44
Qwen2.5-3B (AR)
3B
34.32
13.15
16.32
34.99
25.90
3.96
4.16
Qwen2.5-3B (NAR)
3B
10.12
12.59
13.70
25.64
3.47
2.35
3B
43.39
24.00
23.37
33.78
34.82
4.25
Pretrain+
3B
57.63
39.30
45.68
32.38
43.76
46.10
3.90
4.23
models. Tables 2 and group results by model scale, distinguishing efficient models from large ones. Notably, Moshi does not support ASR, and the official releases of I.L.MA-mmi, SL.MM Omni and GLM-4-Voice lack ASR prompting, hence no ASR results are reported. Mini-Omni and SpeechGPT exhibit poor gencralization to Chinese ASR tasks, as they are trained solely on English speech. Among efficient models, Pretrain+TCT achieves state-of-the-att performance across Audio- QA and ASR, while remaining competitive on AAC. It substantially outperforms 0.5B baselines such Mini-Omni on Audio-QA QA and ASR, and SLAM-Omni reports higher AAC scores (54.52 on Clotho, 50,46 on MACS), its official implementation relies on separate 7B Vicuna model fine-tuned specifically for AAC. Notably, Pretrain+TtT also exceeds several 7B-scale models on some tasks. It outperforms SpeechGPT on all ASR benchmarks and on AE. and LQ. tasks in Audio-QA. and surpasses Moshi on WQ. and AE. tasks. These results demonstrate that our hybrid AR-NAR design enables a compact 3B model to match or exceed some significantly larger systems on several benchmarks.
Audio-QA. QA. While SLAM-Ommi
Beyond the standard benchmarks, we further validate our model on URO-Bench, a comprehen- sive S2S benchmark that assesses speech understanding, reasoning, and oral conversation across basic and pro difficulty levels (Yan et al. (025). As shown in Table among efficient models, Pretrain+TYT achieves the best performance across both basic and pro difficulty levels. Compared to large models, Pretrain+TiT outperforms Moshi and SpeechGPT across all task categories, and achieves comparable performance to VITA-Audio and LLaMA-Omin on pro oral conversation tasks, while remaining competitive pro reasoning tasks. While GLM-4- Voice achieves the strongest overall performance, the performance gap is reasonable given their ~3 x model size. The percep- tual quality of both TtT and Pretrain+TtT falls within the 3.89-4.25 range (NMOS & UTMOS), confirming consistently good audio synthesis quality. However, Kimi-Audio exhibits notably lower perceptual quality despite its strong task completion performance. We attribute this degradation to language consistency issues observed in its generations. Kimi-Audio frequently produces mixed Chinese-English audio or generates Chinese audio for English tasks. While the semantic content may be correct, such cross-lingual inconsistencies significantly degrade perceptual audio quality.
CONCLUSION
In this work, we introduce unified framework that combines AR text generation with NAR au- dio diffusion. By explicitly respecting the asymmetry between text and audio dependencies, our framework bridges the strengths of AR and NAR modeling within single Transformer. We fur- ther propose simple yet effective strategies to mitigate train-test discrepancies, enabling robust and flexible audio generation. Experiments demonstrate clear improvements over strong AR and NAR baselines. Our results highlight the im- portance of modality-awee aware design for building scalable and effective S2S systems.
Audio-QA ASR, AAC and URO-Bench benchmarks
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was supported in part by Beijing Natural Science Foundation under Grant No. L252010;; in part by NFSC under Grant No. 62477025; in part by Key Laboratory of Smart Education of Guangdong Higher Education Institutes, Jinan University (202LLSYS003) and in part by the Excel- lent Graduate Student Cultivation Program of Jinan University under Grant No. 2024CXB040.
REFERENCES
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. In Proceedings of the 34th Annual Conference Neural Information Processing Systems, virtual Event, December 2021.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Shar- ifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: language modeling approach to audio generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:2523-2533, 2023.
George EP Box, Gwilym M Jenkins, Gregory C Reinsel, and Greta M Ljung. Time series analysis: forecasting and control. John Wiley & Sons, 2015.
Junyi Chen, Shuming Shen, Andi Chen, Wen Wu, Jiantao Kang. Haohe Li, Weijiang Zhou, Yi Ren, Yanmin Qian, Xipeng Qiu, et al. Speecht5: Unified-mode modal encoder-decoder pre- pre-training for spo- ken language processing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, Dublin, Ireland, May 2022.
Wenxi Chen, Ziyang Ma, Ruiqi Yan, Yuzhe Liang. Xiquan Li, Ruiyang Xu, Zhikang Niu, Yanqiao Zhu, Yifan Yang, Zhanxun Liu, al. Slam-omni: cinbreecontrollable voice interaction sys- tem with single-stage training. In Findings of the 63th Annual Meeting of the Association for Computational Linguistics, Vienna, Austria, July 2025a.
Zhichao Chen, Haoxuan Li, Fangyikang Wang, Odin Zhang, Hu Xu, Xiaoyu Jiang, Zhihuan Song, and Hao Wang. Rethinking the diffusion models for missing data imputation: A gradient flow perspective. In Proc. Adv. Neural Inf. Process. Syst., volume 37, pp. 112050-112100 12100, 2024.
Zui Chen, Tianqiao Liu, Mi Tian, Weiqi Luo, Zitao Liu, et al. Advancing mathematical reasoning in language models: The impact of problem-solving data, data synthesis methods, and training stages. In The Thirleenth International Conference on Learning Representations, 2025b.
Fengxiang Cheng, Haoxuan Li, Fenrong Liu, Robert Van Rooij, Kun Zhang, and Zhouchen Lin. Empowering L.MMs with logical reasoning: A comprehensive survey. In Proc. Int. Joint Conf. Artif. Intell., pp. 1 1-9, 2025.
Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. Qwen2-audio technical report. arXiv preprint arXiv:240..10759, 2024.
Trung Dang, David Aponte, Dung Tran, and Kazuhito Koishida. Livespeech: Low-latency zero-shot text-to-speech via autoregressive modeling of audio discrete codes. arXiv preprint arXiv:2406.02897, 2024.
Brian A Davey and Hilary Priestley. Introduction to lattices and order. Cambridge university press, 2002.
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: speech-text foundation model for real-time dia- logue. arXiv preprint 3XXv:2410.0000::: 2024.
Ding Ding, Zeqian Ju, Yichong Leng. Songxiang Liu. Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, et al. Kimi-audio technical report. arXiv preprint arXiv:2504 18425, 2025.
11

---

Published as a conference paper at ICLR 2026
Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. In 2020 IEEE International Conference on Acoustics. Speech and Signal Processing, Barcelona, Spain, May 2020.
Harishchandra Dubey, Ashkan Aazami, Vishak Gopal, Babak Naderi, Sebastian Braun, Ross Cut- ler, Alex Ju, Mehdi Zohourian, Min Tang, Mehrsa Golestaneh, et al. Icassp 2023 deep noise suppression challenge. IEEE Open Journal of Signal Processing, 5:725-737, 2024.
Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. Llama-omni: Seamless speech interaction with large language models. In Proceedings of the 13th International Conference on Learning Representations, Singapore, April 2025.
Guhao Feng, Yihan Geng, Jian Guan, Wei Wu, Liwei Wang, and Di He. Theoretical benefit and limitation of diffusion language model. Proceedings of the 39th Annual Conference on Neural Information Processing Systems, San Diego, USA, December 2025.
Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, et al. Scaling diffusion language models via adaptation from autoregressive models. In Proceedings of the 13th International Conference on Learning Repre- sentations, Singapore, April 2025.
Ailin Huang, Boyong Wu, Bruce Wang, Chao Yan, Chen Hu, Chengli Feng, Fei Tian, Feiyu Shen, Jingbei Li, Mingrui Chen, et al. Step-audio: Unified understanding and generation in intelligent speech interaction. arXiv preprint arXiv:2502. 1946, 2025.
Yu Kang, Tianqiao Liu, Hang Li, Yang Hao, and Wenbiao Ding. Self-supervised audio-and-text pre-training with extremely low-resource parallel data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 10875-10883, 2022.
Jungil Kong, Jaehyeon Kim, and Jackyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. In Proceedings of the 34th Annual Conference Neural Information Processing Systems, Virtual Event, December 2020.
Gen Li and Changxiao Cai. A convergence theory for diffusion language models: An information- theoretic perspective, arXiv preprint arXiv:2505.21400 2025.
Hang Li, Wenbiao Ding, Yu Kang, Tianqiao Liu, Zhongqin Wu, and Zitao Liu. Ctal: Pre-training cross-modal transformer for audio-and-lanne and-language representations. In Proceedings of the 2021 conference on empirical methods in natural language processing, pp. 3966-3977, 2021.
Tianpeng Li. Jun Liu, Tao Zhang, Yuanbo Fang, Da Pan, Mingrui Wang, Zheng Liang, Zehuan Li, Mingan Lin, Guosheng Dong, et al. Baichuan-audio A unified framework for end-to-end speech interaction. arXiv preprint arXiv:2502.172333 2025a.
Xueyi Li. Zhuoneng Zhou, Zitao Liu, Yongdong Wu. and Weiqi Luo. A synergistic multi-agent framework for camouflage attack on large language models. IEEE Transactions on Audio Speech and Language Processing, 34:299-310, 2025b.
Alexander Liu, Sang-gil Lee, Chao-Han Huck Yang. Yuan Gong, Yu-Chiang Frank Wang, James R Glass, Rafael Valle, and Bryan Catanzaro. Uniwav: Towards unified pre-trainin trainiiinng for speech representation learning and generation. Proceedings of the 13th International Confer- ence on Learning Representations, Singapore, April 2025a.
Tianqiao Liu, Zui Chen, Zitao Liu, Mi Tian, and Weiqi Luo. Expediting and elevating large language model reasoning via hidden chain-of-thought decoding. arXiv preprint arXiv:2409.0856/ 2024.
Tianqiao Liu. Zui Chen, Zhensheng Fang, Weiqi Luo, Mi Tian, and Zitao Liu. Matheval: A compre- hensive benchmark for evaluating large language models on mathematical reasoning capabilities. Frontiers of Digital Education, 2(2): 16, 2025b.
Zuwei Long, Yunhang Shen, Chaoyou Fu, Heting Gao. Lijiang Li, Peixian Chen, Mengdan Zhang, Hang Shao, Jian Li, Jinlong Peng, et al. Vita-audio: Fast interleaved cross- modal token generation for efficient large speech-language model. arXiv preprint 2rX 2025.
12

---

Published as a conference paper at ICLR 2026
Justin Lovelace, Varsha Kishore, Yiwei Chen, and Kilian Weinberger. Diffusion guided language modeling. In Findings of the 62nd Anmual Meeting of the Association for Computational Linguis- fics, Bangkok, Thailand, August 2024.
Irene Martín-Morató and Annamaria Mesaros. What the ground truth? reliability of multi- annotator data for audio tagging. In Proceedings of the 29th European Signal Processing Confer- ence, Dublin, Ireland, August 2021.
Shivam Mehta, Ruibo Tu, Jonas Beskow, Éva Székely, and Gustav Eje Henter. Matcha-tts: A fast its architecture with conditional flow matching. Proceedings of the 2025 IEEE International Conference on Acoustics, Speech and Signal Processing, Seoul, Republic of Korea, April 2025.
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. In Proceedings of the 39th Annual Conference Neural Information Processing Systems, San Diego, USA, December 2025.
Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. arXiv preprint urXiv:2406.00 03736, 2024.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and llya Sutskever. Robust speech recognition via large-scale weak supervision. Proceedings of Machine Learning Research, 202:28492-285188 2023.
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level train- 2015.
ing with recurrent neural networks. arXiv preprint
Yi Ren, Jinglin Liu, Xu Tan, Zhou Zhao, Sheng Zhao, and Tie- Liu. A study of non- autoregressive model for sequence generation. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, Virtual Event, July 2020.
Paul K Rubenstein, Chulayuth Asawaroengchai, Due Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al. Audiopalm: A large language model that can speak and listen. arXiv preprint arXiv:2306. 2023.
Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hi- roshi Saruwatari. Utmos: Utokyo-sarulab system for voicemos challenge 2022. arXiv preprint arXiv:2204.02152, 2022.
Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language mod- els. In Proceedings of the 38th Annual Conference on Neural Information Processing Systems, Vancouver, BC, Canada, December 2024.
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and general- ized masked diffusion for discrete data. In Proceedings the 38th Annual Conference on Neural Information Processing Systems, Vancouver, BC, Canada, December 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 11805 2023.
Chengyi Wang, Sanyuan Chen, Yu Wu, Ziyang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. IEEE Transactions on Audio, Speech and Language Processing, 33:705-718, 2023.
Tsung-Han Wu, Joseph E Gonzalez, Trevor Darrell, and David M Chan. Clair-a: Leveraging large language models to judge audio captions. In Proceedings of the 2025 IEEE Automatic Speech Recognition and Understanding Workshop, Honolulu, HI, USA, December 2025.
Zhifei Xie and Changqiao Wu. Mini-omni: Language models can hear, talk while thinking in streaming. arXiv preprint 2408 16725, 2024.
13

---

Published as a conference paper at ICLR 2026
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang. Yang Fan, Kai Dang, et al. Qwen2.5-omni technical report. arXiv preprint arXiv:2503.20215, 2025.
Ruiqi Yan, Xiquan Li, Wenxi Chen, Zhikang Niu, Chen Yang, Ziyang Ma, Kai Yu, and Xie Chen. Uro-bench: A comprehensive benchmark for end-to-end spoken dialogue models. In Findings of the 2025 Conference on Empirical Methods in Natural Language Processing, Suzhou, China, November 2025.
Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09330 2025a.
Ling Yang. Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. Mmada: Multimodal large diffusion language models. In Proceedings of the 39th Annual Conference Neural Information Processing Systems, San Diego, USA, December 2025b.
Jiacheng Ye, Jiahui Gao, Shansan Gong, Lin Zheng, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Beyond autoregression: Discrete diffusion for complex reasoning and planning. In Proceedings of the 13th International Conference on Learning Representations, Singapore, April 2025.
Runpeng Yu, Xinyin Ma, and Xinchao Wang. Dimple: Discrete diffusion multimodal large language model with parallel decoding. arXiv preprint arXiv: 16990, 2025.
Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang. Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. Gloc4-voice: Towards intelligent and human-like end-to-end spoken chathot. arXiv preprint arXiv:2412.02612, 2024.
Aohan Zeng, Zhengxiao Du, Mingdao Liu, Lei Zhang, Shengmin Jiang, Yuxiao Dong, and Jie Tang. Scaling speech-text pre-training with synthetic interleaved data. In Proceedings of the 13th International Conference on Learning Representations, Singapore, April 2025.
Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abil- ities. In Findings the 2023 Conference Empirical Methods in Natural Language Processing. Singapore, December 2023.
Shuai Zhao, Meihuizi Jia, Luu Anh Tuan, Fengjun Pan, and Jinming Wen. Universal vulnerabili- ties large language models: Backdoor attacks for in-context learning. Proceedings of the 2024 Conference on Empirical Methods in Natural I Language Processing, Miami, Florida, USA, November 2024.
Shuai Zhao, Meihuizi Jia, Zhongliang Guo, Leilei Gan, XIAOYU XU, Xiaobao Wu, Jie Fu, Feng Yichao, Fengjun Pan, and Anh Tuan Luu. A survey of recent backdoor attacks and defenses in large language models. Transactions on Machine Learning Research, pp. 1-28, 2025.
Dan Zhu, Tianqiao Liu, and Zitao Liu, Statschartmwpp A dataset for evaluating multimodal math- ematical reasoning abilities on math word problems with statistical charts. In Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 19944,12944, 2025.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 MATHEMATICAL. DERIVATION
A.1.1 DERIVATION OF THE TRAINING OBJECTIVE UPPER BOUND
Recall from Eq. that the joint distribution factors as:
M
T.n.l
(0(
II
pa (mm Aems (Am TsmAAcn
m==1
-=1
Taking the negative logarithm of both sides gives:
log (1))))))
Σ
log po (mm. i Acmolmme Σ
log Po (Am Tsms Acm)
==1
= LAR +
Σ
(- log (m
(10)
m==
By Eq.6 Gladd its summation over m, we have:
(a)(#) (- log Tssm Acm)).
(11)
Therefore, combining both components:
M
(aR(F) + Lso(x) 2 (AR(2)
Σ
(oloão Acm)) log (2));
(12)
which establishes Eq. 8 This confirms that our practical training objective Lumiedd is valid upper bound on the true negative log- liglliiel,oodd, enabling tractable optimization while preserving consistency with the target joint distribution (1))).
A.2
RELATED WORK
A.2.1
AUDIO-LANGUAGE MODEL PRETRAINING
Recent success of LLMs has been demonstrated through their exceptional performance in complex reasoning tasks, particularly in logical inference (Cheng al. 2025 Liu al. 2024) and mathemat- ical problem-solving (Chen et al.) 2025b; Liu et 20025b). Catalyzed by these advancements. there has been paradigm shift towards general-purposs MLLMs capable of processing and generating information across diverse modalities (Zhu et al. 2025), along with broader considerations such as
reliability and safety (Zhao et al.) 2024] Li et al., 2025b Zhao et al.) 2023). Under this paradigm, recent advances in end-to-end audio- language models have moved beyond traditional cascaded chitectures toward unified multimodal frameworks (Chen et al. 2022: Wang et al. (203)). Represen- tative works include Moshi, which achieves real-time duplex speech conversation through hierarchi- cal Transformer architectures (Défossez et al. 2024): GL.M-4-Vicce, which builds upon GLM-4-9B for robust Chinese and English speech processing (Zeng et al., (024); and VITA-Audio, which in- troduces a lightweight multiple cross-modal token prediction (MCTP) module for fast audio-text generation with significantly reduced first-token latency (Long et al. 2025). More recent efforts have focused on scaling and production readiness: Step-Audio presents 130B -oB--prameter unified speech-text model with generative speech data engine and instruction-drin
fine control across di- alects, emotions, singing, and RAP (Huang et al.) 2025). while Baichuan-Aude features text-guided aligned speech generation with multi-codebook discretization to preserve both semantic and acous- tic information (Li et al. 2025a). UniWav proposes the first unified encoder-decoder framework that jointly learns representation encoders and generative audio decoders for both discriminative and generative speech tasks (Liu et al. 2025a).
key limitation shared by these approaches is their reliance on uniform AR objectives for both text and audio tokens, which overlooks the distinct dependency structures of these modalities. Our work addresses this gap by proposing a hybrid AR-NAR framework that respects the inherent asymmetries between text and audio generation.
15

---

Published as a conference paper at ICLR 2026
A.2.2 DISCRETE DIFFUSION MODELS
Discrete diffusion models have emerged as a compelling alternative to AR generation, offering NAR approaches that can generate entire sequences in parallel (Chen et al. 2024). The foundational work of D3PMs generalized diffusion processes to discrete data through flexible transition matri- ces, with absorbing processes that progressively mask tokens proving particularly effective (Austin jet al., 2021). This framework has since evolved through both theoretical advances and practical improvements. From theoretical perspective, recent work has deepened our understanding of dis- crete diffusion dynamics. Ou et al.] (2024) revealed that absorbing diffusion's 's concrete score can be expressed as time-independent conditional probabilities, leading RADD, which a reparam- eterized model that removes explicit time conditioning while establishing connections to any-order AR generation. Building on this foundation, Li & Cai (2025) formally characterized convergence rates, proving that KL divergence decays at Q(1/T) with bounds scaling linearly with token mutual information. However, [Feng et al.] (2025) identified fundamental trade-off: while masked dif- fusion achieves near-optimal perplexity in constant steps, sequence-level tasks like reasoning may require steps linear in sequence length. Practical advances have focused on training efficiency and application domains. Shi et al. (202)) reformulated the variational objective weighted integral of cross-entropy losses, unifying prior approaches while achieving state-of-the-an results that even surpass comparable AR baselines. For complex reasoning tasks where AR models struggle with sub- goal imbalance. Ye et (2025) demonstrated that multi-granulariin diffusion modeling can achieve near-perfect accuracy by prioritizing harder subgoals during training. The scalability challenge has been addressed through innovative adaptation strategies. Rather than training from scratch, [Gong al.) (2025): Nie al.] (2025) showed that pretrained AR models can be efficiently converted to diffusion models via continual pre-training, maintaining competitive performance while enabling parallel generation. Meanwhile, hybrid approaches are gaining traction: Lovelace et al. (2024) com- bined diffusion-based latent proposals with AR decoding for controllable generation, while Yang jt al. (2055b) developed MMaDA, a unified multimodal diffusion foundation model that processes text, images, and reasoning within a single architecture.
A.3 AR MODELING & ABSORBING DISCRETE DIFFUSION
A.3.1 AR MODELING
AR models are a fundamental class of generative models that factorize the joint probability distribu- tion of sequence x = (x+, ,..,,,., into a product of conditional probabilities, based on the chain rule:
L p(z) II ppaal E=
(13)
where x: = (z'...... ...... 1) represents the tokens preceding the current token x'. This factoriza- tion imposes a sequential, causal structure on the generation process. Such models, typically imple- mented with Transformer decoders, are trained by minimizing the negative log-likelihood (NLL) of the data, which corresponds to a cross-entropy loss at each position.
A.3.2 ABSORBING DISCRETE DIFFUSION
Discrete diffusion models offer NAR alternative for sequence generation. We focus on absorbing discrete diffusion (Austin et al. 2021 Ou et al. 2024), which involves a forward corruption process and learned reverse denoising process.
The forward process is a continuous-time discrete Markov chain that corrupts clean sequence my over time interval [o, 7). Its dynamies are governed by time-dependent transition rate matrix Q: = att)( Q, where ott) is a positive noise schedule, For absorbing diffusion, the constant matrix Q abs is defined as:
Forward Process
if o'' = M and + [M].
Qabs (x + x²)=
if x'' = x + [M], otherwise.
(14)
0,
16

---

Published as a conference paper at ICLR 2026
This structure dictates that any token * [M] transitions to special mask token [M] at rate of (t)). The state [M] is an absorbing state because the transition rate out of it is zero (i.e., Q*M(([ + x') 0 for all z'), meaning once token is masked, it remains masked. Over time, the sequence converges to a fully masked state. The probability that a token is masked by time is
given by (tt) = 1 - e 1o a(s)ds
The reverse process is also continuous-time Markov chain that learns to denoise a corrupted sequence Iy back towards the clean data zo. Its reverse transition rate matrix Q, is related to the forward rate matrix by:
Reverse Process
Q xz) (c().) -
2t St: ft
(x: + 2x)
p((z)) E Qr( k),
(15)
The term ps (fx) (azz) is known the concrete score. Since the forward process only allows transitions to the [M] state, the only non-trivial reverse transitions are from [M] back to a vocabulary token. This simplifies the learning task to modeling the score for these specific denoising transitions.
TimeeInnependent Score and the Denoising Objective A key theoretical insight for absorbing diffusion is that the concrete score analytically decomposes into known, time-dependent scalar and a dime-independent conditional probability over the clean data (Ou et al. 2024). Specifically, for transition that unmasks position i from [M] token v, the score is:
Pe U,,.. Pr ,'
((1)
Ivis
(16)
M
(())
clean conditional probability
concrete score
where Evis denotes the set of unmasked (visible) tokens in the corrupted sequence. This decom- position is crucial because decouples the time-dependent dynamics from the data distribution. implies that the model go does not need to learn a complex function of time t. Instead, its sole objective is to learn to approximate the clean conditional distribution Po (o( which is a static, time-indeppndent property of the data. The learning task is thus simplified to a denoising objective: given a corrupted sequence with some tokens masked, predict the original tokens for the masked positions based on the visible context.
Equivalence to Any-Order AR Modeling This denoising perspective reveals a profound con- nection to AR modeling. standard AR model learns to predict a token based on fixed, causal context. The diffusion model, through its denoising objective, learns to predict token given an arbitrary context of unmasked tokens. This ability to condition on any subset of the context the defining feature of any-order AR model (AO-ARM).
In fact, the principled training objective for the diffusion model, known as the X-denoising cross- entropy loss, mathematically equivalent to the training objective of an AO-ARM (Ou al. 2024). which averages the prediction loss over all possible permutations (or orderings) of the sequence:
Cao( BRTUU
log (1 =(1) (z, ((!)).
(17)
where π is random permutation of the token indices. Therefore, training an absorbing discrete diffusion model is equivalent to training powerful ensemble of AR models that can operate in any order. This inherent flexibility is what enables parallel, NAR generation at inference time and makes a suitable choice for modeling source-dependent modalities like audio.
A.4 BLOCK-WISE MASKED DIFFUSION GENERATION FOR AUDIO TOKENS
For NAR audio generation, we employ a block-wise denoising approach adapted from (Nie al. 2025). Unlike full-sequence diffusion, it processes audio in fixed-length blocks, balancing paral- lelism and controllability.
As detailed in Algorithm [ the model generates audio in fixed-size blocks of length B, where each block is progressively denoised over T steps using absorbing discrete diffusion process.
17

---

Published as a conference paper at ICLR 2026
At each denoising step the model predicts tokens for all currently masked positions in parallel. The algorithm then selectively commits the most confident predictions (determined by predicted probability or random sampling) while remasking the remaining positions for further refinement. This progressive denoising continues until all positions the current block are decoded. Crucially, if an (EOA) token is generated within block, decoding terminates immediately at that position, truncating the remainder and seamlessly returning control to the AR text generation mode.
Algorithm Block-wise Masked Diffusion for NAR Audio Generation
Require: Context tokens NIXL,, max generation length Imax E N, 1: Sampling steps 7 E N, block length B E N, temperature T 2 O,
2: CFG scale y 2 remasking strategy R (Low.confidence, random),
3: Special token IDs: mask maskk end-of-audio- E.
Ensure: Generated token sequence s E Nix with L < Le + Lmax
4: Initialize s f-6
Start from context
5: while |s| < [c| + m.max do
6:
(-pad(s, B, value = mmmus)
Append B mask tokens
7: Mbleck + {i x, = mmask A 2 }
b. Masked block indices
8:
{me)i=) f- schedule([Moacc I)
Tokens to decode per step
9:
for = 1 to T do
10:
M₁ - {i
mmask) }
Current mask positions
11:
if > 0 then
Xuncond x; Xuncond [M t- mmaak
Unconditional input b. Batched forward
13:
Leond, Luncond t- model ([x; Xuncona)
L Luncond + (7 1) (Lcond Luncond)
15:
else
L model (x)
17:
end if
< arg (ggax(umbel((47))
Gumbel sampling
19:
if R = lowlloowooooo now_confidenee then
20:
p <- so ffmtma c; + [x[
D Confidence = predicted prob
21:
else if R random then
22:
c, + Uniform(0,
for Mr
23:
end if
24:
c, t- - x for i < |s|
b Protect context tokens Only update masked positions Select nh most confident/random tokens
25:
x; + x; for $ Mr
26:
K, + Topk(fc; i Me), = nd)
27:
x; t x; fol Kr,
D Commit tokens sequence
28:
end for
b+ b |s] + B]
> Extract generated block
30:
& # 0 then
31:
min i b E E s < (s, b return s
Early termination at first end token
32:
end if
33:
- [s]s)b)
Append full block
34:
end while
35: return s
A.5 ILLUSTRATION OF TRAINING LOSS AND ATTENTION DESIGN
Figure (a) shows the training loss of our framework: AR loss is applied to text spans, while NAR loss (based on discrete diffusion) used for audio spans. Although we employ discrete diffusion, our framework is extensible to other NAR generation methods. Figure 3(bb) visualizes the attention pattern described Section 3.4
A.6 DATASET DETAILS
Table provides a summary of the training datasets, with detailed examples provided in the Ap- pendix A.9.T During training, we aim construct balanced corpus that supports effective learning
18

---

Published as a conference paper at ICLR 2026
Pronept
Pre-trained with Expanded Vocabulary
Systom
Prompt
Response
Speech
(a)
(b)
Figure 3: Training loss and attention design. (a) Training pipeline. Starting from a pretrained text
LLM, expand the vocabulary with audio tokens and control symbols. Text spans AR cross- entropy loss while audio spans use NAR diffusion loss, sharing single Transformer backbone. (h) Attention pattern. Text spans follow causal attention (left-to-right), while audio spans bidirec- tional attention within spans but causal attention across spans, enabling parallel audio generation while preserving cross-modal dependencies.
across multiple tasks. Specifically, we randomly sample one million instances from the ASR dataset, the TTS dataset, and the audio chat dataset respectively. In addition, we create bilingual interleaved text and audio data, ensuring that Chinese and English are represented in approximately equal pro- portions. To build the audio chat corpus, we rely on the text-to-audio dataset Voice Assistannntt together with the text-based datasets OpenHermes-2.. and - TTS model, namely Cosy Voice2, to convert text into synthetic audio so as to enrich the training data. To further enhance cross- cross-modal modal alignment between text and audio, we follow prior work ZZeng et al., (225)) and supplement the training corpus with interleaved text and audio data derived from the large-scale pretrained corpus FineWeb-Edd This strategy not only expands task coverage but also strengthens the model's ability to jointly learn from and align textual and acoustic modalities. The evaluation datasets are shown Table 7
and we employ
A.7 EVALUATION DETAILS
A.7.1 EVALUATION TASKS
URO-Bench We leverage URO-Bench for more comprehensive evaluation of our proposed method against existing baselines (Yan et al. 2025). JIRO-Bench is specifically designed for audio- in, audio-out tasks, directly simulating real-world conversational scenarios. In this framework, the spoken outputs from each model are first transcribed into text using Whisper-large-v3 (Radford et al.) 2023), and the resulting transcripts are then evaluated for correctness, coherence, and task alignment. This evaluation employs a hybrid scoring framework comprising three components: (1) an LLM-as- a-judge (originally implemented via commercial LLM APIs) to assess semantic correctness and task alignment; (2) rule-based metrics for automatic WER computation; and (3) a fine-tuned emotion- aware model to evaluate the appropriateness of affective expression in spoken responses. Together, these components ensure that model outputs are judged not merely on surface-level textual fidelity, but on semantic accuracy, transcription quality, and emotional coherence.
URO-Bench structures its evaluation across two difficulty levels: Basic and pro. Each level com- prises three distinct categories: Understanding tasks, Reasoning tasks, and Oral Conversation tasks. This hierarchical design facilitates a fine-graines assessment of model capabilities across increasing levels of linguistic and cognitive demand, covering both single-round and multi-round scenarios. In experiments, report results the English subset of limited access to the original evaluation APIs (Gemini Flash and GPT-4o-mini), we substitute the L.LM-as-a-judge component with thinking model Qwen3-30B-A3R, while keeping all other com-
evaluation set. Due to
19

---

Published as a conference paper at ICLR 2026
Table Summary of datasets used training.
Dataset
Language
Samples
Task Type
Emilia zh
Chinese 500000
ITS
Emilia.en
English
500000
TTS
AISHELL2
Chinese
ASR
AISHELL3
Chinese
ASR
Common Voice
Chinese, English
ASR
GigaSpeech
English
ASR
600000
LibriSpeech
English
ASR
MLSSEnng
English
ASR
PeopleSpeech
English
ASR
VoxPopuli
English
ASR
WenetSpeech
Chinese
400000
ASR
VoiceAssistaat-400K
English
Audio Chat
Openllermes-2.5
English
1000000
Audio Chat
FirefIyTTrain-.IMM
Chinese
Audio Chat
MathInstruct
English
262039
Text Chat
MACS
English
AAC
Clotho-v2
English
AAC
Nonspeech7k
English
59282
SEC
VocalSound
English
SEC
CochlScene
English
ASC
Chinese- Finesse-Fineweb-Euu (Skypile)
Chinese English
1500000
Interleaved Data
Fine
1500000
Interlcaved Data
Total
6321321
ponents, including the Whisper-base based ASR pipeline and rule-based scoring, identical to the original implementation.
Furthermore, the benchmark integrates a perceptual quality evaluation mechanism. We employ the strong UTMOS for the UTMOS score and DNSMOS for the NMOS score evaluation, enabling the joint assessment of content accuracy and acoustic quality (Saeki et al., 2022 Dubey et al.) 2024)) Importantly, none of the URO-Bench data was used during training validation, ensuring unbi- ased assessment of generalization.
Audio-QA Task In addition to URO-Bench, we also evaluate our model with the Audio- QA tasks established by Kimi-Audio (Ding et al. 2025) Previous evaluation framework in Kimi-Audio as- sesses Audio-QA performance using the text portion of interleaved outputs, which overlooks the fact that the audio output of an end-to-end speech model more directly reflects its ability to gener- ate natural and semantically faithful responses. To address this limitation, we evaluate Audio-QA directly on the audio outputs of our framework by first applying an ASR model transcribe the generated audio into text, where Whisper-large-y is used for English audio and Paraformer- zh for Chinese audio, with comparison of ASR performance across different models provided in Table 5) The transcribed text is then combined with the original QA queries and the ground truth answers and passed a large scale reasoning model, Swee3-33BB-A3e which serves as LLM-as-a-Judce model to determine whether the response semantically matches the reference and to provide either correctness label or a graded score. We report the average accuracy or score on the benchmark, and this evaluation pipeline provides more faithful assessment of our model's 's audio-to-audia QA ability in realistic conversational scenarios where speech serves as the output modality.
ASR Task To assess the model's capability in aligning speech with textual representations, evaluate it on the ASR task, where the model generates text transcriptions from input audio and performance is measured using WER. A lower WER indicates more accurate recognition, which reflects not only strong ASR ability but also effective cross modal consistency achieved by our hybrid AR-NAR modeling framework.
20

---

Published as a conference paper at ICLR 2026
Table WER performance of different ASR models on Chinese (zh) and English (en).
Model
WER-zh (4) WER-en (1)
Whisper-Large-v3
0,5054
0.2167
Paraformer-zh
0.1028
0.3946
AAC Task To assess the model's capacity comprehend complex acoustically challenging au- dios, we evaluate its audio captioning (AAC) performance on two established benchmarks: Clotho- v2 (Drossos et al. 2020) and MACS (Martin-Morató & Mesaros 2021). We input audio clips and generate corresponding textual captions. The quality of these captions is then evaluated using Qwen3-30B-A3B Yang et al. 2025a), guided by the evaluation prompt introduced in CLAIR-A (Wu et al. 2025), which emphasizes semantic relevance, completeness, and naturalness. The judge as- signs score on 0- 100 scale, where higher scores indicate better caption quality.
A.7.2 EVALUATION DATASETS
URO-Bench We use the English portion of performance. As detailed in Table the benchmark consists of 10 basic tasks and 12 pro tasks. The basic tasks include oral conversation, reasoning, and 2 understanding tasks, while the pro tasks comprise 4 understanding, 4 reasoning, and oral conversation tasks. The final score is obtained by first averaging the model's performance on each dataset, and then averaging these scores within each (difficulty, category) group.
Yan (2025) evaluate our model's
Table 6: Evaluation datasets used from URO-Bench.
Dataset
Task /Evaluation Aspect
data nums
Category
Basic tasks
AlpacaEval
Authentic, open-ended dialogue
199
Oral Conversation
CommonEval
Authentic, open-ended dialogue
200
Oral Conversation
WildchatEval
Real-world conversation
Oral Conversation
StoralEval
Deduce morals from a given story
201
Reasoning
Summary
Summarize given story statement
118
Oral Understanding
TruthfulEval
Factual questions about life English listening questions
470
Reasoning
GvokaoEval
303
Understanding
Gsm8kEval
Practical mathematical problems
582
Reasoning
MLC
Mathematics, logic, and common sense
Reasoning
Repeat
Repeat the user's 's words verbatim
252
Understanding
Pro Tasky
CodeSwitching-en
Understand code switching sentences
70
Understanding
GenEmotion-en
Respond in specified tone
54
Oral Conversation
GenStyle-en
Respond specified style
Oral Conversation
MLCpro
Difficult mathematical, scientific questions
91
Reasoning
Safety-en
Reject answering privacy-related questions Sing, recite poems, read tongue twisters
24
Reasoning
SRT-en
43
Oral Conversation
UnderEmotion-enn
Understand the speaker's mood
137
Understanding
Multilingual
Respond in multiple languages Comprchension general ambient sounds
1108
Oral Conversation
ClothoEval-cn
265
Understanding
MuChoEval-cn
Comprehension of music
311
Understanding
MrBenchEEvllen
Multi-round spoken dialogue
190
Reasoning
Speaker
Multi-speaker multi-round dialogues
Reasoning
Audio-QA, ASR and AAC Tasks We evaluate model performance on a diverse set of benchmarks covering both Audio Question Answering (Audio-Qu QAA, Automatic Speech Recognition (ASR), and automatic audio caption (AAC) tasks. For Audio uudio-QA, we use four datasets: AlpacaEval, Trivi- aQA, and WebQuestions (English), along with LLaMAQuestions (English), assessing cross-lingual reasoning and comprehension from speech. For ASR, we include five datasets: Fleurs-zh/en (mul- tilingual), AISHELL-1/2, and WenetSpeech (all Chinese), covering varied domains, accents, and recording conditions to robustly measure transcription accuracy. For AAC, we include two datasets: Clotho-v2 (English) and MACS (English), covering natural audios collected from various environ-
21

---

Published as a conference paper at ICLR 2026
mental sound clips, not limited to human- to-human dialogue, which helps assess the model's ability to understand the environment rather than simple language processing. Dataset details are summa- rized in Table
Table 7: Evaluation datasets used for Audio QAn, ASR and AAC tasks.
Dataset
Language
Task Type
Abbreviation
AlpacaEval
English
Audio-QA QA
AE.
LLaMAQuestions
English
Audio-QA
LQ.
TriviaQA
English
Audio-Qu QA
TQA.
WebQuestions
English
Audio-QA
WQ
Fleurs-zh
Chinese
ASR
Fzh.
AISHELL-2
Chinese
ASR
A2.
AISHELL-1
Chinese
ASR
Al.
peeetSpeecc-testmeeeeee
Chinese
ASR
WS_m WS.m.
WenetSpeech-test
Chinese
ASR
WS
Fleurs-en
English
ASR
Fen.
Clotho-v2
English
AAC
Clo.
MACS
English
AAC
MACS
A.8 BASELINES
We compare our TtT model with the following state-of-the-ar ofttherrrt large audio-language models to eval- uate its effectiveness:
Moshi Défossez et al. 2024h: It unifies streaming speech and text understanding within a single
AR framework, aligning acoustic and linguistic representations for low-latency real-time dialogue and robust multimodal instruction following.
SpeechGPT Zhang et al. (023): It incorporates discrete speech tokens into a single language model and follows three-stage training pipeline enable unified speech-text understanding and cross-modal instruction following within one framework.
Kimi-Audio Ding et al. (025)): It uses a multi-task training pipeline to align speech, text and semantics through contrastive and generative objectives, enabling robust fnslroction-folllwiing and long-form audio dialogue understanding.
VrTA-Audio Long et al. (0025): It tackles the latency bottleneck in LSLMs by introducing a fast interleaved decoding mechanism and dynamic token predictor, allowing efficient and streaming- capable audio response generation.
LLaMA-Omni Fang et al. 2025): extends a unified language model to support real-time speech
understanding and generation by integrating low-latency audio streaming, codec-based tokeniza- tion, and tightly aligned speech-text representations for seamless multimodal interaction.
EIL-4-Voicc Zeng et al. (2024): introduces a unified end-to-end spoken language model that interleaves speech and text modalities using a supervised speech tokenizer and joint train- ing paradigm, enabling high-quality spoken dialogue generation.
Mini-Omni Xie & Wu (2024): It enables real-time speech interaction by generating text and audio model, using text-instructed parallel decoding and lightweight
tokens in parallel within training pipeline to preserve the base model's reasoning ability.
SLAM-Omai Chen et al. 2025a):: enables end-to-end spoken dialogue by modeling text and semantic audio tokens in parallel within single model, supporting zero-shot timbre control and low-latency voice interaction through single-stage training.
A.9 TRAINING DETAILS
We train our model using the Adam W optimizer with a global batch size of 2048, a learning rate of 2e -5 and weight decay factor of 1 The learning rate follows cosine decay schedule
22

---

Published as a conference paper at ICLR 2026
with linear warmup ratio of 0.01. Training incorporates three stochastic strategies: (1) BANOM with probability 0.3; (2) PPM with ratio 0.3; (3) SST with probability 0.3. During inference, the model alternates between AR text decoding and NAR diffusion- based audio generation, where text decoding uses nucleus sampling with & = 10 and = 0.95, and audio spans are generated with 200 diffusion steps, a block length of 32 tokens, and total diffusion span length of 640 tokens under classifier- free guidance with scale 0. Since different training strategies lead to varying convergence speeds, reported results are based on checkpoints where training loss has converged. All experiments are conducted on 4 nodes with 8 NVIDIA A100 GPUs per node using the DeepSpeed runtime.
A.9.1 DATA FORMAT OF TRAINING DATA
To enable unified training across diverse tasks, we transform all datasets into a consistent input- output format. On the hand, this standardization allows the model seamlessly integrate het- erogeneous modalities such as speech, text, and interleaved audio-text sequences. On the other hand, unified design is essential for supporting our training strategies, including batchwise AR NAR objective mixing, prefix preservation masking, and stochastic span truncation. These strategies rely on shared representation operate across modalities representative examples of the adopted data formats as follows, covering ASR, TTS, audio chat, text chat, AAC, SEC, ASC, and interleaved text-audio data, as shown in Figures
consistent way. For clarity, we provide
A.10 USAGE OF LLM
In this paper, the LLM is employed solely for text refinement, including correcting typos, fixing spelling errors, and enhancing readability. It is not used for generating research ideas, producing results, or creating original content.
Automatic Speech Recognization (ASR) Data Format
"messages":
"role": "system",
"content": "You Automatic Speech Recognition (ASR) model. The will provide you with an audio input. Your task is to transcribe the audio into text and output the result in an interleaved format: generate 13 text tokens followed by 26 audio tokens, and repeat this pattern until the transcription is complete."
"role": "user",
"content": "<SOA>AUDIO Ser
"role": "assistant",
"content": "TEXT SerrrSneeeeeeee__ Sequencee_1SOAAAAUU 1<S0A>AUDI0 Sequence 1<E0A>
TEXT Sequence Sequenee_2 2=S0:=2U0OOAAAA Sequenee_ 2<E0A><EOS~"
Figure 4: Example of ASR data format.
23

---

Published as a conference paper at ICLR 2026
Text-to-speech (TTS) Data Format
"messages": [
role": "system",
"content": "You are Text to Speech (TTS) model. The user will provide you with text
input. Your task is to transcribe the text into audio and output the result in interleaved format: generate 13 text tokens followed by 26 audio tokens, and repeat this pattern until the transcription complete."
"role": "user",
"content":
"role": "assistant",
[ "content": "TEXT Seaueeceeeeeee___ee S I=SOA>AUDIO SSeeueSeeeeeeeee 1<EOA>
SEXT_Sequenee222 040A 2=E0A><EOS>*
Figure Example TTS data format.
Audio Chat Data Format
"messages":
"role": "system",
"content": "You are an audio-to-audie chat model. The user will provide you with audio input. Your task is to respond in interleaved format: generate 13 text tokens followed by 26 audio tokens, and repeat this pattern until the response is complete."
}.
"role": "user", "content": CAADAAUDIO_Seuenee-FCCCC
"role": "assistant",
"content": "TEXT" SecTT_Sqqeeee__sSeee --SAAIAUSSSSAAAAAASAAASAA BeeeAuuuOOSSSSuueeeeeeeee 1<E0>>>>>
TEXT SOXT__Squence_221 S 2-SOA~AUDIO S - 2=E0A><E0S~"
Figure 6: Example of audio chat data format.
24

---

Published as a conference paper at ICLR 2026
AAC/SEC/ASC Data Format
"messages":
rolee: "system",
"content": "You are helpful audio model. The user will provide you with text-based instruction and an audio input. Your task is to follow the instruction based on the audio and output the result in an interleaved format: generate 13 text tokens followed by 26 audio tokens, and repeat this pattern until the transcription is complete."
"role": "user", "content": SOAX_SSuuuueesSSAAAoooouuueeeno
"role": "assistant",
"content": "TEXT Sequence 1<SOA>AUDIO Sequeeeeeeeeeeen 2<E0A><00>>
TEXT Dequenee_1SSA==AUDIII
Sequunceeeeeeee 1<E0A>
Figure 7: Example of AAC/SEC/ASC data format.
Interleaved Data Format
"text": "TEXT [rruunce_-SSAAAAUIII Sequence _<tuenne_1EEA>>>
TEXT S A-SOAA 2 2<E0A><EOS>
Figure 8: Example of interleaved data format.
Text Chat Data Format
"messages": [
"role": "system",
"content": "You are helpful model."
"role": "user",
"content": "TEXT Sequence"
"role": "assistant".
"content": "TEXT Sequence=EOS~"
Figure 9: Example of text chat data format.
25


Reviews:
[{"review_id": "RUbjVBsZ64", "reviewer": "Reviewer_m8aZ", "summary": "This paper introduces Text-to-Talk (TtT), a multimodal large language model (MLLM) that unifies autoregressive (AR) text generation with non-autoregressive (NAR) audio synthesis via discrete diffusion within a single Transformer architecture. The core motivation is the identified fundamental asymmetry between text and audio modalities: text generation relies on strong target-target (causal) dependencies, while audio generation is primarily driven by source-target dependencies. To address this, the authors propose a hybrid training objective, using a standard AR cross-entropy loss for text spans and an any-order AR loss (equivalent to absorbing discrete diffusion) for audio spans. The paper provides a theoretical justification, showing that the combined training objective is an upper bound on the desired joint distribution. Furthermore, three specialized training strategies—BANOM, PPM, and SST—are introduced to mitigate train-test discrepancies inherent in this hybrid paradigm. Extensive experiments on Audio-QA and ASR tasks demonstrate that TtT significantly outperforms strong AR-only and NAR-only baselines, validating the effectiveness of the proposed approach.", "questions": "See weakness. The main concerns lie in evaluation about tasks(training with AAC but not evaluated), benchmark(without common benchmarks),  and comparison (without comparing with other models).", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper identifies a clear and compelling problem—the mismatch between a uniform AR objective and the distinct dependency structures of text and audio. The proposed hybrid AR-NAR framework is a principled and innovative solution to this problem.\n2. The work is not merely empirical. It provides a solid theoretical grounding by framing the model within a partial-order factorization and proving that the practical training objective serves as a tight upper bound on the negative log-likelihood of the theoretical joint distribution. This strengthens the validity of the approach.\n3. The three proposed strategies (BANOM, PPM, SST) are effective in addressing specific train-test mismatch issues. The ablation study clearly demonstrates their individual and critical contributions to the final performance, with SST showing a particularly dramatic impact on conversational ability.", "weaknesses": "1. Mismatch Between Training and Evaluation Tasks: The model was trained on a diverse set of tasks, including Automated Audio Captioning (AAC) using datasets like Clotho-v2 and MACS. However, the evaluation completely omits any assessment of this capability. It is unclear whether the model has effectively learned to understand and describe general, non-speech audio, which is a significant limitation given its training objective. \n2. Lack of Audio Quality Metrics: The Audio-QA evaluation, while innovative, is limited to semantic correctness. There is no objective or subjective evaluation of the audio quality (e.g., MOS - Mean Opinion Score) or prosody/naturalness of the generated speech, which is a crucial aspect of a speech-generation model.\n3. Absence of Standard Conversational Benchmarks: The evaluation is conducted on custom-held splits of generic QA datasets. The model is not benchmarked against established, standardized benchmarks for conversational AI (e.g., URO-Bench[1]), making it difficult to compare its dialogue capabilities directly with other state-of-the-art spoken dialogue systems.\n4. Limited Comparative Analysis: The experimental results only compare TtT against its own backbone (Qwen2.5) trained with AR or NAR objectives. There is no comparison with other recent, open-source, end-to-end audio-language models such as Mimi-Omni[2], SLAM-Omni[3], Moshi[4], GLM-4-Voice[5], or VITA-Audio[6]. This lack of external benchmarking makes it challenging to gauge the true competitive standing of the proposed method.\n5. Clarity on \"Source\" Context: The term \"source-target\" for audio is sometimes ambiguous. While it's clear the primary source is the corresponding text, the model can also condition on previously generated audio tokens within the same span (as allowed by the any-order AR). The paper could more precisely delineate the relative importance of the textual context versus the intra-span audio context.\n\n[1] URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Models\n\n[2] Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming\n\n[3] SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Training\n\n[4] Moshi: a speech-text foundation model for real-time dialogue\n\n[5] GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot\n\n[6] VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Model", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2XAeob2t6h", "reviewer": "Reviewer_wZ81", "summary": "This paper proposes Text-to-Talk (TtT), a unified audio-text generation framework that combines autoregressive (AR) text generation with non-autoregressive (NAR) audio diffusion within a single Transformer backbone (Qwen2.5-Base). The model introduces a unified AR–NAR training objective based on absorbing discrete diffusion, and a modality-aware attention mechanism that enforces causal decoding for text and bidirectional context modeling for audio.", "questions": "**1. Regarding Lack of Comparison with Strong Baselines**\n\nCould the authors provide results comparing TtT with existing speech-language models such as SALMONN or SpeechGPT to verify whether the proposed approach truly surpasses these baselines? (For audioQA or ASR)\n\n**2. Regarding Evaluation Scope and Applicability**\n\nDo the authors plan to evaluate their model on a broader range of speech-language tasks (e.g., speech captioning, translation, or dialogue) to test whether the proposed framework can generalize beyond Audio-QA and ASR?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "**1. Unified AR–NAR Training Framework**\n\nThe paper introduces an elegant and theoretically grounded method that unifies AR text generation and NAR audio diffusion in a single Transformer. This hybrid objective bridges two fundamentally different generation paradigms under a coherent formulation.\n\n**2. Modality-Aware Architectural Design**\n\nThe proposed modality-aware attention effectively reflects the distinct properties of text and audio—maintaining causal decoding for text while allowing bidirectional modeling for audio spans. This design shows a strong understanding of each modality.\n\n**3. Well-Designed Training Objective and Inference Gap Mitigation**\n\nThe authors incorporate a carefully structured training objective and propose several strategies (e.g., BANOM, PPM, SST) to reduce the training–inference discrepancy. These design choices demonstrate practical awareness of the challenges in aligning AR and NAR generation and help improve stability and sample quality during inference.", "weaknesses": "**1. Lack of Comparison with Strong Speech-Language Baselines**\n\nThe paper does not include comparisons with existing speech-language foundation models such as SALMONN or SpeechGPT, which already perform Audio-QA and ASR tasks. Without such baselines, it remains unclear whether the proposed approach truly outperforms existing speech-language models or if the observed improvements are merely relative to a text-only baseline.\n\n**2. Limited Evaluation Scope**\n\nThe evaluation focuses only on Audio-QA and ASR, whereas prior Speech LLMs (e.g., SALMONN, SpeechGPT) cover a broader set of speech reasoning and generative tasks such as speech captioning, translation, and dialogue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hI4Hef7Vg7", "reviewer": "Reviewer_mjLU", "summary": "This paper addresses the limitation of existing multimodal audio-text models that adopt unified autoregressive (AR) training while ignoring inherent dependency differences between text (target-target causal dependence) and audio (semantic reliance on source text). It proposes the Text-to-Talk (TtT) framework, which integrates AR text generation and non-autoregressive (NAR) audio diffusion within a single Transformer, leveraging absorbing discrete diffusion for a unified training objective, a modality-aware attention mechanism, and three training strategies to mitigate train-test discrepancies. Experiments show TtT outperforms AR/NAR variants of Qwen2.5 on Audio-QA and ASR tasks, but key shortcomings remain: no open-source code/Demo, insufficient comparison with field SOTA models, and lack of subjective evaluations for audio generation.", "questions": "1. What's the specific open-source timeline? Can this paper provide a temporary repo or interactive Demo (e.g., Hugging Face Spaces) during rebuttal to verify reproducibility and audio quality through a display loop?\n\n2. Can this paper add subjective evaluations (e.g., 1-5 MOS scoring, blind listening) to assess audio usability using a display loop?\n\n3. What are TtT's core differences from existing AR-NAR works (e.g., CosyVoice)? Can this paper prove its advantages with experiments utilizing a display loop?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1. **Logically adapts to modal essence**: Precisely captures the inherent dependency differences between text (target-target causal dependence) and audio (semantic dependence on source text), with theoretically sound design of the unified training objective based on absorbing discrete diffusion, filling the gap in modeling \"modal dependency asymmetry\".\n\n2. **Targeted architecture design**: The modality-aware attention mechanism and unified modeling within a single Transformer avoid error propagation and modal separation in traditional cascaded models, well-suited for the practical scenario of \"interleaved text-audio generation\" in speech conversations.\n\n3. **Consistent internal experiments**: Ablation experiments validate the effectiveness of the three training strategies and architectural components, while scaling experiments (1.5B vs 3B) show a performance improvement trend, demonstrating logical internal experimental design.", "weaknesses": "1. **Lack of verifiability**: No interactive Demo or immediate open-source code/data, only a \"subsequent open-sourcing\" statement—hinders reproducibility and audio effect verification, undermining research credibility.\n\n2. **Insufficient baselines**: Only compares Qwen2.5’s AR/NAR variants, excluding other recent field SOTA models—fails to prove industry positioning or highlight hybrid architecture innovation.\n\n3. **Missing subjective evaluations**: Relies solely on quantitative metrics (WER, LLM scoring), ignoring MOS-like standard subjective indicators—cannot assess actual audio usability.\n\n4. **Inadequate hyperparameter/deployment analysis**: No ablation for block-wise diffusion hyperparameters, no exploration of complex scenarios (long conversations, low-resource languages)—unclear deployment potential.\n\n5. **Limited novelty**: AR-NAR+LLM-diffusion ideas are not original; fails to clarify core differences from existing works or prove adaptation irreplaceability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VkEbpeeLgE", "reviewer": "Reviewer_JGXG", "summary": "This paper proposes to combine AR and NAR modeling in SpeechLLMs, based on the motivation that text token generation has a strong dependence on previously generated text token, while speech token generation is strongly dependent on text tokens. To make the model work, they also proposed 3 training tasks/schemes to mitigate training-inference discrepancy. Experiments show the effectiveness of modeling text token with AR method and audio tokens with NAR method.", "questions": "1. what is the text in AAC, SEC, ASC?\n\n2. does any-order AR means the latency of the model is higher? can a user talk to the model in real time?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Novelty. Autoregressive modeling for text tokens and non-autoregressive modeling for audio tokens all in the same Transformer model is a novel methodology contribution. The proposed training tasks make sense and are empirically verified to work well with the model.", "weaknesses": "1. the motivation is confusing, in particular, \"target-target\" and \"source-target\" needs more rigorous explanation - source-target language is used usually in seq2seq scenario such as machine translation, while for autoregressive LLMs, source and target are the same sequence.\n\n2. The input format of AudioLLMs are not settled yet, for example there is Moshi's time-aligned interleaving and there is GLM's alignment-free fixed-length interleaving. I think the paper used the later, but it would make it easier to follow if the data format is stated clearly in early section (e.g. section 2)\n\n3. experiment part is a bit weak because there is no comparison with SotA models on audio benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sBnhyf78FV", "reviewer": "Reviewer_YsXg", "summary": "Prevailing SLM adopt an interleaved text-audio sequence for next-token prediction. (e.g. GLM4-Voice [1], Kimi-Audio [2]). In the mean time, CV research shows promising results on adopt AR for text prediction and NAR for image prediction, such as Transfusion [3] and Show-O [4], compared to modeling both text and image tokens with the same AR objective. This paper thus research the possibility to replace the AR prediction on the interleaved audio token spans with NAR and bidirectional discrete diffusion. The main argument is simple: \"AR for text+NAR for audio\" > \"AR for text and audio\" by a large margin.\n\n1. GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot\n2. Kimi-Audio Technical Report\n3. Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model\n4. Show-o: One Single Transformer to Unify Multimodal Understanding and Generation", "questions": "- Stochastic Span Truncation (SST) is unclear and seems wrong. I understand the motivation of such design, but it does not seem correct to me. If you randomly crop the last audio span (while the complete text sentence is already generated), does it mean that the generation of the final audio span does not adhere to the previously generated text sentence?\n\n**Note**\n\nIf all my concerns and questions are resolved I will consider raising the score to 6.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. Given their experiments, AR+NAR significantly outperforms AR.\n2. The ablation study on the train/test mismatch is clear and effective.", "weaknesses": "## Major Weakness\n1. The paper only compares its models to self-established baselines. However, it is unclear how the trained model performs relative to existing SLM systems. It is not necessary to outperform all existing systems, as the training resources differ, but such comparisons would still help readers understand whether the baseline AR model is completely failing or serves as a reasonable baseline. Comparisons with models such as GLM4-Voice, Kimi-Audio, and LLaMA-Omni are required. If the AR baseline significantly lags behind these models, the conclusions may not generalize and could undermine the significance of the claimed contributions (e.g., conclusions drawn from small models may not hold for large models).\n2. The paper should provide a demo page for readers to understand the quality of the generated speech compared to the pure AR approach.\n\n## Minor Weakness\n1. No human evaluation on the generated speech (both semantic coherence and speech quality)\n2. The novelty is minor, as the CV domain already has extensive work demonstrating that a single Transformer can benefit from both AR and NAR training. Extending this idea to speech thus seems natural.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)