NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction
Authors: Qichao Wang
Ziqiao Meng
Wenqian Cui
Yifei Zhang
Pengcheng Wu
Bingzhe Wu
Irwin King
Liang Chen
Peilin Zhao
Venue: ICML 2025 poster (2025)
Keywords: Spoken dialogue language modeling
Autoregressive
Streaming
Decoder-only Transformer
Primary area: applications->language_speech_and_dialog

Abstract: Inspired by the impressive capabilities of GPT-4o, there is growing interest in enabling speech language models (SLMs) to engage in natural, fluid spoken interactions with humans. Recent advancements have led to the development of several SLMs that demonstrate promising results in this area. However, current approaches have yet to fully exploit dual-channel speech data, which inherently captures the structure and dynamics of human conversation. In this work, we systematically explore the use of dual-channel speech data in the context of modern large language models, and introduce a novel generative modeling paradigm—Next-Token-Pair Prediction (NTPP)—to enable speaker-independent dual-channel spoken dialogue learning using decoder-only architectures for the first time. We evaluate our approach on standard benchmarks, and empirical results show that our proposed method, NTPP, significantly improves the conversational abilities of SLMs in terms of turn-taking prediction, response coherence, and naturalness. Moreover, compared to existing methods, NTPP achieves substantially lower inference latency, highlighting its practical efficiency for real-time applications.  Demo and code can be found at https://audio-3059.pages.dev.

NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Paad Prediction
Qichao Wang" Ziqiao Meng" 1 Wenqian Cui Yifei Zhang" Pengcheng Wu
Bingzhe Wu
Irwin King'
Liang Chen Peilin Zhao 16
Abstract
Overlop
Backchannel
1 [t]]]] one (|lll1/lll [Mi]] |||i||
[ |
Inspired by the impressive capabilities of GPT- 4o, there growing interest in enabling speech language models (SLMs) to engage in natural, fluid spoken interactions with humans. Recent advancements have led to the development of several SLMs that demonstrate promising results in this area. However, current approaches have yet fully exploit dual-channel speech data, which inherently captures the structure and dy- namics of human conversation. In this work, we systematically explore the use of dual-channel speech data in the context of modern large lan- guage models, and introduce a novel genera- tive modeling paradigm- Noxeiign--e diction (NTPP)-to enable indeeeneen dual-channel spoken dialogue learning using decoder-only architectures for the first time. We evaluate our approach standard benchmarks, and empirical results show that our proposed method, NTPP, significantly improves the con- versational abilities of SLMs in terms of turn- taking prediction, response coherence, and natu- ralness. Moreover, compared to existing methods, NTPP achieves substantially lower inference la- tency, highlighting its practical efficiency for real- time applications. Demo and code can be found at wtttt:://// /tt////ddo//////// 3059. www.paagesee...
(c) Reffective Pause
(0) Interruption
Let thens for csampte, you can story
You ta
Stlence...
Karghe, more enough
||||[[0|]| - ( [|]]||
|||111110000 www ww (|0111001]||
Figure I. The dual-channel speech encapsulates various conversa- tional turn taking events, including: (a) Overlap, (b) Backchannel, (c) Pause, and (d) Interruption. These events intermingled within the single-channel audio stream but could be explicitly ob- served in the dual-channel audio stream.
1. Introduction
Pre-
The emergence of large language models (LLMs). espe- cially the GPT series as referenced in (Patel et al., 2023; OpenAL, 2023; 2024), has significantly revolutionized the realm of artificial intelligence. These potent language mod- els (LMs) derive their capabilities from pretraining on vast text corpora, utilizing decoder-only transformer architec- tures, and are steered by a next-token prediction (NTP) objective function. Recently, there's been a surge of inter- est in merging LLMs with other modalities, such as images (Radford et al., 2021; L. et al., 2023; Liu et al., 20236), audio (Zhang et al., 2023a; Hassid et al., 2023), protein sequences (Lin al., 2022; Madani et al., 2023), and more. Among these modalities, audio or speech data is particularly crucial it allows LLMs to engage in real-time vocal interactions with humans. The recently introduced GPT-4o model (Ope- nAI, 2024) demonstrates exceptional proficiency in handling real-time interactions with users in conversational contexts. During the demo presentation, was capable of generat- ing genuine emotional responses and engaging users with prompt reactions. However, these functionalities present additional challenges, the model must accurately interpret the unique audio information embedded in human speech while performing inference with minimal delay.
"Equal contribution. This work is done when the first two authors work as interns in Tencent Al Lab. 'Tencent NNational University of Singapore 'The Chinese University of Hong Kong "Nanyang Technological University 'Shenzhen University Shhanghai Jiao Tong University. Correspondence to: Ziqiao Meng zzq- @ nus. edu.sg. Peilin Zhao <ma- sonzhao tencent.com.
A wide range of advanced speech language models (SLMs) (Xie & Wu, 2024; Zhang et al., 2023a; Fang et al., 2024; Hassid et al., 2023; Rubenstein al., 2023; Nguyen et al.,
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pan Prediction
both speakers. Second, it adopts decoder-only architecture, which provides improved learning and parameter efficiency compared to models requiring additional encoders (Dao et al., 2022). Third, NTPP eliminates the need for voice ac- tivity detection (VAD) module, learning diverse turnttaking behaviors in a fully data-driven manner. Fourth, it maintains a single KVCache (Pope et al., 2023b), enabling greater memory and inference efficiency. Table summarizes these advantages in comparison to existing methods.
Table 1. Comparisons to existing dual-channel spoken dialogue
generative models.
Model
Speaker-Indeenneen
Fnender-Piee
VAD-Free
Single KVCache
dGSLM
/
LSLM
Moshi
NTPP (Ours)
<
2024; Fathullah et al., 2024; Nachmani et al., 2024) has been developed to enable real-time voice interactions with human users. These models typically rely on single-channel audio data and some of them focus on aligning audio and text streams. However, the potential of dual-channel speech data has been somewhat under-explored. Dual-channel speech, which records the audio channels of two speakers inde- pendently, offers distinct advantages over single-channel data. Notably, it can explicitly capture various conversa- tional dynamics, such as overlaps, pauses, and interruptions, providing richer representation of real-world interactions. Some examples of these turn-taking events are illustrated in Figure 1. These dynamics can help train SLMs to engage in more natural and fluent conversations with human users across diverse scenarios.
We conduct comprehensive experiments to evaluate the performance of our approach across multiple dimensions. First, we assess continual dialogue generation by provid- ing preceding conversational context, following established benchmarks (Nguyen et al., 2023). We further evaluate response success rates during interruption events in syn- thesized streaming multi-turn conversations. The results demonstrate that our method enables SLMs to more effec- tively model the distribution of turn-taking events in spoken dialogue. In addition, we perform human evaluations to assess the mcaningfulness and naturalness of the generated responses. To test speaker independence, we permute the input channels of different speakers and observe the robust- ness of model performance; our NTPP exhibits the highest stability among all baselines. Finally, we measure inference latency to determine whether the model can generate timely and coherent cesponses-excuddiie initial non-informative tokens. Our findings show that NTPP delivers faster re- sponse times, with performance remaining strong, particu- larly as the number of conversation turns increases. To sum up, our contributions can be summarized follows:
dGSLM (Nguyen et al., 2023) was the first to introduce textless generative spoken dialogue modeling for simulat- ing dual-channel speech using a Siamese two-tower trans- former architecture. More recently, LSLM (Ma et al., 2024) proposed token fusion strategies, in which dual-channel audio tokens are combined and fed into causal trans- former. Moshi (Défossez et al., 2024), in contrast, presents a text-based multi-channel speech sequence model that aligns multi-scale audio streams with textual streams in parallel. However, these approaches generally either rely on an ad- ditional encoder-randomm! selecting one speaker's chan- nel as the input condition or lack speaker-independenne,, meaning the learned distribution is not permutation invariant with respect to speaker order.
We introduce a novel next-token-pit prediction (NTPP) paradigm for generative spoken dialogue mod- eling, implementing a decoder-only architecture with innovative design enhancements.
We develop compatible solutions for both VQ and RVQ speech tokenizers, enabling a broad range of SLMs to effectively learn from dual-channel speech using our proposed method.
In this research, we propose a novel dual-channel speech autoregressive generative model based on an innovative paradigm called next-token-pair prediction (NTPP), utiliz- ing decoder-only transformer architecture. The model is trained to predict the next pair of speech tokens conditioned on previously generated spoken dialogues. This approach capitalizes on the time-aligned structure of dual-channel speech sequences, enabling more precise representation of the generative distribution in spoken dialogues. To expand its applicability in advanced speech language models, we ex- tend our solution from vector quantization (VQ) tokenizers (van den Oord et al., 2017) to the more advanced residual vector quantization (RVQ) (I.ee et al., 2022) tokenizers.
We comprehensively evaluate NTPP, highlighting its strengths in conversational event simulation, speaker independence, and inference latency, among others.
2. Related Works
Speech Language Models (SLMs). Recent advance- ments in SLMs have focused on integrating LLMs (OpenAI, 2023; Dubey et al., 2024; Chu et al., 2024) to unify au- dio and text processing (Ao et al., 2022; Tang et al., 2022; Wang et al., 2023; Rubenstein et al., 2023). Some models, such as SpeechTS (Ao et al., 2022) and SpeechNet (Tang et al., 2022), adopt an encoder-decoder framework to handle
Compared to existing approaches, NTPP offers four key advantages. First, instead of modeling a conditional dis- tribution, NTPP directly estimates the joint distribution of
2

---

NTPP: Generative Speech Language Modeling for Spoken Dialogue via Next-Token-Pail Prediction
Speaker Channel A - -ff1---0011--111 Ila--
Speaker Channel [10 Ii
Speaker Channel
Tokenizer
Tokenizer
Tokenbee
Dual-channel Transfomer
Encoder
Decoder
De-tokenizer
De-tokenizer
Speaker Channel B uI -1l 1l P(S",s*) = P(St, of s4 (3
Speaker Channel B
-1 [
De-tokenizer
Speaker Channel B 1l |
Σ [ (a) Turn-based Dialogue Modeling
P(S)
18-.....5..5
P(S*15*)= Σ P se sf. (b) NTP with context encoder
PGf 16..
(c) Our NTPP learnig paradigm
Figure 2. An illustration of three different generative models for spoken dialogue is shown: (a) Turn-based dialogue modeling, as formulated in Eq. 4, which is commonly used in cascading and multi-modal approaches; (b) The NTP paradigm with a context encoder architecture, adopted by models such as LSLM, Moshi, and similar variants; (c) Our NTPP paradigm, which employs decoder-only transformer. Replacing the decoder-only architecture with an decoder-decoder Siamese transformer yields the dGSLM.
various specch-related tasks. However, these approaches require specialized pre-processing and post-processing mod- ules tailored to different input and output modalities. In con- trast, models like VioLA (Wang et al., 2023), AudioPaL.M (Rubenstein et al., 2023), SpeechGPT (Zhang et al., 2023a), and SpeechGen (Wu al., 2023) utilize decoder-only trans- formers, representing both discrete audio and text tokens within shared vocabulary. Building on these advance- ments, our work leverages SLMs as pre-trained foundation models, further refining them through continual pre-training on the dual-channel speech.
continuous speech signals, x E IRT" with time length 7", are firstly transformed into sequence of discrete speech tokens S = ($1, ST) = 2(x) (T < T') using a quantizer Q. The quantization operation Q is mapping each latent feature f. where f = {x) E IR xd is downsampled latent feature of derived from an encoder E, the code index s; of its nearest embedding vector:
arg min fel12, vE[V]
(1)
where denotes the ith embedding vector of the learnable codebook 2 E TR VRVxd containing [V] vectors. We refer the readers for the detailed VQ techniques to (van den Oord et al., 2017). If the VQ-tokenizer is adopted, then LLMs are trained via the typical NTP objective:
Spoken Dialogue Language Models. Recently, there has been growing interest in spoken dialogue language mod- cling, inspired by the advancements GPT-4o (OpenAI, 2024). Works such as LLama-Ommi (Fang et al., 2024) and Mini-Omni (Xie & Wu, 2024) generate voice-based dia- logue data from text-based question- -nnswerng pairs. These SLMs are trained on both speech and text sequences, with inference optimized for parallel processing. However, these approaches resemble multi-modal models (Liu et al., 2023b; Alayrac et al., 2022; OpenAI, 2023) and are not well-suited for handling real-time, streaming spoken interactions. A promising yet relatively underexplored direction dual- channel speech language modeling. dGSLM (Nguyen al., 2023) addressed dual-channel speech sequence generation prior to the emergence of modern LLMs, relying on con- ventional encoder-decoder architectures. In contrast, LSLLM (Ma et al., 2024) and Moshi (Défossez et al., 2024) leverage LLMs to model dual-channel speech autoregressively, em- ploying speaker fusion strategies and rQ-transformer (Lee al., 2022) connections, respectively.
p(81, 82, ... ST) II (1 81,
Another popular quantization technique used for speech tokenizers RVQ (Lee et al., 2022) since it can maintain higher reconstruction quality. Specifically, each feature f. estimated by 1) codes in residual manner such that f; 2,1 + 2,2 t.. where each Zıd is indexed Sia- Therefore, the latent feature is represented by two-dimensional array of indices S E TxD, Then the probability distribution p(S) is factorized as
D p(sta Sen,
p(s1, 82 ST)
LLMs trained continually using Eq.2 or Eq.3 effectively cap- ture the underlying generative distribution of speech tokens, which are seamlessly integrated into the text vocabulary.
3. Preliminaries
SLMs. SLMs are usually obtained via continually pre- training LLMs on large-scale single-channel speech se- quences. To fit the NTP learning paradigm in LLMs, input
Generative Spoken Dialogue Modeling. The spoken di- alogue a pair of speech signals x²), containing two speakers' conversations (speakers a and b). By applying the
3

---

NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Paile Prediction
Training SLM with autoregressive transformers on the dual-channel speech sequence
Cross Entropy
Embedcing
Channel
81
S4
3
Sg
Embedding
Fair-wise Causal Mask
Postiin pf
Dual-channel Autoregressive Transformer
Embedding
Codebook
Embedding
1s
1z
kg
4
Embedding Lookup
Token Pair Embedding Operation
= Tx2 5x2 10
Figure 3. The illustration of the autoregressive transformer for learning the dual-channel speech sequence, with the token pair embedding operation (left), the overall architecture (middle) and the pair-wise causal masking mechanism (right).
previously mentioned quantization techniques, (x", x²) can be converted to a pair of speech token sequences (S",S*) S*). Existing popular approaches model dialogue in a sequential generation manner, p(55) (S)), treating one speaker sequence (assume a) given condition:
4. Methods: Next-Token-Pane Prediction
4.1. NTPP Dual-Channel Generative Modeling
Existing spoken dialogue models are mainly learning the conditional distribution p( (S"154) (or p(S*1(")) Eq. 4 and Eq. 5. In this work, we propose a novel approach, prediction (NTPP), that explicitly
shown in
called next-token learns the joint distribution of speaker sequences p(S", S*) using the decoder-only transformer. Specifically, the model learns to predict the next token pair (s% sy) time step t conditioned on the previously generated token pairs from step 1 step 1:
PSS* = p
(4)
As previously mentioned, this approach is limited han- dling multi-turn (even one-turn) conversations and cannot support real-time interactions with human users, it fails to leverage the time-alignment property inherent in human speech conversations. Recently, some works have started to focus on the full duplex capabilities of spoken dialogue language models. Moshi (Défossez et al., 2024) and LSLM (Ma et al., 2024) leverage different architectures follow- ing the NTP manner to learn the conditional distribution p(S* (S)) of the dual-channel speech:
p(S 8h) p(s 22 ....... S7 91. S2 ...... s
II M189 89,0
8 +1).
(6)
Unlike Eq. 4 and Eq. 5, the above Eq. 6 is learning to predict both sf and se during the training process. Then we decompose the distribution p(s $6157 s;-, sf sf) by assuming a conditional independence between of and s at each step such that
T II p( p 1:1 1
P(S
p(s st 16155_11. 82 81, --1.... 2/ $ p( 1.1 ( 1 s
Moshi employs the cQ-transformer to encode both sf-1:1 and - into a conditional latent representation for pre- dicting s; LSLM, on the other hand, explores various token fusion strategies to merge of and of at each time step t. Although LSLM adopts decoder-only architecture, still models S)) as its training objective focuses solely on predicting Shh In contrast, dGSLM employs a two-tower Siamese transformer with an encoder-decoder architecture to model the joint distribution p(S", 5*), thereby enabling speaker-indes independent learning. In this work, we achieve the same goal using a decoder-only architecture NTPP. A comparison of these approaches illustrated in Figure 2.
We illustrate this conditional independence and the dialogue distribution modeling Figure 2. The probability distri- bution E and Eq.7 adheres tive bias that a person's 's speech is influenced by both his previous statements and what he has heard in the past. This approach naturally incorporates mutual dependence, as both p( s/-1a)) and "f s/-1a) ) are mod- eled. Additionally, the aime-alinmment property of the two speaker sequences is preserved through the joint prediction of (sf,5)) at each time step f.
fundamental induc-

---

NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Paie Prediction
S
b
P1,1
114
2.2
2,1
2.2
2.3
h 2,4
Depth Embedding
Cyclic
(a) Token Pair Embedding
Shared Positional Embedding
Shared Channel Embedding
Figure 4. The illustration of two modified components RVQ-based dual-channel transformer: (a) Token pair embedding operation (including eyclic depth embedding) and (b) RVQ causal masking mechanism.
4.2. Autoregressive Dual-channel Speech Transformer
Pair-wise Causal Masking. In the standard LLM atten- tion mechanism, causal masking ensures that cach token can only attend to previous tokens, preventing any access to future tokens. Consequently, the masking matrix M is structured as a lower-triangular matrix. In our dual-channel sequence setting, the key distinction lies in the diagonal of the masking matrix. Specifically, the 2 2 block- wiie diag- onal entries m in M E 22Tx2T follow a pair-wise causal masking strategy. Within each block m, only the diagonal entries remain unmasked, while all other entries are masked. This enforces the constraint that st and sy cannot attend to each other at any given time step t.
The remaining challenge is how the model can learn ,(S",S") as defined in Eq. 6 and Eq. 7 in decoder-only architectures. To address this, we propose the autoregres- sive dual-channel speech transformer. The two speech sequences are rearranged in an interleaved order: 5 = ((sf. (5 $ ..., (sf, ))).. At each time step t, the model predicts pair of tokens St = (sf st). This design requires only minimal modifications adapt the decoder- only transformer architecture of L.LMs. Specifically, two essential components are adjusted to accommodate the se- quence of token pairs: the token pair embedding operation and the pair-wise causal attention masking mechanism.
The solution described above applies to the simple VQ tok- enizer case, as illustrated in Figure 3. We omit the training loss function here, as it nearly identical the NTP train- ing loss, with the only difference being the inclusion of an additional loss term for predicting the second speech channel sequence.
Token Pair Embedding. The token pair embedding op- eration is used to transform each index token 8, back to continuous latent embedding. For token pair S,, there are three important latent embeddings: codebook embedding Zt, positional embedding Pt and channel embedding c, zł and 2f can be easily retrieved from the codebook Z by querying token indices si, se respectively. For positional embedding, we inherit the rotary positional encoding (Su et al., 2024) to indicate which time step that each token belongs to. Note that each token pair st and s share the same positional embedding such that Pe = p since they are aligned at the same time step t. Compared to the conventional SL.M architecture, we have two aligned speech sequences instead of one single sequence. Hence, to inform the model about the speaker role of each sequence, we additionally include a channel embedding e for each token to help the model distinguish which speaker generates the token. C₂ is a sim- ple one-hot encoding c, one-hor(id), where id is either b. Following the implementation of Llama 3 (Dubey et al., 2024), we add both positional embedding and channel embedding to the query q and key vectors k derived in the attention mechanism. Then the token embedding operation for each token pair (st, st? is as follows:
4.3. Generalizing Solutions to RVQ Tokenizers
As mentioned in the preliminary section, the RVQ-tokenizer is widely adopted in SLMs for achieving higher recon- struction quality. It is a non-trivial challenge to extend our solutions to the RVQ-tokenizer since each speech sequence becomes 2D array of codes S E RTxD instead of a one-dimensional index token sequence. How to maintain the decoder-only transformer architecture to learn p(Sa S*) remains tricky issue. To solve this issue, we flatten S into one-dimensional sequence 50/b (( ... &1.)); (+2/1 ($2.11-., 22.D), (ST.1 ST.D)) (ei- ther or b) with sequence length T D. In this way, the dual-channel speech sequence can be re-arranged as
a/b
S = = St Sq, 8t, = ((.,,).....$.1)))) si.....iip......... .... (s7.1.... St........... STD))
si,p), (s) (sT.1.... ST.)))
(10)
This sequence is similar to the VQ-based interleaving se- quence and just additionally contains D residual depth to- kens for each time step f and each speaker channel.
q Wq |2. z + [p. pll + [c c.
(8)
k- Wx [2 x + [P] Pil + |cf, cf],
(9)
where We and Wx k are projection matrices for queries q and keys respectively.
RVQ Token Pair Embedding. The codebook embedding 2₁ the channel embedding c₁ and the positional embedding
5

---

NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pail Prediction
Pr remain the same operation as VQ-based solution. The same P. is shared by Si and Sp. The same channel emhed- ding C₁ is shared by st,d for all d. One challenging problem brought by RVQ-tokenizer is how to identify the depth of each token. For example, when the ith token is input to the model, how could the model know the depth of the token (range from to D)? To alleviate this issue, we introduce the cyclic depth embedding
Loma 3.1 8B -B-0
Gemma-77
12
= Taxt
10
Text
10
Training Loss
8
Perploxity
follows:
d= = (sin((2 * i)/ D), c2s * i))
(11)
Note that this embedding is cycled with step length D. That is, dj di+D for every position (In VO-tokenized sequences, i = t; While in RVQ-tokenized sequences, t = i/DD) due to D depth tokens for each channel).
106k 200k
300k 400k
Step
Figure 5. Comparison of training loss curves across different mod- els. Solid lines show the training progress for different foundation models. Dashed lines represent an ablation study comparing a model trained with textual data (w Text) versus without (w/o Text).
RVQ Causal Masking. The causal masking strategy for RVQ-based tokenized sequences closely resembles that of VQ- VQ-based tokenized sequences. However, since the depth increases from 1 to D (transitioning from VQ to RVQ), the 2D x 2D biock-wise diagonal entries m of the masking ma- trix M E (IT are specifically adjusted. The upper triangular part of m remains masked to ensure that current tokens cannot attend to future tokens -his includes pre- venting shatlow-depth tokens St.4 from attending to deeper- depth tokens st,dd for each step t and each channel). Ad- ditionally, the bottom-left D x D submatrix is masked to ensure that sf. and sy do not attend to each other.
we use Librosa to upsample it to 16kHz for consistency with our training setup.
Baselines We evaluate NTPP's performance by comparing it with established generative speech systems. In terms of turn-taking statistics within generated dialogues, we com- pared NTPP specifically with dGSLM (Nguyen et 2023), as it represents a comparable full full-duplex generative model trained on the Fisher dataset. Following dGSLM's setting (Nguyen et al., 2023), we include a cascaded baseline which consists of an Automatic Speech Recognition followed by a text-based language model and a Text-To- Speech (TTS) module. Following the settings of (Nguyen et al., 2023), we select wav2vec2- large et al., 2020), Kenl.M(Heafield, 201 1), and Google TTS API as the mod- ules respectively. For assessing the meaningfulness and naturalness of the interaction, we extend our comparison to include Moshi (Défossez et al., 2024) and SyneLLM (Pope et al., 2023a). Notably, SyncLLM is a full-duplex dialogue agent designed for real-time, overlapping speech interac- tions through the joint, streaming processing of speech input and output.
The extended solutions in the section 4.3 RVQ-tokenized sequences are illustrated in Figure 4. Note that we omit the special start tokens during discussions for simplicity.
model,
5. Experiments
5.1. Dataset and Baselines
Dataset. Our NTPP is trained using a two-stage pipeline. In the first stage, we establish the SLM with foundational speech capabilities by training the model on three speech datasets, totaling approximately I 140,000 hours. This phase focuses on both speech understanding and synthesis. Un- like other models (Fang et al., 2024; Xie & Wu, 2024) that require additional text alignments, our approach follows textless learning paradigm. This eliminates the need for speech-text alignment, reducing data preprocessing require- ments and significantly increasing the amount of available training data. In the second stage, we equip our SLMs with the ability to listen and speak simultaneously through NTPP training. For this, we leverage the Fisher dataset (Cieri et al., 2004), which contains 2,200 hours of phone conversations between randomly paired participants discussing predefined topics. A key advantage of the Fisher dataset is that each speaker's s audio is recorded on separate channels, provid- ing high-quality dual-channel speech streams essential for NTPP training. Since the original audio sampled at 8kHz,
5.2. SLM Training & Implementation Details
Audio Tokenizer Token Vocoder. We train an RVQ speech tokenizer based on (Zeghidour al., 2022), which encodes each second of audio into 40 discrete tokens from a codebook of size 4096. Due to the limitations of the single- speaker token vocoder presented in (Kong et al., 2020), we train multi-speaker HiFi-GAN to decode speech signals from discrete tokens. The HiFi-GAN architecture consists of a generator and multiple discriminators. The generator uses look-up tables to embed discrete representations and the embedding sequences are up-sampled by a series of
hhtps://iibross.orrgttt..
6

---

NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token- Pari Prediction
Table Turn-taking statisties (event occurrences and durations/minn) for generated dialogues, compared to ground truth using mean absolute differences ([A) between predicted and ground truth values. NTPP model results cover different temperature settings [0.1, 0.5, 0.9]. Lower | values indicate better performance, with the best results highlighted in bold.
Model
Number of occurrences / min
Cumulated duration /min
A
A Pause
Aapp
Averlap
ATPU
Pausel
Aaapp
AOverlap
dGSLM w/o CA
3.9
2.9
3.6
1.
12.1
8.3
1.4
2.5
dGSLM
1.6
3.4
2.
2.9
4.6
3.6
1.8
1.9
LSLM
2.2
3.6
2.4
3.2
4.1
3.4
1.5
2.3
Cascaded
4.1
7.0
7.4
6.5
4.3
5.5
0.9
3.6
NTPP..
1.4
2.1
2.0
1.
3.2
2.5
1.2
2.1
P
1.5
1.9
1.8
1.5
2.9
3.0
0.9
2.2
PTP
1.3
2.3
1.5
0.9
3.3
2.8
1.4
1.9
blocks composed of transposed convolution and a residual block with dilated layers. The speaker embedding is con- catenated to each frame in the up-sampled sequence. The discriminator features Multi-Period Discriminator and Multi-Scale Discriminator, which have the same architec- ture as (Kong et al., 2020).
interference.
5.3. Turn-Taking Event Distribution
Following the experimental setting in dGSLM (Nguyen et al., 2023), we evaluate the dialogue systems with turn- taking capabilities using corpus-level statistics (Ardila et al., 2019) and testing on the Fisher dataset (Cieri et al., 2004). We evaluate the linguistic quality and turn-taking dynam- ics of generated dialogues using various models, as shown in Table 2. The detailed evaluation settings are in the Ap- pendix C.2. LSLM (Ma et al., 2024) integrates speaker chan- nels at the embedding layer and only predicts the speech tokens from the assistant channel, demonstrating notable reduction in the number of Inter-Pausal Units (IPUs) and gaps, indicating smoother transitions between speakers. The LGSLM (Nguyen et al., 2023), particularly with the cross- attention module, shows significant decrease in the cumu- lative duration of pauses and gaps, suggesting more fluid and continuous dialogue. Comparatively, NTPP exhibits balanced performance with moderate reductions in both the number and duration of turn-taking events, highlighting their potential for generating natural and coherent dialogues. These findings underscore the importance of model archi- tecture in optimizing dialogue flow.
LLM Backbones. We leverage three well-known LLMs as the backbone of our SLM: LLaMA 3.1-8B (Dubey et al., 2024), Mistral-7B-v0.1 (Jiang et al., 2023a), and Gemma- 2-9B (Team et al., 2024). We evaluate the performance of SLMs obtained by training these LLMs with the NTP objec- tive large-scale single-channel audio dataset. Perplexity on the test set is used as the evaluation metric. Figure 5 shows the training loss curves over time for all three mod- els. Each model demonstrates a consistent downward trend, indicating effective learning. Mistral-7B and Gemma show comparable learning dynamics, while LLaMA 3.1 knnow for its strong reasoning capabilities in ext-achiives lower training loss more quickly. This finding supports our hypoth- esis that stronger text-based models serve as more effective initializations for continual speech learning, consistent with the perspective of treating audio as a new language.
Perplexity curves highlights that during the pre-training phase, the NTPP model demonstrates superior performance when trained exclusively audio data ("w/o Text"" com- pared to when trained with both audio and its ASR text tran- scriptions ("w Text"). Specifically, the audio-only approach leads to significantly faster convergence and consistently lower perplexity-a measure indicating better predictive ca- pability of the model. The results show that the audio-only setting leads significantly faster convergence and consis- tently lower perplexity throughout training. This indicates that eliminating the ASR transeript-whitn which may introduce recognition errors or redundancy-allows the model to fo- cus on more informative acoustic cues, thereby facilitating more stable and efficient learning. In contrast, the inclu- sion of textual supervision appears to slow optimization and result in higher perplexity, suggesting potential modality
5.4. Interruptions & Reflective Pause
We further develop a comprehensive evaluation framework comprising 400 diverse conversational scenarios specifically designed to systematically capture natural dialogue dynam- ics, with an emphasis on pauses and interruptions. These scenarios are carefully crafted using GPT-4 to reflect the complexity and nuance of human interactions. We then use ChatTTS(cha. 2024) to synthesize high-quality speech from the generated text, effectively mimicking the acoustic characteristics of real-world conversations. As shown in Figure 6, NTPP demonstrates closer alignment with human reference judgments, which are meticulously established through manual annotation by human evaluators, in both
7

---

NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Paie Prediction
speaking-up and interruption scenarios when acting as listener. For instance, informing its response strategy, NTPP employs Voice Activity Detection (VAD) and considers silence state to have been reached after 200ms of contin- uous non-speech is detected, facilitating its decisions on when to speak or acknowledge a pause. This suggests that NTPP more effectively captures the subtleties of human conversational behavior compared other models.
390
Latanoy (mo
300
230
200
130
of tum 2aiing
16
Figure 7. Our method (blue) demonstrates lower latency growth compared to Moshi's linear degradation((ed), maintaining response times below perceptual thresholds (220 ms) across all rounds.
Cerraced
Algineeeee Augg Label
5.7. Speaker Independence
To assess the innreeerindeeeedeenc of various models, we conduct speaker-swapped evaluation. All models are first trained on the dual-channel Fisher dataset (Cieri et al., 2004) using the canonical speaker order. We reverse the input speaker sequence without any additional fine-tuning or adap- tation. This setup allows us to test whether a model's per- formance remains stable when the roles of the two speakers are exchanged- an essential indicator of robustness and generalization in dialogue turn-taking modeling. We eval- uate this on both the training and test sets. As shown in Table 4, both dGSLM and NTPP exhibit minimal variation in key turn- turn-taking metries under speaker-swapped condi- tions-nearle zero variation on the training set and consis- tently low variation on the test set-demonstrating strong speaker-indeppedent behavior. Moshi shows substantial de- viations in metrics such as the number and duration of IPUs, pauses, gaps, and overlaps. These findings suggest that Moshi relies on sreaker-conditiooed generation (due to its modeling of conditional distributions), resulting in degraded performance when the input speaker order is reversed.
Figure 6. NTPP, when acting a listener, shows closer alignment with Human Reference judgments both speaking up and inter- rupting scenarios.
5.5. Human Evaluation
We follow the evaluation protocol from (Veluri et al., 2024) and conduct human study involving 25 annotators with native-level English proficiency. Adopting the Mean Opin- ion Score (MOS) framework, we use 5-point Likert scale to evaluate the Naturalness (N-MOS) of turn-taking and the Meaningfulness (M-MOS) of the generated dialogue con- tent. Table 3 presents a comparison of NTPP with various baselines in terms of both naturalness and meaningfulness, We also include performance comparisons on the out-of- distribution Candor test set (Reece et al., 2023) to assess generalization.
22.5
200-
5.6. Inference Latency
In multi-turn interactions, latency is a crucial metric for assessing the performance of speech interaction models, as it captures the time between receiving the end of input speech and the start of output speech, directly impacting user experience. As shown Figure 7, our NTPP consistently achieves lower inference latency than Moshi, especially as the number of turn-taking rounds increases. We attribute this advantage to NTPP's efficient memory usage: it maintains a single KV Cache, while Moshi requires two separate KV Caches for the two separate transformers. This difference becomes increasingly significant as conversational context grows longer.
30c
4Dx
206
40
Figure 8. Comparative performance analysis. (a) Ablation study of NTPP model training stages, illustrating perplexity versus training steps for the full two-stage model and variants lacking either stage one stage two. For better visualization. we trim the stage training steps to be the same as stage two. (b) Comparison of VQ and RVQ model training loss function of training steps.
8

---

NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Paie Toke-n-Per-ir Prediction
Table Human evaluation results across different SLMs. Meaningfulness (Meaning.) and Naturalness (Nat.) scores (ranging from 5) represent mean estimates and their standard errors (shown in parentheses), reported both overall and separately for the Fisher and CANDOR datasets.
Model
Overall
Fisher
CANDOR
Meaning.
Nat. f
Meaning. t
Nat. f
Meaning.
Nat.
dGSLM
1.38 (0.10)
3.85 (0.12)
1.82 (0.09)
4.10 (0.13)
1.51 (0.12)
2.85 (0.18)
SyneLLM
3.85 (0.06)
4.10 (0.05)
4.10 (0.04)
4.33 (0.08)
3.85 (0.09)
3,91 (0.08)
Moshi
3.90 (0.07)
3.95 (0.06)
3.20 (0.10)
3.90 (0.08)
3.95 (0.08)
3.95 (0.08)
NTPP
3.95 (0.04)
4.15 (0.06)
4.10 (0.06)
4.42 (0.06)
4.05 (0.04)
4.05 (0.10)
GT
4.90 (0.01)
4.95 (0.02)
4.90 (0.03)
4.90 (0.04)
4.90 (0.02)
4.95 (0.02)
Table 4. Linguistic quality and turn-taking metries under speaker-swapped inference. Reported values denote the absolute difference between deviation metrics AM (e.g. AIPU) under the original and swapped speaker orders, computed as | Munginll Lower values indicate higher robustness speaker order permutation.
A Mawppell
Split
Model
Number of Occurrences min
Cumulated Duration min
AIPU
Paasse
Gapp
Avvelapp
AIPU
PaPse
AGapl
AOverlap
dGSLM
0.05
0.14
0.04
0.06
0.09
0.15
0.11
0.09
Train
Moshi
0.43
0.32
0.29
0.29
0.30
0.56
0.55
0.58
NTPP
0.03
0.07
0.07
0.05
0.06
0.18
0.14
0.06
dGSLM
0.20
0.15
0.18
0.21
0.39
0.32
0.41
0.32
Test
Moshi
0.43
0.38
0.37
0.39
0.57
0.62
0,84
0.68
NTPP
0.18
0.14
0.19
0.20
0.35
0.38
0.45
0.24
5.8. Ablation Studies
plan to explore synthetic data strategies for generating high- quality dual-channel speech data.
We further investigate the importance of two-stage train- ing and the use of an RVQ tokenizer in this section. As shown in Figure 8 (a), our two-stage training strategy con- sistently achieves lower perplexity compared to the one- stage approach, which omits pretraining on single-channel audio. This suggests that pretraining on single-channel au- dio provides a strong foundation, significantly improving performance on subsequent dual-channel speech learning. As expected, omitting the second-stage NTPP training on dual-channel speech also leads to performance degradation. Figure 8 (b) compares training loss between VQ and RVQ tokenizers, with RVQ yielding consistently lower loss, high- lighting the importance of developing extended solutions tailored to this tokenizer.
7. Conclusion
In this work, we introduce a novel spoken dialogue gener- ative model based on the NTPP paradigm. To effectively capture the dynamics of human conversations, we design decoder-only dual-channel transformer that models the joint distribution of two speaker channels. Our approach includes both VQ-tokenizer and RVQ-tokenizer versions, significantly enhancing the real-time conversational capabil- ities of diverse SLMs. Through extensive evaluations across multiple benchmarks, we demonstrate the effectiveness and superiority of our method in generating natural and coherent spoken dialogus,, paving the way for more advanced and interactive speech-based based AI systems.
6. Limitations and Future Works
One major limitation is the limited availability of large- scale dual-channel speech data. Unlike single-channel au- dio, which can be sourced from the vast amount of open- source data available online, dual-channel speech data re- quires either additional channel separation operations on single-channel audio or meticulous collection from real- world human conversations. We hope our work will inspire the community to gather large-scale dual-channel or even multi-channel spoken dialogue datasets. In the future, we
Impact Statement
This paper aims to advance the field of SLMs. By enhancing SLMs through our approach, we enable more natural and seamless spoken interactions with human users, benefiting various domains such as voice-based personal assistants, customer service chatbots, and online education with voice interactions. However, potential negative societal con- sequence is the risk of misuse in telecom fraud, as our

---

NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction
approach improves the naturalness of Al-generated conver- sations. To mitigate this risk, further advancements in Al safety techniques and fraud detection systems are necessary.
Baevski, A., Auli, M., and Conneau, A. Wav2vec 2.0: Learning the structure of speech from raw audio. Meta AI, 24, 2020.
Borsos, Z.,, Marinier, R., Vincent, D., Kharitonov, E., Pietquin, O., Sharifi, M., Roblek, D., Teboul, O., Grang- ier, D., Tagliasucchi, M., and Zeghidour, N. Audiolm: A language modeling approach to audio generation. IEEE ACM Trans. Audio Speech Lang. Process., 31:2523-2533, 2023.
Acknowledgements
The research is supported by the Tencent Al Lab RBFR2024004.
References
Casanova, E., Weber, J., Shulby, C. Júnior, A. C., Gölge, E., and Ponti, M. A. Yourtts: Towards zero-shot multi- speaker TTS and zero-shot voice conversion for everyone. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, vol- ume 162 of Proceedings of Machine Learning Research, pp. 2709-2720. PMLR, 2022.
Chattts, 2024. URL
com/.
Alayrac, J., Donahue, J., Luc, P., Miech, A., Barr, I., Has- son, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring. R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J. L.., Borgeaud, S., Brock, A., Nematzadch, A., Sharifzadeh, S., Binkowski, M., Barreira, R., Vinyals, O., Zisserman, A., and Simonyan, K. Flamingo: visual language model for few-shot learning. In Advances in Newral Information Processing Systems 35: Annual Conference on Neural In- formation Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
Chang. H., Zhang. H., Jiang, I.., Liu, C., and Freeman, W. T. Maskgit: Masked generative image transformer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022. New Orleans, LA, USA, June 18-24, 2022, pp. 11305-11311 IEEE, 2022.
Chen, S., Liu, S., Zhou, L.,, Liu, Y., Tan, X., Li, J., Zhao, S., Qian, Y., and Wei, F. VALL-E 2: Neural codec lan- guage models are human parity zero-shot text to speech synthesizers. CoRR, abs/2406.053700 2024.
Anastassiou, P. Chen, J., Chen, J., Chen, Y., Chen, Z.., Chen, Z., Cong, J., Deng, L., Ding, C., Gao, L., Gong, M., Huang, P., Huang, Q. Huang, Z.,, Huo, Y., Jia, D., Li, C., Li, F. Li, H., Li, J., Li, X., Li, X., Liu, L., Liu, S., Liu, S., Liu, X., Liu, Y., Liu, Z., Lu, L.,, Pan, J., Wang, X., Wang, Y., Wang, Y., Wei, Z., Wu, Yao, C., Yang, Y., Yi, Y., Zhang. J., Zhang. Q., Zhang, S., Zhang, W., Zhang, Y., Zhao, Z. Zhong, D., and Zhuang, X. Seed-tts: A family of gigh-tuulity versatile speech generation models. CoRR, abs/2406.004300, 2024.
Chu, Y., Xu, J., Yang, Q., Wei, H., Wei, X., Guo, Z., Leng, Y., Lv, Y., He, J., Lin, J., Zhou, C., and Zhou, J. Qwen2- audio technical report. CoRR, abs/2407.10759, 2024.
Cieri, C., Graff, D., Kimball, O., Miller, D., and Walker, K. Fisher english training speech part I transcripts. Philadel- phia: Linguistic Data Consortium, 2004.
Copet, J., Kreuk, F., Gat, I., Remez, T., Kant, D., Synnaeve, G., Adi, Y., and Défossez, A. Simple and controllable music generation. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Ao, J., Wang, R., Zhou, I.,, Wang, C., Ren, S., Wu, Y., Liu, S., Ko, T., Li, Q., Zhang, Y., Wei, Z., Qian, Y., Li, J., and Wei, F. Speechts: Unified-modal encoder-decoder pre- training for spoken language processing. In Proceedings of the 60th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 5723-5738. Association for Computational Linguistics, 2022.
Dao, Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. Flashat- tention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Pro- cessing Systems 35: Annual Conference on Neural In- formation Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022.
Ardila, R., Branson, M., Davis, K., Henretty, M., Kohler, M., Meyer, J., Morais, R., Saunders, L., Tyers, F. M., and Weber, G. Common voice: A massively-mullllnguaal speech corpus. arXiv preprint arXiv: 19t:::191...0 1912.06672, 2019.
Arora, S., Futami, H., Jung, J., Peng, Y., Sharma, R. S., Kashiwagi, Y., Tsunoo, E., and Watanabe, S. Univer- slu: Universal spoken language understanding for diverse classification and sequence generation tasks with a single network. CoRR, abs/2310.02973. 2023.
Défosse7, A., Mazaré, I.., Orsini, M., Royer, A., Pérez, P., Jégou, H., Grave, E., and Zeghidour, N. Moshi: a speech-text foundation model for real-time dialogue. Technical report, Kyutai, September 2024. URL http: //kyutai, .or...rggi..... .pdf.
10

---

NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pan Prediction
Deshmukh, S., Elizalde, B., Singh, R., and Wang. H. Pengi: An audio language model for audio tasks. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NewrlPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023.
Fu, C., Lin, H., Long, Z., Shen, Y., Zhao, M., Zhang, Y., Dong, S., Wang. X., Yin, D., Ma, L.., Zheng, X., He, R., Ji, R., Wu, Y., Shan, C., and Sun, X. Vita: Towards open-source interactive omni multimodal 1lm, 2024.
Gao, Z., Li, Z., Wang, J., Luo, H., Shi, X., Chen, M., Li, Y., Zuo, L., Du, Z., and Zhang, S. Funasr: A fundamental end-to-end speech recognition toolkit. In 24th Annual Conference of the International Speech Communication Association, Interspeech 2023, Dublin, Ireland, August 20-24, 2023, pp. 1593-1597. ISCA, 2023.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A.,
Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravanku- mar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Rozière, B., Biron, B.. Tang, B., Chem,, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConnell, C.,, Keller, C., Touret, C., Wu, C., Wong, C., Ferrer, C. C., Nikolaidis, C., Al- lonsius, D., Song, D., Pintz, D., Livshits, D., Esiobu, D., Choudhary, D., Mahajan, D.. Garcia-Olano, D., Perino, D.. Hupkes, D., Lakomkin, E., AlBadawy, E., Lobanova, E., Dinan, E., Smith, E. M., Radenovic, F., Zhang, F., Synnaeve, G., Lee, G., Anderson, G. I.., Nail, G., Mialon, G., Pang, G. Cucurell, G., Nguyen, H., Korevaar, H., Xu, H., Touvron, H., Zarov, I., Ibarra, I. A., Kloumann, I. M., Misra, I., Evtimov, I., Copet, J., Lee, J., Geffert, J. Vranes, J.. Park, J., Mahadeokar, J.. Shah, J., van der Linde, J., Billock, J., Hong, J., Lee, J., Fu, J., Chi, J., Huang. J., Liu, J., Wang. J.. Yu, J., Bitton, J., Spisak, J., Park, J., Rocca, J., Johnstun, J., Saxe, J., Jia, J., Alwala, K. V. Upasani, K., Plawiak, K., Li, K., Heafield, K., Stone, K., and et al. The Ilama 3 herd of models. CoRR, abs/2407.21783, 2024.
Hassid, M. Remez, T., Nguyen, T. A., Gat, I., Conneau, A., Kreuk, F., Copet, J., Défossez, A., Synnaeve, G., Dupoux, E., Schwartz, R., and Adi, Y. Textually pretrained speech language models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NewrIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Heafield, K. Kenlm: Faster and smaller language model queries. In Proceedings of the sixth workshop on statisti- cal machine translation, pp. 187-197, 2011.
Huang, R., Huang, J., Yang, D., Ren, Y., Liu, L., Li, M., Ye, Z., Liu, J., Yin, X., and Zhao, Z. Make-an-audio, Text-to- audio generation with prompt-enhanced diffusion models. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 13916-13932. PMLR, 2023.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. I., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.0682. 2023a.
Elizalde, B., Deshmukh, S., Ismail, M. A., and Wang, H. CLAP learning audio concepts from natural language supervision. In IEEE International Conference on Acous- tics, Speech and Signal Processing ICASSP 2023, Rhodes Island, Greece, June 4-10, 2023, pp- 1-5. IEEE, 2023.
Jiang, Z., Ren, Y., Ye, 7.,, Liu, J., Zhang, C., Yang, Q. Ji, S., Huang, R., Wang, C., Yin, X., Ma, Z., and Zhao, Z. Mega-tts: Zero-shot text-to-speech scale with intrinsic inductive bias. CORR, abs/2306.03509, 2023b.
Esser, P., Rombach, R., and Ommer, B. Taming transformers for nighhresoluttin image synthesis. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 12873-12883. Computer Vision Foundation / IEEE, 2021.
Kharitonov, E., Vincent, D., Borsos, Z., Marinier, R., Gir- gin, S., Pietquin, O., Sharifi, M., Tagliasacchi, M.. and Zeghidour, N. Speak, read and prompt: High-fidelity text-to-speech with minimal supervision. Trans. Assoc. Comput. Linguistics, 11: 1703-1788 2023.
Fang, Q., Guo, S., Zhou, Y., Ma. Z., Zhang, S., and Feng, Y. Llama-omni: Seamless speech interaction with large lan- guage models. arXiv preprint NXX1:2449900666 2024.
Kong, J., Kim, J., and Bae, J. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. In Larochelle, H. Ranzato, M.. Hadsell, R., Balcan, M., and Lin, H. (eds.). Ad- vances in Neural Information Processing Systems, volume 33, pp. 17022-17033. Curran Associates, Inc., 2020. URL / htwp:oeceeee.nn.n............... cc//sppee_fileespppprr//2/////1 2a411680996644222220066164d03323
Fathullah, Y., Wu, C., Lakomkin, E., Li, K., Jia, J., Shang- guan, Y., Mahadeokar, J., Kalinli, O., Fuegen, C., and Seltzer, M. Audiochatlamma: Towards general-purpose speech abilities for llms. In Proceedings of the 2024 Conference of the North American Chupter of the Associ- ation for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pp. 5522-5532. Association for Computational Linguistics, 2024.
pdf.

---

NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Paie Prediction
Kong, Z., Ping, W., Huang, J., Zhao, K., and Catanzaro, B. Diffwave: A versatile diffusion model for audio synthesis. In 9th International Conference Learning Representa- tions, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
generation with latent diffusion models. In International Conference on Machine Learning, ICMI. 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceed- ings of Machine Learning Research, pp. 21450-21474 PMLR, 2023a.
Kreuk, F., Synnaeve, G., Polyak, A., Singer, U., Défosse7, A., Copet, J., Parikh, D., Taigman, Y., and Adi, Y. Audio- gen: Textually guided audio generation. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenRe- nitw.net, 2023.
Liu, H., Li, C.. Wu, Q., and Lee, Y. J. Visual instruction tuning. In Advances in Neural Information Processing Systems 36: Annual Conference
Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023b.
Liu, W. Wang. J., Wang. H., Li, R., Deng, Z.,, Zhang, Y., and Qiu, Y. D-separation for causal self-explanation. Advances in Neural Information Processing Systems, 36: 43620 43633, 2023c.
Le, M., Vyas, A., Shi, B., Karrer, B., Sari, L.., Moritz, R., Williamson, M., Manohar, V., Adi, Y., Mahadeokar, J., and Hsu, W. Voicebox: Text-guided multilingual univer- sal speech generation at scale. In Advances in Neural In- formation Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Liu, W., Deng, Z., Niu, Z., Wang, J., Wang, H.,, Zhang, Y., and Li, R. Is the mmi criterion necessary for interpretabil- ity? degenerating non-causal features plain noise for self-rationalizattio. Advances in Neural Information Pro- cessing Systems, 37:117636-117633 2024.
Lee, D., Kim, C., Kim, S., Cho, M., and Han, W. Autore- gressive image generation using residual quantization. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18.24, 2022, pp. 11513-11522. IEEE, 2022.
Ma, Z., Song, Y., Du, C., Cong, J., Chen, Z., Wang, Y., Wang, Y., and Chen, X. Language model can listen while speaking. CoRR, abs/2408 020022, 2024.
Madani, A., Krause, B., Greene, E. R., Subramanian, S., Mohr, B. P., Holton, J. M., Olmos, J. L., Xiong, C., Sun, Z. Z., Socher, R., Fraser, J. S., and Naik, N. V. Large language models generate functional protein sequences across diverse families. Nature Biotechnology, pp. 1- 8, 2023. URL https: /pii. www.nticscooola.........c org / Ctrp/orrrussaa 2563044
Li, J., Li, D., Xiong, C., and Hoi, S. C. H. BLIP: boot- strapping language-image pre-training for unified vision- language understanding and generation. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 12888- 12900. PMLR, 2022.
Nachmani, E., Levkovitch, A., Salazar, J., Asawaroengehai, C., Mariooryad, S., Skerry-Ryan, R. J., and Ramanovich, M. T. Lms with voice: Spoken language modeling beyond speech tokens. CoRR, abs/2305. 15255, 2023.
Li, J., Li, D., Savarese, S., and Hoi, S. C. H. BLIP-2: bootstrapping language-imase pre- pre-training with frozen image encoders and large language models. In Inter- national Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 19730- 19742. PMLR, 2023.
Nachmani, E., Levkovitch, A., Hirsch, R., Salazar, J., Asawaroengchai, C., Mariooryad, S., Rivlin, E., Skerry- Ryan, R. J., and Ramanovich, M. T. Spoken question answering and speech continuation using spectrogram- powered LLM. In The Twelfth International Conference on Learning Representations, ICI. 2024, Vienna, Aus- tria, May 7-11, 2024. OpenReview.net, 2024.
Li, T., Tian, Y., Li, H., Deng. M., and He, K. Autoregressive image generation without vector quantization. CoRR, abs/2406.1 1838, 2024.
Lin, Z. Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., dos Santos Costa, A., Fazel-7arandi, M., Sercu, T., Candido, S., and Rives, A. Evolutionary-scat prediction of atomic level protein structure with a language model. bioRxiv, 2022. doi: 00.101/2022.0..2050009999
Nguyen, T. A., Kharitonov, E., Copet, J., Adi, Y., Hsu, W., Elkahky, A., Tomasello, P., Algayres, R., Sagot, B., Mo- hamed, A., and Dupoux, E. Generative spoken dialogue language modeling. Trans. Assoc. Comput. Linguistics, 2023.
Nguyen, T. A., Muller, B., Yu, B., Costa-jussà, M. R., EI- bayad, M., Popuri, S., Duquenne, P., Algayres, R., Mav- lyutov, R., Gat, I., Synnaeve, G., Pino, J., Sagot, B., and
Liu, H., Chen, Z. Yuan, Y., Mei, X., Liu, X., Mandic, D. P., Wang, W., and Plumbley, M. D. Audioldm: Text-to-audio
12

---

NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pait Prediction
Dupoux, E. Spirit-Im: Interleaved spoken and written language model. CoRR, abs/2402.057555, 2024.
Zilka, L.,, and Frank, C. H. Audiopalm: A large language model that can speak and listen. CoRR, abs/2306.12925, 2023.
OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023.
Schwaller, P., Laino, T., Gaudin, T., Bolgar, P., Hunter, C. A., Bekas, C., and Lee, A. A. Molecular transformer: A model for uncertainty-calibrre diction. ACS Central Science, 100911172--5888 2019.
OpenAI. 2024. URL hello-gpt-40/.
chemical reaction pre-
Patel, A., Li, B., Rasooli, M. S., Constant, N., Raffel, C., and Callison-Burch, C. Bidirectional language models are also few-shot learners. The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
Shen, K., Ju, Z., Tan, X., Liu, E., Leng, Y., He, L., Qin, T., Zhao, S., and Bian, J. Naturalspeech 2: Latent diffu- sion models are natural and zero-shot speech and singing synthesizers. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Open 2024.
Pope, R., Douglas, S., Chowdhery, A., Devlin, J.. Bradbury, J., Heek, J., Xiao, K., Agrawal, S., and Dean, J. Efficiently scaling transformer inference. In Proceedings of the Sixth Conference on Machine Learning and Systems, MLSys 2023, Miami, FL, USA, June 4-8, 2023. misys.org. 2023a.
Shi, C., Xu, M., Zhu, Z., Zhang, W., Zhang, M., and Tang, J. Graphaf: a flow-based autoregressive model for molec- ular graph generation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury. J., Heek, J., Xiao, K., Agrawal, S., and Dean, J. Efficiently scaling transformer inference. In Proceedings of the Sixth Conference on Machine Learning and Systems, MLSys 2023, Miami, FL, USA, June 4-8, 2023. mlsys.org. 2023b.
Siuzdak, H. Vocos:
Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Ma- chine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8763. PMLR, 2021.
Su, J., Ahmed, M. H. M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurvcompuating, 568: 12766, 2024. doi: 10. 101GJ..EEUCCO..223..22200 URL https://///// /doi. org 10.. http:///.neuucunna nww.om..cmm...mm... www.:/ccc................ 2023 127063.
Su, Y., Lan, T., Li, H., Xu, J., Wang, Y., and Cai, D. Pandagpt: One model to instruction-follow them all. CoRR, abs/2305. 2023.
Radford, A., Kim, W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. Robust speech recognition via large-scale weak supervision. In International Confer- ence on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 28492-28518. 88. PMLR, 2023.
Sutskever, I., Vinyals, O., and Le, Q. V. Sequence to se- quence learning with neural networks. In Advances in Neural Information Processing Systems 27: Annual Con- ference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pp. 3104 32,2, 2014.
Reece, A., Cooney, G., Bull, P., Chung, C., Dawson, B., Fitzpatrick, C., Glazer, T., Knox, D., Liebscher, A., and Marin, S. The candor corpus: Insights from large multimodal dataset of naturalistic conversation. Science Advances, 9(1 :)aadf31997 2023.
Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., and Zhang, C. SALMONN: towards generic hear- ing abilities for large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRe- view.net, 2024.
Rubenstein, K., Asawaroengchai, C., Nguyen, D. D., Bapna, A., Borsos, Z., de Chaumont Quitry. F., Chen, P. Badawy, D. E., Han, W., Kharitonov, E., Muckenhirn, H., Padfield, D., Qin, J., Rozenberg, D., Sainath, T. N., Schalkwyk, J., Sharifi, M., Ramanovich, M. T., Tagliasac- chi, M., Tudor, A., Velimirovic, M., Vincent, D., Yu, J., Wang, Y., Zayats, V., Zeghidour, N., Zhang, Y., Zhang, Z.,,
Tang, R., Kumar, K., Yang, G., Pandey, A., Mao, Y., Belyaev, V., Emmadi, M., Murray, G. C., Ture, F., and Lin, J. Speechnet Weakly supervised, end-to-end speech recognition at industrial scale. In Proceedings of the 2022 Conference on Empirical Methods Natural Language
13

---

NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Paie Prediction
Processing: EMNLP 2022 Industry Track, Abu Dhabi, UAE, December 7- 11, 2022, pp. 285-293. Association for Computational Linguisties, 2022.
Xie, Z. and Wu, C. Mini-omni: Language models can hear, talk while thinking in streaming, 2024. URL https: / //arxiv.
16725.
Team, C. Chameleon: Mixed-modal early-fusion foundation models. CoRR, abs/2405.098/1 2024.
Yang, D., Yu, J., Wang, H., Wang, W., Weng, C., Zou, Y., and Yu, D. Diffsound: Discrete diffusion model for text- to-sound generation. IEEE ACM Trans. Audio Speech Lang. Process. 31:1720-1733, 2023.
Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, I., Rivière, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.0827
Yang, D., Liu, S., Huang, R., Weng, C.,, and Meng, H. Instructtts: Modelling expressive TTS in discrete la- tent space with natural language style prompt. IEEE ACM Trans. Audio Speech Lang. Process., 32:0e99,,3:221133 32:2913-29255. 2024.
2024.
Tian, K., Jiang, Y., Yuan, Z., Peng, B., and Wang. L. Visual autoregressive modeling: Scalable image generation via next-scale prediction. CoRR, abs/2404.02905, 2024.
You, J., Ying, R., Ren, X., Hamilton, W. L., and Leskovec, J. Graphrmn:
van den Oord, A., Vinyals, O., and kavukcuoglu, k. Neural discrete representation learning. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://procceedingg. neurips. cc Seaaper_fllesspppperr////aaaaeep 7a98af (rtBaafeeeecaaaaceeeeeee00003030 Paper. pdf.
Generating realistic graphs with deep auto- regressive models. In Proceedings of the 35th Interna- tional Conference on Machine Learning, ICMI. 2018, Stockholmsmässsan Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 5694-5703, PMLR, 2018.
Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., and Tagliasacchi, M. Soundstream: An end-to-end neural audio codec. IEEE ACM Trans. Audio Speech Lang. Process. 30:495-507, 2022.
Veluri, B., Peloquin, B. N., Yu, B., Gong. H., and Gol- lakota, S. Beyond aurnbased interfaces: Synchronous Iims as dullex dialogue agents. arXiv preprint arXiv:2409. 15594, 2024.
Zhang, D., Li, S., Zhang, X., Zhan, J., Wang, P., Zhou, Y., and Qiu, X. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abili- ties. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singupore, December 10, 2023, pp. 15757-15773. Association for Computational Linguistics, 2023a.
Wang, T., Zhou, L.,, Zhang, Z., Wu, Y., Liu, S., Gaur, Y., Chen, Z., Li, J.. and Wei, F. Viola: Unified codec lan- guage models for speech recognition, synthesis, and trans- lation. CoRR, abs/2305.16107 2023.
Weissenborn, D., Täckström, O., and Uszkoreit, J. Scal- ing autoregressive video models. In 8th International Conference on Learning Representations, ICLR 2020, Ad- dis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Zhang, Y., Han, W., Qin, J., Wang. Y., Bapna, A., Chen, Z., Chen, N., Li, B., Axelrod, V., Wang, G., Meng, Z., Hu, K., Rosenberg, A., Prabhavalkar, R., Park. D. S., Haghani, P., Riesa, J., Perng, G., Soltau, H., Strohman, T., Ramabhadran, B., Sainath, T. N., Moreno, P. J., Chiu, C., Schalkwyk, J., Beaufays, F., and Wu, Y. Google USM: scaling automatic speech recognition beyond 100 languages. CoRR, abs/2303.01037, 2023b.
Wu, H.. Chang, K., Wu, Y., and Lee, H. Speechgen: Un- locking the generative power of speech language mod- els with prompts. CoRR, abs/2306.02200 10.48 1044850/ARRIVV.33660022
2023. doi: URL https: //doi. 2306. 02207.
Zhou, C., Yu, L., Babu, A., Tirumala, K., Yasunaga, M., Shamis, L., Kahn, J., Ma, X., Zettlemoyer, L.,, and Levy, O. Transfusion: Predict the next token and diffuse images with one multi-modal model, 2024.
org/10. 4
Wu, S., Fei, H., Qu, L., Ji, W., and Chua, T. Next-gpt: any-to-any multimodal LLM. In Forty-first International Conference on Machine Learning, ICML. 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
Zhou, X., Wang, J., Cui, Z., Zhang, S., Yan, Z., Zhou, J., and Zhou, C. Mmspeech: Multi-modal multi-task encoder- decoder pre-training for speech recognition. In 24th An- nual Conference of the International Speech Communi- cation Association, Interspeech 2023, Dublin, Ireland, August 20-24, 2023, pp. 4943 4943-4947. ISCA, 2023.
Xie, J., Mao, W. Bai, Z., Zhang, J. Wang, W., Lin, K. Q., Gu, Y., Chen, Z., Yang, Z., and Shou, M. Z. Show-o: One single transformer to unify multimodal understanding and generation, 2024.
14

---

NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pan Prediction
A. Other Related Works
Autoregressive Generative Models. The autoregressive generative modeling has achieved remarkable success natural language processing, giving rise to variety of powerful LLMs (Sutskever et al., 2014; OpenAI. 2024; 2023; Patel et al., 2023). Inspired by these LLMs, numerous studies have examined the application of autoregressive modeling in other domains, such images (van den Oord al., 2017; Esser et al., 2021; Li et al., 2024; Tian et al., 2024; Lee et al., 2022; Chang et al., 2022), graphs (You et al., 2018), videos (Weissenborn et al., 2020), molecules (Shi et al., 2020; Schwaller et al., 2019) and protein sequences (Madani et al., 2023; Lin et al., 2022). The core idea of autoregressive modeling is to iteratively generate an entire sequence by predicting each part from the preceding segments, a property that makes it particularly well-suited for audio generation and inherently strong in reasoning tasks (Liu et al., 2023c; 2024).
Multi-Modal LLMs. Multi-modal Large Language Models (MM-LLMs) strive to incorporate knowledge from diverse modalities. A key category of MM-LLMs concentrates on developing connectors (Li et al., 2022; 2023; Liu et al., 2023b; Alayrac et al., 2022) that identify knowledge alignment across various modalities. An alternative strategy (Team, 2024; Zhou et al., 2024; Xie et al., 2024) merges all modalities into a cohesive sequence of tokens and utilizes LLMs to sequentially generate them using modified attention masks. These methods (Wu et al., 2024; Su et al., 2023; Fu et al., 2024) even integrate audio as an imput modality, and by simply combining text and audio through MM-LLM techniques, they can address one-direction conditional generation tasks such as speech-to-text translation (e.g.. ASR and spoken language understanding) (Radford et al., 2023; Zhang et al., 2023b; Deshmukh et al., 2023; Arora et al., 2023; Tang et al., 2024; Chu et al., 2024; Zhou et al., 2023; Baevski et al., 2020; Gao et al., 2023) and text-to-speech translation (e.g. TTS) (Elizalde et al., 2023; Liu et al., 2023a; Huang et al., 2023; Nachmani et al., 2023; Yang et al., 2023; Kreuk et al., 2023; Borsos et al., 2023; Copet
et al., 2023; Chen et al., 2024; Anastassiou et al., 2024; Jiang et al., 2023b; Kong et al., 2021; Shen et al., 2024; Casanova et al., 2022; Siuzdak, 2024; Yang et al., 2024; Kharitonov et al., 2023; Le et al., 2023). However, these methods are limited to multi-turn (or even single-turn) QA tasks (where the model produces an answer only after the question is completed, as signaled by the voice-activity-detecion (VAD) module, e.g. special tokens, button pressing and hard tunr-taking interval threshold.) and thereby struggle with real-time voice interaction tasks, which is the primary focus of our work.
B. Streaming Conditional Inference
In order to simulate a real-time user-assistant commu- nication scenario, our speech LMs improved by NTPP should be proficient in conducting conditional inference with streaming user voice input. In this inference set- ting, one speaker's 's voice input is provided as the user, and the model is assigned the task of inferring the other audio channel. This creates a situation that resembles a constrained generation problem. If the inference process strictly follows the training process, then the model should predict st immediately after receiving the speaker's 's voice input sf at time f. However, due to the VQ-VAE tokeniza- tion mechanism, it's not feasible to receive just single speech token from the speaker channel during the stream- ing inference. This is because the VQ- VAE requires complete continuous speech signal input with a specific time length. Therefore, we need to determine when the model should start generating spoken responses upon re- ceiving streaming user input speech tokens. Specifically, the inference process follows the chunk-wise style, con- taining a predetermined À number of tokens. As long as the number of user input tokens reaches À (a chunk of speaker input is given), our model begins to generate predictions until the number of predicted tokens also reaches X (a chunk is filled). This procedure is repeated until the end of user voice inputs (e.g.. the conclusion of the voice- service). Here we omit the RVQ-verision inference mechanism since the only difference is the inclusion of additional depth tokens (if A = 5 in the VQ-based inference, then À = 5/ in the RVQ-based inference).
Streaming Inference with conditional user input
Chunk
Streaming Input
$2
k4
k2
kg
kg
kg
Last
v
v/
vig
V/
VE
Next
Chunk
k
k-
k.
k2
Chunk
Attention
v
v2
9
Figure 9. The figure illustrates the chunk-wise streaming inference process. Within each chunk, (sf, s2, s3 si sg) represents the pro- vided speaker sequence. Their corresponding keys and values are stored in the KV-cache. NTPP sequentially predicts tokens (s), 8h, s),, sh. sg) based on generated query vectors, which are di- rected to the Key-Value (KV) cache through attention computations. Once chunk is filled, the inference proceeds to the next chunk.
15

---

NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token- Tokeen-airr Prediction
C. More Implementation Details and Hyper-Parameter Settings
C.1. Hyper-Paramett Settings
Our model is trained on 16 A100 GPUs, utilizing a cosine annealing learning rate scheduler with minimum learning rate of 4e-6 and maximum learning rate of 4e-4. Each training epoch consists of 40,000 steps, with batch size 64 for each step. During fine-tuning, we use learn rate from 4e-6 5e-5.
C.2. Dialogue Turn-Taking Statisties Evaluation
Turn
IPU
Cranapp
IPU
P.
IPU
IPU
IPU
Turn
Turn
Figure 10. Illustration of tur-takkin events: IPU (Inter-Pauue Pausal Unit), Turn (for Speaker A and Speaker B, resp), P(Within-Speaker Pause), Gap and Overlap.
Our model generates two audio channels at the same time, allowing us to use basic Voice Activity Detection (VAD) tools on the output to gather turn-taking metrics. According to the settings in (Nguyen et al., 2023), an Inter-Pausal Unit (IPU) is a continuous speech segment within one speaker's ' channel, bordered by VAD-detected silences longer than 200ms on both ends. Silence defined as the lack of voice signals on either channel, while overlap refers to segments where voice signals are detected both channels. Silences can be further divided into gaps (between IPUs of different speakers) and pauses (within the same speaker's IPUs). Consecutive IPUs by the same speaker, separated by pause, are merged into single turn. Our analysis will focus measuring the duration distribution of IPUs, gaps, pauses, and overlaps in both the training corpus and the dialogues generated by our various models.
C.3. Reflective Pause Audio Dataset
Prompt for reflective pause
"Hmm..., this question is complicated, need to think about it." "Let me recall, uh..., ycs, we went to the park that day." "You know, that... oh, yes, it's the new restaurant." "I remember he mentioned it,
be last Friday."
"This matter, um... I think we need to discuss it again." "Let me think about it, uh.... yes, that's it." "I'm not sure, um.... maybe need to confirm it again." "This question, um.... think we solve it this way." "Let me think about it again, uh..., yes, remember it." "The one you mentioned, um..., seem to have some impression."
"We need to deal with the budget issue of this project. Um... this problem is a bit complicated, need to think about it." "Do you remember the last time we met? Let me recall, uh..., yes, we went to the park that day."
"Have you heard about the restaurant? You know, that... oh. yes, that new restaurant." Friday."
"When did he tell you news? remember mentioned uh.... it seems to
"Do you have any suggestions about this plan? This matter, uh.... I think we need to discuss it again."
"Can you give me an example? Let me think about it, uh.... ycs, that's 'ss's it."
"Are you sure this data is correct? I' I'm not
uh..., I may need to confirm it again." "How should we deal with this emergency? This problem, uh.... think we can solve it this way."
"Can you explain this concept again? Lei me think about it again, uh.... yes, remember
"Do you know what he is talking about? The one you said, uh.... seem to have some impression."
16

---

NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Paile Prediction
Prompt for GPT score
Content (1-5 points):
point: The response is largely irrelevant, incorrect, fails address the user's query. may be off-topic or provide incorrect information.
2 points: The response is somewhat relevant but lacks accuracy or completeness. It may only partially answer the user's question or include extraneous information.
points: The response is relevant and mostly accurate, but may lack conciseness include unnecessary details that don't It contribute the main point.
points: The response relevant, accurate, and concise, providing a clear answer to the user's question without unnecessary claboration.
5 points:
The response is exceptionally relevant, accurate, and to the point. It directly addresses the user's query in a highly effective and efficient manner, providing exactly the information needed.
Style (1-5 points):
point: The response is poorly suited for speech interaction, possibly including structured elements like lists or heing overly complex, disjointed, or difficult to understand.
The response is somewhat suitable but may be too long, too short, awkwardly phrased, making it less effective in speech interaction context.
2 points:
points: The response is generally suitable for speech interaction, it may have minor issues with length, clarity, or fluency that detract slightly from the overall effectiveness.
4 points: The response is well-suited for speech interaction, with appropriate length, clear language, and natural flow. It is casy to understand when spoken aloud.
points: The response is perfectly suited for speech interaction. is ideal length, highly clear, and flows naturally, making it easy to follow and understand when spoken.
Below are the transcription of user's instruction and models' response:
### [Instruction]: {instruction]
### [Response]: {response}
After evaluating, please output the scores JSON format: ("content": content score, "style": style score). You don't need to provide any explanations.
D. Case Study
Scenario: user engages in a conversation with Parrot, describing an object and asking the medel to identify it. User: Please listen to my description of an object below, and say its name when you have guessed it. The description is: it has four legs, flat surface, and is often used for dining working... guess it might be a table.
Parrot:
Figure Case study of NTPP interrupt human speaking correctly and timely.
To intuitively understand the differences in responses from our models, provide an example in Figure 11. In this scenario. NTPP interrupts the user at the precise moment it has gathered enough information to make an accurate prediction. This capability is a significant departure from current models that would typically wait for the user to finish speaking before responding. The ability to interject appropriately not only demonstrates the model's advanced comprehension skills but also enhances the fluidity and naturalness of the interaction.
17


Reviews:
[{"review_id": "oPDfhB3EzF", "reviewer": "Reviewer_eH4e", "summary": "The paper presents a method for improving the conversational capabilities of speech language models using dual-channel spoken dialogue learning. It introduces a Next-Token-Pair Prediction (NTPP) approach within a decoder-only transformer architecture, enabling the model to predict both speakers' next speech tokens simultaneously. The study evaluates this method using benchmarks and analyzes its performance in turn-taking dynamics, interruptions, and response timing, comparing with existing models such as Moshi and dGSLM.\n\n## update after rebuttal\nI have read the authors' reactions to my review. My recommendation was already very positive and I'm keeping it.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the value of our work. Here are our responses:\n\n**Q1: Textless Pretraining.**\n\nThanks for your question. In our paper, \"textless pretraining\" specifically refers to the fact that neither the first-stage single-channel speech pretraining nor the second-stage dual-channel learning incorporates text alignment, unlike some existing SpeechLM models. We recognize the potential ambiguity and will provide a clearer explanation in the camera-ready version.\n\n**Q2: Section 5.5 Further Clarifications.**\n\nIn our study, we aimed to assess the performance of existing audio models in real-world scenarios, particularly focusing on their ability to discern when to start and stop speaking. This is crucial for improving user experience, as current models often rely on Voice Activity Detection (VAD) to determine speech initiation, which can lead to interruptions whenever any sound is detected.\n\nWe generated 200 thoughtful pauses (e.g., reflective pause or conversational hesitations like \"Hmm... let me think...\") and 200 interruptions (e.g., abrupt interjections: “But wait—[interrupted]”) using GPT-4o, ensuring contextual diversity. Audio was synthesized via ChatTTS with explicit silence annotations. We compared the performance of our model against two other models, Cascaded and Moshi, by measuring the proportion of instances where the VAD model correctly identified whether speech should occur. \n\nTo ensure alignment between generated audio and labels, we employed human judge as the \"gold standard\". The closer this proportion was to human judgment, the better the model's performance. Figure 7 shows that our model is closer to human performance than Cascaded and Moshi. \n\nWe have also added a demonstration of the corresponding audio effects on the [Demo Page](audio-3059.pages.dev).\n\n**Q3: About the two audio samples in demo page.**\n\nWe provide more dialogue examples at [Demo Page](audio-3059.pages.dev), which illustrates the performance of our NTPP model across several key scenarios, including multi-turn dialogues, \"Interruptions & Reflective Pause\" evaluation, and speech continuation. The audio samples demonstrate how NTPP handles these different situations.\n\nEach audio sample on the demo page provides a clear representation of the output dynamics for each channel at any given moment. This setup allows users to observe how the model manages dual-channel audio, particularly in terms of when to initiate and cease speech. By listening to these samples, users can gain insights into the NTPP's ability to handle interruptions and reflective pauses effectively, showcasing its practical application in real-world interactions.\n\n**Q4：About Multi-channel system (more than channels)**\n\nThank you for your suggestions. Expanding our approach to multi-channel conversation modeling is definitely on our roadmap, and our method can naturally be extended to this more complex setting. However, due to the lack of publicly available datasets, we are not yet fully prepared for this extension. Nonetheless, this remains a promising direction for our future work.", "claims_and_evidence": "The claims are well supported by the evidence from the experiments carried out. The details of some of the evaluations are a bit sparse though (see below).\n\nA minor problem is the claim that pretraining is done in a textless fashion. While defensible in a narrow sense, this is misleading as the starting point of the pre-training is a trained textual LLM.", "methods_and_evaluation_criteria": "The evaluations are appropriate in general.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The evaluation in section 5.5 is described only very briefly, lacking most details needed to understand its value. It's not clear what these automatically generated interactions sound like, or how exactly participants were asked to rate them.", "relation_to_broader_scientific_literature": "The paper's main contribution is the introduction of a decoder only modeling for dual channel speech data via next token pair prediction.", "essential_references_not_discussed": "None identified.", "other_strengths_and_weaknesses": "I don't understand what the two audio samples on the demo page are supposed to illustrate. They don't seem related to modeling dual channel audio in any obvious way.", "other_comments_or_suggestions": "It may be useful to discuss how feasible it would be to extend this approach to more than two channels. Conversations with more than two participants are common, and people handle then easily, and thus thus capability would be useful to have in a dialog system.", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iBLYy0VL8g", "reviewer": "Reviewer_muk3", "summary": "This paper proposes a next-token-pair prediction approach for modelling a dual-channel streamable Speech LM. The authors propose to use an autoregressive LM to model both speakers in a conversation, predicting token pairs from both channels at each timestep. The model is trained using a two-stage pipeline and compared against seminal works like dGSLM and Moshi.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Claims and Evidence\n**Q1: The inefficiency of Encoder-decoder architectures**\n\nThe efficiency of decoder-only model is supported by various literature. For example, FlashAttention (Dao et al., 2022) and Parallelized decoding (Kumar et al., 2020). They show how decoder-only models optimize memory and speed compared to encoder-decoder alternatives. \n\n**Q3: StoryCloze linguistic quality evaluation**\n\n| Model |  sStoryCloze$\\uparrow$ |\n|---|---|\n|Spirit-LM| 61.0 |\n|Moshi| 60.9 |\n|NTPP| 61.4 |\n\n# Experiments Designs or Analyses\n**Q1: The comparison between Moshi, NTPP our other approaches.**\n\nWe illustrate the comparison across different approaches through the following table:\n| Different Models | Speaker-Independent | Encoder-free | VAD-free | Single KVCache | End-to-End |\n|---|---|---|---|---|---|\n| dGSLM | $\\checkmark$ |  | $\\checkmark$ | | $\\checkmark$|\n| LSLM |  |  | | $\\checkmark$ |$\\checkmark$|\n| Moshi |  |  | $\\checkmark$ |  |$\\checkmark$|\n| **NTPP(ours)** | $\\checkmark$ | $\\checkmark$ | $\\checkmark$ | $\\checkmark$ |$\\checkmark$| \n\nIn general, only our NTPP simultaneously possess the five important properties compared to other approaches. Moshi is not speaker-independent and not encoder-free. Additionally, it requires two KV caches. \n(1) This Moshi architecture is not “speaker-independent”. dGSLM indicates that the model should learn the joint distribution $p(s^{b},s^{a})$ instead of any conditional distribution $p(s^{b}|s^{a})$ or $p(s^{a}|s^{b})$. To satisfy this property, dGLSM adopts a two-tower transformer that follows the Siamese encoder-decoder architecture to estimate $p(s^{b},s^{a})$. Moshi and LSLM are only learning the conditional distribution $p(s^{b}|s^{a})$, which inherently has the generalization issue when we just simply switch speaker's roles. Also, this problem cannot be easily solved by training the paired data alternatively since learning $p(s^{b}|s^{a})$ and $p(s^{a}|s^{b})$ sequentially is not equivalent to learning $p(s^{b},s^{a})$ directly. (2) Moshi is not an encoder-free model since it adopts the RQ-transofmer architecture. Further clarifications are in the Reviewer Ubh3 Q1 response. (3) Moshi requires storing two separate KV caches for its two decoder-only transformers, reducing memory efficiency. In contrast, our NTPP, as a typical decoder-only architecture, requires only a single KV cache, making it more memory-efficient.\n\n**Q2: Two-Stage Ablation Studies**\n\nWe refer this reviewer to read **Reviewer Ubh3** Q7. \n\n**Q3 & 4: Figure 7 judge and more clarifications on baselines and evaluation metrics in Table 1 & Table 2.**\n\nTo save space, we refer this reviewer to read **Reviewer Ubh3** Q3.\n\n**Q6: Llama-Omni and SpeechGPT Evaluations.**\n\nLlama-Omni and SpeechGPT, as single-turn QA audio model, face incompatibilities with these real-time streaming multi-turn conversational benchmarks. Fisher and CANDOR require real-time handling of dynamic context shifts, interruptions (barge-ins), and mid-conversation pauses—capabilities inherently absent in VAD-dependent architectures.\n\n**Supplementary Material & Suggestions**\n\nWe have updated the demo and code [Page](audio-3059.pages.dev). Upon acceptance, we will release the full code and model weight publicly and create a well-organized project page for it. The reported digit for the cascaded model should be 4.3 instead of 1.3 (a typo). We will promptly correct this in the revised version. \n\n# Questions\n**Q1-Q2: RVQ and Vocoder Implementations.**\n\nRegarding the RVQ tokenizer and Vocoder, we followed the training settings as specified in Soundstream (Neil et al. 2021) and HifiGAN (Jungil et al. 2020), respectively. Additionally, we have updated the training code with detailed training procedures.\n\n**Q3: Cascaded models and Table 1 Explanation** \n\nSince the performance of the Cascaded model relies on VAD, we use \"Cascaded model\" as a general term to represent all non-interactive models, including multi-modal approaches and LLM-based cascading models. This class models do not directly learn the generation of dual-channel speech, so we do not introduce them in details. We will add short descriptions in the revised version. For the interpretation of NTPP subscripts in Table 1, please refer the Q3 response for the Reviewer Ubh3. \n\n**Q4-Q6: Dataset details, Moshi Evaluation Details and Tokenizer comparison**:\n\nWe put the corresponding content in [Page](audio-3059.pages.dev).\n\n**Q7: Inference Latency Analysis**\n\nTo save space, we refer this reviewer to read Q4 response for the Reviewer Ubh3. \n\n**Q8: start tokens.**\nInstruction format:\n```\n<bos><Model_0_0>...<Model_0_k><Human_0_0>...<Human_0_k><eos>\n```\n\n**Q9: Figure 6 Discussions & Q10: Further explanations on positional encoding**\n\nWe put Figure 6 discussions and further explanations on positional encoding in [Page](audio-3059.pages.dev).\n\nWe hope the above responses address your concerns. If you have further questions, please feel free to ask.", "claims_and_evidence": "The paper makes several claims regarding the advantages of their proposed architecture. However, there are some issues where the claims are not fully supported by evidence:\n\n1. Encoder-decoder inefficiency (Lines 70-72): The authors claim that encoder-decoder models are inefficient and not scalable but provide no justification or supporting references. Given that Moshi, which (according to the authors) also follows an encoder-decoder structure, achieves the same goals as this paper, a more detailed explanation and citations are needed to support their claims.\n\n2. (Lines 90-105): The authors list four advantages of their approach, but Moshi already exhibits three of these (points 2, 3, and 4) during pretraining. The differences between NTPP and Moshi should be more clearly articulated. Also, Moshi too uses decoder-only architecture as the inference is done by Helium LLM.\n\n3.The performance of the RVQ-based tokenizer is not compared against Mimi or other streaming-compatible tokenizers, making it difficult to assess its effectiveness.\n\n4. The performance improvements claimed in Table 1 and 2 cannot be properly verified due to insufficient details about baseline models (including the \"cascaded\" model) and evaluation methodology.\n\n5. The paper highlights that its approach outperforms Moshi in inference latency as number of turn-taking increase, but since Moshi uses a Helium LLM with a smaller context window than Llama 3.1 8B, this might be an unfair comparison.", "methods_and_evaluation_criteria": "The authors use relevant benchmarks like IPU, but some important details are missing:\n\n1. The 14k-hour dataset composition is not clearly specified: Which three datasets were used? What is their language composition? Are they purely single-channel or multi-channel datasets?\n2. The paper compares against Mistral-7B and Gemma-7B, but Qwen 2.5 would have been a more appropriate choice due to its stronger audio capabilities.\n3. The paper does not use StoryCloze or ZeroSpeech Challenge. Other works like dGSLM and Moshi use these, so their absence weakens the evaluation. \n4. The distinction between different NTPP variants (0.1, 0.5, and 0.9) in Table 1 is never explained, making it impossible to interpret these results meaningfully. There's also no mention of cascaded model anywhere in the main text.\n\nMany more crucial details are missing from the paper which make it difficult to assess the validity of claims.", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "1. The comparison against Moshi is unclear, especially given that the open-source Moshi checkpoint is fine-tuned on one speaker and always begins with a welcome message. The paper doesn't specify how this was handled while conducting evals.\n2. The \"one-stage\" vs. \"two-stage\" ablation study in section 5.8 lacks clear definition. It's not clear to me what the one-stage approach is - details are missing from the paper.\n3. Regarding Figure 7 - it's not clear who the \"judge\" is or the methodology used, making it difficult to assess the objectivity of these results.\n4. Vocoder streaming compatibility is not addressed. How did the authors make the vocoder compatible for streaming?\n5. Figure 6 is included but never discussed in the text and lacks a legend, making its purpose and meaning unclear.\n6. Table 2 has limited comparisons. I would have also wanted to see comparison against Llama-Omni or SpeechGPT.", "relation_to_broader_scientific_literature": "There has been a huge interest in speech LMs, especially ones that can process dual-channels like Moshi, GPT 4o, etc. These models have application in Voice Agents and hence, this paper too tries to make a contribution towards such models. I appreciate the authors' intention towards open-sourcing the codebase and checkpoints.", "essential_references_not_discussed": "-", "other_strengths_and_weaknesses": "## Strengths:\n- Proposed architecture looks simple to implement and can also work with new LLMs out-of-the-box. \n\n## Weaknesses:\n\n- Key implementation details are missing, making it difficult verify the evals and draw comparisons against Moshi.\n- There are several references and clarity issues. Moreover, figure 6 is added but never discussed. [See \"Questions for Authors\" section below]\n- I also have concerns around RVQ and Vocoder [See \"Questions for Authors\" section below]\n- No standard benchmarks like StoryCloze or ZeroSpeech challenges were used for linguistic quality evaluation", "other_comments_or_suggestions": "1. Lines 352-353:  Text says Appendix 1, but this seems to be missing (likely Appendix C.2). The reference should be fixed, and additional missing details should be included.\n2. In Table 1, cascaded model has the lowest delta IPU of 1.3s yet the NTPP's values are highlighted.\n3. Line 186, column 2. Typo: \"modelling Figure 1\" Wrong reference. Should be Figure 3.5.", "questions_for_authors": "1. The RVQ implementation has ambiguity: it's unclear how Z_a and Z_b are handled in the RVQ case, specifically whether W_q is multiplied by d (calculated in equation 12) and how the model learns different embedding values for different speech tokens.\n2. What are the three speech datasets that make up the 14,000 hours of training data? Please provide details about their language compositions and other characteristics.\n3. What is the \"cascaded model\" referenced in Table 1, and how was it implemented? Also, please clarify what NTPP 0.1, 0.5, and 0.9 represent in this table.\n4. How did you evaluate Moshi given that the open-source checkpoint is fine-tuned on one speaker (moshiko / moshika) and always begins with a welcome message?\n5. How does your RVQ tokenizer compare to other streaming-compatible tokenizers like Mimi in terms of performance? What was the data used to train it?\n6. Is your vocoder streaming-compatible? What are its encoding/decoding rates and how many frames of audio are synthesized in each time step?\n7. Could the latency differences with Moshi be attributed to differences in context window size rather than architectural advantages of your approach?\n8. What start tokens are used, and how exactly did you prepare the pretraining and finetuning data?\n9. What does Figure 6 represent, and why is it not discussed in the main text?\n10. Lines 293-294 say \"To encode the relative positional information of tokens, all three leverage rotary positional encoding (Su et al., 2024a).\" - Can authors elaborate more on this? Isn't equation 8, 9 and 12 being used for embeddings?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aLLyPCuCeB", "reviewer": "Reviewer_Ubh3", "summary": "This paper introduces PARROT, a system designed to handle dual-channel spoken dialog using large language models (LLMs). Authors highlight importance of capturing conversational features such as overlaps, pauses, and interruptions, to provide more realistic spoken interactions.\nBuilding upon previous work like dGSLM and Moshi, the authors explore how to utilize dual-channel speech data within modern LLMs. \nThe core of their approach is a Next-Token-Pair Prediction (NTPP) paradigm, where a decoder-only transformer is trained to predict the next pair of speech tokens based on past dialog (it predicts at time t the next token pair (a,b) of both channels: user channel and bot channel). Their dual-channel transformer architecture can also benefit from recent KV-cache optimizations for lower inference latency.\nThey compare PARROT with existing methods (dGSLM & Moshi) along metrics/tasks such as conversation event simulation, interruption response success rate, and inference latency.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1, Why we classify Moshi as a encoder-decoder model** \n\nThanks for raising this important and insightful question. While Moshi utilizes the Helium LLM as its temporal transformer—a decoder-only architecture—the inclusion of the RQ-Transformer introduces a spatial transformer component, which deviates from a standard decoder-only structure for several reasons (see Figure 3 on page 13 and Figure 1 on page 7 in Moshi’s paper). First, the RQ-Transformer employs a spatial transformer that encodes input structure information before the autoregressive decoding process. This encoding step explicitly transforms the input data, making it functionally similar to an encoder stage. Second, instead of directly processing tokens in a purely decoder-based manner, the RQ-Transformer constructs a context vector through the spatial transformer. This context vector resembles the latent representations typically computed by encoders in encoder-decoder architectures. Overall, Moshi’s architecture can be viewed as “two decoder-only architectures connected via an encoder-decoder module.” Consequently, unlike a purely decoder-only approach like our NTPP, Moshi requires maintaining two separate KV caches for its two decoder-only transformers. Therefore, Moshi is not a standard decoder-only architecture; instead, we classify it as an encoder-decoder model from a dual-channel modeling perspective. We refer the reviewer to our response to Reviewer muk3 for further details on this matter.\n\n**Q2:The major weaknesses of Moshi architectures compared to our NTPP and previous methods**\n\nWe refer this reviewer to read the Experiments Designs or Analyses Q1 response for the Reviewer muk3 for further details about this question. \n\n**Q3: Further Clarifications on Table 1**\n\nWe followed the evaluation settings for quality and statistical analysis of generated dialogues used in Moshi[^1]. Specifically, we selected 1,000 random 10-second prompts from the Fisher dataset and utilized NTPP and other baseline models to generate dual-channel speech continuations. For each prompt of NTPP, we generated 32 continuations across three different temperature settings [0.1, 0.5, 0.9], as temperature significantly impacts the results.\n\n**Q4: Inference Latency Explanations**\n\nWe employ Voice Activity Detection (VAD) as the latency measurement benchmark, calculating system response delays exclusively during effective speech token segments. This approach more accurately simulates real-world conversational user experience. Our experimental results demonstrate that the latency improvement in multi-round dialogue primarily stems from our novel Next-token-pair Prediction mechanism: By independently calculating the attention values of dual-channel tokens, our method achieves fine-grained time alignment. In contrast, Moshi’s strategy of directly summing dual-channel token embeddings risks introducing inter-channel information interference. \n\n**Q5: Further Clarifications on Human Evaluation Experiments.**\n\nWe followed the evaluation settings of Moshi and conducted the evaluation study with 25 annotators who possess native-level English proficiency. These annotators were recruited through a combination of academic networks and online platforms, ensuring a diverse range of backgrounds and experiences. Each annotator was thoroughly briefed on the evaluation criteria and the objectives of the study. We adapted the Mean Opinion Score (MOS) protocol, utilizing a 5-point Likert scale, to assess two key aspects: the Naturalness (N-MOS) of turn-taking and the Meaningfulness (M-MOS) of dialogue content. This approach allowed us to gather quantitative data on the perceived quality of the generated dialogues.\n\n**Q6: More Representative Demos.**\n\nWe provide more dialogue examples at [Demo Page](audio-3059.pages.dev), which illustrates the performance of our NTPP model across several key scenarios, including multi-turn dialogues, \"Interruptions & Reflective Pause\" evaluation, and speech continuation. The audio samples demonstrate how NTPP handles these different situations. \n\n**Q7: More Explanations on Two-Stage Training.**\n\nWe describe our two-stage training process in Section 5.1 (Dataset). The first stage involves training a text-based LLM on single-channel speech token sequences. In the second stage, we introduce our newly proposed NTPP paradigm for dual-channel speech learning, building upon the SpeechLM obtained in the first stage. The ablation results in Figure 9 aim to support a simple claim: the first-stage training is essential before the second-stage NTPP training. We have further clarified this ablation study with a slightly modified figure [Figure 9](audio-3059.pages.dev/figure9). This updated figure includes three curves: Full Two-Stage NTPP (NTPP), NTPP without the first stage (NTPP w/o-1), and NTPP without the second stage (NTPP w/o-2).\n\nPlease feel free to ask if you have any further question.", "claims_and_evidence": "Yes they mostly are but there is a problematic claim i describe below:\n\nlines 68 & 134 => the paper's comparison to previous work (e.g., Moshi) may need revision for accuracy.\n\nI think there is a potential issue in the paper’s positioning of prior work. Specifically, Moshi is inaccurately classified as following an encoder-decoder architecture when it actually employs a decoder-only model (Moshi is powered by Helium, a 7B parameter transformer-based language model designed for spoken dialogue. Helium follows a decoder-only structure, similar to GPT models, meaning it generates responses autoregressively without a separate encoder).\n\n=>this is to me an important misunderstanding of a related work which weakens the paper positioning;", "methods_and_evaluation_criteria": "Yes they are (although i have a few concerns expressed in the 'experiments & design' section)", "theoretical_claims": "The paper is mostly experimental, formalizations mostly recap previous work", "experimental_designs_or_analyses": "-The results in Table 1 need more detail—it's unclear what 0.1 or 0.9 represent in NTPP_0.1 or NTPP_0.9. Additionally, it's difficult to assess whether the differences shown in Table 1 (compared to dGSLM) and Figure 7 (compared to Moshi) are significant or meaningful. A clearer explanation   would help in understanding the impact.\n\n-Inference latency is better => ok but where does it come from ? From the authors’ architecture or from leveraging  more recent KV-cache optimization techniques ?\n\n-The most convincing exp to me  is the human evaluation, which measures turn-taking naturalness and content meaningfulness. However, more details on the evaluation process would be helpful—such as how listeners were recruited, their backgrounds, etc.", "relation_to_broader_scientific_literature": "It's not clear how original this work is compared to Moshi (Defossez, 2024). Their Next-Token-Pair Prediction (NTPP) method is new, but the idea of handling dual-channel speech in a single sequence using subsequences is not (done in Moshi technical report for instance).  Authors should more clearly explain what sets their approach apart.", "essential_references_not_discussed": "ok", "other_strengths_and_weaknesses": "i covered them already in the previous sections", "other_comments_or_suggestions": "no", "questions_for_authors": "section 5.8 ablation:\n-can you elaborate more on what the two stage training approach is exactly (that seems to be better) => should be more detailed ?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}]

Author rebuttal:

Meta-review: This paper proposes a method to enhance the conversational abilities of speech language models through dual-channel spoken dialogue learning. The core innovation is the Next-Token-Pair Prediction (NTPP) approach, implemented within a decoder-only Transformer, which allows the model to simultaneously predict the next speech tokens for both speakers. 

All reviewers agreed that the method offers valuable contributions to the spoken language modeling community. However, concerns were raised about its similarity to prior work, particularly Moshi. The authors addressed most of these concerns during the rebuttal, which is greatly appreciated. I encourage the authors to incorporate the additional results into the final version and to more clearly articulate how their approach differs from previous methods—especially those involving speech-text token mixing in Speech LMs and more exact definition of the encoder-decoder vs. decoder only architectures.
Decision: Accept (poster)