Parrot: Seamless Spoken Dialogue Interaction with Double-Channel Large Language Models

Authors: Qichao Wang, Ziqiao Meng, Wenqian Cui, Yifei Zhang, Pengcheng Wu, Bingzhe Wu, Zibin Zheng, Irwin King, Liang Chen, Peilin Zhao
Primary area: foundation or frontier models, including LLMs
Keywords: Speech Language Models, Generative Spoken Dialogue Language Modeling

Abstract:
Recent advancements in large language models (LLMs) have demonstrated significant potential in enhancing real-time spoken interactions. Presently, open-source methodologies predominantly depend on intermediate generative text-based translations to manage real-time spoken dialogues. However, these techniques often struggle with providing seamless interactions that involve real-time streaming audio inputs. In this research, we unveil an innovative spoken dialogue language model, Parrot, distinguished by its unique pre-training and supervised fine-tuning (SFT) pipeline. This pipeline deviates from conventional methodologies by utilizing both single-channel audio data and double-channel spoken dialogue data to train the textless speech language model. During pre-training, we transmute single-channel audio input into a sequence of discrete tokens, thereby instructing the LLM to identify audio tokens via next-token predictions. In the SFT phase, we pioneer a novel approach to double-channel generative spoken dialogue language modeling with a unique ``next-token-pair prediction" objective, facilitating the LLM's comprehension of natural human conversations. Our inventive pipeline equips the LLM to produce spoken interactions that are more natural and fluid than those generated by previous text-based approaches, as substantiated by thorough evaluations.

Decision: reject
Reviewer scores: 3, 5, 3, 3, 1

Reviews and discussion:
<thread>
creation date:
1729738374105

thread_type:
official_review

<thread_object>
creator: reviewer_2fhe

<content>
summary:
This work presents an approach to spoken dialogue interaction through the development of a large language model (LLM) named Parrot. The authors propose a two-stage training pipeline that leverages single-channel audio data for pre-training and double-channel audio data for supervised fine-tuning (SFT). The key innovation lies in the "next-token-pair prediction" paradigm, which aims to enhance the model's ability to comprehend and generate natural human conversations in real-time.

strengths:
1. This work explores an interesting topic, namely dual-channel speech input modeling. Previous work has focused on user-assistant turn-based interactions, but in real interactions, immediate processing is required. Therefore, there is a need for a listening channel to continuously process user's speech input.

2. The author made reasonable reviews and citations to related work.

weaknesses:
1. The writing of this paper is very poor, with some tables and figures not being referenced in the main text, and inconsistent statements in the context, making the entire article very difficult to understand. (see Questions for details)

2. The structure design of the model and "next-token-pair prediction" paradigm are not well-motivated, as there is a significant gap between training and inference stages.

    -  The author inputs listening tokens and speaking tokens as pairs into the LLM, which doubles the context length. LSLM [1] has demonstrated the effectiveness of modeling double-channel through embedding fusion and is more efficient in context. Therefore, the author needs to prove the effectiveness of their modeling approach
    -  In the training process, the prediction of the next token depends on the previous predicted tokens. But in the inference process, according to section 3.3, listening tokens are sent to LLM in chunks. At this point, predicting the next speaking tokens no longer depends on the previous predicted speaking tokens. This inference method will disrupt the casual modeling during training.

3. The lack of evaluation details.

      -  In section 4.5.1, LLAMA-Omini and SpeechGPT do not have the ability to interrupt and pause. How does Reflective Pause and Interruption caculated?

      -  In section 4.5.2, all these evaluation metrics cannot reflect the linguistic quality of the model. For an open-ended instruction-following task, there is no evaluation of response quality.

      -  In section 4.6, the evaluation of speech-to-text tasks, such as ASR and ST, is discussed. However, this model is a textless speech-to-speech model and does not have text generation capability, so it is unclear how these tasks are evaluated.

4. Some results are very strange, which it‘s hard for me to believe that it's real.

      -  According to table 1, the model is only trained on English-Only speech data, but in figure 8(c), the model can perform Chinese ASR task on aishell, SER task on MELD, and sound-related task on AIRBench-Sound. This is very strange.
      -  In appendix A.4.2, the larger latency, the worse performance. There is no explanation for this strange results.

questions:
1. In section 4.1, the authors claim to have used 14,000 hours of single-channel for pretraining and 2,200 hours of double-channel for sft. However, in table 1, there are over 70,000 hours of single-channel data and 2,000 hours of double-channel data. At the same time, InstructS2S-200K is a single-channel data, how do you use it in stage 2?

2. In section 4.5.3, the author claims that the embedding of two channels is in different spaces, but in Figure 7, there are two completely different feature spaces, with the right side appearing consistent with the author's statement, so what is the left side of the figure?

3. In Appendix A.6.3, the author lists "prompt for gpt score," but there is no mention in the main text of the need for gpt evaluation.

4. How much data was used to train the speech tokenizer? And what's the output rate of the speech tokenizer?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730100096575

thread_type:
official_review

<thread_object>
creator: reviewer_ubhb

<content>
summary:
The Parrot framework aims to address double-channel spoken dialogue modeling by implementing a pipeline that includes pre-training on single-channel audio and fine-tuning on double-channel audio data. The framework introduces a "next-token-pair prediction" approach within a decoder-only model architecture. However, the proposed solution lacks substantial originality and leaves questionable details for evaluation, which ultimately weakens the paper.

strengths:
The framework presents a relevant approach to double-channel spoken dialogue, aiming to improve latency by avoiding text generation stages, which could contribute to real-time applications.

weaknesses:
* Minimal Novelty: The framework essentially serves as a decoder-only adaptation of dGSLM combined with a textually pre-trained model (TWIST), offering limited technical innovation. This incremental change does not justify the need for a separate model or paper.

* No Human Evaluation: The absence of human assessment significantly limits the validity of the framework's claims about improving conversational fluidity and natural interaction, which are central to spoken dialogue applications.

* Lack of Established Benchmark Comparisons: Despite the existence of standard benchmarks like ZeroSpeech [1] and StoryCloze [2] for textless spoken language models, the paper does not include comparisons with these datasets. This omission raises concerns about the thoroughness of the experimental validation.

* Poorly Defined Evaluation Methodology: The evaluation details, especially for the reflective pause and interruption response accuracy (Section 4.5.1), are incomplete. Key information, such as the evaluation metric definitions like interaction accuracy, is missing, making it hard to verify the claimed improvements.

* Insufficient Explanation of Key Evaluation Components: See the comments below.

1. Zerospeech 2021 benchmark, https://arxiv.org/abs/2011.11588
2. StoryCloze, https://arxiv.org/abs/2305.13009

questions:
* L473-476: The discussion on "layer-wise" and "consistent" channel embeddings is unclear. These terms appear only once and lack explanation, leaving their meanings and relevance ambiguous to the reader.
* L1162-1187: The purpose of the GPT score is not clear. It is mentioned in the appendix, but its usage and significance are not explained in the main text, making it difficult to understand its role in the evaluation.
8 483-505: Since AIR-bench questions are in text format, it is unclear how the proposed model, which is audio-based, handles text input. Without further clarification, it is difficult to interpret the evaluation results accurately.
* Section A.1: The essential distinctions between the proposed method and closely related prior work should be concisely summarized in the main related works section. The current related works section lacks sufficient comparison, especially with the most relevant prior methods.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730601847502

thread_type:
official_review

<thread_object>
creator: reviewer_w4cg

<content>
summary:
The paper introduces a spoken dialogue model, Parrot, which leverages large-scale single-channel audio data for pre-training and moderate-scale dual-channel dialogue data for supervised fine-tuning. The authors also propose a “next token-pair prediction” approach for spoken dialogue language modeling. The study claims that Parrot facilitates more natural and fluid conversations compared to Dialog GSLM and traditional cascaded spoken dialogue systems.

strengths:
1. The authors state that they will open-source their training and inference framework, which would be a valuable contribution to the community, especially since existing end-to-end spoken dialogue models either do not disclose their training methodologies (e.g., Moshi) or lack sufficient documentation (e.g., Mini-OMNI).
2. The authors also evaluate their approach on turn-taking properties, such as recognizing when the user has paused and determining appropriate moments to interrupt the user.

weaknesses:
1. The paper lacks specific details on the cascaded system used for evaluation. DialogGSLM used a relatively weak cascaded system in their work, so it would strengthen this study if the authors evaluated a cascaded system with state-of-the-art ASR, LLM, and TTS models, such as Hugging Face's Speech-to-Speech (https://github.com/huggingface/speech-to-speech), for a fairer comparison. Additionally, latency in cascaded systems can be minimized by parallel threading each module, as demonstrated in the Hugging Face repository.

2. Given the primary objective of this work is to enable engaging and naturally fluid conversations, a human study would be valuable for a thorough evaluation—similar to assessments in text-based dialogue systems. When motivating their approach, the authors point out limitations in cascaded methods; more analysis is needed to clarify if these limitations genuinely impact user experience in human-AI conversations. User study could focus on user satisfaction compared to baseline systems as well as other auxiliary metrics such as naturalness of turn-taking and semantic coherence of response.

3. Although the code is public, it lacks sufficient documentation, and I was unable to run their demo. For example, an inference.py file appears to be missing. 

4. The paper would benefit from further discussion on design choices:

a. I was curious about the choice to use a single codebook. Table 4’s synthesized audio quality results lack clarity on the dataset used. Multiple codebooks often improve results—did the authors experiment with this option?

b. The motivation for using next-token pair prediction, rather than multi-stream prediction as seen in Moshi, is also unclear. While streaming audio generation is effective, the paper doesn’t address the length limits for audio modeling. Given the potential length of audio sequences, does the model implement techniques to reduce sequence length?

c. The authors use only human-human conversation data for supervised fine-tuning. Adding human-AI conversation data, even synthetic, could be beneficial as there are nuance differences in how humans communicate with AI versus other humans.

5. Clarity

a. Section 4.5.1 is difficult to follow. My understanding is that the ground truth is generated using GPT-4. Did the authors verify this ground truth with human judgments? I would recommend the authors to clarify their methodology for generating and validating the ground truth data, and include this information in the paper.

b. Section 4.5.3 presents an interesting observation. Do the authors have any intuition as to why this occurs?

6. In addition to evaluating turn-taking properties, the paper could also benefit from evaluating "speaking-while-listening" capabilities (https://arxiv.org/pdf/2408.02622).

7. Did the authors assess whether the LLM undergoes catastrophic forgetting when fine-tuned on audio data? For example, is Parrot still capable of instruction-following or answering factual questions at a level comparable to LLAMA 3.1?

8. In section A.1, the paper makes claims about prior works that lack experimental or discussion-based support. For instance, the statement that RQ transformers reduce model efficiency is not substantiated. Unsubstantiated claims should be avoided, as they could lead to incorrect community conclusions.

questions:
Check weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730699663451

thread_type:
official_review

<thread_object>
creator: reviewer_8t93

<content>
summary:
**Paper Summary**:
- The authors present a textless spoken dialogue language model and corresponding training pipelines. The work involves two training stages: during pre-training, the LLM model is used to instruct the prediction of the next audio token, and in SFT, a double-channel mechanism is applied to predict the next token pair. Ablation studies demonstrate that this method yields more natural and fluid dialogue generation compared to baselines.

strengths:
**Summary Of Strengths**:
- Comprehensive Presentation: The paper includes all necessary sections, diagrams, and tables to demonstrate their contribution, the usefulness of the work, while also acknowledging its limitations.
- Research Direction: The application of LLMs for audio token prediction and the double-channel mechanism are interesting and challenging directions in spoken language modeling.

weaknesses:
**Summary Of Weaknesses**:
- Novelty and Clarity: In the contribution summary, the authors list (1) the Parrot model and its innovative pre-training and SFT pipeline, (2) the paradigm of double-channel spoken language modeling, and (3) the evaluations. In my opinion, textless spoken language models are not particularly rare, especially in machine translation tasks (e.g., [Seamless: Multilingual Expressive and Streaming Speech Translation](https://arxiv.org/abs/2312.05187), [UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units](https://arxiv.org/abs/2212.08055)). Additionally, the pipeline and the double-channel modeling mechanism appear to be two perspectives on the same concept. It is difficult to consider the potential readiness for future exploration and evaluation as innovative contributions.
- Limitations: Besides the limitation regarding the inability to integrate audio tokens, further analysis of the audio tokenizer should be elaborated on (e.g., how it impacts computational efficiency or downstream inference latency). More case studies should also be included to demonstrate the effectiveness of this work, as A.5 seems unfinished.

questions:
As listed in the limitation section
</content>
</thread_object>

</thread>
<thread>
creation date:
1730799004331

thread_type:
official_review

<thread_object>
creator: reviewer_z65b

<content>
summary:
This paper introduces Parrot, an audio LLMs models designed for modeling two-channel dialog audio. Parrot is built by fine-tuning an off-the-shelf text LLM on tokenized audio. At first, a single channel audio is used, as given in multiple standard datasets (eg LibriLight). Next, it is fine-tuned on a dialog dataset, Fisher. Here, the model is trained to simultaneously predict two tokens, one for each channel. According to the evaluation, this leads to a more natural dialog flow than in baseline model, dGSLM.

strengths:
* The paper proposes a simplification of a two-tower approach described by dGSLM, by using a single Transformer predicting two tokens.

weaknesses:
1. In a few places, the paper misrepresents related work. Examples:
  * "the academic community primarily utilizes open-sourced models (Zhang et al., 2023a; Xie & Wu, 2024; Rubenstein et al., 2023; Huang et al., 2024; Wang et al., 2023a; Nachmani et al., 2024; Wang et al., 2023b) following a cascading approach." For instance, [Rubenstein et al., 2023] and [Nachmani et al., 2024] are not cascaded models. They are also not open-sourced.
 * "Much of the prior research has utilized the encoder-decoder architecture to enhance pre-training (Borsos et al., 2023; Lakhotia et al., 2021; Kharitonov et al., 2022; Polyak et al., 2021; Chen et al., 2023; 2022; Hsu et al., 2021; Zeghidour et al., 2022; Defossez et al., 2023; Agostinelli et al., 2023; Ao et al., ´ 2022; Tang et al., 2022; Wu et al., 2023).". ** The first three models are decoder-only LMs, and perhaps many others.

2. The evaluation of the proposed model is limited and is insufficiently described. 
 * S4.5.1 evaluates some pause-prediction accuracy metrics, but the metrics are not really introduced in the text. These metrics are evaluated on some synthetic data --- is there a reason a hold-out subset of Fisher is not used?
 * AudioQA evaluation and comparison to AudioQWEN-2 is only mentioned in a single sentence without any discussion or description. At the same time, it does require some discussion. What are the metrics used? The AudioQA figure indicates some reasonable performance on CoVost2 and FLEURS, I assume better than AudioQWEN-2. However, the training data does not include speech-to-speech translation examples nor contains non-tier1 languages. Should we assume the model somehow picked it up from purely text base model?
* Audio-QWEN2 reports 90+% accuracy on VocalSound. In Figure 8c it is reported to be below 80%. What is the reason for the difference?

3. The paper should at least mention Moshi https://arxiv.org/abs/2410.00037

questions:
* I would appreciate it if some of the weaknesses related to describing the evaluation study are resolved.

 * "encodes each second of audio into 30-50 discrete tokens from a codebook of size 2048." Why is the token rate variable?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a novel spoken dialogue model, named Parrot, that is pre-trained with single-channel audio data and then fine-tuned using two-channel dialogue data. While the reviewers list the strengths of the proposed work as usefulness for real applications and release of the code, they also list several weaknesses, such as lack of clarity about experimentation details (e.g., is GPT-4 generating the ground truth), lack of clarity on evaluation methodology and comparisons to benchmarks, and so on. Based on these reviews, the weaknesses overweigh the strengths, the work could be improved to tackle these suggestions, especially the questions related to the evaluation.