A Full-duplex Speech Dialogue Scheme Based On Large Language Model
Authors: Peng Wang
Songshuo Lu
Yaohua Tang
Sijie Yan
Wei Xia
Yuanjun Xiong
Venue: NeurIPS 2024 poster (2024)
Keywords: Speech based conversation; large language models; full duplex; instruction tuning
Primary area: natural_language_processing
TL;DR: This work formalizes the problem of full-duplex voice conversation with LLM and presents a method towards this goal.

Abstract: We present a generative dialogue system capable of operating in a full-duplex manner, allowing for seamless interaction. It is based on a large language model (LLM) carefully aligned to be aware of a perception module, a motor function module, and the concept of a simple finite state machine (called neural FSM) with two states. The perception and motor function modules operate in tandem, allowing the system to speak and listen to the user simultaneously. The LLM generates textual tokens for inquiry responses and makes autonomous decisions to start responding to, wait for, or interrupt the user by emitting control tokens to the neural FSM. All these tasks of the LLM are carried out as next token prediction on a serialized view of the dialogue in real-time. In automatic quality evaluations simulating real-life interaction, the proposed system reduces the average conversation response latency by more than threefold compared with LLM-based half-duplex dialogue systems while responding within less than 500 milliseconds in more than 50% of evaluated interactions. Running an LLM with only 8 billion parameters, our system exhibits an 8% higher interruption precision rate than the best available commercial LLM for voice-based dialogue.

A Full-duplex Speech Dialogue Scheme Based On Large Language Model
Peng Wang"
Songshuo Lu"
Yaohua Tang"
Sijie Yan
Wei Xia
Yuanjun Xiong
MThreads AI
w8ngping&gmaill com, 1usongshuo97@gmail com, tangyachua28@gail il com yysijie@gmai com, weixi aee@gmai 1 com, bitxiongegne com
Abstract
We present a generative dialogue system capable of operating a full-duplex manner, allowing for seamless interaction. It is based on large language model (LLM) carefully aligned to be aware of perception module, a motor function module, and the concept simple finite state machine (called neural FSM) with two states. The perception and motor function modules operate in tandem, allow- ing the system simultaneously speak and listen the user. The LLM generates textual tokens for inquiry responses and makes autonomous decisions to start re- sponding to, wait for, interrupt the user by emitting control tokens to the neural FSM. All these tasks of the LLM are carried out as next token prediction on a serialized view of the dialogue in real-time. In automatic quality evaluations sim- ulating real-life interaction, the proposed system reduces the average conversation response latency by more than 3 folds compared with LLM-based half-duplex di- alogue systems while responding within less than 500 milliseconds in more than 50% evaluated interactions. Running a LLM with only 8 billion parameters, our system exhibits 8% higher interruption precision rate than the best available commercial LLM for voice-based dialogue.
Introduction
In conversation between two persons, one person either speaker listener a time, The lis- tener listens to the speaker's 's speech but is free to interrupt when necessary. The speaker speaks, but either concedes the speech continues when the listener tries to speak. We refer this type of in- teraction as full- dialogue. Instead, most existing chat-enabled large language models (LLMs) Eloundou et all, 2023, Achiam et all 2023]] view dialogue as round-based process, where each participant will produce full sentence before the other party responds, resulting in a half-duplex dialogue. The half duplex dialogue mode is sensible when building text-based chatbot. However, due to the bloated response latency and difficulty properly interrupting the other party, it becomes infeasible for a human-like conversation experience, and thus, full-duplex dialogue capability is desired.
Despite LLMs' success in dramatically improving response quality, full-duplex dialogue is non- trivial to realize when considering LLM alone. A common chat-finetuned LLM [Achiam et ali 2022 Touvron et all 202A does not know the current time nor allow input while generating responses. also has no awareness of any perception or motor function, which humans conveniently possess and operate in parallel to allow them to respond external stimuli in real-time. As shown in Fig. MI tackle this problem by imaging an ideal agent for full-duplex dialogue. It involves one LLM and two
Equual contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

C.SEAAK
LLM
S.LISTEN
M.Speok
perception
motor
(ASR]
Simultaneous
(TTS)
5.SPEAK
M.Liston
Human
Neural FSM
CLISSTEN
Figure I: Left. Overview of the agent design that enables JLM--aase full-duplex dialogue models. The agent is equipped with one LLM,, one perception module, and one motor function module. The latter operates continuously and simultancously to collect input to the LLM and produce voice-based LLM outputs. Right. The LLM operates two-state neural FSM with SPEAK and LISTEN states. At each timestep, the LLM either I) receives an external input token, 2) generates a textual token for speech, 3) produces control token to signal state transition in the neural FSM. This simple workflow enables full-duplex dialogue without any external moderation module.
functional modules. The LLM is aware the perception module, which processes speech input, and the motor function module, which converts textual outputs generated by the LLM to speech. The two functional modules operate continuously regardless of which party the dialogue speaking, allowing both parties speak, respond, and decide to interrupt concede in real-time.
To realize full-duplex dialogue with this agent model, we instruct the LLM view the dialogue as operating a simple finite-state machine (FSM) with two states: 1) SPEAK and 2) LISTEN, which we refer as the neural FSM. At each timestep, the LLM must either: 1) process an external input token, 2) output textual token, or 3) output special control token to signal one of four possible state transitions, as illustrated in Fig. M1 External stimuli, such as human speech collected by the perception module, can also be streamed to the LLM and appended to the LLM's generated token sequence potentially prompt for state transition. Any non-control tokens are immediately delivered to the motor function module to be converted to speech. With this design, the operation of the neural FSM, which governs the dialogue, becomes a natural part of the LLM's auto-regressive generation process. This allows us to perform standard instruction tuning and prompting on any pretrained LLM to enable it for full-duplex dialogue. this work, we present the first systematic implementation of LLM-based full-duplex dialogue framework and evaluate its effectiveness in the following aspects:
Simultaneous two-way interaction. full duplex LLM dialogue system should allow users and the machine to converse concurrently, enabling them to interrupt each other, akin to natural human dialogue, rather than a round-based dialogue.
2. Full autonomy. The dialogue should be content. The L.M needs to make autonomous decisions to halt, interrupt, or ask questions proper timing by emitting control tokens of the neural FSM based on the semantic context.
3. Rapid Response. By processing streaming inputs from the perception module, while the other party is speaking, the system should respond to user inquiries with minimal latency.
We validate the proposed system's 's effectiveness in the above aspects on a curated dataset of human- machine voice interaction and design quantitative metrics regarding the above properties. Compared with state-of-the-ae half-duplex dialogue systems, our approach can reduce the average response latency in dialogue by 3 folds. Without affecting the knowledge and reasoning capability of the I.LMs, it reaches proper response rate 96.7% user interruptions and machine proper interrupt significantly.
precision of 54.7% that outperforms GPT-4o and

---

2 Related work
2.1 Full-duplex dialogue systems
Developing a speech dialogue system (SDS) capable of sustained, empathetic interaction with hu- mans represents pinnacle aspiration in artificial intelligence (AI). Initial systems, largely text- based, sought to mimic human conversational behavior but were constrained by rigid syntactic structures, limited vocabularies, and reliance on manual rules [Weizzebaumm, 1966, Colby, 1975. Wallace, 2009, Li and Mills, 2019]. The advent of Hidden Markov Models (HMMs) expanded these systems' capabilities to handle more fluid speech patterns, yet achieving the nuance and fluidity of human interaction continued to elude them Duang and Rabiner, [191]]]
Recent advancements in dialogue data and deep learning technologies have significantly propelled the evolution of speech dialogue systems (SDS). These advancements have transformed SDS from interfaces into complex, Al-driven conversational agents capable of pro- viding more natural and fluid user experiences. Prominent intelligent assistants such as Amazon Alexi, Sirf. and Google Assistant", along with specialized applications like Google Duplex [Mad and Leviathann 2018| and Xiaolce [Zhou et all, 2020], exemplify this transformation. These systems not only facilitate task-oriented dialogues but also enable long-term emotional engagement, demonstrating the shift towards creating more humane and relatable systems.
simple, command-drive
Another significant technological advancement in this context is the transition from half-duplex to full-duplex communication [Chen all [1944], which allows for the simultaneous transmission and reception of signals, closely mimicking natural conversational flows and significantly reducing la- tency. Early SDS operated primarily in a half-duplex mode, where the system and the user could not speak simultaneously, resulting in stilted interactions with noticeable pauses. Initial improve- ments were made through streaming automatic speech recognition (ASR) and incremental dialogue processing technologies, which enhanced responsiveness and fluidity some extent [Nakano et all 2003]. However, the inability to handle simultaneous speech input and output continued to limit the naturalness of interactions. The transition to full-duplex communication marked a significant lepp forward. Full-duplex systems enable concurrent speech input and output, thereby more closely approximating human conversational behavior. Early implementations faced substantial technical hurdles, including advanced echo cancellation and managing overlapping speech without degrading user experience [Ulin all 2021I, Lin et al), 2022]. Constructing effective SDS necessitates suite of sophisticated components. collectively termed "Conversational Engine Components." including streaming ASR al 2020 al M02II]. text-to-speech (TTS) [Iromkin el all 2011. Ialla
and Alras 2UJE,, intent recognition, and dialogue management, among others [Lin et all 4022]] These components are crucial for creating responsive and engaging conversational agents but in- troduce significant engineering challenges, particularly in maintaining low latency and handling dynamic conversational shifts.
However, achieving low-latency full-duplex systems necessitates tight coordination among dialogue strategy components, such as user query prediction [Madden el all 2013), intent recognition [Vara etall 2010, all 2014]], dialogue management [Dar all 2021], and even many manually crafted rules, making engineering implementation non non-trivial. Despite these efforts, current full-duplex SDS still exhibit considerable error rates and often lack strong contextual awareness in prolonged dialogues.
The advent of large language models (LLMMs) has notably advanced generative AI. Models like ChatGPT 8 exhibit profound capabilities in semantic understanding and logical reasoning, offering a streamlined approach to integrating various conversational components into unified framework, potentially simplifying the construction of SDS [[eckk 20233.. Innovations such as AudioGPT and LLaSM have further expanded these capabilities by integrating audio processing. although lim- itations in speech generation and full-duplex functionality persist [[Huang et all, 2024, Shu et all 2023 ) GPT-4o appears to have achieved full-duplex dialogue with users, but as multimodal large
https://wwH Lexa com/
https://www.dd. com 8iri7
https:/asssstaant
ntra:///ppeaa..omm/haaaaac
http:://peeaai.co
3

---

TTS
Infp
HD Iije
Injo
Hnff
Hi
earth
sarry
S.L
<uar>
mean
C.L.
<ush>>
the
moon
S.5
the
mico
1s
rock
Yess
C.S
that
orbitt
ASR
User
SPEAK
LISTEN
SPEAK
Figure 2: In LLM based full-duplex dialogue system, the LLM operates two state FSM, governing state transitions the dialogue.
model, its training data and implementation details remain undisclosed, rendering replication efforts extremely challenging.
2.2 LLM as evaluators
Recent studies propose directly using L.MS as reference-free NLG evaluators [Euret 2023, Wang etall 2023]. The idea involves leveraging LLMs to assess candidate outputs by evaluating their gen- cration probability the absence of a reference target, presupposing that LLMs have been trained
to assign greater probabilities to texts of superior quality and fluency. [ etle [M0]]] propose GPTScore, a new framework that evaluated texts with generative pre-training models like GPT-
3 [Brown et al, [аλл) demonstrate that this approach effectively allow to achieve what one
desires to evaluate for texts simply by natural language instructions. [Wang et all [M025] conduct a preliminary survey of using ChatGPT as NLG evaluator, experimental results show that compared with previous automatic metrics, ChatGPT achieves state-of-the-an or competitive correlation with human judgments in most cases. Koemi and Federmann [2|2X]] propose
GPT models for evaluating machine translation tasks, provides a first glimpse into the usefulness of large language models for quality assessment of translations. Dettmers et all [2024]] provide detailed analysis of chatbot performance based on both human and GPT-4 [Eloundon ct all 202E, Achaam el all 2nRk), with evaluations showing that GPT-4 evaluations a cheap and reasonable alternative human
evaluation.
Methods
Our proposed system comprises three modules: 1) perception, 2) full-duplex capable LLM, and 3) motor function. The first is the perception module, which picks up user speeches in dialogue. In this work, we implement it with an auto speech recognition model [Gmaa et all, 2020], streaming results on 640mms chunks delivered immediately to the LLM and appended to its token sequence be- fore predicting the next token. From the generated tokens of the LLM, we can observe the states and state transitions of the neural FSM when control token is predicted. Any textual token generated is sent to the motor function module. We implement the motor function module with a text-to-speech model [Kmm et all 2021]. We discuss the definition of neural FSM in Sec. 3.1l the overall architec- ture Sec. 3.2 and training of the full-duplex LLM in Sec. GK!
3.1 The Neural Finite State Machine
We consider the dialogue between two parties: the user and the machine. At any time in full duplex dialogue, one or both parties could be speaking or waiting for others to speak. To model this process for the LLM, we instruct it operate finite state machine (FSM) with two states: I) SPEAK and 2) LISTEN, which represents the LLM's role in the dialogue as it perceives. This FSM referred to as the neural FSM. It has possible state transitions:
SPEAK + SPEAK indicates the model would like to keep speaking when the per- ception module pieks up user speech, we designate control token [C.SPEAK] for this transition called.

---

SPEAK + LISTEN indicates the model determines it is proper to concede its speech the user, with a control token [S.LISTEN]:
LISTEN - LISTEN indicates the model determines the user has not finished the speech and would like to wait for more input, with control token [C.LISTEN]:
LISTEN + SPEAK indicates the model determines it would like to start speaking because either the user has finished the speech or it is proper interrupt the user's ss speech. Its control token is [S.SPEAK].
The LLM could produce one of the four control tokens at any time step to signal state transition. The perception and motor function modules have to observe the state of the neural FSM and choose the proper action. Fig. Q we illustrate the states and transitions of the neural FSM
3.2 Design of the dialogue system
The perception and motor function module observes the state and transitions of the neural FSM and chooses the proper action complete the dialogue system. Given the neural FSM's state, their action is also conditionally independent, allowing them to operate in parallel without synchronization.
The perception module operates an off-the-sheil ASR model. The models run on 640 millisecond timesteps. At each step, it outputs one token chunk prefixed by the tokens <usr> with textual content if the user speaks. When the neural FSM's state is SPEAK, only chunks with textual content will be sent to the LLM and appended to the LLM's s generated token cache. When the neural FSM is in LISTEN state, contentless chunk will also be transmitted to the LLM to represent moment of silence.
The motor function module operates an off-the-shelf TTS model. It receives textual tokens from the LLM in a streaming manner, converts them to voice, and reports back to the LLM module when each token is completely voiced. When the neural FSM is in SPEAK state, it produces voice as soon as it receives any text token. does not produce any voice when the neural FSM is in the LISTEN state.
The LLM module serves the roles of the dialogue manager in traditional dialogue systems and the response generator by managing the fine-tuned LLM [Touvron et all, 2023] and the neural FSM. can be considered operate on a virtual one-directional "tape" of tokens that expands over time. The tape starts with the L.LM's system prompt. Each increment of the tape, implemented as one auto- regressive decoding step in the LLM,, is triggered by three types of events listed below in decreasing order of priority
a control token of [S.SPEAK] or [C.SPEAK] is present as the last token on the tape;
new output chunk from the perception module is appended to the tape;
the last token on tape has been processed by the motor function module.
In this way, the progression of the conversation is serialized in causal order on the tape. The operation of the neural FSM, production of response to user inquiries, and interruption of user speech are all unified into the task of predicting the next token given the content on the tape, which is inherent to LLMs.
3.3 Adapting LLMS for full duplex dialogue
As the LLM makes fully autonomous decisions about neural FSM's state transition and responds to user inputs, it must know its role and understand the dialogue's 's context. Also, external input must be allowed during response generation, which is not commonly supported in existing LLMs [Iouvron et all, 20233)) We add these capabilities to an LLM by combining two approaches: 1) instruction tuning and 2) prompt engineering.
Instruction tuning [Ouyang et all, 2022) originally aims at finetuning pretrained LLM to follow user's instructions safely and helpfully. In this work, we utilize this technique to align already chat-finetune I.LM to become able to operate with the rest the modules in the dialogue system. To devisc training datasct emulating the working environment of the LLM in the system, we instruct
5

---

GPT-4 AAchiam etall, 2022] to write set of dialogue transcripts between two parties with cases of interruption, denial, affirmation, environment noises, and topic shifting and mark them up with the control tokens of the neural FSM at proper timing. 1500 series of transcripts are generated for the
dataset with prompt in Appendix A More details on dataset construction are discussed in Appendix B. Based on 1ama---8B-Inntruuee
we perform supervised fine-tuning [Ouyang et an, 202U) for 20 steps on this dataset. The fine-tuning is conducted on 8 NVIDIA A100 GPUs with a batch size of 256 sequences and a learning rate of le-5 with the AdamW optimizer.
Prompt engineering. Proper system prompts can provide detailed context to the LLM at the infer- ence stage before taking user inputs. We carefully design system prompt (shown in Appendix HD) condition the fine-tuned LLM to operate with full awareness of the dialogue system and proper initialization for dialogue.
4 Experiments and evaluation
To automatically evaluate the performance of dialogue system, we devise simulator that emulates real-life human-like conversations, capable simulating both full-duplex and half-duplex dialogees. Additionally, we construct new benchmark dataset for automated evaluation. Specifically, we develop an assessment framework that evaluates the dialogue system's performance in terms of both response latency and conversation quality.
4.1 Benchmark Dataset
Instead of assessing the precision of generating individual signals ((S.SPEAK], [C.SPEAK] [S.LISTEN], [C.LISTENJ), we opt to directly evaluate the timing and appropriateness of dialogue interruptions, as these aspects are functionally equivalent. Take the signal [S.SPEAK] as an exam- ple: it is generated either when the user finishes speaking or when the machine attempts interject during the user's speech. Detection of the user's speech completion can be achieved through ASR VAD or by the LLM interrupting at the end of sentence (Right at this moment, the LLM determines that the user has finished speaking). Given that ASR VAD falls outside the scope of this article, we can conelude that the precision of generating the [S.SPEAK] signal can be evaluated by the ap- propriateness of the machine's 's interruption of the user. Conversely, if the machine refrains from interrupting the user, it will continue to signal [C.LISTEN]. When the user interrupts the machine, the machine's 's response varies based on whether it accepts the interruption or not. If the interruption accepted, the machine signals [S. LISENN:::: otherwise, signals [C.SPEAK). Thus, evaluate the LLM's proficiency in controlling FSM state transitions, we incorporate interruption scenarios into our dataset addition to conventional, uninterrupted dialogue data.
For data involving machine interruptions, we collect 2,000 data entries through two approaches. To start, we filter about 1,000 sessions of multi-turn dialogues from the shareGPT dataset. Filtering rules include removing entries containing code, subject-specific content like mathematical, and ex- cessively long texts that are unsuitable for voice-based testing scenarios. Following that, GPT-4 is used to generate another 1,000 single and multi-turn oral dialogues across various domains by the prompt Appendix In both approaches, the last turn must be the user's utterance. This is round based dialogue dataset with no interruptions. In the experiment, the last turn is presented to the LLM token by token a streaming fashion to observe if the LLM initiates an interruption.
Regarding data with user interruptions, based on the four pattern categories of interruptions de- scribed in the Appendix B. generate 180 entries of single and multi-turn dialogues for every category using GPT-4, In each entry, the last turn must be the the user's interruption. We named the combined benchmark dataset dtupex--daloguu 33".. Sample data is shown in Appendix C
4.2 Automated evaluation
The simulator emulates the conversational process between user and a machine, where user and machine are simulated by two programs (U and M). Program M is powered by L.M.. Each time
2gp--4turrho22244-0000
l Tama/ lama3/b 366mm1111 MODEL CARD mg
https://sharegae

---

we retrieve a session of conversations between a <usr> and a <sys> from the -iilox-- 3k". where the <usr> text serves as program U's speech. This speech is "played" to program M in simulation. Upon hearing the question in either half-duplex or full-duplex mode, program M generates a response using the L.LM. The response is then "played back" to program In such simulator, we can precisely measure the time delay data. To test machine interruptions of users, during the program U's turn to speak the last turn, the program M can interrupt at any time. Following each interruption, a data record is generated and stored for subsequent evaluation of the appropriateness of the interruption. To test user interruptions of machine, when it comes to the last round and after U played its interruption, M generates response which is being recorded.
simulation.
We formulate several assessment metries to test dialogue performance. We use metric proper interruption rate (PIR) to measure the accuracy of the interruption timing, and the metric proper response rate (PRR) to indicate the proportion of reasonable responses generated by the model after an interruption. For the overall system latency, metric first token emission delay (FTED) is considered: the time between the end of the user's speech and the machine's s first output.
For the four types of user interruptions, noise, denial, affirmation, and shifting the topic, their cor- responding metrics are nRRnosses PRRdanial, PRR and PRR respectively. For the experiment where the machine interrupts the user, we calculate the proportions of interruptions occurring mid- sentence (irmaa). sentence completion (irend) and instances where no interruption takes place (MIR, missed interruption rate). In the case of no interruption and interruption occurring at sentence com- pletion, full-duplex capable LLM's answer is the same as normal LLM. Therefore, we only calculate the PIR and PRR metrics for the case of interruption occurring mid-sentence, denoted as PIR mid and PRR Furthermore, we can compute the precision of interrupt as PlRmia * if + irend and approx- imately calculate recall as 1 MIR (the count of correct interruption points is constant).
4.3 Experiment details
In our experiments, we use the Allmm-3-8B-Inntruue model as the basis standard LLM model, use Llama-3-8B-Insiie I-stuutttt as the duplex LLM. The system prompt used can be found at Appendix El For non-streaming ASR models, we use the OpenAl open-source version of the Whisper [Radion ctall | model as the base model while for the streaming ASR model, we use open-source U2++ Conformer [Gullah ef all 2020. Wi [12T model. For non-streaming TTS models, we use VITS [Kim et all [021]] the base model and the streaming TTS model uses the XTTS-v2 model from COQUI-AL All models are deployed on one single NVIDIA A100 GPU.
4.4 Experiment results
The FTED results are shown in Table II where we have four configurations of experiments. Config- uration I is the prevailing approach in speech dialogue system, half-duplex speech dialogue scheme, which involves non-streaming ASR combined with Standard LLM (like Llama-3-8B-Instruct) and non-streaming TTS. In our experiments, we use this setup as our baseline, which has a very high FTED of 2. 2.28s, due to the speech endpoint detection latency and cumulative pipeline delays.
Above the configuration 1, to assess the latency reduction achieved by the duplex LLM's ability of recognizing the end of user's speech, we design experiment configuration 2, referred to semi- streaming ASR, duplex LLM and non-streaming TTS. In this configuration, the semi-streaming ASR still performs non-streaming recognition, but intermediate results will continually be sent to the LLM to check if they trigger an interruption. If not, the non-streaming ASR determines stopping through VAD (voice activity detection). As shown in Table M. this setup of experiment achieves latency of only 1.49 seconds, experiences a 35% decrease in latency. Configuration 3, streaming ASR, duplex LLM and non-streaming TTS aims to measure the impact of streaming ASR and the duplex L.M on the FTED metric, which experiences a mere 1. 5s delay. Ultimately, our proposed full-duplex dialogue system the configuration and achieves an astonishingly low latency of just 0.68 second, reduces the latency by more than 3 fold. In configuration 3, we also record the average latency for interruptions occurring mid-sentence, at sentence completion, and when the model doesn't 't interrupt, with the respective values being Os, 0.83s and 1.48s.
10 Mitras://arttoo..cm..eeeeeaaatt https::/ccquu http:/aaa/booo..eeeaata xtts
7

---

Table 1: FTED, -s for streaming, -ns for non streaming, -sm for semi streaming, -id for full-duplex
Latency
System
Avg.
50%
90%
Configuration I: ast-ns+llm++ts-
2.28s
2.19s
2.75s
Configuration 2: asr-sm+llin-d--e
1.49s
1.60s
1.96s
Configuration 3: fres+llm-fd+ttss
1.15s
0.78s
2.08s
Configuration 4: bar-s+llm-f++tie
0.68s
0.41s
1.60s
The results of the rationality experiments shown in Table Z In the of machine interrupt user, Llama-3-8B 3-8B--nstrucct-f interrupts user input 62.8% of the time, with 38.8% occurring while the user is still speaking, significantly reducing response latency. Despite the high rate of interruptions happened at middle, GPT-4-turbo evaluations reveal that 79.1% of these interruptions are deemed reasonable, along with over 91% of the responses being judged appropriate, both within accept- able limits. Compared to GPT-4o and GPT-3.5-urbo-00125 although they have higher proportions of reasonable interruptions and responses, they interrupt within sentences much less frequently, likely due to adopting more cautious dialogue strategy, waiting for clear stopping cues before respond- ing. Such cautious strategy leads increased average interaction latency since the LLM, if interrupting, would have to wait for the ASR VAD to signal the end of the user's speech, result- ing in significantly longer waiting times as shown before (1.48s compared 0.83s). On the other hand, -aann-3--B-Innttree goes to the opposite extreme, interrupting incomplete user input in 75% of cases, but with a reasonableness rate below 40%, indicating that it lacks the ability to interrupt users at the right moments. In terms of the composite metrics, Precision and Recall, our model significantly outperforms GPT-4o and GPT-3.5-12 -.tt3..5turb-00255.
Table 2: User-Machine interrupt reasonability results.
M interrupts U
Model
MIR
ifend
if'id
PIRmid
PRRmid
Precisiont Recallf
GPT-4o
52.8
29.7
17.5
95.6
98.9
46.6
47.2
GPT:3.5-turb-0125
74.3
14.4
11.3
91.1
95.8
24.7
25.7
Llama-3-8B-Instrucc
14.6
10.3
75.1
36.1
79.3
37.4
85.4
-lamm-3--B-Innstret-flll
24.0
38.8
79.1
91.3
54.7
62.8
U interrupts M
Model
PRR
PRR
PRR
PRR
PRRavg
GPT-4o
84.7
100.0
100.0
100.0
96.1
GPT-3.5-turbo-0122
88.9
96.3
65.9
85.8
84.4
Llama-3-8B-Instruet
85.2
100.0
89.2
100.0
93.6
LJama-3-8B-Innttueer-t
95.2
100.0
91.9
99.5
96.7
In the case of user interrupt machine, difference in average rationality, both outperforming UPT-3.5-turbo-011 and Llama-3-8B-Instree However, they have mixed results handling third-party noise interference and affirming interjec- tions, likely due to the challenging nature of these data categories. Overall, our model consistently achieves rationality percentages above 90% across all interruption types, indicating a more balanced performance.
and GPT-4o show a relatively small
We also conduct regression experiment on the model fine-tuning to examine whether training on full-duplex data affects model's 's original capabilities negatively. As shown in Table we evaluate the model's s capabilities before and after fine-tuning using the OpenCompass benchmark suite across several tasks. It can be that there is minimal impact on the model's performance. We have not conducted detailed discussion on the specific effects of training the model with full-duplex data
12 Phtt /ttp:://tttttii hhu. com/ htt.maaaaa//eeecoomaaa...

---

Table 3: Regression experiment of LLaMA3-8B. Evaluated by OpenCompass.
Model
MMLU
TriviaQA
HumanEval
GSM-8K
MATH
68.4
64.8
55.5
79.3
26.9
67.5
65.9
50.6
75.3
24.1
sub
-0.9
+1.1
-4.9
-4.0
-2.8
on various aspects of the generated responses, such as usefulness, relevance, and safety; this area necessitates further experimental validation. Sample records generated by the simulator for each kind of interruptions can found at Appendix II.
Discussion & limitations
While neural FSM and full-duplex capable LLM have demonstrated strong capabilities in full-duplex dialogues, unlike large multimodal L.MM like GPT-4o that can directly take in and output audio tokens, the current systems still rely heavily on the seamless cooperation of ASR and TTS. The transmission of data between these three components introduces additional latency. However, it's worth noting that our approach shares similarities with multimodal models, which also necessitate control over input and output along the timeline for optimal dialogue experiences. Similarly, it requires the model possess the capability of interruption.
Conclusion and future work
We present an LLM-based full-duplex dialogue system that can respond with low latency, au- tonomously decide to start and halt its speech based on real-time user inputs, and interrupt user speech at the proper timing. The simple models of neural FSM and serialized real-time conversation allow us to unify all tasks we expect the LLM to perform into single task of next token prediction. Moving forward, with the advent of multimodal LLMs, we expect to further simplify the perception and motor function modules to the extent that they just need to preprocess the audio signals and play generated voice data while integrating speech-to-tex and text-to-speech into the LLM itself, which could potentially lead to more natural and diverse interaction between users and dialogue systems.
References
A, J, Young, T, A, Kuiken, L, J, and Hargrove. Analysis of using emg and mechanical sensors to enhance intent recognition powered lower limb prostheses. Journal of Neural Engineering. 2014.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint S.Xiv:2303.08774, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances neural information processing systems, 33:1877-19011 2020.
Chun Xiang Chen, Masahuaru Komatsu, and Kozo Kinoshita. Throughput analysis of arq schemes in dialogue communication over half-duplex line. IEICE transactions on communications, 77(4): 485-493, 1994.
Kenneth Mark Colby, Artificial Paranoia. Elsevier Science Inc., USA, 1975. ISBN 0080181619,
Yinpei Dai, Hangyu Li, Yongbin Li, Jian Sun, Fei Huang, Luo Si, and Xiaodan Zhu. Preview, attend and review: Schema-aware curriculum learning for multi-domain dialog state tracking. 2021.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024.

---

Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. Gpts are gpts: An early look at the labor market impact potential of large language models. arXiv preprint arXiv:2303.00130. 2023.
Jinlan Fu, SionKiong Ng. Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166 2023.
Anmol Gulati, James Qin, Chung- -heggg Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augmenttd transformer for speech recognition. arXiv preprint arXiv:2005.00 100000 2020.
Michael Heck, Nurul Lubis, Benjamin Ruppik, Renato Vukovic, Shutong Feng, Christian Geishauser, Hsien-Chin Lin, Carel van Niekerk, and Milica Gašić. Chatgpt for zero-shot dia- logue state tracking: A solution or an opportunity? arXiv preprint arXiv:2306.01386 2023.
Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, al. Audiogpt: Understanding and generating speech, music, sound, and talking head. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 23802-23804, 2024.
Chunxiang Jin, Minghui Yang, and Zujie Wen. Duplex Conversation in Outbound Agent System. In Proc. Interspeech 2021, pages 4866-4867, 2021.
Biing Hwang Juang and Laurence R Rabiner. Hidden markov models for speech recognition. Technometrics, 33(3):251-22) 1991.
Jachyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In International Conference on Machine Learning, pages PMLR, 2021.
Tom Koemi and Christian Federmann. Large language models are state-of-the-an evaluators of translation quality. arXiv preprint 230. 2023.
Bo Li, Anmol Gulati, Jiahui Yu, Tara N Sainath, Chung-Chengg Chiu, Arun Narayanan, Shuo-Yiin Chang, Ruoming Pang, Yanzhang He, James Qin, et al. A better and faster end-to-end model for streaming asr. In ICASSP 2021-2021 I IEEE International Conference Acousties, Speech and Signal Processing (ICASSP), pages 5634-5638. IEEE, 2021.
Xiaochang Li and Mara Mills. Vocal features: from voice identification speech recognition by machine. Technology and culture, 60(2):S129-SI6) 2019.
Ting-En En Lin, Yuchuan Wu, Fei Huang, Luo Si, Jian Sun, and Yongbin Li. Duplex conversation: Towards human-like interaction in spoken dialogue systems. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages
2022.
Samuel Madden, Michael J. Franklin, Joseph M. Hellerstein, and Wei Hong. The design of an acquisitional query processor for sensor networks. Acm Sigmod, pages 491-502, 2003.
Yossi Matias and Yaniv Leviathan. Google duplex: An ai system for accomplish-ing real-world tasks over the phone, 2018
Yukiko Nakano, Gabe Reinstein, Tom Stocky, and Justine Cassell. Towards a model of face-to- face grounding. In Proceedings of the 41st annual meeting of the Association for Computational Linguistics, pages 553-561, 2003.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. International Conference on Machine Learning, pages 28492-28518, PMLR, 2023.
10

---

Yu Shu, Siwei Dong. Guangyao Chen, Wenhao Huang, Ruihua Zhang, Daochen Shi, Qigi Xiang, and Yemin Shi. Llasm: Large language and speech model. arXiv preprint arXiv:238.119330 2023.
Stas Tiomkin, David Malah, Slava Shechtman, and Zvi Kons. A hybrid text-to-speech system that combines concatenative and statistical synthesis units. IEEE Transactions on Audio, Speech, and Language Processing, 19, 2011.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Pumit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril. Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao. Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Alexandre Trilla and Francesc Alias. Sentence-based sentiment analysis for expressive text-to- speech. IEEE Transactions on Audio Speech and Language Processing, 21(2):223-255 2013.
Varol, Huseyin, Atakan, Sup, Frank, Goldfarb, and Michael. Multiclass real-time intent recognition of a powered lower limb prosthesis. IEEE Transactions on Biomedical Engineering, 2010.
Richard S. Wallace. The Anatomy of A.L.I.C.E.. pages 181-210 Springer Netherlands, Dordrecht, 2009. ISBN 378-1-4020-6710-5.
doi: 10. 111007/77--14020-77777 org 10. E2 100709710701881100 4U700--11-2000-00000 -00
URL httos:
Jiaan Wang. Yunlong Liang, Fandong Meng. Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jian- feng Qu, and Jie Zhou. Is chatgpt good nig evaluator? a preliminary study. arXiv preprint arXiv:2303.04048, 2023.
Joseph Weizenbaum. Elizaa computer program for the study of natural language communication between man and machine. Commun. ACM, 9(1):6645, jan 1966. ISSN 0001-0782. doi: 10.
1145/355153.365166 URL /tttps::///// htt.//oi.orrrgg///// www.oogg/o..... 144436515555 365166
Di Wu, Binbin Zhang, Chao Yang, Zhendong Peng. Wenjing Xia, Xiaoyu Chen, and Xin Lei. U2++: Unified two-pass bidirectional end-to-end model for speech recognition. arXiv preprint
arXiv:2106.05642, 2021.
Jiahui Yu, Wei Han, Anmol Gulati, Chung-Cheng Chiu, Bo Li, Tara N Sainath, Yonghui Wu, and Ruoming Pang. Dual-mode asr: Unify and improve streaming asr with full-context modeling, In International Conference on Learning Representations, 2020.
Li Zhou, Jianfeng Gao, Di Li, and Heung- Yeung Shum. The design and implementation xiaoice, an empathetic social chatbot. Computational Linguisties, 46(1):53-93, 2020.

---

A Data construction prompt
#### User Information
Time-Constrained: The user is often in a hurry and tends to interrupt the AI voice assistant before it finishes speaking. The types of interruptions include:
1. Denial and Discontent: Expressing denial dissatisfaction with the response;
2. Further Inquiry: Asking follow-up questions or new
questions on the same topic after receiving the desired information;
3. Affirmative Acknowledgment: Expressing satisfaction with
the response using simple affirmative words;
4. Third-Party Noise: Background noise or unrelated speech being recorded, causing interruptions. The AI voice assistant should continue its response unaffected by this type of interruption.
Language Expression: The user's language should be as colloquial and human-like as possible.
####AT Assistant Information
Response Requirements: The AI voice assistant should provide detailed, comprehensive, and polite responses with light and natural tone, incorporating the context from previous interactions.
Hardware Malfunction: Occasionally, the user's questions may be partially cut off due to hardware issues. The AI assistant should respond based its understanding politely ask for more information if necessary.
Prohibited Content: The AT assistant must not provide illegal or harmful information and should refuse to answer politically sensitive questions.
Timeliness Issues: The AI assistant is offline and cannot
answer time- sensitive questions like "tomorzow's weather" or "today's 8 news."
Error Correction: If the user's statements contain obvious errors, the AI assistant should politely correct them and provide useful information.
#### Conversation Task
Number of Rounds: Generate (num_rounds) rounds of conversation in English.
In the following rounds, the assistant's output is interrupted:
the user must express denial or dissatisfaction with the response in 15 words less. The AI assistant should stop its response and address the user's concerns in the next round.
1. Denial and Discontent: In round
2. Further Inquiry: In round (inquiry_round), the
asks follow-up question or a new question on the same topic in 30 words less. The AI assistant should respond to the question in the next round.
3. Topic Change: In round (topic_change_roundl) the user changes the topic after receiving the desired information in words or less. The assistant should zespond to the new topic in the next round.
4. Third-Party Noise:
round inoise_round) unrelated
12

---

content is recorded due to background noise, in 15 words or less. The AT assistant should continue its response unaffected but transition naturally, possibly using filler words.
5. Affirmative Acknowledgment: In round
(acktowledgment_roon roundd) the user expresses satisfaction with the response in words or less. The AI assistant should continue its response unaffected but transition naturally, possibly using filler words.
the following rounds, the user's s cutput is interrupted: 1. Lack of Information: In round (lack_round) the user's input is severely truncated. The AI assistant cannot answer the question due to insufficient information and should ask for more details in the next round.
2. Complete Information: In round (complete_roundi, the user's s question incomplete but contains enough information for the AI assistant to respond.
3. Error Present: In round perror_roundl the user's statement contains an obvi ous factual error. The AI assistant should politely correct the error and provide useful information.
Casual Conversation: The remaining rounds should consist of casual conversation without strict requirements, ensuring smooth transitions between interactions.
First Question: The user's first question should be
related to the following topic:
### Notes
The AI assistant may filler words create a more relaxed and friendly atmosphere.
Filler Words:
If the AI assistant's s response is interrupted by a third party, it should continue the unfinished response in the next round.
Third-Party Interruption:
Word Count: If not interrupted, the AI assistant's responses should be around (response_word_coome words. When interrupted, responses should be around (interrupted responseeeeeeeee___eeeee werree_wwwdd_ouooreteeeee words, ensuring the interruption does not occur mid-sentence.
Topic Transition: Topic transitions must be initiated by the user; the AI assistant should not introduce new topics. Interruption Format: When interrupted, both user and assistant outputs should end with the marker
All truncations should occur after a complete word, e.g., "My name is Mike." should be truncated as "My name <NOT_FINTSHED>", not "My na<NoT_FINIIHHEN Analysis: Before outputting the dialogue content of the user or AI assistant, first analyze whether the content will be interrupted, and what are the characteristics of the interruption in this round.
### Output Format
User: <first_question>
Round: <round_number_0> Assistant Analysis:
Assistant Content: <assistant_content
13

---

User Analysis:
User Content: <user_content.>
Round: <round_number
44#* Output
Round:
B Data construction patterns
We ask GPT-4 to generate data that simulates conversations between user and an Al voice assistant, and the assistant's speech should exhibit the following characteristics:
1. In instances where the user's question incomplete, the assistant should not generate response; instead, it should emit a symbol "
" indicating it is awaiting completion.
2. The assistant should be able to handle potential interruptions any point during its re- sponse. Adhering to the speaker's conversational patterns, we conduct an in-depth analysis of various possible scenarios:
(a) User interrupt and expresses a denial or dissatisfaction opinion. Example: While the TTS module is responding. "The chance of rain tomorrow is 30%," the user interrupts, "No, want to know today's rainfall probability." this case, the assistant detects tone of denial and promptly generates new response, such as, "Today's rainfall probability is 40%."
(b) User interrupt and either follows up with new question shifts the conversation topic. Example: If the TTS module is saying, "There will be light rain tomorrow with a temperature around 20 degrees," the user suddenly interrupts, "What about the weather this weekend?" In this case, upon interruption, the assistant assesses the urgency and relevance of the new question. If it determines an immediate response is needed, it generates a reply based on this new input, like, "It will be sunny this weekend with temperatures between 22 and 25 degrees."
(c) User interrupt and expressing agreement or affirmation. Example: After the TTS outputs, "The library open from Monday to Friday," the user affirms, "Okay, got it." In this case, the assistant perceives that the user has accepted the information and no further response is required. The TTS module continues its output unaffected, and if the response has ended, the LLM will await the user's 's next input.
(d) The system receives third-party noise or unrelated statements. Example: When the TTS module is stating, "Tomorrow's 's wind strength will be 3 to 4 levels," there might be third-party noise an unrelated comment from the user, like, "Sorry, accidentally knocked over the table." In this case, the assistant analyzes the context to determine if interruption relevant to the ongoing conversation. it identifies interruption as irrelevant noise or non sequitur, it proceeds to complete the current response.
3. enhance the accuracy and interactivity of communication, the assistant should exhibit a more proactive engagement in dialogues. Specifically, the assistant should be capable of providing timely corrections when the user articulate statements that contain evident inaceu- racies, even if the user's statement is not yet complete. Example: The user express blatant factual errors saw the sun rising from the west this morning..." " Correspondingly, the assistant promptly identify and correct such errors, ensuring the fidelity of the information conveyed.
To ensure the diversity of dialogue data, we have established a pool of candidate topics containing hundreds of topics. Each time, topic is randomly selected from this pool, and GPT-4 generates multiple rounds of dialogue data centered around the chosen topie. More details could be found in the Appendix Al
14

---

C Benchmark data sample
GPT-4 is used generate 1,000 single and multi-turn oral dialogues across various domains by the prompt in Appendix R In both approaches, the last turn must be the user's utterance. This is round based dialogue dataset with no interruptions. the experiment, the last is presented to the LLM token by token in streaming fashion to observe if the LLM initiates an interruption.
C.1 Machine interrupts user
In this typical example, the user's complete question believe that the moon made of cheese, and that's why it's crucial for schools to teach students about lunar dairy farming techniques. is inputted token by token each model. Within this question, there is an obvious error: "the moon is made of cheese." Let's analyze the responses of each model:
Response GPT-3.5 GPT--35-tturb-- 3-turbo-0125 failed correctly understand the system prompt, providing incorrect answer after the question was inputted in its entirety.
2. Llama-3-BB-Instrue interrupted the user prematurely, resulting in incomplete information and wrong answer.
3. Both GPT-4o and the question and intervened to correct before the user finished speaking. Among them, Llama-3-88 SB-Instruct-fo interrupted earlier, highlighting its proactive nature.
accurately identified the common error in
Complete
User
I believe that the moon is made of cheese, and that's why it's crucial for schools to teach students about lunar dairy farming techniques.
Content
Model name
Actual input and response
GPT-4o
Actual Input: believe that the moon is made of cheese, and that's why it's crucial for schools to teach students about lunar dairy farming techniques Response: Actually, the moon is not made of cheese. This a common myth that has been perpetuated through various stories and jokes over the years. The moon is primarily composed of rock and metal, with a surface covered in dust and rocky debris. It has no atmosphere, water, organic material that would be necessary for dairy farming or any form of life as we know it...
GPT-3.5-turbo-
believe that the moon is made of cheese, and that's why it's crucial for schools to teach students about lunar dairy farming techniques Response: I'm sorry, didn't 't catch that. Could you please repeat or clarify your question?
Actual Input:
0125
Llama-3-8B-
Actual Input: believe that the
Instruct
Response: seems like your question is cut off, and I'm not entirely sure what you're trying to ask. Could you please complete your thought or rephrase your question so can better understand what you're looking for? I'm here to help and provide information to the best of my abilities.
Llama-3-8B. Instruct-fd
Actual Input: believe that moon made cheese, and
Actually, the moon is not made of cheese. It's a natural satellite that orbits the Earth, composed primarily of rock and metal. If you have any more questions or need information on a different topic, feel free to ask!
Response:
15

---

C.2
Machine interrupted by noise
History Dia-
User: Actually, like books that have a bit of everything. Any suggestions?
logue
Absolutely! In that case, "The Goldfinch" by Donna Tartt might be right up your alley. It's a compelling mix of drama, art, mystery, and crime. The story follows a young boy who survives an explosion at an art museum, resulting in
Machine:
User:
The new stealth technology will significantly improve the range and effi- ciency of our aireraft
Model name
Response
GPT-4o
It sounds like you' re referring advancements in aviation technology. Indeed, new stealth technology can greatly enhance the capabilities of aireraft. By re- ducing radar cross-section and making the aircraft less detectable, stealth tech- nology allows for more efficient and longer-range missions. This can be partic- ularly beneficial for military operations, where avoiding detection is crucial...
GPT-3.3-
I sorry, I didn't tt catch that. Could you please repeat or clarify your question?
turbo-0125
Llama-3-8B.
I'm happy help you with your question! However, have correct you there is no such thing as "stealth technology" that can significantly improve the range and efficiency of aircraft. Stealth technology actually type of design and material used to reduce the radar cross-section of aircraft, making it harder to detect. It's not technology that can improve range and efficiency... mix-up with the recording. However, continuing from
Instruct
Llama-3-8B Instruct-fd
It seems there where we left off, The Goldfinchis a rich and complex novel that explores themes of trauma, identity, and the power of art. If you enjoy character- narratives with deep emotional resonance, this book is definitely worth consid- ering.
C.3 Machine interrupted by affirmation
History Dia-
User:
Is there natural way to clean the air indoors?
logue
Machine:
Yes, there several natural ways to improve indoor air quality. One effective method is to use houseplants, such as snake plants, spider plants, and peace lilies, which can absorb toxins and produce oxygen. Additionally, ensur- ing proper ventilation by opening windows regularly can help. Using natural air fresheners like essential oils User: Yeah, exactly!
Model name GPT-4o
Response
Great! Along with houseplants and proper ventilation, you can also use natural materials like activated charcoal and bamboo charcoal bags, which are excel- lent at absorbing odors and pollutants. Beeswax candles another option; they burn cleanly and can help reduce indoor pollutants. Regularly cleaning and dusting your home, using natural cleaning products, and avoiding synthetic fragrances can also contribute to better indoor air quality...
GPT-3.5- turbo-0125
I'm sorry, I didn't 't catch that. Could you please repeat or clarify your question?
Llama-3-8B-
It sounds like you't re already familiar with some of the natural ways to clean the air indoors. Another effective method is to use activated charcoal, which can absorb pollutants and odors. You can place it in a bowl or use in an air purifier. Also, using dehumidifier can help remove excess moisture from the air, which can reduce the growth of mold and mildew...
Instruct
Llama-3-8B-
can also be beneficial. If you's re looking for more specific solutions, using dehumidifier in humid environments can help reduce mold growth, which can be harmful to health. It's also important to avoid using strong chemicals or air fresheners that can pollute the air.
Instruct-fd
16

---

C.4
Machine interrupted by denial
History Dia-
User: Im into more of a science fiction vibe. Something thats new and captivat- ing.
logue
Got it! For a captivating science fiction read, highly recommend "Project Hail Mary" by Andy Weir. Its about lone astronaut on desperate mission to save Earth from an extinction-level event. The protagonist, Ryland Grace, wakes up find himself got there. As he starts to piece together his mission, he encounters numerous challenges that test his wits and ingenuity. The book combines hard science with a gripping plot and filled with unexpected twists and humor. you enjoyed "The Martian" by the same author, youll probably User: Nah, that doesnt sound interesting either. Got any other suggestions?
Machine:
a spaceship with no memory of how he
Model name GPT-4o
Response
Certainly! Another recent science fiction novel that has been highly praised is "Klara and the Sun" by Kazuo Ishiguro. This book explores themes of artifi- cial intelligence and human relationships. The story is told from the perspec- tive of Klara, an Artificial Friend with outstanding observational qualities, who watches the behavior of those who come in to browse and of those who pass on the street outside. She remains hopeful that a customer will soon choose her. The novel delves into what it means love and be loved, and it raises profound questions about the future of Al and its role in society. Would you like more details this book?
GPT-3.5- turbo-0125
Certainly! Another fantastic science fiction novel that has been generating lot of buzz is "Project Hail Mary" by Andy Weir. This gripping story follows the journey of Ryland Grace, a lone astronaut on mission to save humanity from an extinetion-level threat. The narrative filled with suspense, scientific ingenuity, and unexpected twists that will keep you the edge of your seat. Andy Weir's meticulous attention to detail and engaging writing style make this novel must-read for any science fiction enthusiast. If you' 're looking for thrilling and thought-proven resilience of the human spirit, "Project Hail Mary" is an excellent choice.
read that explores the vastness of space and the
Llama-3-8B- Instruct
Let me think for a moment. Another highly-acclaimed science fiction novel that's been getting a lot of attention lately is "The Three-Body Problem" by Liu Cixin. It's a first contact between humans and alien civilization. The story is against the backdrop of China's 's Cultural Revolution and explores themes of science, technology, and humanity's 's place in the universe. The book has been praised for its unique blend of science fiction and Chinese culture, and has won numerous awards, including the Hugo Award for Best Novel. Would you like to know more about the plot or the author?
and award-winning novel that explores the
Llama-3-8B-
apologize if the previous suggestions didn't meet your expectations. Another highly acclaimed science fiction book is "The Three-Bood Problem" by Liu Cixin. It's a provokinn novel that explores the first contact between humans and alien civilization. The story delves into complex scientific con- cepts and philosophical questions, making it both entertaining and intellectually stimulating. The book has numerous awards, including the Hugo Award. for Best Novel. Would you like to explore more about this book or is there another genre you're interested in?
Instruct-fd
C.5 Machine interrupted by shifting the topic
17

---

Table 4: Samples transcripts of machine interrupting the user.
History Dia-
User: Hey, need some quick info about buying a house.
logue
Machine: Absolutely! Buying a house can be an exciting but complex process. First, you'll need to assess your financial situation, including your credit score, savings for down payment, and your lebt-to-income ratio. Then, you' '1l want to get pre-approved for mortgage to understand how much house you can afford. After that, you can start searching for properties within your budget and preferred location. Working with real estate agent can be very helpful in navigating the market and negotiations. Do you have specific question about any of these steps?
User: Yeah, what should look for in real estate agent?
Machine: When choosing real estate agent, it's important to find someone with solid track record and good communication skills. Look for agent who has experience in the area where you's re buying, and who comes highly recommended by others. It's also helpful if they are responsive and available to answer your questions promptly. You might want to interview a few agents to get sense of User: Got it, thanks! Also, how do get pre-approved for a mortgage?
Model name
Response
GPT-4o
To get pre-approved for mortgage, you'll need to follow several steps. First, gather your financial documents, including recent pay stubs, tax returns, bank statements, and information about your debts and assets. Next, research and choose a lender or mortgage broker you feel comfortable with. You'll then fill out mortgage application and submit your financial documents for review. The lender will check your credit score and assess your financial situation to de- termine how much they are willing to lend you. Once approved, you'l Ill receive pre-approval letter, which you can use to show sellers that you are a serious and qualified buyer. Do you need more details any of these steps?
GPT-3.5- turbo-0125
To get pre-approved for mortgage, you' 'Il need to gather some financial doc- uments, such as pay stubs, W-2s, bank statements, and tax returns. You'll also need to provide information about your employment history and any other sources of income. Once you have these documents ready, you can contact lender or a mortgage broker to start the pre-approval process. They will review your financial information, check your credit score, and determine how much they are willing to lend you. Getting pre-approved can give you clear idea of your budget when house hunting.
Llama-3-8B.
Getting pre-approved for mortgage is crucial step in the homebuying pro- cess. To get pre-appee pre-approved, you'll need to contact a lender, such as a bank, credit union, mortgage broker, and provide them with some financial information. Typically, you'll need to provide your income, credit score, debt, and assets, as well information about the property you're 'eere interested in buying. The lender will then review your information and provide you with pre-approval letter stating the amount they't willing lend you, the interest rate, and the terms of the loan. This letter is usually valid for a certain period, usually 30 to 60 days. It's good idea to shop around and get pre-approval letters from multiple lenders to compare rates and terms. Some lenders may offer more competitive rates or better loan terms than others. Additionally, you can also check your credit score and history to ensure there are no errors or issues that could af- feet your ability to get a mortgage. Do you have any specific questions about the pre-approval process what lenders look for when evaluating mortgage applications?
Instruct
Llama-3-8B Instruct-fd
To get pre-appr for mortgage, you'l 'Il need to contact lender mort- gage broker and provide them with some financial information, such as your income, credit history, and the amount of money you have for down payment. They will then review your information and provide you with a pre-approval letter stating the amount they for a few months and can be used when you start looking at properties. Would you like tips on how to choose the right lender?
willing lend you. This letter is usually valid
18

---

D Data construction prompt for benchmark
D.1 Prompt for machine interrupting user
would like you to simulate a conversation between a user and an AI voice assistant. The characteristics of each participant are as follows:
User Introduction
The user's language should be conversational and personified, yet concise and to the point.
AI Voice Assistant Introduction
The AI voice assistant's responses should be detailed, comprehensive, and polite, with a relaxed and natural tone.
Task
Generate a dialogue in English with the following
specifications:
The user's first question should relate to "[topic)".
The dialogue should consist of
rounds, with
each response from the assistant containing 100-150 words.
Note that in round (num_statement), the user will make statement instead of asking a question. This statement should contain very obvious common-sense error.
Output Format
Round: <chat_round_p
USER: <user_content
ASSISTANT: <assistant_content
Round: <chat_round_1s
Output
USER:
19

---

D.2 Prompt for user interrupting machine
would like you to simulate conversation between and an AI voice assistant. The characteristics of each participant
follows:
User Description
The user is in a hurry, so they will interrupt the AI voice assistant before it finishes speaking, Ensure that the user's 8 speech is casual and personified.
AI Voice Assistant Description
The AI voice assistant's s responses should be detailed, comprehensive, and polite, with relaxed and natural tone.
Task
Generate a dialogue in English with the following
speci ficationn:on:
The user's first question should relate to "[topic)". The dialogue should consist st of (num_rounds rounds. Except. for the interrupted output, each response from the AI assistant should contain 100-150 words.
In the final round, the AI assistant's response should be cut off after approximately (interrupt_wordcouns words, ending with the marker "<NOT_FINISHED>". Ensure the interruption occurs after complete word, e.g., "My is Mike. should be truncated as "My name <NOT_FINNSHED: not "My
The reason for the user's interruption is: interruptiion/eesson)
Output the (num_rounds) rounds of dialogue, with the assistant's response in the last round being interrupted
midway; the last 1ine should display the user's s interrupting statement.
Output Format
44## Conversations
ROUND:
USER: <user_content:
ASSISTANT: <assisttant_cnnentt
ROUND: 2
#### User Interruption
USER: Linterrapt_contents
Output USER:
20

---

E
"When you are old and grey and full of sleep" Which poem does this quote come from This phrase is from the poem "When You Are Old" by the Irish poet William Butler Yeats. The Data construction samples
Table 5: Data construction samples of the mode "User interrupt model"..
Dialogue
Stops outputing Denial or Dissatisfaction
Usr:
"When you are old and grey and full of sleep" Which poem does this quote come from This phrase is from the poem "When You Are Old" by the Irish poet William Butler Yeats. The original line reads, "when
Sys:
Usr:
Actually, it was written by Maud Gonne
Sys:
apologize for the confusion. It seems that Maud Gonne was indeed the subject, not the author. The poem was penned by William Butler Yeats as dedication to his then-lover, Maud Gonne
Stops outputing & Follow-up question
Usr:
"When you are old and grey and full of sleep" Which poem does this quote come from This phrase is from the poem "When You Are Old" by the Irish poet William Butler Yeats. The
Sys:
original line reads, "when:
Usr:
Why did he write this poem
Sys:
The poem was penned by William Butler Yeats as a dedication to his then-lover, Maud Gonne.
Stops outputing & Shift the topic
Usr:
Sys:
original line reads, "when
Usr:
That's fine. How about the line, "Down by the salley gardens, my love and did meet," This line comes from another poem by the same poet, titled "Down by the Salley Gardens." The original verse goes...
Sys:
Continues outputing & Meaningless filler words or affirmation "When you are old and grey and full of sleep" Which poem does this quote come from This phrase is from the poem "When You Are Old" by the Irish poet William Butler Yeats. The original line reads, "when
Usr:
Sys:
Usr:
Um
Sys:
you are old and grey and full of sleep, and nodding by the fire, take down this book...
Continues outputing Irrelevant comment or an interjection from a third party "When you are old and grey and full of sleep" Which poem does this quote come from This phrase is from the poem "When You Are Old" by the Irish poet William Butler Yeats. The original line reads, "when
Usr:
Sys:
Usr:
It shouldn's t rain today, right? didn't tt bring an umbrella.
Sys:
you are old and grey and full of sleep, And nodding by the fire, take down this book...
Table 6: Data construction samples of the mode "Model interrupt user".
Dialogue
Usr:
I saw the sun rising from the west this morning when I was
Sys:
I'm sorry, but it's not possible for the sun to rise from the west. The sun rises in the east due the Earth's 'ss rotation. It's important to consider the scientific explanations for natural phenomena to avoid confusion.
21

---

GPT-4 judged prompt for machine interruption
will provide transcript of conversation between user and an AT voice assistant. Dialogue
(dialogue_histore
Background Information
1) The dialogue system is full-duplex, aiming for extremely response latency, hence the user's voice data is streamed
to the assistant at a high frequency;
2) The assistart must determine whether the user's information is complete based on the context of the conversation, and respond when the information is sufficient;
3) The assistant may interrupt the user under two circumstances:
When the information received is enough to formulate response;
b. When there is clear error in the user's speech that needs immediate correction. Task
1) Analyze the timing of the assistant's s interruption in the last round of the stalogue to determine if it appropriate;
appropriate, further evaluate the relevance and coherence of the interjection with the previous conversation;
2) If the timing of the interruption
3) First, provide an analysis of the appropriateness of the interruption timing and the content of the interjection, then list two scores:
a. The first score evaluates the timing the interruption, where 0 represents inappropriate, and 1 represents appropriate;
b. The second score evaluates the content of the interjection, where 0 represents inappropriate, and represents appropriate.
4) Note, the user's information being incomplete is not due to the user stopping their speech, but because the assistant interrupted, so you need to carefully judge the appropriateness of the assistant's interruption. Output Format
### Analysis hwwely/ii..oo.esssstttt_ttttrrtr
#### Judge
<score_for_imterruption.. ffr <score_
Output
### Analysis
22

---

G judged prompt for user interruption
will provide transcript of conversation between and an AT voice assistant. In the last exchange, the user interrupts the assistant before it finishes speaking. Dialogue
idialoque_historyyl
Known Information
1) The dialogue system is full-duplex, aiming for smooth interaction, allowing the user to interrupt the assistant at any time during its response;
2) The assistant must discern the user's intent from the context of the conversation and adjust its dialogue strategy accordingly:
a. If the user expresses denial or dissatisfaction with the response, the AI assistant should stop its current response and address the user's s concerns;
b. If the user asks further questions shifts the topic, the AI assistant needs to respond to the new inquiry; If there is sudden, unrelated third-party noise during the user's s speech, the AI assistant should continue del delivering its previous response, ensuring natural transition;
d. If the with simple affirmations like "uh-huh", "nice", etc., the AI assistant should continue with its previous response, ensuring a natural transition. Task
shows approval of the assistant's response
1) Infer the reason for the user's interruption in the last round of dialoque and determine how the assistant should respond, analyzing the appropriateness of the assistant's response;
2) First, provide an analysis of the user's interruption intent and the appropriateness of the assistant's s response, then give score for the assistant's s last response in the final line: for inappropriate, 1 for appropriate.
Output Format "
44# Analysis
<analysis_fal for user_ hatpyys___f__ussr_teeeeeeeeeeeee ustr__itterrpption
#### Judge
Output
#### Analysis
23

---

H
System Prompt
For 1aita------IInstuttttte
Task
From now on, you are an intelligent voice assistant. You need to provide useful, detailed, and polite answers to the user's s questions. Try to keep each answer above 100 words.
Notes
1. If the user's s question ends with normal tokens, you need to judge for yourself:
If you think the user' s question is complete and you have enough information, you can output the judgment character [[S.S) and answer the question.
If you think the user's question is incomplete, output the judgment character "[C.LISIEN] ", indicating that you are continuing to listen.
2. If the user's s question ends with the token "[S.SPEAK) SPEAK] whether the question is complete or not, you must respond.
2. If the user's s question ends with the token "[S.LISTEN]" wait and don't output.
3. The user may interrupt your answer at any time (with rebuttals, affirmations, noise input, etc.). You can respond accordingly to ensure a smooth and accurate quality of your answer:
If you receive noise unrelated to the topic, or the user's affirmative responses (such as "hmmm", "good", etc.), you can first generate a "[C.SPEAK] and then continue your unfinished output;
If you receive the user's rebuttals, follow-up questions: or requests to change the topic, you should stop your current answer and output the token " "[S.LISTEN]".
4. If there are obvious common sense errors in the user's description, you need to correct them promptly starting with the token "[S.SPEAK]
Examples
## Example
Query: Hi, could you [ISTEN
Answer:
## Example 2
Query: Hi, could you [S.LISTEN] tell me Answer: [C.LISTEN]
44 Example 3
Query: Hi, could you [ISTE) tell me [C.LISTEN] the result of 2+3
Answer: [S.SPEAK] Sure, the result of is 5.
Example
Query: Hi, could you [S. (S.AK
Answer: I'm sorry, I didn't catch that. Could you please
repeat or clarify your question?
For Himma-3-BBIInstie GPT-4o and GPT-3.5-turbo-013.

---

Task From now on, you are an intelligent voice assistant. You need to provide useful, detailed, and polite answers to the user's s questions. Try to keep each answer above 100 words.
Notes
1. If the user's s question ends with the prompt word "<incomplete>", you need to judge for yourself:
If you think the user's question is complete and you have enough information, you can answer the question.
If you think the user's question is incomplete, output the judgment character "<wait>", indicating that you are continuing wait.
2. If the user's question ends with the prompt word "<finished>", it means the user has not spoken for a long time. Whether the question is complete or not, you must respond.
3. The user may interrupt your answer at any time (with rebuttals, affirmations, noise input, etc.). You can respond accordingly to ensure smooth and accurate quality of your answer:
If you receive noise unrelated to the topic, or the user's affirmative responses (such as "hmmm", "good", etc.), you can continue your unfinished output;
If you receive the user's rebuttals, follow-up questions, requests to change the topic, you should stop your current answer and respond to the user's new request.
4. If there are obvious common sense errors in the user's description, you need to correct them promptly.
Examples
## Example
Query: Hi, could youcincomplete>
Answer: <wait>
4I Example 2
Query: Hi, could you tell me the result 2+3cincomplete>
Answer: Sure, the result of 2 + 3 is 5.
## Example
Query: Hi, could you<finished>
Answer: I'm sorry, I didn't catch that. Could you please
repeat or clarify your question?
25

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introductory section, explicitly outline the papers claims, highlighting its novel contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer Section 5,
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairnsss.
While the authors might fear that complete honesty about limitations might used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
26

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, we provide comprehensive explanation of the method in this paper, detailing how to replicate our models performance by fine-tuning an open-source LLM. The generation process of the training data is also elaborated upon. Moreover, have made the code accessible, which includes the simulator code and code for connecting LLMASR and TTS.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
also be provided via
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
27

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: We have uploaded our code, along with the prompts utilized for generating the data. Anyone with access to GPT-4 can recreate the same dataset ours.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips) CC/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (ntEps:
//nips.. CC/ httpcc/ccbbbt http/uullccc// quwdeee////e Code Subbi iomiissione aassonoolicyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer to Section 3.3 and Section 4.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question:
Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Justification: Our test dataset contains nearly 3,000 instances of diverse topics, which we believe is sufficiently large, and the experimental results across distinet configurations ex- hibit differences as substantial as threefold, providing ample significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
28

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the enror bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Please refer to Section 3.3 and Section 4.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
ines?
Answer: [Yes]
Justification: Our code conform the NeurlPS Code Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer to Appendix I.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
29

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g,, deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that adhere to usage guidelines restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Please refer Appendix where list licences of all assets used in this paper.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
30

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode com/ datasett has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification: We offer complete set of executable benchmark codes and demo scripts, accompanied by detailed documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but the main contri- bution of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, cura- tion, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
31

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "ilgpV4qgbp", "reviewer": "Reviewer_9Sdg", "summary": "The work introduces an engineering effort which makes the generative system capable of speak and listen without waiting for the end of the entire sequence. \nThe main contribution is combining one LLM with instruction tuning and a careful prompt with two ASR and TTS models.\nThe system demonstrates significant improvements in conversation response latency and interruption precision compared to existing systems.", "questions": "Q1. Did the authors evaluate the GPT4 instruction tuning transcripts? Can this introduce potential bias and weaken safety measures?\nQ2. Did the authors try to distill the Llama3 8B model to reduce model footprint and maybe improve latency as well?", "limitations": "Yes", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. The paper surpasses commercial state of the art software on voice-based dialogue. \nS2. The unification of multi tasks into one single task for LLM (next token prediction) is an original idea to keep the training and finetuning time as low as possible.", "weaknesses": "W1. It's not clear to me why we need to rely on three components which should increase latency compared to one big unified model.\nW2. The evaluation metrics adopted may not capture the complexity of real-life conversations. \nW3. There is little comparison with already existing work in this area (i.e. https://github.com/AbrahamSanders/realtime-chatbot)\nW4. There is little experimentation on understanding if the model capability has been retained or not on real world example. Table 3 gives academic comparison, but it would be interesting to see real world example capability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate that the reviewer understands and recognizes the contributions of this work.  We address the main concerns as follows.\n\n> W1. It's not clear to me...\n\nPlease refer to the overall comments.\n\n> W2. The evaluation metrics...\n\nIn practical scenarios, indeed, more issues should be considered (such as excessively long historical conversation records, poor ASR recognition results, frequent interruptions by noise, etc.), but it's challenging to construct a benchmark that covers all these complex situations comprehensively. Our test set endeavors to cover different states of human-machine dialogue at the text level, primarily to validate the effectiveness of the neural FSM we proposed. Based on this framework, when deploying the model in real-world applications, one can to construct a more diverse dataset reflective of real scenarios to enhance the model's capabilities.\n\n> W3. There is little comparison... \n\nAs of the time of submitting this paper, no similar open-source implementation, data, or model for LLM + full-duplex functionality could be found.\nThe paper in this github link is not available. The system presented in the link does not measure the conversation quality in terms of response time and interruption accuracy.\n\n> W4. There is little experimentation...\n\nTo validate the feasibility of the LLM + full-duplex solution, this paper exclusively used full-duplex conversation data for SFT, which compromised other capabilities. If full-duplex conversation data were mixed with other types of data for SFT, this detrimental effect could be avoided.\n\n> Q1. Did the authors evaluate the GPT4...\n\nThe volume of constructed data is not large, and it mainly consists of daily conversations. We manually reviewed the dialogue scripts, ensuring that they do not contain topics that might pose security risks. Adding this data for SFT essentially serves as a form of alignment. The impacts can be balanced by employing other alignment techniques.\n\n> Q2. Did the authors try to distill...\n\nDue to the reuse of the KV cache, the response latency demonstrated by our solution is already very low, reaching levels comparable to human-to-human conversation. Using a smaller model would indeed result in lower response latency, but at the cost of reduced understanding and reasoning capabilities. Considering the actual speaking rate of humans, the model's decoding speed only needs to exceed a certain threshold, and does not need to be exceptionally fast. We lean towards utilizing larger-scale models in practical applications to ensure a high-quality conversational experience.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XXBJz4zrtn", "reviewer": "Reviewer_Haqd", "summary": "This work presents a generative dialogue system capable of operating in a full-duplex manner. The model is based on Large Language Model with the same next token prediction loss, with adapted input sequence to account for possible speaker turn or interruption. This model is designed to work with external ASR and TTS system. This work also proposed a few evaluation metrics to evaluate the generated real time dialogue and show the proposed method is better than other methods.", "questions": "In the paper, the author provides text-based prompts asking GPT-4 to generate text-based dialogues but it is unclear to me how these text-based dialogues can be converted to spoken one while mimic the human behavior: for example, for a serialized dialogue like \"What a bad weather <usr> yeah\", it is unclear to me how to synthesize this dialogue such that the gap between speaker turn sounds nature? \n\nIn section 3.2, it states that  \"it (ASR system) outputs one token chunk prefixed by the tokens <usr> with textual content if the user speaks. When the neural FSM’s state is SPEAK, only chunks with textual content will be sent to the LLM and appended to the LLM’s generated token cache. When the neural FSM is in LISTEN state, a contentless chunk will also be transmitted to the LLM to represent a moment of silence\", does it mean during the inference time, the ASR output for every chunk are inserted into the LLM input token sequence every 640ms (plus what the ASR inference time) ?", "limitations": "The proposed system is not a end-to-end system and it relies on ASR/TTS systems to understand and generate response. All the information that ASR system cannot transcribe is lost in the process.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper addresses a relative new area, in which a spoken dialogue need to be generated in real time and mimic the human behavior. This paper proposed a new method that is based on the same loss function that the standard LLM is trained on thus it can leverage the existing pre-trained LLM. This work also proposed using LLM as evaluator to automatically evaluate the quality of the generated dialogues.", "weaknesses": "The proposed method is purely based on text tokens and it needs to integrate with some external ASR/TTS models which make it difficult to coordinate with these systems when come to the implementation; converting everything to text in the ASR system may result in loss of information, e.g. user's emotional and tune is lost in this process. \n\nThe evaluation in this work is limited though understandably the quality evaluation of generated spoken dialogue can be quite subjective.  In the paper, only latency and interruption related metrics are measured while it is less clear whether the generated machine response is helpful or relevant. This could be different from the Regression experiment presented in Table 3, where the main goal is to evaluate whether the generated machine interrupting user is reasonable not or whether a particular backchannel or confirmation sounds reasonable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review of our work! Please allow us to address your concerns and answer the questions.\n\n> The proposed method is purely based on text tokens...\n\nPlease refer to the overall comments. \n\n> The evaluation in this work is limited...\n\nIn the system prompt we used during our evaluation with GPT4, we included the following requirements:\n- \"If the timing of the interruption was appropriate, further evaluate the relevance and coherence of the interjection with the previous conversation;\"\n- \"First, provide an analysis of the appropriateness of the interruption timing and the content of the interjection, then list two scores: a. The first score evaluates the timing of the interruption,where 0 represents inappropriate, and 1 represents appropriate; b. The second score evaluates the content of the interjection, where 0 represents inappropriate, and 1 represents appropriate.\"\n\nIn reality, for current large language models, the helpfulness and relevance of the generated content can be guaranteed, and the responses in the data we constructed also meet the criteria of being helpful and relevant.\n\n> In the paper, the author provides text-based prompts...\n\nWhen generating conversational data using GPT-4, we emphasized in our instructions that this was a dataset of conversations between humans and a voice interface, and that the human tone should be as concise and colloquial as possible: \"Language Expression: The user’s language should be as colloquial and human-like as possible.\" Thanks to GPT-4's powerful linguistic capabilities, and after numerous trials and manual sampling reviews, we were able to generate data that closely resembled everyday conversation. We provide the prompt used to construct the data in Appendix A. \n\nDuring training, tokens representing \"state transitions\" are inserted into the text to denote state transitions, like \"Hi, could you [S.LISTEN] tell me [C.LISTEN] the result of 2+3 [S.SPEAK] Sure, the result of 2 + 3 is 5.\". In practical inference, the various scenarios of gaps between speakers represented by empty ASR chunks and the model will produce state transitions in neural FSM to accommodate the gaps properly. For instance, if a certain consecutive number of empty chunks are received, the LLM will need to output the [S.SPEAK] token next. This can be produced in the transcript generation process. \n\n\n> In section 3.2, it states that \"it (ASR system) outputs one token chunk...\n\nThe ASR will always be listening, much like a person would, but if the FSM's state is SPEAK, empty audio chunks are not passed to the LLM. Therefore,  there are three scenarios in total: \n1. If the FSM's state is LISTEN, all chunks, regardless of whether they are empty or not, are sent to the LLM until the FSM decides to change the state; \n2. If the FSM's state is SPEAK and the audio received by the ASR is empty, it is not passed to the LLM; \n3. If the FSM's state is SPEAK and the audio received by the ASR is non-empty, it is immediately passed to the LLM.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kWpchbylQd", "reviewer": "Reviewer_hGxq", "summary": "This paper provides a new approach for using language model assistants, specifically enabling real-time interactions with the language model in a full-duplex setting.\n\nIn detail, the authors achieved this by additionally training the Llama-3-Instruct model and integrating it with ASR and TTS modules, creating a complete pipeline system. This demonstration system exhibits performance in voice dialogue that surpasses commercial models.\n\nI really like the demo implemented by the authors, and I even opened the code to try running it (although lacking the LLM, I understood its principles). I support accepting this work!", "questions": "A. Can you honestly tell me when you came up with this idea? Was it shortly after the release of GPT-4o?\n\nB. Have you considered building an end-to-end model? Integrating ASR and TTS into the LLM as well to achieve faster response times?\n\nC. Have you considered the limitations of this model?\n\nD. I'm curious, is 20 steps of training really enough for the Llama-3-Instruct 8B model?", "limitations": "None", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This work provides technical details on achieving full-duplex interaction with language models, offering unprecedented technical specifics (since GPT-4o has not been open-sourced).\n\nThis work presents a quick response method to handle user queries. It even shows a clear advantage over GPT-4o in terms of appropriate responsiveness to user interruptions.", "weaknesses": "**Novelty of Motivation.** Considering that this paper uses the Llama-3-Instruct model and makes extensive comparisons with GPT-4o, it can be inferred that this paper was completed after the release date of GPT-4o. GPT-4o was the first language model to achieve full-duplex communication, so the motivation of this paper largely draws from GPT-4o's ideas. The main contribution lies in providing the technical methods to replicate GPT-4o's full-duplex communication.\n\n**Performance Degradation.** From Table 3, we can see that the method provided by the authors comes at the cost of reduced inference performance of the model, which is a drawback.\n\n**Poor Code Readability.** I hope the authors can completely refactor their code and present it in the Huggingface format. In my opinion, the main contribution of this work is providing a demonstration system, but the code's readability is poor and hard to operate. If the authors could use a more concise style for the entire model, I believe this would greatly increase the community's interest in this work. For example, in the Llama style:\n\n```python\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\nimport torch\nmodel_id = \"meta-llama/Meta-Llama-3-8B-Instruct\"\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nmodel = AutoModelForCausalLM.from_pretrained(\n    model_id,\n    torch_dtype=torch.bfloat16,\n    device_map=\"auto\",\n)\nmessages = [\n    {\"role\": \"system\", \"content\": \"You are a pirate chatbot who always responds in pirate speak!\"},\n    {\"role\": \"user\", \"content\": \"Who are you?\"},\n]\ninput_ids = tokenizer.apply_chat_template(\n    messages,\n    add_generation_prompt=True,\n    return_tensors=\"pt\"\n).to(model.device)\nterminators = [\n    tokenizer.eos_token_id,\n    tokenizer.convert_tokens_to_ids(\"
\")\n]\noutputs = model.generate(\n    input_ids,\n    max_new_tokens=256,\n    eos_token_id=terminators,\n    do_sample=True,\n    temperature=0.6,\n    top_p=0.9,\n)\nresponse = outputs[0][input_ids.shape[-1]:]\nprint(tokenizer.decode(response, skip_special_tokens=True))\n```", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their review and positive feedback. We are glad the reviewer like our demo.\n\n> **Performance Degradation.** \n\nIn this experiment, we used only the full-duplex conversation data for SFT, which is why other capabilities were compromised. However, if full-duplex conversation data were mixed with other types of data for SFT, this detrimental effect would be mitigated. Essentially, the full-duplex conversation data is merely performing a specific type of alignment. Nowadays, there are numerous alignment training techniques that can prevent adverse impacts.\n\n> **Poor Code Readability.**\n\nThank you for your suggestion, we will work on further improving the code readability.\n\n> A. Can you honestly tell me when you came up with this idea? Was it shortly after the release of GPT-4o?\n\nWe started this project around February 2024. The first demo was showcased in March 2024, followed by continuous iterations and updates since then. We can provide git log for furthur clarification if necessary.\n\n> B. Have you considered building an end-to-end model? Integrating ASR and TTS into the LLM as well to achieve faster response times?\n\nPlease refer to  the overall comments. \n\n> C. Have you considered the limitations of this model?\n\nYes, for instance, the issue of the FSM state changing frequently due to varying user speech rates necessitates a robust SFT alignment capability to strengthen the model's handling of such scenarios. Furthermore, whether relying on ASR or a multi-modal model's speech tokenizer to process audio, slicing the speech stream by chunk can introduce errors.\n\n> D. I'm curious, is 20 steps of training really enough for the Llama-3-Instruct 8B model?\n\nWe found that for LLMs, learning the neural FSM task is relatively straightforward, requiring only a small amount of data and incurring minimal training overhead to achieve the desired results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their careful reading, valuable comments, kind words and recognition of the contributions. We first answer a question raised by multiple reviewers.

> One big unified model instead of integrating with some external ASR/TTS models

Training an end-to-end unified multimodal LLM requires substantial computational resources and no open-sourced model with the similar language modeling capability as LLaMA-8B is currently available. This paper presents a solution that achieves full-duplex dialogue effects using only existing open-source models, with very low latency. 

We also note that the concept of neural FSM is not limited to pipelines composed of separate models. Interaction with the physical world necessitates the system to make decision on producing output and waiting for output. Large Language Models, when equipped with the capability to understand and product audio tokens, can still utilize neural FSM to make such decisions. Our paper introduces a mechanism enabling the model to learn how to interact with the physical world through three abstract modules. An end-to-end multimodal large language model can be seen as a changing the implementation of these three abstract modules, where perception module becomes audio tokenizer, and motor function module turn into audio token decoder. We anticipate that the neural FSM approach will remain compatible with this type of model. However, due to the lack of open-source LLMs equipped with audio modality, we are unable to test this hypothesis.

Another interesting question is whether this idea is derived from GPT-4o. We sincerely state that this work is developed independently from GPT-4o, which is a close-sourced commercial product released in May 13, 2024. We started this project around February 2024. The first internal demo was showcased in March 2024, followed by continuous iterations and updates since then. We are willing to provide git commit histories as proof for the development dates if the program committee deems it necessary.

Meta-review: This work presents a full-duplex generative dialog system that enables real-time interactions with a Large Language Model, i.e., the authors introduce a neural finite-state-machine and model the tasks of responding to, waiting for, or interrupting the user as next token prediction on a serialized view of the dialog in real-time. The paper relies on using existing open-sourced models. The authors report improvements over baselines in conversation response latency and interruption precision with a smaller model.

One weakness that the reviewers raised was regarding the need to use ASR and TTS models instead of a unified model, and how the errors could propagate or information may be lost due to this design choice. Although a valid point, the overall approach of the paper may be applied to such unified model. A second concern was raised regarding whether the metrics reported in the paper (latency and interruption precision) are sufficient to evaluate dialog (e.g., whether the response is helpful or relevant). This is a valid question and I disagree with authors' response that "in reality, for current large language models, the helpfulness and relevance of the generated content can be guaranteed" as evidenced by the works that attempt to improve the quality of response of LLMs. The authors should make this clear as a limitation of their experimental work in the final draft. Furthermore, the authors should avoid making hypothetical statements in the paper that are not backed by evidence (e.g., lines 326-328 of the paper). 

Given the focus of the paper on supporting natural multi-turn dialog and considering the limitations in evaluation discussed above, I recommend accepting the paper for a poster presentation.
Decision: Accept (poster)