Beyond the Turn-Based Game: Enabling Real-Time Conversations with Duplex
  Models
Authors: name: Xinrong Zhang; name: Yingfa Chen; name: Shengding Hu; name: Xu Han; name: Zihang Xu; name: Yuanwei Xu; name: Weilin Zhao; name: Maosong Sun; name: Zhiyuan Liu
Published: 2024-06-22 03:20:10

Abstract: As large language models (LLMs) increasingly permeate daily lives, there is a
growing demand for real-time interactions that mirror human conversations.
Traditional turn-based chat systems driven by LLMs prevent users from verbally
interacting with the system while it is generating responses. To overcome these
limitations, we adapt existing LLMs to duplex models so that these
LLMs can listen for users while generating output and dynamically adjust
themselves to provide users with instant feedback. % such as in response to
interruptions. Specifically, we divide the queries and responses of
conversations into several time slices and then adopt a
time-division-multiplexing (TDM) encoding-decoding strategy to
pseudo-simultaneously process these slices. Furthermore, to make LLMs
proficient enough to handle real-time conversations, we build a fine-tuning
dataset consisting of alternating time slices of queries and responses as well
as covering typical feedback types in instantaneous interactions. Our
experiments show that although the queries and responses of conversations are
segmented into incomplete slices for processing, LLMs can preserve their
original performance on standard benchmarks with a few fine-tuning steps on our
dataset. Automatic and human evaluation indicate that duplex models make
user-AI interactions more natural and human-like, and greatly improve user
satisfaction compared to vanilla LLMs. Our duplex model and dataset will be
released.