Voila: Voice-Language Foundation Models for Real-Time Autonomous
  Interaction and Voice Role-Play
Authors: name: Yemin Shi; name: Yu Shu; name: Siwei Dong; name: Guangyi Liu; name: Jaward Sesay; name: Jingwen Li; name: Zhiting Hu
Published: 2025-05-05 15:05:01

Abstract: A voice AI agent that blends seamlessly into daily life would interact with
humans in an autonomous, real-time, and emotionally expressive manner. Rather
than merely reacting to commands, it would continuously listen, reason, and
respond proactively, fostering fluid, dynamic, and emotionally resonant
interactions. We introduce Voila, a family of large voice-language foundation
models that make a step towards this vision. Voila moves beyond traditional
pipeline systems by adopting a new end-to-end architecture that enables
full-duplex, low-latency conversations while preserving rich vocal nuances such
as tone, rhythm, and emotion. It achieves a response latency of just 195
milliseconds, surpassing the average human response time. Its hierarchical
multi-scale Transformer integrates the reasoning capabilities of large language
models (LLMs) with powerful acoustic modeling, enabling natural, persona-aware
voice generation -- where users can simply write text instructions to define
the speaker's identity, tone, and other characteristics. Moreover, Voila
supports over one million pre-built voices and efficient customization of new
ones from brief audio samples as short as 10 seconds. Beyond spoken dialogue,
Voila is designed as a unified model for a wide range of voice-based
applications, including automatic speech recognition (ASR), Text-to-Speech
(TTS), and, with minimal adaptation, multilingual speech translation. Voila is
fully open-sourced to support open research and accelerate progress toward
next-generation human-machine interactions.