Making Flow-Matching-Based Zero-Shot Text-to-Speech Laugh as You Like
Authors: name: Naoyuki Kanda; name: Xiaofei Wang; name: Sefik Emre Eskimez; name: Manthan Thakker; name: Hemin Yang; name: Zirun Zhu; name: Min Tang; name: Canrun Li; name: Steven Tsai; name: Zhen Xiao; name: Yufei Xia; name: Jinzhu Li; name: Yanqing Liu; name: Sheng Zhao; name: Michael Zeng
Published: 2024-02-12 02:58:10

Abstract: Laughter is one of the most expressive and natural aspects of human speech,
conveying emotions, social cues, and humor. However, most text-to-speech (TTS)
systems lack the ability to produce realistic and appropriate laughter sounds,
limiting their applications and user experience. While there have been prior
works to generate natural laughter, they fell short in terms of controlling the
timing and variety of the laughter to be generated. In this work, we propose
ELaTE, a zero-shot TTS that can generate natural laughing speech of any speaker
based on a short audio prompt with precise control of laughter timing and
expression. Specifically, ELaTE works on the audio prompt to mimic the voice
characteristic, the text prompt to indicate the contents of the generated
speech, and the input to control the laughter expression, which can be either
the start and end times of laughter, or the additional audio prompt that
contains laughter to be mimicked. We develop our model based on the foundation
of conditional flow-matching-based zero-shot TTS, and fine-tune it with
frame-level representation from a laughter detector as additional conditioning.
With a simple scheme to mix small-scale laughter-conditioned data with
large-scale pre-training data, we demonstrate that a pre-trained zero-shot TTS
model can be readily fine-tuned to generate natural laughter with precise
controllability, without losing any quality of the pre-trained zero-shot TTS
model. Through the evaluations, we show that ELaTE can generate laughing speech
with significantly higher quality and controllability compared to conventional
models. See https://aka.ms/elate/ for demo samples.