NVSpeech: An Integrated and Scalable Pipeline for Human-Like Speech
  Modeling with Paralinguistic Vocalizations
Authors: name: Huan Liao, name: Qinke Ni, name: Yuancheng Wang, name: Yiheng Lu, name: Haoyue Zhan, name: Pengyuan Xie, name: Qiang Zhang, name: Zhizheng Wu

Paralinguistic vocalizations-including non-verbal sounds like laughter and
breathing, as well as lexicalized interjections such as "uhm" and "oh"-are
integral to natural spoken communication. Despite their importance in conveying
affect, intent, and interactional cues, such cues remain largely overlooked in
conventional automatic speech recognition (ASR) and text-to-speech (TTS)
systems. We present NVSpeech, an integrated and scalable pipeline that bridges
the recognition and synthesis of paralinguistic vocalizations, encompassing
dataset construction, ASR modeling, and controllable TTS. (1) We introduce a
manually annotated dataset of 48,430 human-spoken utterances with 18 word-level
paralinguistic categories. (2) We develop the paralinguistic-aware ASR model,
which treats paralinguistic cues as inline decodable tokens (e.g., "You're so
funny [Laughter]"), enabling joint lexical and non-verbal transcription. This
model is then used to automatically annotate a large corpus, the first
large-scale Chinese dataset of 174,179 utterances (573 hours) with word-level
alignment and paralingustic cues. (3) We finetune zero-shot TTS models on both
human- and auto-labeled data to enable explicit control over paralinguistic
vocalizations, allowing context-aware insertion at arbitrary token positions
for human-like speech synthesis. By unifying the recognition and generation of
paralinguistic vocalizations, NVSpeech offers the first open, large-scale,
word-level annotated pipeline for expressive speech modeling in Mandarin,
integrating recognition and synthesis in a scalable and controllable manner.
Dataset and audio demos are available at https://nvspeech170k.github.io/.