Vox-Profile: A Speech Foundation Model Benchmark for Characterizing
  Diverse Speaker and Speech Traits
Authors: name: Tiantian Feng, name: Jihwan Lee, name: Anfeng Xu, name: Yoonjeong Lee, name: Thanathai Lertpetchpun, name: Xuan Shi, name: Helin Wang, name: Thomas Thebaud, name: Laureano Moro-Velazquez, name: Dani Byrd, name: Najim Dehak, name: Shrikanth Narayanan

We introduce Vox-Profile, a comprehensive benchmark to characterize rich
speaker and speech traits using speech foundation models. Unlike existing works
that focus on a single dimension of speaker traits, Vox-Profile provides
holistic and multi-dimensional profiles that reflect both static speaker traits
(e.g., age, sex, accent) and dynamic speech properties (e.g., emotion, speech
flow). This benchmark is grounded in speech science and linguistics, developed
with domain experts to accurately index speaker and speech characteristics. We
report benchmark experiments using over 15 publicly available speech datasets
and several widely used speech foundation models that target various static and
dynamic speaker and speech properties. In addition to benchmark experiments, we
showcase several downstream applications supported by Vox-Profile. First, we
show that Vox-Profile can augment existing speech recognition datasets to
analyze ASR performance variability. Vox-Profile is also used as a tool to
evaluate the performance of speech generation systems. Finally, we assess the
quality of our automated profiles through comparison with human evaluation and
show convergent validity. Vox-Profile is publicly available at:
https://github.com/tiantiaf0627/vox-profile-release.