Introduction
Recent progress in generative spoken language modeling (Lakhotia et al., 2021 ; Borsos et al., 2023a ; Wang et al., 2023 ; Kharitonov et al., 2023 ; Borsos et al., 2023b ; Rubenstein et al., 2023 ; Le et al., 2023 ; Vyas et al., 2023 ; Hassid et al., 2024 ; Nguyen et al., 2024 ) brought models that can generate speech in a wide range of voices, prosody and recording conditions as found in large-scale datasets of natural speech Kahn et al. ( 2020 ); Pratap et al. ( 2020 ); Ardila et al. ( 2020 ) .

These generative models are typically evaluated in terms of faithfulness to their inputs, i.e. speaker identity and transcript (Borsos et al., 2023a ; Wang et al., 2023 ; Kharitonov et al., 2023 ) . Thus it remains unknown how diverse their outputs are, since those metrics neither capture variability nor take into account factors of the perceived speech diversity (i.e., recording conditions, prosody, styles, and accents). Being able to measure speech diversity would also help detect mode collapse when finetuning models (Kirk et al., 2023 ) , selecting training & inference hyperparameters, and optimizing for human feedback (Cideron et al., 2024 ) Moreover, such measurements are vital when building synthetic data or mixing existing natural datasets.

In this work, we focus on measuring acoustic diversity of a set of speech utterances, i.e. measuring the diversity in the distribution of voices, genders, accents, emotions, and background noise. Throughout the paper, we use the term “facets” to describe these different aspects of acoustic diversity.

We propose a set of metrics, which we collectively refer to as MAD Speech ( M easures of A coustic D iversity of Speech). We build these metrics by combining a pretrained speech representation and an aggregation function that is used to calculate the diversity of a sample in the representation space. We experiment with a set of off-the-shelf continuous speech representations such as HuBERT Hsu et al. ( 2021 ) , Wav2Vec-BERT Chung et al. ( 2021a ) , and SoundStream Zeghidour et al. ( 2021 ) as well as with SpeechSim, a lightweight representation model related to TRILL Shor et al. ( 2020 ) and COLA Saeed et al. ( 2021 ) . We explore the average pairwise cosine dissimilarity and Vendi Score Friedman and Dieng ( 2022 ) as aggregation functions.

Further, we learn per-facet projection models with the goal of highlighting contributions of each facet “independently” from others. Such a setup is beneficial since it (a) provides a better insight into the relative contributions of different aspects of diversity, and (b) allows to avoid fusing of the different facets in the same metric with uncontrolled relative importance.

To evaluate the proposed metrics, we build a collection of datasets with known ground-truth acoustic diversity levels. We achieve that by sub-sampling standard speech datasets, while controlling for some measurable parameter that reflects the diversity of the samples along a particular facet (e.g., number of unique voices). Repeating that for all facets, we build a suite of datasets that can be used to measure to what extent a particular metric is capable of measuring acoustic diversity.

Finally, we run a set of studies to measure the impact of various improvements proposed in the literature on the acoustic diversity: fast non-autoregressive decoding of SoundStorm replacing the auto-regressive semantic-to-acoustic model of AudioLM Borsos et al. ( 2023a , b ) and Best-of-K decoding of SPEAR-TTS to increase quality of generated audio Kharitonov et al. ( 2023 ) . We also investigate how changes in the temperature sampling affect the acoustic diversity and compare several off-the-shelf TTS systems. These studies highlight that acoustic diversity changes non-trivially across the considered scenarios and it must be taken into account when selecting a model and a sampling setup.

The suite of MAD Speech metrics will be made publicly accessible.