RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
SPEAR-TTS: Best-of-K decoding
Kharitonov et al. ( 2023 ) leveraged the stochasticity of the temperature decoding to improve audio quality of the generated speech in an AudioLM-like model. Namely, they proposed to sample K examples for the same input and then select the sample with the highest acoustic quality as the output. Now we check if that approach brings a penalty to the acoustic diversity.

Since SPEAR-TTS is only stochastic in the semantic-to-acoustic token decoding, we use the same task and AudioLM model as above. Again, for each semantic token sequence, we sample 128 acoustic token sequences that correspond to it. Next, we randomly split them in groups of K\in\{1,2,4\} and, within each group, select one with the highest acoustic quality score, relying on the same model as Kharitonov et al. ( 2023 ) . After this filtering, each semantic token sequence is mapped into 128, 64, or 32 audio samples (for K=1, 2, and 4). In the first two cases, we sample 32 sequences so that all sets have the same size.

From Table 10 see that, except for the gender facet, higher K always brings a diversity penalty.
Text Synthesis: Influence of Temperature
Temperature is a parameter aiming at controlling the sampling distribution. As the sampling temperature approaches zero, samples for a fixed input become increasingly deterministic; while a high temperature introduces diversity in the sampling process. A priori, one can expect that this token-level diversity can translate to higher-level, per-utterance acoustic diversity.
In this experiment we use a single-stage fully auto-regressive TTS system that maps grapheme text representations to a sequence of acoustic tokens, akin to VALL-E Wang et al. ( 2023 ) . This model is trained on a combination of Multilingual LibriSpeech Pratap et al. ( 2020 ) , LibriLight Kahn et al. ( 2020 ) and other public data. Table 11 shows that voice diversity is by far the most affected by small variations of temperature while gender and background noise are barely impacted. Samples with low temperature are also less diverse in terms of emotion and accent.
Acoustic diversity in Text-to-Speech systems
Next, we measure acoustic diversity of several publicly available TTS systems. Here, our goal is to show that MAD Speech brings a new dimension to the evaluation of real-world TTS systems.
We compare the following systems:

Bark TTS is a generative text-to-audio system that is based on a decoder-only Transformer model that operates on discretized speech representations, akin to AudioLM Borsos et al. ( 2023a ) and VALL-E Wang et al. ( 2023 ) . Bark TTS synthesizes utterances in a random voice and acoustic conditions.

StyleTTS 2 combines a style diffusion model with speech with WavLM-based adversarial training Chen et al. ( 2022a ) for an end-to-end training of a multi-component system. We use a checkpoint trained on LibriTTS Zen et al. ( 2019 ) . The samples are conditioned on a female voice, but randomize timbre and prosody. 1 1 1 https://colab.../Inference_LibriTTS.ipynb

Tortoise TTS combines auto-regressive and diffusion-based components Betker ( 2023 ) . For training, it pooled LibriTTS Zen et al. ( 2019 ) , HiFiTTS Bakhturina et al. ( 2021 ) , and public data sourced from audiobooks and podcasts.

FastSpeech 2 consists of a Transformer-based encoder that embeds phoneme-based text representation, a variance adapter that predicts pitch, energy, and duration values, and a non-autoregressive decoder. We use a publicly available re-implementation trained on LJSpeech Ito and Johnson ( 2017 ) . 2 2 2 https://github.com/ming024/FastSpeech2

As some of these systems are deterministic, we adopt a different setup and synthesize all transcripts in shuffled LibriTTS test-clean and calculate the average acoustic diversity across batches of audio of size 128. We report the results in Table 12 .

We see that Bark and Tortoise TTS are the most diverse across all facets, likely due to training on more diverse internet data. Interestingly, Tortoise TTS tends to be more diverse that Bark in emotions while less or equally diverse in other facets. StyleTTS 2 is considerably less diverse in the gender facet — just as one would expect, given that the generation is restricted to female voices. FastSpeech 2 is the least acoustically diverse due to a combination of (a) training on a very homogeneous dataset, and (b) having a mostly non-autoregressive architecture.

All in all, the comparisons we carried out in this Section show that previously proposed changes lead to non-trivial and sometimes even non-monotonic changes in per-facet acoustic diversity. Equally, the sampling temperature, a frequently tuned hyperparameter, also affects the resulting acoustic diversity. We also found that MAD Speech can be useful in detecting acoustic diversity trends in off-the-shelf TTS systems. Overall, we believe those findings highlight the necessity for including acoustic diversity metrics, such as MAD Speech, in the standard evaluations.
Limitations and Broader Impact
By building MAD Speech, we make it possible to holistically assess progress in generative speech and reveal possible hidden biases in the acoustic scenes. Yet, our metrics are machine-learned models themselves and are limited by the nature of the data we used to build them. Specifically, the public datasets we use are limited to English language. The labels in the data can be noisy and only represent a limited reflection of the world.
Conclusions & Future work
We introduced MAD Speech, a set of metrics for evaluating different aspects of acoustic diversity in speech. MAD Speech metrics are calculated in two steps: (1) a set of utterances is mapped to an embedding space, (2) acoustic diversity is estimated by comparing the resulting embeddings with an aggregation function (cosine dissimilarity or Vendi Score). Calculating embeddings is done by firstly applying a general-purpose representation model with a subsequent metric-specific projection. We focus on five facets of acoustic diversity: voices, gender, emotion, accents, and background noise.

In order to validate the proposed metrics, we built a collection of datasets with a controlled level of acoustic diversity. Having these sets allows us to evaluate a metric by calculating its Spearman rank correlation with the ground-truth acoustic diversity.

Our empirical study demonstrated that our proposed metrics have a highest agreement with the ground-truth diversity levels when compared to baseline approaches and that they are insensitive to variations of other facets of diversity.

Finally, we highlighted a necessity of controlling acoustic diversity when developing new models and approaches. To this end, we took some recent modelling improvements and demonstrated that they, in fact, affect acoustic diversity of speech. Moreover, we show that even such a standard hyperparameter as sampling temperature has effect on speech diversity. We hope that our metrics would provide a useful tool for a holistic evaluation of developments in the field of speech generation.

Extending MAD Speech to cover more languages and facets is a promising directions of future work.

Acknowledgements The authors are grateful to Aren Jansen and Salah Zaiem for their feedback.
(a) SpeechSim/Gender, avg. pairwise dissim.; (b) SpeechSim/Voice, Vendi score.; Changing the proportion of female voices and number of distinct speaker voices simultaneously. Grey area represents 95% CI.
Voice diversity. Average Spearman correlations ( \pm standard error) between number of distinct speakers and diversity scores induced by speech representations. Male (Female) refer to samples with male-only (female-only) voices. Best results are in bold, second best results are underlined.
Gender diversity. Average Spearman correlations between proportion of female voices and diversity scores induced by speech representations.
Emotion diversity. Average Spearman correlations between the classes entropy in EmoV and Expresso and diversity scores induced by the speech representations. SpeechSim/Emotion-Expresso (resp. Speech/Emotion-EmoV) refers to SpeechSim emotion head trained on Expresso (resp. EmoV).
Accent diversity. Average Spearman correlations between the accent class entropy and diversity scores induced by speech representations.
Background noise diversity. Average Spearman correlations between the number of different classes of noise and diversity scores induced by speech representations.
Changing voice and gender diversity in the opposite directions. Average Spearman correlation between the gender diversity and the diversity scores induced by speech representations.
Changing emotion and speaker diversity in the opposite directions. Average Spearman correlation between emotion diversity and diversity scores induced by speech representations.
Changing accent and voice diversity in the opposite directions. Average Spearman correlation between accent diversity and diversity scores induced by speech representations.
Semantic-to-acoustic tokens resynthesis task. The proportion of utterances (%) where the AudioLM stage 2 model generates samples that are more diverse than those generated by SoundStorm.
Best-of-K filtering on the semantic-to-acoustic tokens resynthesis task. The proportion (%) of utterances where Best-K with K\in\{1,4\} have higher diversity scores than with K=2 .
T\in\{0.7,0.9\} have higher diversity scores than with T=0.8 .
Acoustic diversity scores for off-the-shelf TTS systems (higher is more diverse). We report Vendi score for all facets except for gender, which uses cosine dissimilarity.