RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Diversity in generative models
Different diversity metrics have been proposed for evaluating generative models in computer vision (Heusel et al., 2017 ; Naeem et al., 2020 ; Sajjadi et al., 2018 ; Kynkäänniemi et al., 2019 ) and natural language processing (Zhang et al., 2020 ; Li et al., 2016 ; Zhu et al., 2018 ; Shen et al., 2019 ; Dušek et al., 2019 ; Shu et al., 2019 ) . However these metrics are either computed on raw inputs for text (Zhu et al., 2018 ; Shen et al., 2019 ; Alihosseini et al., 2019 ) or based on underlying representations specific to text and images (Heusel et al., 2017 ; Cífka et al., 2018 ; Tevet and Berant, 2021 ) which makes them unsuitable for evaluating acoustic diversity in generative speech models.
Diversity metrics for generative audio
Already the first paper proposing a generative spoken language model (GSLM) recognized the importance of measuring speech diversity (Lakhotia et al., 2021 ) . However, since GSLM is not able to generate acoustically diverse content (e.g., it supports only a predefined set of speakers), the focus of Lakhotia et al. ( 2021 ) was on measuring the diversity on the transcript level. In a follow-up paper, Kharitonov et al. ( 2022 ) proposed a modification of GSLM that has a more expressive prosody yet their focus was on improving and measuring within-utterance variability of prosody. To the best of our knowledge, our work is the first to study across-utterance acoustic diversity in speech generation.
Speech representations
Learning compact and powerful speech representations is an extremely prolific research area (e.g., van den Oord et al., 2018 ; Schneider et al., 2019 ; Baevski et al., 2020 ; Saeed et al., 2021 ; Hsu et al., 2021 ; Kharitonov et al., 2021 ; Chung et al., 2021b ; Baevski et al., 2022 ; Chen et al., 2022b ) . Often, generative audio systems are built on top of such representations by appropriately discretizing them Lakhotia et al. ( 2021 ); Polyak et al. ( 2021a ); Kreuk et al. ( 2022 ); Kharitonov et al. ( 2022 ); Borsos et al. ( 2023a ); Rubenstein et al. ( 2023 ); Défossez et al. ( 2023 ) . This informs our choice of representations to experiment with: HuBERT (Hsu et al., 2021 ) (used in the GSLM family of models (Lakhotia et al., 2021 ; Polyak et al., 2021b ; Kreuk et al., 2021 ; Kharitonov et al., 2022 ; Nguyen et al., 2023 ) ) and Wav2Vec-BERT (Chung et al., 2021b ) (used by the AudioLM lineage (Borsos et al., 2023a ; Kharitonov et al., 2023 ; Borsos et al., 2023b ; Agostinelli et al., 2023 ) ). We complement these off-the-shelf representations by SpeechSim, a variant of TRILL (Shor et al., 2020 ) and COLA (Saeed et al., 2021 ) that learns to contrast pairs of speech chunks. In turn, we also learn projection models that map SpeechSim representations into embeddings that are specialized for each specific facet of diversity.
Learning Representations
We take a two-steps approach for measuring diversity of a set of speech utterances. First, we encode raw waveform audio in a compact representation and, afterwards, we estimate the diversity of a set of utterances in this embedding space. Hence, it is vital that the information about the relevant properties of speech is preserved in the embedding space, while the irrelevant details are omitted. We calculate embeddings in a hierarchical way: at first, we map audio to general-purpose embeddings and then apply lightweight specialized projection models that are tailored to capture a specific facet. As a result, our approach allows us to get reliable per-facet diversity metrics.

SpeechSim We aim to learn speech representations where acoustically similar audio segments are close to each other in the embedding space. Following Saeed et al. ( 2021 ) , we train a self-supervised contrastive model with positive pairs coming from non-overlapping fixed-length chunks of the same speech utterance. The main idea being that such speech segments are likely to be acoustically similar while being semantically different — as there is no overlap between positive pairs. We refer to this model as SpeechSim . SpeechSim provides a speech representation that are sensitive to changes in acoustic diversity, however, it has all types of acoustic properties mixed up together. Indeed, our experiments in Section 7 show that the vanilla SpeechSim embeddings are strongly biased towards speaker voice. We therefore learn projection models (heads) on top of SpeechSim embeddings to extract different types of acoustic information.

Learning Projections We want to extract the signal relevant to each individual facet from our pretrained embedding while suppressing information relevant to other facets. Thus, we train lightweight projection models on top of pretrained SpeechSim embeddings for each facet. To this end, we assume that we have labelled data and train these models with a contrastive objective where positive pairs are sampled from the same class w.r.t the facet.
From Speech Representations to Diversity Metrics
Once a set of speech utterances is encoded with one of the representations (Section 3 ), we can measure the diversity within the set in a variety of ways. We focus on two measures: mean pairwise dissimilarity and Vendi Score (Friedman and Dieng, 2022 ) .
Mean pairwise dissimilarity
Denoting the sample size as n and embeddings as {\bm{e}} , we calculate:

1-\frac{1}{n(n-1)}\sum_{i eq j;i,j\leq n}\frac{{\bm{e}}_{i}^{T}{\bm{e}}_{j}}{||{\bm{e}}_{i}||\cdot||{\bm{e}}_{j}||} (1)

We subtract the mean cosine similarity from 1, as it is more convenient to measure diversity (dissimilarity) rather than similarity. This way, an agreement with the ground-truth diversity would result in a positive correlation (see Section 5 ).
Vendi Score
Friedman and Dieng ( 2022 ) proposed to measure diversity of a sample as the exponential of the Shannon entropy of the eigenvalues of the pairwise similarity matrix, normalized by the number of vectors. In practice, we built it by computing pairwise cosine similarity and dividing by the number of examples. Formally, denoting the eigenvalues of the normalized similarity matrix as \{\lambda_{k}\}_{k=1}^{n} , Vendi Score is calculated as:

\exp\left[-\sum_{i=1}^{n}\lambda_{i}\log\lambda_{i}\right] (2)

Eq. 2 is well-defined, as the normalized similarity matrix is positive semi-definite and 0\leq\lambda_{i}\leq 1 .
Our Approach
We break down the acoustic diversity into the following facets: speaker voice, gender, emotion (or style), accent, and background noise. For each of these facets, we repeatedly sample subsets from a large dataset of natural speech utterances while systematically increasing the level of diversity in a controllable way, as further described below. Next, we calculate Spearman’s rank correlation (Spearman, 1904 ) between diversity metric scores and the ground-truth diversity level’s rank. Metrics with a higher rank correlation are better at representing a particular facet of the acoustic diversity.

In the following, we describe the general template for building a series of datasets of controllable diversity, using voice diversity as a reference. We fix the sample size a priori and we define voice diversity as the number of distinct speakers in a set of examples. To build one dataset, we sample examples from a natural dataset such that we obtain the required number of speakers and have every speaker represented by the same number of utterances. We create a series of datasets by varying the number of distinct speakers. We also ensure that other facets do not differ from one dataset to another as much as possible so that we make sure we indeed test speaker voice diversity.

This process is similar for other facets. We determine gender diversity as a proportion of female voices; background noise diversity is characterized by the number of different background noise classes; and emotion (accent) diversity is determined by computing the entropy of the distribution of the different emotion (accent) classes.
Speakers voices
We build evaluation (validation) sets from LibriTTS (Zen et al., 2019 ) test-clean & test-other (dev-clean & dev-other). Each set contains 200 utterances of a single gender (either all male voices or all female voices) to test whether a particular metric is sensitive to the voice diversity independently from the gender diversity. We vary the number of distinct voices in \{5,10,15,20,25,33\} . For each diversity level, we randomly sample 100 sets using different seeds.

To test the influence of gender diversity for the metrics that target voice diversity, we generate additional data where the voice and the gender diversity are pushed in the opposite directions: having a low number of speakers with uniform gender distribution, and a single gender (selected randomly) when the number of speakers is high.
Gender
We build evaluation (validation) sets from LibriTTS (Zen et al., 2019 ) test-clean & test-other (dev-clean & dev-other). Each set contains 100 utterances. We change the ratio of female voices from 0.0 to 1.0 in increments of 0.1.

We built two types of evaluation samples, one with equal number of different speaker voices in the samples and another one where number of different speaker voices is maximum (resp. minimum) when there is few (resp. lots of) gender variation to test if the gender metric is speaker-sensitive.
Emotion
We build evaluation (validation) sets from 40% (10%) of EmoV (Adigwe et al., 2018 ) randomly sampled and test (dev) splits of Expresso (Nguyen et al., 2023 ) .
We built two types of samples: (1) with equal number of different speaker voices and (2) with a single speaker voice for datasets with high entropy but all speaker voices—same number of utterances per speaker—for datasets with low entropy. Speakers in test and dev sets are identical to those found in the training set. Emotion classes match in train, dev, and test.
Accent
We build evaluation (validation) sets from test (dev) sets of VCTK (Veaux et al., 2016 ) . Accent classes are common between train, dev, and tests sets. Again, we built two types of samples (1) with equal number of different speaker voices and (2) with a unique speaker voice for high-entropy datasets but all speaker voices— equally represented—for low-entropy datasets. Speakers in test and dev sets are distinct from those found in the training set.
Background Noise
We build evaluation and validation sets by sampling 20% and 10% of AudioSet, respectively (Gemmeke et al., 2017 ) .
We select examples that have exactly two tags, one of them being “Speech”. We treat the second tag as a class of the background noise. Next, we construct a sample by we randomly selecting 100 examples with varying number of distinct noise classes (1, 5, 10, 25, 50, or 100).
Data
To train SpeechSim, we use LibriLight (Kahn et al., 2020 ) , an unlabelled dataset containing approximately 60,000 hours of self-recorded audiobooks. To train projection models on top of SpeechSim, we use different datasets depending on the target facet. Specifically, the projection models related to speaker voices and gender are trained on LibriTTS train-other (Zen et al., 2019 ) . The projector related to emotion is trained on EmoV (Adigwe et al., 2018 ) and Expresso (Nguyen et al., 2023 ) train splits. For Expresso, we selected audios with a single speaker and removed narration and enunciated classes. Expresso and EmoV datasets have only the neutral class in common.

Next, the accent projection model is trained on VCTK-train (Veaux et al., 2016 ) , which contains 12 English accents spoken by 110 speakers. Finally, the background noise projection model is trained on AudioSet (Gemmeke et al., 2017 ) . Here, we select all examples that are tagged as “Speech” and which have between 1 and 3 additional tags as we found examples with more tags to be too noisy. We randomly sample 70% of the selected examples to build the train set, the remaining 30% are randomly split as dev and test.
SpeechSim
The input to SpeechSim is a 6-seconds chunk of audio with sampling rate of 16kHz. We compute the mel-spectrogram using a window length of 512, hop length of 256 and 156 bins. This produces 376 temporal frames of dimension 156. The model is based on a small ViT architecture (Dosovitskiy et al., 2020 ) , containing 12 layers with each layer having 6 attention heads, embedding dimension of 512, and a feed-forward layer of dimension 1024 (25M parameters). The final embeddings are averaged across the time axis and are projected into a 192-dimensional vector.
We train SpeechSim with the semi-hard triplet loss (Schroff et al., 2015 ) , using a total batch size of 3840. We train the model for a total of 10^{5} steps.
Projection models
We train projection models on top of SpeechSim embeddings in a contrastive manner. The positive pairs are constructed by taking two examples with the same labels (e.g., same emotion). Except for the gender projector, all models have 2 fully connected layer (of size 256 and 128) with a GELU activation (Hendrycks and Gimpel, 2016 ) in-between. In the case of the gender projector, we found that a 4-layers network works better (dimensions of 256, 256, 128 and 128). Dropout rate of 0.1 is applied to SpeechSim input representations. We trained the projection models using Adam (Kingma and Ba, 2014 ) and use a learning rate of 1e-4, and a batch size of 128. We also apply a weight decay of 1e-3 for projection models related to speaker voices, emotion and background noise and 1e-4 for other facets. We select hyperparameters using validation loss.

All embeddings that we use in our experiments have a single vector representing the entire utterance. To achieve that, we average embeddings over the time axis. We found this to perform better than max-pooling.
Wav2Vec-BERT
We experiment with representations provided by a 600M-parameter Wav2Vec-BERT model Chung et al. ( 2021a ) . We take embeddings of intermediate layers, with each utterance represented as a sequence of 1024-dimensional vectors corresponding to 40ms frame.
HuBERT
We use HuBERT embeddings, produced by the Base model Hsu et al. ( 2021 ) (95M parameters), trained on LibriSpeech Panayotov et al. ( 2015 ) . We follow Lakhotia et al. ( 2021 ) and use embeddings extracted from the 6th layer. This model has a framerate of 20 ms.
SoundStream
Another type of representations found to be useful in AudioLM is SoundStream Zeghidour et al. ( 2021 ) . While Wav2Vec-BERT representations were found to be high-level, SoundStream representations are fine-grained, faithfully representing minute details of speech Borsos et al. ( 2023a ) . We use a version of SoundStream with 60 residual quantization layers, each quantized into 1024 tokens and 200 ms frames. The encoder has around 12M parameters.
For uniformity, we use dequantized representations.
This model operates on 24kHz audio.
Training from scratch
To evaluate the contribution of the pretrained model SpeechSim, we also trained a model from scratch for each facet on the labelled data directly. These models have the same architecture as SpeechSim but with 6 layers. They are trained with the same contrastive objective used for training the projection models.
Evaluating MAD Speech
In this Section, we use the collection of per-facet diversity benchmarks (Section 5 ) to assess which speech representations are more suitable for estimating diversity of a set of speech utterances when only one facet is changed (Section 7.1 ). Next, we run a series of experiments with two facets changed in the opposite directions (Section 7.2 ).
Voice Diversity
From Table 1 we firstly notice that, across all columns, SpeechSim shows higher correlation scores than off-the-shelf embedding models and the models trained from scratch. This supports our decision to rely on SpeechSim as a basis for a per-facet models. In turn, further voice specialization of SpeechSim (SpeechSim/Voice) results in similar scores. Finally, on comparing average cosine vs. Vendi Score aggregation, we observe that often Vendi Score shows higher correlations.
Gender diversity
In Table 2 we report correlation scores for the gender diversity. Again, we split in two groups: male-voice and female voice-dominant. From the results, we notice that the correlations showed by the non-specialized embeddings are extremely weak. However, both specialized models reach very high correlations (e.g., up to 0.946 for SpeechSim/Gender). All in all, gender-specific projector gets the higher correlation score. In this setup, Vendi Score underperforms w.r.t. the average cosine dissimilarity.
Emotion diversity
In Table 3 we report average Spearman correlations for the tested representation models, when tested on EmoV and Expresso separately. From these results, we see that, generally, SpeechSim performs better than other general-purpose representations across all configurations. Equally, the specialized projection of SpeechSim for the emotion facet, SpeechSim/Emotion performs best across all representations. As with voice diversity, Vendi Score gets higher correlations than average cosine dissimilarity.

To make sure that our models can measure diversity of emotions beyond that is covered by the (limited) label sets of EmoV and Expresso, we run an experiment where the projection model is trained on one dataset and tested on another (the only common label is “neutral”).
Table 3 shows that SpeechSim finetuned on Expresso does transfer to EmoV classes as it obtains a high Spearman correlation. However, SpeechSim finetuned on EmoV gets a Spearman correlation equal to 0.321.
Digging deeper, we additionally compute the correlation of the SpeechSim/Emotion-EmoV scores averaged across seeds instead of the average correlation of all scores as computed in Table 3 . In this, SpeechSim/Emotion-Emov gets a correlation of 0.964 which means that SpeechSim finetuned on EmoV generalizes well on average.
Accent diversity
We report average correlations in Table 4 . Here, again, we notice that SpeechSim has higher correlations than other non-specialized embedding models. Both specialized model perform similarly good with perfect correlations. As before, Vendi Score leads to better correlations across all setups.
Background noise diversity
We report our results in Table 5 . We see that generally all representations have somewhat low levels of correlation. SpeechSim/Noise combined with Vendi Score obtains the second highest correlation, following HuBERT.

We additionally compute the correlation of the SpeechSim/Noise scores averaged for all seeds instead of the average correlation of all scores as computed in Table 5 . In this setup, SpeechSim/Noise gets a Spearman correlation of 1.00, indicating that it works very well in average.
Are facets mixed within metrics?
Above we assessed how good the proposed scores are in detecting changes in a single factor of diversity with the rest fixed. However, one can expect that in a real-life scenario many factors can change simultaneously. In this Section, we verify whether this behavior takes place.
Gender and Voices
We design a sequence of samples such that the voice and gender diversity are changing in the opposite directions. We continuously change the proportion of female voices from 0.0 to 1.0: the gender diversity grows at first, peaks when the ratio is 0.5, and then starts to decrease. While the gender diversity increases, we reduce the number of speakers (from 30 to 2) and when the gender diversity starts to fall, we increase the number of speakers back (from 2 to 30).

We report results in Table 6 . From this results we see that SpeechSim/Gender is positively correlated with the gender diversity. In contrast, the SpeechSim/Voice metric tracks the speaker diversity and hence is negatively correlated with the gender diversity. Finally, the original SpeechSim embeddings have negative correlation with the gender diversity. Moreover, its correlation coefficient is smaller than that of SpeechSim/Voice. From that we conclude that the vanilla SpeechSim measure overemphasizes voice over gender diversity.

We further illustrate this experiment in Figure 1 , reporting SpeechSim/Gender and SpeechSim/Voice scores as the female voice ratio changes. From Figure 1 we see that SpeechSim/Gender is almost invariant to the changes in the number of speakers. Conversely, SpeechSim/Voice is relatively insensitive to changes of gender diversity.
Emotions and Voices
We run a similar experiment, examining interactions between emotion and voice diversity within metrics. This time, we change the number of speakers to be 1 or 4. We report the results in Table 7 . We observe that SpeechSim/Emotion is unaffected by the changes in the number of speakers, while SpeechSim/Voice follows the speaker diversity and hence it has a negative correlation with the emotion diversity score. SpeechSim focuses on the changes in the speaker diversity, thus having a negative correlation with the gender diversity.
Accents and Voices
In this experiment, we use 7 and 30 voices. The results are reported in Table 8 . We see that the projected metric is unaffected by changes in voices, while the vanilla version of SpeechSim is focused on the speaker facet.

Overall, the experiments reported in this Section support the necessity of introducing per-facet projection models. While SpeechSim embeddings might provide reasonably good correlations with the ground-truth diversity when only one facet is changed, they implicitly mix all facets into a single score.
We find this inconvenient, as (a) changes in one facet can mask changes in another, (b) the user might want to control the trade-off explicitly.
Hence, per-facet models are preferrable.
Comparing Generative Models
In this section, we show that despite that the acoustic diversity was mostly ignored in the literature when proposing new models and approaches, it does change.

We start by assessing whether the introduction of SoundStorm (Borsos et al., 2023b ) to the AudioLM framework Borsos et al. ( 2023a ) lead to any hidden penalties in acoustic diversity (Section 8.1 ). Next, we examine the effect of the Best-K sampling technique used by Kharitonov et al. ( 2023 ) on acoustic diversity (Section 8.2 ). We study how temperature changes acoustic diversity of the synthesized speech (Section 8.3 ) and compare acoustic diversity of off-the-shelf TTS models (Section 8.4 ).
Semantic-to-Acoustic Token Conversion: SoundStorm vs. AudioLM
Generating acoustic tokens from semantic tokens is a core part of AudioLM Borsos et al. ( 2023a ) . This step is believed to be responsible for modelling the acoustic details of the generated audio, such as voice, background noise, etc. Borsos et al. ( 2023b ) proposed to implement it with a non-autoregressive Transformer, yielding the SoundStorm model that dramatically improved inference time. However, it is unclear whether this change harms acoustic diversity of the generated audio.

We take 32 male- and 32 female-voice utterances from hold-out LibriTTS test-clean and represent them as sequences of semantic tokens. Next, we convert these semantic token sequences to acoustic token sequences using both models, repeating the process 128 times. We convert the acoustic tokens into audio using the corresponding SoundStream codec Zeghidour et al. ( 2021 ) . Finally, for each combination of a source utterance and a model, we calculate the diversity of the produced audio samples. We use exact same SoundStorm and AudioLM models as Borsos et al. ( 2023b ) .

In Table 9 we report, for each facet of the acoustic diversity, the ratio of utterances where AudioLM generated higher-diversity audio than SoundStorm. We see that SoundStorm produces audio that is more diverse in voices, emotions, accents, and background noise, while being behind is the gender diversity. We hypothesize SoundStorm managed to exploit speaker details that “leak” in the semantic tokens, hence biasing the produced outputs.