TTSDS2: Resources and Benchmark for Evaluating Human-Quality Text to Speech Systems
Authors: Christoph Minixhofer
Ondrej Klejch
Peter Bell
Venue: ICLR 2026 Oral (2026)
Keywords: speech synthesis
distributional analysis
objective evaluation
Primary area: datasets and benchmarks
TL;DR: With TTSDS2, we introduce a metric and benchmark for TTS, covering 14 languages, which consistently correlates with human judgements.

Abstract: Evaluation of Text to Speech (TTS) systems is challenging and resource-intensive. Subjective metrics such as Mean Opinion Score (MOS) are not easily comparable between works. Objective metrics are frequently used, but rarely validated against subjective ones. Both kinds of metrics are challenged by recent TTS systems capable of producing synthetic speech indistinguishable from real speech. In this work, we introduce Text to Speech Distribution Score 2 (TTSDS2), a more robust and improved version of TTSDS. Across a range of domains and languages, it is the only one out of 16 compared metrics to correlate with a Spearman correlation above 0.50 for every domain and subjective score evaluated. We also release a range of resources for evaluating synthetic speech close to real speech: A dataset with over 11,000 subjective opinion score ratings; a pipeline for recreating a multilingual test dataset to avoid data leakage; and a benchmark for TTS in 14 languages.

Published as a conference paper at ICLR 2026
TTSDS2: RESOURCES AND BENCHMARK FOR EVALU- ATING HUMAN-QUALITY TEXT TO SPEECH SYSTEMS
Christoph Minixhofer, Ondrej Klejch, Peter Bell
Centre for Speech Technology Research
University of Edinburgh
(christoph. minixxofer, o. kle klejch, peter peeter.belljp (ealljjeed.. ac. uk
ABSTRACT
Evaluation of Text to Speech (TTS) systems is challenging and essource-intensiin Subjective metries such Mean Opinion Score (MOS) are not easily comparable between works. Objective metrics are frequently used, but rarely validated against subjective ones. Both kinds of metrics challenged by recent TTS systems capable of producing synthetic speech indistinguishable from real speech. In this work, we introduce Text to Speech Distribution Score (TTSDS2), a more robust and improved version of TTSDS. Across range of domains and languages, it is the only one of 16 compared metrics correlate with Spearman correlation above 0.50 for every domain and subjective score evaluated. We also release range of resources for evaluating synthetic speech close to real speech: A dataset with over 11,000 subjective opinion score ratings; pipeline for recreating multilingual test dataset to avoid data leakage; and a benchmark for TTS in 14 languages.
INTRODUCTION
1.1 BACKGROUND AND IMPACT OF TTS EVALUATION
Text Speech models have significantly advanced recently. achieving a level of quality where synthetic speech can be indistinguishable from real speech (Eskimez et al.) 2024). Since subjective evaluation of TTS using listening tests is difficult and resource-intensinn
some recent works have
partially or fully replaced subjective metrics with objective ones (Cooper et al. (024). Recent TTS systems report buman-level quality, with listeners sometimes not able to distinguish between real and
synthetic speech (Chen et al. 2024 Wang al. 2024 Li et al. 2023 Shen et al. 2024).
The domains of TTS have shifted as well. Read audiobook speech (Zen et al. 2019; Pratap et al. 2020) used to be the standard training data for TTS. but many modern systems datasets instead, For example, the Emilia dataset contains "diverse and spontaneous speaking styles, including breathing, pausing, repetitions, changes in speed, and varying emotions" (He et al. 2024). Multilingual TTS has seen advances too (Liao et al.) 2024). but at the time of writing, no public TTS benchmarks for more than single language exist. The jump in the quality of recent TTS makes difficult to say if objective metrics continue to reliably predict continually updated human ratings.
train scraped
However, the speed of progress for synthetic speech generation outpaces most evaluation efforts: in this work we provide the first public evaluation of 20 systems published between 2022-2024 which controls for speaker identities and dataset domains. We are also the first to provide this kind of comparison beyond English, with a total number of 14 languages, with our pipeline extensible to cover more languages the future. Providing this comparison between systems can improve efficiency when building new systems or extending existing ones. TTSDS2 also gives us information as to how close synthetic speech is to real speech with state-of-the-ar models. Our work's 's impact on advancements in TTS could be used for positive applications such improving synthetic voices for people who are at risk of losing the ability to speak through illness. Research in this field can also increase risks, such identity theft by use of synthetic speech however, believe that good evaluation practices can also help assess these risks accurately by providing information about the generative capabilities of current and future systems.

---

Published as a conference paper at ICLR 2026
1.2 CURRENT STATE OF TTS EVALUATION AND SELECTED SYSTEMS
Subjective ratings collected through listening tests are standard practice in TTS evaluation, the most commonly-used rating being Mean-Opinion-Scorte (MOS). However, pairwise comparison tests, in which listeners rate the preference of one sample over another on a numeric scale, are gaining popularity, as they can lead to significant results with fewer listeners (Cooper et al. 2024). A popular example of these A/B metrics is Comparative MOS (CMOS), A preference test is also frequently conducted to assess systems' ability to replicate a specific speaker, and is called Speaker Similarity MOS (SMOS). Recent systems often achieve MOS or CMOS scores close to or surpassing real speech, making evaluation more challenging (Wang et al., 2024 Li et al. 2023; Lyth & King, 2024). This has lead to wide range of objective metrics used in recent works, which we describe in more detail in Section 1.4 Additionally, any variant of subjective evaluation not comparable between works, as the listeners and surveys differ substantially however, many TTS systems have released their code and weights, making evaluations like ours possible.
In this paper select 20 open-source, wpen-weight TTS systems for evaluation, released between 2022 and 2024, and covering 14 languages. See Appendix [A more details about these systems. should be noted that we only compare voice-cloning TTS systems which use speaker reference and transcript to control the output. All but two of our selected systems reported subjective evaluation metries; however, due different listeners and datasets, these are not comparable between systems. Objective evaluation is frequent as well. especially for ablation experiments and architecture variants (Casanova al., 2024). We also find that three systems in this set reported parity with human evaluation, which we define as MOS or CMOS within 0.05 of the ground truth values. Four systems surpassed ground truth scores, meaning listeners preferred the synthetic to the real speech recordings, with respect to the questions outlined in these tests.
1.3 TTSDS2: DISTRIBUTIONAL, ROBUST AND MULTILINGUAL
The Text-to-Speedt Distribution Score (TTSDS) was introduced by Minixhofer et al. (2024) as an objective metric using perceptual factors (such as Speaker Identity, Intelligibility, Prosody) and scores each system based on how close several distributions representing each perceptual factor are to a real data reference in comparison to noise. The average of these scores was shown correlate with subjective ratings.
In this work, we extend and validate TTSDS using 20 TTS systems, all published in 2022 later. We increase the robustness of TTSDS scores across 4 differing domains, and increase robustness across languages, leading to the updated TTSDS2. We compare correlation to subjective listening test results with 16 open-source metrics and find only TTSDS correlates with >> 0.5 in all cases, with an average of 0.67. We additionally publish a pipeline which continually recreates multilingual Yobeube dataset and synthesises the samples to provide an up-to-date, uncontaminated and automated ranking of TTS systems across 14 languages- we ensure this pipeline can be easily extended as more languages are covered by modern TTS systems.
1.4 OTHER OBJECTIVE METRICS
Objective evaluation of synthetic speech can be grouped into four broad families. In addition to these families we also distinguish intrusive and non-intrusive metrics. Intrusive metrics require some ground truth speech of the same speaker as a reference. Non-intrusive metries are reference-fres When the reference does not need to contain the same lexical content, it is described as non- non-matching.
The oldest group consists of intrusive metries that compare each synthetic utterance to a matching reference. Percepiual Evaluation of Speech Quality (PESQ) (ITU-T 200T) and Short-Time Objective Intelligibility (STOI) (Taal al.) (01 and Mel-Cepstrul Distortion (MCD) are the known representatives. They were designed for telephone or enhancement scenarios rather than TTS, and require access to the ground-trui waveform.
bassd reference metries:
To predict scores directly, researchers train neural networks that map a single audio signal to an estimated MOS. MOSNet (Lo et al.) 2019) introduced the idea, and was followed by UTMOS (Sacki al. 2022), its SSL-based successor UTMOSy2 (Baba al.) (024). and NISQA-MOS (Mittag et al.] 2021). SQUIM-MOS (Kumar et al., 2023) additionally grounds its prediction by requiring non-matching reference of the ground truth speech. These methods
MOS-prediction networks:
2

---

Published as a conference paper at ICLR 2026
Table I: Feature set used for TTSDS compared to TISDS2.
Factor
Features (TTSDS)
Features (TTSDS2)
WavL.M Chen activations (actv.)
Hubert Hsm
M21
GENERIC
wav2vec 2.0 Baoski ai
20X0
HuBERT Hu et 2021
(base) actv.
wav2vec 2.0
(base) actv.
VoiceFixer+PEST
2121
ENVIRONMENT
WADA SNR Kim Stem
2008
(Factor removed)
d- Vector Wan et 2018
d- V-eettor Wan 2018
SPEAKER
WeSpeaker Wang al. 202bb
WeSpeaker Wang
WORLD FO Morise 2016
HuBERT His vi al.
2021
token length
PROSODY
Prosody embeddings Wallbndge ot
WORLD FO Morise
2016
2025
HuBERT (Bu ct 202]
Allosaurus L et 202
speaking-ree -aat
speaking-rate
Prosody embeddings
Wallbridge et.
wav2vee 2.0 Baevaki
20207
WER
wav2vec 2.0 Baevski
ASR actv.
INTELL.
Whisper Radion cl 3025
(small) WER
whisper (Rdiond et 2025
(small) ASR actv.
report in-domain correlations; however, recent VoiceMOS challenges (Huang et al. 2024) show that correlation with subjective ratings drops out-of-domain.
Inspired by the image domain's ' Fréchet Inception Distance (FID) (Heusel et al. 2017), audio researchers proposed measuring entire corpora rather than single files. Frécher Audio Distance (FAD) (Kilgour et al. 2019) compares embeddings and has since been adapted for TTS (Shi al., 2024). Distributional metrics require set of references which do not need to correspond to the synthetic data. The authors of these metrics state the need for thousands of samples, which may be why they have not found more widespread adoption.
Distributional metries:
Multi-dimensional perceptual metrics: Recent work argues that no single score can capture everything listeners care about. Audiobox Aesthetics predicts Production Quality, Complexity, Enjoyment, and Usefulness scores for audio (Tjandra et al. (025)..
Other metries Often reported are also Word Error Rate (WER) and Character Error Rate (CER), computed on Automatic Speech Recognition (ASR) transcripts, as well as Speaker Similarity com- puted as the cosine similarity between speaker representations of synthetic speech and non-mhgching reference.
TTSDS2
The task of synthetic speech generation is inherently one-to-many, meaning there is no single ground truth for any given text. In the following, we denote synthetic variables using to avoid overly relying on sub- or superscripts. We therefore frame its evaluation as a problem of distributional similarity. Let S denote a speech signal and R be transformation function that extracts a specific feature representation R(S), such as one of the features shown in Table [ Our objective is to quantify how closely the empirical distribution of features from synthetic dataset, P(R(Š) matches that of real dataset, (()(S) D), while remaining distinct from various noise distributions, | NOISSE) where NOISS is drawn from NOISE which contains uniform noise, normally distributed noise, all ones and all zerof[] We choose noise instead of alternate anchors since it is agnostic to domain, speaker and other confounding factors. An example distribution for the one- dimensional pitch feature can be seen in Figure I
pNOISE (R(Š)
To achieve this, TTSDS2 employs a factorised evaluation framework, assessing distributional similar- ity across the following perceptually- mottvvtte aspects of speech: (i) GENERIC: Overall distributional similarity, via SSL embeddings. (ii) SPEAKER: Realism of speaker identity. (iii) PROSODY: Pitch, duration, and rhythm quality. (iv) INTELLIGIBILITY: Uses ASR-derived features.
"available wh. wtt.co//aaasattsttttteeeeee htt::/ooiieeeeeeeee
3

---

Published as a conference paper at ICLR 2026
Each factor is evaluated using multiple feature representations (see Table D The scores for these features are averaged to produce a factor score, and these factor scores are in turn averaged to yield the final TTSDS2 score. Thus, TTSDS2 is distributional metric by design, it compares entire distributions rather than individual samples. Through its use of factor scores, also functions as multi-dimensional perceptual metric, providing interpretable insights into specifie speech attributes, as categorised in Section 1.4
Computing Wasserstein distances To compare feature distributions, we use the 2- Wasserstein distance (W2), which is intuitively understood the Earth Mover's Distance (FMD). This aligns with its common application computer vision, such as the Fréchet Inception Distance (FID) (Heusel al. 2017). The W2 distance is well-suited for this task due to its desirable properties: it is symmetric (unlike Kullback-Leilerr divergence (Kullback] 195T)) and can differentiate between non non-overlapping distributions (unlike
divergence (Lin 1991 Kolouri et al. 2019)).
For high- diseensinni vectors where distributions are approximated by multivariate Gaussians, the squared 2- Wasserstein distance (also known the Fréchet distance) between real dataset D and its synthetic counterpart D is given by:
W2 WD, 1 -ll +Tr Tr TT++++++++2 -
where Σ and i., En are the mean and covariance matrices of the real and synthetic feature distri- butions, respectively. In the one-dimensional case, the squared distance possesses closed-form solution based the inverse Cumulative Distribution Functions (CDFs), C-1 and Č-!:
W2, = ff -
formulated in (Kolouri et al. 2019).
To normalise distances across different features and factors, we compute a score, ranging from 0 (identical to noise distribution) to 100 (identical to the real reference distribution). For any synthetic speech feature distribution P where the lexical content need not match the reference compute its W2 distance to each distractor noise dataset in the set NOISE and denote the minimum as WONSI piois
W2 (D, DNOISE
The distance the real speech distribution ((R(S) is denoted as RREAL Using these terms, the normalized similarity score for feature is defined as:
WOISI
TTSDS2(D, D, D. NOISE = 100 X WREAL WOISS
(I)
Equation Equation[Iyeedds yields scores between and 100, where values above 50 indicate stronger similarity to real speech than to noise. The final TTSDS2 score is the unweighted arithmetic mean of the factor scores. While each factor score requires the computation five Wasserstein distances, due to the ensembling effect of several representations, only small number of samples is required, with 50-100 samples being sufficient, which not the case for distributional scores with a single latent representation Kilgour al. (2019).
Updated factors and features This work proposes modifications to the original TTSDS framework make it more robust to several domains for each of its factors. To ensure our metric does not overfit to specific features and to maintain high baseline scores for real data, all feature selections were finalised prior to running any correlation experiments with human judgments. We validated feature robustness by splitting each ground-truth dataset in two and computing the TTSDS score of one half against the other. Any candidate feature that scored below 95 on average, or exhibited a high standard deviation across datasets, was excluded. Following these selection criteria, we updated the feature sets. INTELLIGIBILIT experiments, these WER features resulted in low scores for real data across domains recognition models' final-layer activations instead. For PROSODY, TTSDS originally used (a) the WORLD pitch contour (Morise al. 2016), (b) masked-prosody-eoee model embeddings (Minixhofer
TTSDS originally relied on Word Error Rate (WER). In preliminary
speech
4

---

Published as a conference paper at ICLR 2026
Table 2: Mean over datasets of MOS, CMOS,
SMOS and the corresponding TTSDS2 score.
0028
System
MOS
CMOS
SMOS
TISDS2
Truth
3.70::0.06
0.0010.13
4.37
E2-TTS
3.11:0.13
00223:0:18
4.37
91.73
0.88=0.18
4.01+0.115
F5-TTS
3.3340.14
-0.34=0.18
4,10:10..15
91.16
Derah
3.2810.14
-0.17.0.17
FishSpeech
3.24:0.15
-0.43=0.211
3.58::0.20
89.88
TorTo
2.73
Noise Ground Truth
VoizeCraft
3.15:0..15
-0.44:0.211
3.66:20.117
88.30
0.010
WhisperSpeech
3.1240.155
Synthetic
HierSpacilt+
StyleTTS2
3.08:20.17
-0.86:0022
-0.6610.23
2.931:0.18
3/48::0.21
88.63
0.006
Pheme
2.99::0.17
-1.00::0.22
3.35::0.11 18
88.84
292::0.16
-1.21 -0:25
VALL-E
2.90:00144
-0.60:0.22 2.25
3.43::0.1 19
83.59
GPTSaVITS
2.81:0.14
-0.57=00.21
3.22-0.17
22
0.000
XTTS
2.77:00117
-0.73:10.26
2.57:10.1 18
$8.20
FD (H2)
Vnice
2.49:0.14
-1,23-80.21
2.18-8..1
38
Bark
2.9900114
-1.12=0.23
2.51=+0.17
85.21
Figure 1: Distribution of Fo in TTSDS for
ParleTTTS
-1,19:00118
2.93+00117
KK
NanuraSpoecch2
2.05+0.12
-1.42=0.21
2.06::0.16
81.71
ground-truth, synthetic, and noise datasets.
SpeechTS
198.0.15
-1.56:0.226
2.632:0.19
84.84
et al., 2024), and (c) token lengths (in frames) extracted from HuBERT (Hsu et al. 2021). We found that the token-length features lead to low scores for real speech. We instead compute the utterance-level speaking rate by dividing the number of deduplicated HuBERT tokens in utterance by the number of frames. We do the same for the multilingual phone recogniser Allosaurus (Li eet al., 2020), also included in the original. GENERIC uses the same HuBERT (Hsu et al., 2021) and wav2vec 2.0 (Baevski et al. 2020 features as the original, but we also add WavLM (Chen eet al. 2022) features for increased diversity. The factors and their features shown in Table For multilingual use, we replace HuBERT with mHuBERT-144 (Boito et al.) 2024), and wav2vec 2.0 with its XLSSR-3 counterpart (Conneau et al., 2021).
CORRELATIONS WITH LISTENING TESTS ACROSS DATASETS
We now outline how we validate TTSDS to correlate with human scores across a variety of datasets.
3.1 DATASETS FROM READ SPEECH TO CHILDREN'S ' SPEECH
Since most systems are still trained using audiobook speech, and audiobook speech is easier to synthesize due to its more regular nature (He et al. 2024), we use samples from the LibriTTS (Zen) al. 2019) test split as baseline. Since LibriTTS is filtered by Signal-to-Noise Ratio (SNR), only contains clean, read speech. In the remainder of this work, we refer to this as CLEAN. For all datasets, utterances between and 30 seconds with a single speaker are selected. The remaining datasets alter this baseline domain in the following ways:
NOISY is created by scraping LibriVox recordings from 2025 (to avoid their occurrence in the training data) without SNR filtering. This tests how evaluation is affected by noise present in the recordings.
WILD is created by scraping recent YouTube videos and extracting utterances, which tests the metrics' ability to generalize to diverse speaking styles and recording conditions. Its data collection and processing are inspired by Emilia (He et al., 2024). We scrape 500 English-langure YouTube videos uploaded in 2025 using 10 different search terms which emphasise scripted and conversational speech alike. We perform Whisper Diarization (Ashral 2024) to isolate utterances.
KIDS is subset of the My Science Tutor Corpus (Pradhan et al. 2024 and contains children's conversations with a virtual tutor in an educational setting. This tests evaluation metrics can generalize to data rarely encountered during the training.
For all systems, we select 100 speakers at random, with two utterances per speaker. We then manually filter the data to exclude content which is (i) difficult to transcribe (ii) potentially controversial
5

---

Published as a conference paper at ICLR 2026
or offensive. This leaves us with 60 speakers for each dataset. The first utterance by each speaker is used as the reference provided to the TTS system, while the transcript of the second utterance is used as the text to synthesize. This way, we can evaluate both intrusive and non-intrusive metrics. We use matching speaker identities to eliminate any possible preferences of listeners of one speaker over another, and avoid systems scoring highly merely because of a set of speakers is closer the reference than for other systems.
3.2 COLLECTING HUMAN JUDGEMENTS ACROSS SYSTEMS AND DATASETS
We recruit 200 annotators using Prolific which annotate the cround-truth and synthetic data for 20 TTS systems across the aforementioned datasets, in terms of MOS, CMOS and SMOS. Annotators are screened to be native speakers from the UK or the US and asked to wear headphones in quiet environment. Any that fail attention checks are excluded. To keep survey duration around 30 minutes, each annotator is assigned to one dataset only, resulting in 50 listeners per dataset. For MOS, there are pages with samples each. one of which is always the ground truth, while the others are selected at random. For CMOS and SMOS, 18 comparisons between ground truth and a randomly selected system's 's sample are conducted. To avoid any learning or fatigue effects if a certain measure is always asked first or last, the order of the three parts of the test is varied from annotator to annotator. The median completion time was 32 minutes and the annotators were compensated with $10, resulting in an hourly wage of ze $19. For both MOS and CMOS, we instruct annotators to rate the Naturalness of the speech. MOS and SMOS, in line with recommendations of (Kirkland et al. 2023). are evaluated on a 5-point scale ranging from Bad Excellent. CMOS is evaluated on full point scale ranging from -3 (much worse) to 3 (much better). We collect a total of 1,846 anonymized ratings and utterances, of which we publish 1 1,222, excluding the ground truth utterances due to licensing. The ratings can be accessed at f co/ dataaaaaa datasets seeets/ttsadss /tt./tstssssl/t// /11 steninggteenteee
While we use this data to validate if TTSD2 aligns with human ratings, future work could use it for improving MOS prediction networks, since, to the best of our knowledge, all publicly available datasets of this size use TTS systems which have not reached human parity (Huang et al. 2024 Cooper et al. 2022 Maniati et al. 2022).
Tjandra et al. 2025
3.3 EVALUATED OBJECTIVE METRICS
We use the VERSA evaluation toolkit (Shi et al. 2024 for all compared objective metrics, except UTMOSv2, which was not included at the time of writing. For Audiobox Aesthetics we select their Content Enjoyment (AE-CE), Content Usefulness (AE-CU), and Production Quality (AE- PQ) subscores, which they show to correlate with MOS (Taanra et al. (2025). For distributional metrics, we evaluate Fréchet Audio Distance using Contrastive Language- Audio Pretraining latent representations (Wu et al.) 2023). For MOS prediction, we evaluate UTMOS (Saeki et al.) 2022). UTMOSv2 (Baba et al. 2024). NISQA (Mittag et al. 202I). DNSMOS (Reddy et al. 2022). and SQUIM MOS (Kumar et al. 2023), which is the only MOS prediction system we evaluate that requires a non-matching reference. For speaker embedding cosine similarity, which require non- matching reference samples as well, use three systems included in ESPNet-SPK (Jung al.) 2024): RawNet3, SCAPA-TDNE and X- Vectors. We also include some legacy signal- metrics, which are STOI, PESQ, and MCD these are the only ones require matching references. In the next section, we compare these metrics with the subjective evaluation scores.
We evaluate both the original TTSDS metric (Minixhofer et al. 2024) and the updated TTSDS2 version described in Section 2
3.4 CORRELATIONS
For each TTS of the 20 systems, we average human ratings for MOS, CMOS and SMOS for CLEAN, NOISY, WILD and KIDS. These are the "gold" ratings. We now examine the Spearman correlation coefficients (since we deem ranking the systems most important) of these results with the aforementioned metrics across the four datasets. As Table 3 shows, TTSDS2 shows the most consistent correlation across the datasets, with an average correlation of 0.67, surpassing the original by 10% relative. All correlations for TTSDS and TTSDS2 are statistically significant with p < 0.05.
pro1i i foc..cmm

---

Published as a conference paper at ICLR 2026
Table Spearman rank correlations. Colours:
-0.,,
-0.5
0,
0.5,
0.5 ... 1.
Clean
Noisy
Wild
Kids
Metric
MOS
CMOS
SMOS
MOS
CMOS
SMOS
MOS
CMOS
SMOS
MOS
CMOS
SMOS
TTSDS2 (Ours)
0.75
0.69
0.73
0.39
0.54
0.71
0.75
0.71
0.75
0.61
0.50
0.70
TTSDS Minixhofer
2024
0.52
0.61
0.67
0.67
X- V-eector
0.46
0.42
0.56
0.40
0.29
0.77
0.82
0.82
0.62
0.70
0.57
0.75
RawNet3
0.36
0.52
0.37
0.85
0.64
SQUIM
0.68
0.46
0.37
0.48
0.62
0.75
0.79
0.57
0.55
0.45
BCAPA-TDNN
0.36
0.29
0.47
0.29
0.22
0.72
0.81
0.78
0.58
0.69
0.60
0.72
DNSMOS
0.41
0.37
0.22
0.57
0.36
0.22
0.35
0.28
0.03
0.31
0.10
0.28
AE-CE
0.60
0.46
0.32
0.58
0.53
0.21
0.19
0.10
0.11
-0.02
-0.12
-0.10
AI-CU
0.49
0.37
0.30
0.60
0.58
0.13
0.35
0.24
0.22
-0.09
-0.21
-0.13
AE-PQ
0.49
0.21
0.48
0.21
0.16
0.12
-0.08
UTMOSv2
0.39
0.25
0.09
0.36
0.16
0.14
-0.04
0.03
-0.02
FAD (CLAP)
-0,22
0.06
-0.01
0.45
0.30
0.16
-0.03
0.08
0.25
0.12
0.26
0.04
UTMOS
0,51
0.30
0.31
0.47
0.29
0.00
-0.12
-0.12
-0.26
-0.02
-0.18
-0.04
STOI
-0.11
0.01
0.02
-0.06
0.00
0.19
0.07
0.41
0.24
-0.32
-0.08
0.05
PESQ
0.01
-0.16
0.27
-0.34
0.00
0.07
-0.14
0.01
-0.06
-0.08
-0.04
40.38
NISQA
0.05
0.06
-0.21
-0.53
-0.32
-0.33
-0.64
-0.29
-0.27
-0.46
MCD
-0.46
-0.37
-0.27
-0.45
-0.58
-0.74
-0.33
-0.45
-0.51
-0.31
-0.13
-0.38
WER
00.19
0.18
-0.77
-0.11
-0.30
-0.13
-0.28
-0.17
-0.22
-0.45
+0.26
-0.39
4.5
0.58
0,70
4.0
0.75
0.57
0.72
0.72
0.64
0.37
Q
0.71
3.5
3.0
2.5
2.0
15
3.50
4,00
TISDS
SQUIM
Vector
Clean
Nolsy
Wild
Kids
Correlation of three representative objective metries with human MOS across the four datasets. Each colour/marker denotes a domain. Solid line overall least-squares fit; dashed/dotted lines = domain -ssecic fits; each with corresponding Pearson r.
Figure 2:
Speaker Similarity metrics come second, with average correlations of 0.6 for RawNet3 and X- X-Vector Speaker Similarities. Of the MOS Prediction networks, only SQUIM MOS performs well, with an average correlation of 0.57. Following the final Speaker Similarity tested, ECAPA-TDNN, there is a large drop in average correlation, with all remaining averages below 0.3. We note that many of the metrics, including Audiobox Aesthetics and UTMOSv2, still perform well on NOISY and CLEAN, which only contains audiobook speech. Metrics seem struggle most KIDS, which expected, as it is the furthest removed from the most common TTS domains. When averaging the scores across domains, TTSDS2 agrees with MOS and CMOS for the top 4 and bottom systems, as can be seen in Figure
We also investigate the top-performing TTSDS2, Vector, and SQUIM MOS correlations. As Figure shows, some behaviours are not shown in their correlation coefficients alone; TTSDS2 acts the most like continuous scale; both SQUIM MOS and X-Vector Speaker Similarity show some clustering behaviour. Since both SQUIM and Veveot are essentially black boxes, we cannot conclusively state what causes this behaviour, but it could indicate overfitting to specific systems.
3.5 COMPLEMENTARY FACTORS & GENERALISATION
To validate our design choice of equally weighting all factor scores in the final TTSDS2 calculation, conduct ablation study comparing our simple unweighted mean against learned weighting
7

---

Published as a conference paper at ICLR 2026
Table 4: Leave-One-Out Cross- Validation (Gen- eralisation). Evaluation on a held-out domain after training on the other three.
Table 5: Instability of learned weights. Optimal coefficients vary drastically by training domain, occasionally becoming negative.
Held-Out Domain
Simple Mean Learned Weights (LOOCV)
Training Domain
Gen.
Speak.
Pros.
Intel.
(Baseline)
CLEAN
-0.162
0.072
0.048
0.098
CLEAN
0.747
0.645
NOISY
0.066
0.033
0.004
0.071
NOISY WILD KIDS
0.590
0.514
WILD
-0.017
0.072
0.004
-0.020
0.752
0.658
KIDS
0.050
0.101
-0.032
0.026
0.606
0.853
All Combined
-0.045
0.066
0.002
0.019
approach. In the learned setting, factor weights are optimised via linear regression to maximise the correlation with subjective MOS ratings. We evaluate the generalisation capabilities of both aggregation methods using Leave-One-ee Cross-Validation Validation (LOOCV) across our four evaluation domains. As shown in Table the regression model on three domains and evaluated on the fourth held-out domain. The unweighted simple mean outperforms the domain-optimised learned weights in three out of the four unseen domain scenarios, indicating that learned weights tend overfit to their training distributions. We further examine the stability of these learned weights in Table 5 The optimal coefficients exhibit significant variance depending on the training domain. In several configurations, factors even receive negative coefficients (e.g., the GENERIC factor when trained on CLEAN and WILD), despite these individual factors independently demonstrating positive correlations with human judgements. These results highlight the practical advantage of our ensembling approach. The simple mean effectively acts as regulariser, mitigating the inherent noise present in individual feature representations across shifting domains. Crucially, this confirms that TTSDS2 can operate as a fully unsupervised metric, requiring no domain-specific fine-tuning or parameter fitting while maintaining strong and stable cross-domain generalisation.
4 MULTILINGUAL & RECURRING EVALUATION
While the previous Section outlined robustness across datasets in a single language due to the ease of conducting listening tests in English, we extend TTSDS2 for multilingual use, and provide a public benchmark in 14 languages this covers all languages synthesised by at least two systems, and is to be extended as more multilingual TTS systems are released.
As our benchmark should be updated frequently to avoid data leakage, and represent wide range of recording conditions, speakers and environments, we decide to automate the creation of the WILD dataset described in Section 3.1 However, since manual filtering is not feasible in the long term for growing set of languages and evaluation re-runs, we automate the collection process as can be seen in Algorithm|I [ and which describe in detail in the following section.
Here is the updated pipeline section with the requested pseudo-code and additional details.
4.1 PIPELINE
The TTSDS2 pipeline, available at algiithub.. wtwp:tsds/ppppeeii.....oo can be used rebuild the multilingual dataset regularly. The process is outlined in Algorithm | and detailed below.
The pipeline consists of the following automated steps: (i) Data Scraping: Ten keywords (see line 2 of Algorithm [ are translated for each language, and for each term, a YouTube search in the given language. and for time range starting after the last models' publication set is conducted. The results are sorted by views to avoid low-quality or synthetic results and only videos longer than 20 minutes are included. The beginning and end of each video trimmed to avoid intro and outro music or artefacts, and are then diarised using Whisper before. We use FastText (Bojanowski et al., 2016 Grave et al. 2018) language identification on the automatically generated transcripts to filter out videos not in the target language. (ii) Preprocessing: We then extract utterances from the middle of the video and only keep utterances from a single speaker as identified in the previous diarisation step. (iii) Filtering: Next, the utterances filtered for potentially offensive or controversial content. We filter the data using XNLI (Conneau et al., 2018 Laurer et al., 2022) with potentially controversial topics entailment. This leads to large number of falsely filtered
the evaluation
8

---

Published as a conference paper at ICLR 2026
Algorithm 1 TTSDS2 Benchmark Pipeline
Input:
Languages L, TTS Models M
1: AllScores - {)
2: Keywords +- talk show, interview, debate, sports commentary, news, politics, economy, technol-
ogy, science, podcast
3: for each language | in L do
4: Keywords +- Translate (Keywords 1)
5: Videos Search You Yuee/ube language=1, prrii-a-i,,rri mint minDuration==00, max-
Count=250, sortBy=views)
6: Utterances -
7: for each video in Videos do
8: video < Trim Video(start=5, end=5)
9:
DiarizedSegmments- + WhisperDiarrzzaioontteee))) if JanguugeeDD(Aaari//eeeeemme))))) is then
11:
SingleSSpeakerSegmeent <- SacccsiingeSppeeeeeeaaaaaaaaaaaa
12:
max=16))
13:
end if
14:
end for
15:
CleanUtterances FilterConttrverriialUUtttenaaa method=XNLD
16:
CleanUtterances +-
CaapssttallkCCeennttttttaaaa methodd-Pyannote)
CleanUtterances <- CiiierMMusiCCeeanUttteeeaa method=Demucs)
18:
MatchedPairs <- (aeeeSeeeaeePPanseCeeennnnnna
count=50)
ReferenceSet, SynthesisSet <- SpliPPais((attcheeParrs)
20:
for each model m in A do
SyntheticAudio <- Synthesise(m,, ReferenceSet, SynthesisSet)
22:
Score < TaalulaeeTSSDDS2(Symteeeeee AllScores[m, +- Score
ReferenceSet)
23:
24: end for
25: end for
26: Return: AllScores
95
TISDS2 Score
85
Chinase
(m=8)
English
(n=20)
Japanase
(n=4)
Spantsh
(n=5)
Gerann
(n=
Portuguase (n=3)
French
(n=
Russian
2n=
Dutch
(m=3)
Arabic
(n=2)
Italian
(m=4)
Polish
(m=4)
Karean
(n=4)
Turkish
(n=2)
Ground Truth
Figure 3: TTSDS2 scores across 14 languages, n indicates the number of systems per language.
texts, which in our case is not downside, since we only want a small number of "clean" samples. Finally, we use Pyannote speaker diarisation (Bredin) 2023) to detect if there is any crosstalk, and Demucs (Rouard al. 2023) source separation, to detect if there is any background music. Of the remaining utterances. 50 speaker-matched pairs are selected for each language, and split into the REFERENCE and SYNTHESIS set. (iv) Synthesis: For all systems the benchmark, we synthesise the speaker identities in REFERENCE with the text in SYNTHESIS. (v) TTSDS2: We apply multilingual TTSDS2 to arrive at scores for each, and publish the results at Mfttssbbeencmaa be repeated regularly with systems published prior the data range of the collected video data, to eliminate the possibility of data contamination. We plan to expand to more systems and languages each evaluation round.
com). This can

---

Published as a conference paper at ICLR 2026
4.2 MULTILINGUAL VALIDITY OF TTSDS2
While collecting gold MOS labels for 14 languages is out of scope for this work, we verify TTSDS2's applicability to the multilingual case using Uriel+ (Khan et al. 2024), which supplies typological distances for the languages evaluated. We show that if TTSDS2 distances correlate to language distances found by linguists, finding that when comparing the ground truth language datasets to each other using TTSDS2, the scores correlate with the distances with = -0.39 for regular and P 0.51 (both significant with p < 0.05) for multilingual TTSDS2 discussed in Section [ the negative correlations are expected since higher score correlates with a smaller distance, and the higher correlation of multilingual TTSDS2 scores is encouraging. Additionally, ground-truth scores are within narrow range across languages, and scores decrease for lower-resource languages as expected, which is shown in Figure
LIMITATIONS
Since TTSDS2 extracts several features for each utterance and uses CPU-bound Wasserstein distance computations, uses more compute than other methods. Future work could explore more compute- efficient methods such Maximum Mean Discrepancy (MMD) (Gretton et al., 2006 which has shown promising results in computer vision evaluation (Jayasumana al. 2024). While it robustly correlates with human evaluation, it never surpasses Spearman correlation coefficient of 0.8, indicating there either a component of listening tests that is inherently noisy, or not predicted by any objective metric of yet, and therefore TTSDS2 is not equivalent to, nor can it replace, subjective evaluation. Additionally, some modern TTS systems can have failure cases which are not identifiable as such by TTSDS2 such as when the transcript given is not reproduced faithfully. To mitigate this shortcoming, we report the number of utterances with high Word Error Rates for each system at coml. TTSDS2 currently also does not capture the context the utterances were spoken in, and does not include long-form samples, as many systems do not support generation of utterances beyond 30 seconds.
Etsdsbenchmare
ETHICS STATEMENT
To mitigate the risk of generating novel, unattributed speech, the text synthesized for given speaker's voice is always sourced from a separate, distinct utterance previously spoken by that same individual. This ensures no new statements are created. Furthermore, we do not redistribute any of the original ground-truth audio samples, to respect the terms of their original release. We also acknowledge that advancements in TTS technology earry dual-use potential. However, the TTSDS2 metric is designed to evaluate distributions of speech (i.e., entire datasets) rather than individual samples. This characteristic makes poorly suited for the iterative development of single deepfake utterances. Conversely, its distributional nature may offer utility in identifying large-scale synthetic speech campaigns, serving as a potential detection tool. Finally, the selection of the 14 languages was based on the availability of multiple open-source TTS systems. We recognize that this may inadvertently reflect existing biases in speech technology research. By providing an open-source and extensible pipeline, we aim to empower the community to apply and expand this benchmark to wider, more inclusive set of languages.
7 CONCLUSION
In this work, we introduce TTSDS2, a robust, factorised metric that demonstrates consistently high correlation with human judgments across a wide variety of speech domains. Our extensive evaluation shows that out of 16 state-of-the-an objective metrics, TTSDS2 is the only one to maintain a strong Spearman correlation (p > 0.5) in every tested condition, achieving an average of 2 0.67. This consistency holds true not only for clean audiobook speech but also for challenging domains including noisy, conversational, and children's 's speech, where many existing metrics fail to achieve significant correlations. To further advance reproducible and inclusive research, we extend the framework for multilingual use and provide public, recurring benchmark covering 14 languages. This benchmark is supported by an automated pipeline that can continually renew the dataset to prevent contamination and ensure long- long-term relevance. By providing reliable objective measure that aligns closely with human perception, we hope the TTSDS2 benchmark and its associated resources will enhance the efficiency and direction of future research in human-quality text-to-speech synthesis.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, al. SpeechTS: Unified-modal encoder-decoder pre-training for spoken language processing. In ACL, 2022.
Mahmoud Ashraf. Whisper diarization. http:////gtttu.ccoom////eaaeeraa 2024.
Kaito Baba, Wataru Nakata, Yuki Saito, and Hiroshi Saruwatari. The TO5 system for the VoiceMOS challenge 2024: Transfer learning from deep image classifier to naturalness MOS prediction of high-quality synthetic speech. In SLT, 2024.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-super learning of speech representations. NeurlPS, 2020.
James Betker. Better speech synthesis through scaling. arXiv:2305.07243.222 2023.
Marcely Zanon Boito, Vivek Iyer, Nikolaos Lagos, Laurent Besacier, and loan Calapodescu. mHuBERT-147: A compact multilingual hubert model. arXiv:2406.06337, 2024.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. arXiv: 1607, 046666 2016.
Hervé Bredin. pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe. In INTERSPEECH, 2023.
Pawel Budzianowski, Taras Sereda, Tomasz Cichy, and Ivan Vulić. Pheme: Efficient and conversa- tional speech generation. arXiv:2401.00 002839, 2024.
Edresson Casanova, Kelly Davis, Eren Gölge, Görkem Göknar, Iulian Gulea, Logan Hart, Aya Aljafari, Joshua Meyer, Reuben Morais, Samuel Olayemi, et al. XTTS: massively multilingual zero-shot text-to-speech model. arXiv:2406.04994 2024.
Sanyuan Chen, Chengyi Wang. Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li. Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. WavLM: Large-scale self-supervised pre-training for full stack speech processing. J-STSP, 2022.
Yushen Chen, Zhikang Niu, Ziyang Ma Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, and Xie Chen. F5-TTS: A fairytaler that fakes fluent and faithful speech with flow matching. arXiv: 241.::240000888. 2024.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, ACL, 2018.
Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli, Unsupervised cross-lingual representation learning for speech recognition. INTERSPEECH, 2021.
Erica Cooper, Wen-Chin Huang, Tomoki Toda, and Junichi Yamagishi. Data for the VoiceMOS challenge 2022, 2022.
Erica Cooper, Wen-Chin Huang, Yu Tsao, Hsin-Min Wang. Tomoki Toda, and Junichi Yamagishi. A review on subjective and objective evaluation of synthetic speech. AST, 2024.
Jakub Piotr Clapa al. Whisperspeech. hittess//ssthuab.ccc/////ppppspp
Whi ShisppSSpeech 2024.
Sefik Emre Eskimez, Xiaofei Wang, Manthan Thakker, Canrun Li, Chung-Hsicn Tsai. Zhen Xiao, Hemin Yang, Zirun Zhu, Min Tang, Xu Tan, et al. E2 TTS: Embarrassingly easy fully non- autoregressive zero-shot TTS. In SL.7. 2024.
11

---

Published as a conference paper at ICLR 2026
Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. Learning word vectors for 157 languages. In LREC, 2018.
Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex Smola. A kernel method for the two-sample-problem. NeurIPS, 2006.
Haorui He, Zengqiang Shang. Chaoren Wang. Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang, Jiaqi Li, Peiyang Shi, et al. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation. SLT, 2024.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. NeurIPS, 2017.
Wei-Ning Hsu, Benjamin Bolte, Yao- Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. TASLP, 2021.
Wen-Chin Huang, Szu-Wei Fu, Brica Cooper, Ryandhimas E Zezario, Tomoki Toda, Hsin-Min Wang, Jumichi Yamagishi, and Yu Tsao. The VoiceMOS challenge 2024: Beyond speech quality prediction. In SLT, 2024.
ITU-T. Perceptual evaluation of speech quality (PESQ): An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs. Rec. ITU-T P. 862, 2001.
Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar. Rethinking FID: Towards better evaluation metric for image generation. In CVPR, 2024.
Jee-weon Jung, Wangyou Zhang, Jiatong Shi, Zakaria Aldench, Takuya Higuchi, Alex Gichamba, Barry-John Theobald, Ahmed Hussen Abdelaziz, and Shinji Watanabe. ESPnet-SPK: full pipeline speaker embedding toolkit with reproducible recipes, self-supervised front-ends, and off-the-shelf models. In INTERSPEECH, 2024.
Aditya Khan, Mason Shipton, David Anugraha, Kaiyao Duan, Phuong H Hoang, Eric Khiu, A Seza Dogruöz, and En-Shiun Annie Lee. Uriel+: Enhancing linguistic inclusion and usability in typological and multilingual knowledge base. arXiv preprint 2rXiv:2409.18471 2024.
Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fréchet Audio Distance: A reference-free metric for evaluating music enhancement algorithms. In INTERSPEECH, 2019.
Chanwoo Kim and Richard M Stern. Robust signal-to-noiss ratio estimation based on waveform amplitude distribution analysis. In INTERSPEECH, 2008.
Ambika Kirkland, Shivam Mehta, Harm Lameris, Gustav Eje Henter, Eva Szekely, and Joakim Gustafson. Stuck in the mos pit: critical analysis of mos test methodology its evaluation. In SSW, 2023.
Soheil Kolouri, Phillip E Pope, Charles E Martin, and Gustavo K Rohde. Sliced wasserstein auto- encoders. In ICLR, 2019.
Solomon Kullback. Kullback-leible divergence, 1951.
Anurag Kumar, Ke Tan, Zhaoheng Ni, Pranay Manocha, Xiaohui Zhang, Ethan Henderson, and Buye Xu. Porchaudio-Squim Reference-less speech quality and intelligibility measures in torchaudio. In ICASSP, 2023.
Yoach Lacombe, Vaibhav Srivastav, and Sanchit Gandhi, Parler-tts. https://github. com/ 2024.
Moritz Laurer. Wouter van Atteveldt, Andreu Salleras Casas, and Kasper Welbers. Less annotating, more classifying addressing the data scarcity issue of supervised machine learning with deep transfer learning and bert nli. Open Science Framework, June 2022. URL httpp::/oss/st io/
74b8k Preprint.
12

---

Published as a conference paper at ICLR 2026
Sang-Hoon Lee, Ha- Yeong Choi, Seung-Bin Biing-Binnee Kim, and Seong- Whan Lee. Hierspeech++: Bridging the gap between semantic and acoustic representation of speech by hierarchical variational inference for zero-shot speech synthesis. arXi::2311.1245. 2023.
Xinjian Li, Siddharth Dalmia, Juncheng Li, Matthew Lee, Patrick Littell, Jiali Yao, Antonios Anastasopoulos, David R Mortensen, Graham Neubig, Alan W Black, and Metze Florian. Universal phone recognition with a multilingual allophone system. In ICASSP, 2020.
Yinghao Aaron Li, Cong Han, Vinay Raghavan, Gavin Mischler, and Nima Mesgarani. StyleTTS 2: Towards numan-leve text-to-speech through style diffusion and adversarial training with large speech language models. NewrIPS, 2023.
Shijia Liao, Yuxuan Wang, Tianyu Li, Yifan Cheng, Ruoyi Zhang, Rongzhi Zhou, and Yijin Xing. Fish-speech: Leveraging large language models for advanced multilingual text-to-speech synthesis. arXiv:2411.01 2024.
Jianhua Lin. Divergence measures based on the shannon entropy. Transactions on Information theory, 1991.
Haohe Liu, Qiuqiang Kong, Qiao Tian, Yan Zhao, DeLiang Wang, Chuanzeng Huang, and Yuxuan Wang. Voicefixer: Toward general speech restoration with neural vocoder. arXiv:2109.37331 2021.
Chen-Chou Lo, Szu-Wei Fu, Wen-Chin Huang, Xin Wang, Junichi Yamagishi, Yu Tsao, and Hsin-Min Wang. MOSNet: Deep learning-ber based objective assessment for voice conversion. In INTER- SPEECH, 2019.
Dan Lyth and Simon King. Natural language guidance of high-fidelity text-to-speeet with synthetic annotations, 2024.
Georgia Maniati, Alexandra Vioni, Nikolaos Ellinas, Karolos Nikitaras, Konstantinos Klapsas, June Sig Sung, Gunu Jho, Aimilios Chalamandaris, and Pirros Tsiakoulis. SOMOS: The Samsung Open MOS Dataset for the Evaluation of Neural Text-to-Speech Synthesis. In INTERSPEECH. 2022.
Christoph Minixhofer, Ondřej Klejch, and Peter Bell. TTSDS text-to-speech distribution score. In SLT, 2024.
Gabriel Mittag, Babak Naderi, Assmaa Chehadi, and Sebastian Möller. NISQA: A deep enn-self- attention model for multidimensional speech quality prediction with crowdsourced datasets. In INTERSPEECH, 2021.
Masanori Morise, Fumiya Yokomori, and Kenji Ozawa. WORLD: vocoder-based high-quality speech synthesis system for real-time applications. IEICE Transactions on Information and Systems, 2016.
Mylo. Bark voice cloning hubert quantizer.
http::://gituub.com//gg
bark ovoice-clominge HuBERT HuBERT-quantizer 2023.
Puyuan Peng, Po-Yao Huang, Shang-Wer Li, Abdelrahman Mohamed, and David Harwath. Voice- Craft: Zero-shot speech editing and text-to-speech in the wild. In ACL, 2024.
Sameer Pradhan, Ronald Cole, and Wayne Ward. My science tutor (MyST) large corpus of children's conversational speech. In LREC-COLING, 2024.
Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. ML.:: A large-scale multilingual dataset for speech research. In INTERSPEECH, 2020.
Zengyi Qin. Wenliang Zhao, Xumin Yu. and Xin Sun. Openvoice: Versatile instant voice cloning. arXiv:2312.01479, 2023.
Alec Radford, Jong Wook Kim, Tao Xu. Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In ICMI., 2023.
13

---

Published as a conference paper at ICLR 2026
Chandan KA Reddy, Vishak Gopal, and Ross Cutler. Dnsmos p. 835; A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. In ICASSP, 2022.
Simon Rouard, Francisco Massa, and Alexandre Défossez. Hybrid transformers for music source separation. In ICASSP, 2023.
RVC-Boss. Gpt-sovits.
2024.
Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. UTMOS: UTokyo-SaruLab system for VoiceMOS challenge 2022. In INTERSPEECH, 2022.
Siddharth Sharma, Luca Perić, Vatsal Aggarwal, et al. Metavoice. https://github com/
meetavo..eeoo//ettavvicee---eeee 2024.
Kai Shen, Zeqian Ju, Xu Tan, Eric Liu, Yichong Leng, Lei He, Tao Qin, sheng zhao, and Jiang Bian. NaturalSpeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. ICI.R, 2024.
Jiatong Shi, Hye-jin Shim, Jinchuan Tian, Siddhant Arora, Haibin Wu, Darius Petermann, Jia Qi Yip, You Zhang, Yuxun Tang, Wangyou Zhang, et al. Versa: A versatile evaluation toolkit for speech, audio, and music. arXiv:241..16677, 2024.
Suno. Bark. https://gittuu. http:/gittuu..ooosuu.ouoo ai/bark] 2023.
Cees H Taal, Richard C Hendriks, Richard Heusdens, and Jesper Jensen. An algorithm for intelligi- bility prediction of time-frequency weighted noisy speech. TASLP, 2011.
Andros Tjandra, Yi-Chiao Wu, Baishan Guo, John Hoffman, Brian Ellis, Apoorv Vyas, Bowen Shi, Sanyuan Chen, Matt Le, Nick Zacharov, et al. Meta Audiobox Aesthetics: Unified automatic quality assessment for speech, music, and sound. arXiv:2502.0513) 2025.
Sarenne Wallbridge, Christoph Minixhofer, Catherine Lai, and Peter Bell. Prosodic Structure Beyond Lexical Content: A Study of Sepf-Supervised Learning. In INTERSPEECH, 2025.
Li Wan, Quan Wang, Alan Papir, and Ignacio Lopez Moreno. Generalized end-to-end loss for speaker verification. ICASSP, 2018.
Chengyi Wang, Sanyuan Chen, Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, al. Neural codec language models are zero-shot text to speech synthesizers. 3xiv:23301002111 2023a.
Hongji Wang, Chengdong Liang, Shuai Wang, Zhengyang Chen, Binbin Zhang, Xu Xiang, Yanlei Deng, and Yanmin Qian. WeSpeaker: A research and production oriented speaker embedding learning toolkit. In ICASSP, 2023b.
Yuancheng Wang, Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng, Qiang Zhang, Xueyao Zhang, Shunsi Zhang, and Zhizheng Wu. MaskGCT: Zero-shot text-to-speech with masked generative codec transformer.
Ttl::24090007550 2024.
Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg- Kinn-KKrkattrikk, and Shlomo Dubnov. Large-scale contrastive language-audin pretraining with feature fusion and keyword-to-caption augmentation. ICASSP. 2023.
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. LibriTTS: A corpus derived from librispeech for text-to-speech. In INTERSPEECH, 2019.
Xueyao Zhang, Liumeng Xue, Yicheng Gu, Yuancheng Wang. Jiaqi Li, Haorui He, Chaoren Wang, Ting Song. Xi Chen, Zihao Fang, Haopeng Chen, Junan Zhang, Tze Ying Tang, Lexiao Zou, Mingxuan Wang, Jun Han, Kai Chen, Haizhou Li, and Zhizheng Wu. Amphion: An open-source audio, music and speech generation toolkit. In SLT, 2024.
Xueyao Zhang, Xiaohui Zhang, Kainan Peng. Zhenyu Tang, Vimal Manohar, Yingru Liu, Jeff Hwang, Dangna Li, Yuhao Wang. Julian Chan, Yuan Huang, Zhizheng Wu, and Mingbo Ma. Vevo: Controllable zero-shot voice imitation with self-supervised disentanglement. ICLR, 2025.
14

---

Published as a conference paper at ICLR 2026
A EVALUATED TTS SYSTEMS AND PARITY WITH REAL SPEECH
Table 6: Open-source TTS systems, prior evaluation, and results for each system relative to
ground-truth (GT) speech: i accompanied by publication; -tirddparty implementation;
Parity column: Reported MOS/CMOS are close to GT (~), surpassing GT (>) or below GT (<)
System
Year
Objective
Subjective
Parity
Bark Suno 2023 Myla
2023
EE2-TTS Eskimez
2024
F5-TTS Chen al 2024
2024
FishSpeech 1.5 ei 20241
2024
GPT-SoVITS v2 RVC-Hoss 2014
2024
HHierSpech++44 1.1 Loe et 2025
2023
MaskGCT Wanng ctal.
2024 Zhang ot al 2024]
2024
Meta MetaVoice Voice- Sharma et 2024
2024
1* NaturalSpeech 2Shen et 2024
/hang
2023
Open Voice v2 et
2024
ParlerTTS Large 1.0
Lacombee
2024
2024
Pheme (Budzianow
3124
2024
SpeechTss A 2023
2022
SSyyleTTS 2 al. 2023
2023
TorToiSe Betker 2023
2022
1 **VALL- Wang ct
2023a
Zhang ot al. 2024]
2024
Vevo
Zhang et 2025
2024
VoiceCraft- -00t--330 Peng al 2024
2024
WhisperSpeech Medium
2024
1XTTS Casanova
2023
The systems evaluated in TTSDS2 are shown in Table subject to expansion. It should be noted that for ParlerTTS, modification of the codebase was used allow for voice cloning, which could degrade performance. The latest available checkpoint before 01-01-2025, the data collection cut-off, was used for each system, with the exception of XTTSv2, which we experienced difficulties with for grapheme-to-pe to-phoem--e conversion.
Of the 20 systems, 13 were accompanied by research papers, which all but 3 reported subjective and objective evaluation. Of the 13 systems accompanied by papers. report being within 0.05 of
ground truth MOS or CMOS (Eskimez et 2024) Shen et al. 2024 Lee al., (023) or surpassing
ground truth by this margin (Chen al. 2024 Wang et al. 2024
Li et al. 2023 Lyth & King
2024). Of the objective evaluation methods outlined in Section I.4 WER and Speaker Similarity
reported times, followed by UTMOS, CER, Fréchet-type distances, and MCD, which were all reported twice. PESQ and STOI were reported once.
15

---

Published as a conference paper at ICLR 2026
B LISTENING TEST DETAILS
For the precise wording of questions, an example survey can be viewed at Etsdsbenchmarks com/ survey. For WILD "audio books" was replaced with "YouTube"; ; for KIDS with "children's is speech". We also include instructions here:
Listing MOS Instructions
You will be listening to 6 sets of audio samples. In each set, there are text. Some of them may be synthesized
5 audio recordings of the while others may be from audio books. Flease listen to the audio clips carefully, then,
Rate how natural each audio clip scunds
scale of 1 5 with 1
Indicating completely unnatural spesch (bad) and 5 completely natural speech (excellent). Here, naturalness includes whether you feel the speech is spoken by real speaker from a human source.
Some clips may be early, please ignore words that may be cut off when rating the naturalness.
There are some audios for checking your attention through the survey, If certain number of the scores rated unreasonably, you will not paid. Please listen the audios carefully and do not fill out the survey randomly.
You will be listening Listing 2: CMOS Instructions
audio samples. each set, there are two audio recordings (A and B) of the same text. Some of them may be synthesizes while others may be from audio books. Please listen to the audio clips carefully, compare each audio with the reference, and then
Rate how natural is A compared to B on a scale of -3 to 3 with 3 indicating that A is much better than B. Here, naturalness includes whether you feel the speech is spoken by real speaker from human source, as opposed being synthesized by computer.
Some clips may be cut early, please ignore words that may be cut off when rating the naturalness.
There are some audios for checking your attention through the survey. If
a certain number of the scores are rated unreasonably, you will not be paid. Please listen to the audios carefully and do not fill out the survey randomly.
Listing 3: SMOS Instructions
You will be listening to 18 sets of audio samples. In each set, there are two audio recordings (A and of the same text. Some of them may be synthesized while others may from audio books. Please listen the audio clips carefully, compare each audio with the reference, and then
Rate how simi the speaker in B is the speaker in
scale of to 5 with 5 indicating that the speaker in is the same person as the speaker in A. Here, similarity includes whether you feel the speeca is spoken by the same person.
Some clips may be cut early, please ignore words that may be cut off when rating the similarity.
There some audios for checking your attention through the survey. If certain number of the be paid. Please listen to the audios carefully and do not fill out the survey randomly.
rated unreasonably, you will
16

---

Published as a conference paper at ICLR 2026
B.1 MOS, CMOS AND SMOS
The naturalness of this audio is:
(bod)
(poor)
(fair)
(good)
(evcetient)
O
Figure 4: Interface Mean Opinion Score (MOS) listening tests.
Clip
Clip
9:0
The naturalnass of Clip A compared to the natura ines of Clip B is:
(A la slightly worse)
(A
(A much better)
(A la better)
(about same)
siightly better)
worss)
worse)
Figure 5: Interface for Comparison MOS (CMOS) listening tests.
Clip A
Clip
000 3:04
0:30 9:14
46
The speaker similarity between Clip A and Clip B is:
(bod)
(poor)
(fair)
(good)
(axcellent)
Figure 6: Interface for Speaker Similarity MOS (SMOS) listening tests.
In this work, we conduct listening test using the most common methodologies for subjective speech evaluation, namely the Mean Opinion Score (MOS), Comparison MOS (CMOS), and Speaker Similarity MOS (SMOS). For a MOS test, listeners rate isolated audio samples on a 5-point scale from (bad) to (excellent), as shown in Figure CMOS tests present listeners with samples, and B, and ask them to rate their relative naturalness on a 7-point scale from -3 (A is much worse than B) +3 is much better than B), which can be seen Figure 5 This particularly useful for fine-gr ine-grained comparisons when absolute scores may saturate. SMOS is used for voice cloning evaluation and operates similarly to CMOS, but listeners rate the speaker similarity between two clips on a 5-point scale, as illustrated in Figure O
B.2 ETHICS APPROVAT.
This study was certified according to the Informatics Research Ethics Process, reference number 112246. Participants were informed about the purpose and terms of the study using the Participant Information Sheet available here: ttt.dsbbnncharr www.coocc....... wIs. pdf] None the participants withdrew their consent within the 30-day period following the listening test study.
17

---

Published as a conference paper at ICLR 2026
C
DOMAIN-WISE MOS, CMOS, SMOS AND TTSDS2 RESULTS
Table 7: Listening test results in terms of MOS, CMOS and SMOS relative ground truth speech.
SMOS
4,2440.16
4.1910.19
3.11=0.17
3.60.10.19
4.3310.18
3,46+01199
2,1710.11
3.95.10.18
2.20++0.19
3,4610.23
1.9010011 16
263+0.21
2.1740.188
2.76.00.18 18
3.39
26811.220
2.0010.15 15
1.8810.16
2.59-10.18
1.90:10.20 -1.39=0.28
3.3340.27
0.1820.09
-0.42:00.20
-0.03=0-17
-0.9510225
-0.37=:0.20
-0.76=021
-1.17-022
-047.0222
4067=0031
-1.64=023
-1.171025
-1.94-0.30
-1.63
-0.76
-1.02.10.25
-134
-1.37
-1.56:2025
-1.65
-1.75=0.21
Kids
CMOS
3.96400006
3.82:1..133
2222--11
3,49110.14 14
3,261:0.44
3,40+-116
3.07100177
3.2510.17
3.22+0122
2.654..17
2.5910.16
2.774.19
3.0940.15
2.8110.16
263
2.20---119
2.28
2.941014
1.9510.11
2.1140.11
MOS
SMOS
4.51=0.11
4.33
3.65
4.42-0.13 13
4.30= 13
3.40=0.21
2.6 18
1667.0.17
2.57=021
2
266
3.59-0.18
2.00=0.19
1.34
3.98-0.17
2.48-0.17
2.06
3.10=
2.40=0.17
1.77=0.13 0313
1.79
3.67
Wild
CMOS
-0.08
-0. 0.26=00.17
0
0.30-0.32 20
0.16
-0.14=0
-1442
-0.24 10.1
00.98 -0
-0.63
-1.05 26
-1 20
-1.60-0
00.7
27
0093-0222
00.3
0075
21
-109
1.82=0.17
-1.69
-0.88
-0.93
MOS
3.75+0.06
3.40:1.112
3,88+0.12
12
3.52:10.1
3.69
3.18+0011
2.311015
2.87
3.15+0.16
2.91
2.9
2.76-10.22
2.06
2.8910.13
13
2.4410.11
3,12:40.19
3.0410.13
2.89.10114
217:10.11
2.00:0.116
SMOS
4.4==0199
4.20=0.13
445=0.15
402-00.15
4.50=0.12
385=0.19
2.79-0.19
3.33 1.33
290=0.16
3.71=0.18
11222.19
388-
2.54=0.22
3 3,50
424=0.14
261=0.18
253=0.19
245=0.18
3.62-0.17
241
2.61=0118
Noisy
CMOS
10.20:2211
-0.21
-0.06+0-88
0.07 18
0.19:2.116
0048+024
0.02-025
0.3610.16
00224027
00291-226
0.38 23
0934412
-0.1940311
0,641021
-0.330221
0,76400277
-107710.26
-0.74:223
0.84HH188
0.76+:0224
-1.91
3.35
3.00=0.14
3.07=0
3.8
3.27:0.13
3.37=0.17
3,50
3.311
2.08=0115
1,12-0114
3.3320.15
295
2.98=0.19
3.05
28
3.1100.15
2.32=0.15
2.99:0.13
2.6600112
2.2620.17
1.75
MOS
2.
SMOS
4.29=0.14 14
4.45=0.11
4.36 16
4.4=-0.13
353==0.19
330--019
3.69 10
3.
4,04-0021
4.051:115
3.31
3.63-0.18
41441.15
3.67=0.199
2.52-0.16
2.410.16
16
261
2.440.14 14
2.7920.16
Clean
CMOS
0.002:0.11
0..02 18
0. 0.10000118
0.17.00.1
0.36:10.17
0.35++0.20
0.301021
0.68 10.20
1.00
-0.91
0.8110.119
10.19
0.24
11.40+0.20
0.2331.19
19.19
0.0010166
1..4:+0.27
1.7110..5
1.10:40.20
24
-1.24:0.24
3.72+0006
3.4420.12
337=0.12
3,511.14
29010.16
33810015
3.1810.13
3.12+0.16
3.63--.188
1,3310112
3.48+0.166
296+00.4
VALL-E II 2.8410.13
3.09::0.14
268::015
2.3310.13
1.78:20112
1.7920.11
2.0620.144
MOS
Speocht
NaturalSpeseh2
System
Ground Truth
E2-TTS
F5-TTS
MaskGCT
FishSpesch
TorToiSe
VoiceCrall
HierSpeech++
SSyleTTS2
Pheme
OpenVoice
BITSOVITS
XITS
MetaVoice
ParlerTTS
SpeechTS
Vevo
Bark
The Table above shows the MOS, CMOS and SMOS scores derived from the listening tests.
18

---

Published as a conference paper at ICLR 2026
D TTSDS2 FACTORS
Table 8: Pearson correlation (r)) between each factor and MOS.
Dataset
Generic
Speaker
Prosody
Intelligibility
Clean
0.42
0.84
0.38
0.47
Noisy
0.59
0.86
0.46
0.59
Wild
0.53
0.59
0.34
0.58
Kids
0.80
0.70
0.60
0.63
To assess their impact on the overall score, we present their individual TTSDS2 factors' Pearson correlations Table For CLEAN and NOISY, the speaker factor dominates an interesting result, given that paired Speaker Similarity metrics performed poorly for these datasets. For WILD and KIns, the speaker factor is less useful, with Intelligibility and Generic showing similar correlations. Prosody is highest for KIDS, suggesting its value in assessing replication of children's 's prosodic patterns, Overall, while the Speaker factor is generally the most helpful, the others are complementary, especially for the non-read speech in WILD and KIDS. All correlations significant (p 0.05).
E CORRELATIONS BETWEEN OBJECTIVE AND SUBJECTIVE METRICS
Selgectiin
Saljective
Crlad Borywen
Aushobox fasthet
Authbbar
MOS Pradic on
MOS PNNRORY
M00
SFECE
328
Sgrub2ausd
Signa Gasd
MOCH
ASA Eric 10a
as tan AM
STEE
2 tenAD
(a) In-domain (Clean) correlations.
(b) Cross-domain (Noisy, Wild, Kids) correlations.
Figure 7: Significant (p < 0.05) Spearman correlation heatmaps between metrics.
As shown in Figure in the in in-domain setting (CLEAn), TTSDS2 achieves the highest correlation with subjective scores (p 0.79), outperforming both MOS and CMOS in predicting each other. Audiobox Aesthetics also correlates with MOS, though less strongly than previously reported, likely due to differences in evaluation datasets. Among MOS prediction networks, only SQUIM performs well (p- 0.66), while X- X-Vector the only Speaker Similarity metric showing moderate correlation. In the out-of-domain datasets (NoISY, WILD, KIDS), correlation patterns shift: Audiobox Aesthetics weakens, while SQUIM improves (p- 0.67). Speaker Similarity metrics especially RawNet3 show the strongest correlations (p 0.73). TTSDS2 remains strongly correlated (p 0.72), but is no longer the top predictor.
E. NEGATIVE CORRELATIONS WITH MCD AND WER
For Word Error Rates (WER) a negative correlation expected, as higher WER indicates worse TTS performance. However, the correlations observed (see Table 3) are still not sufficient to consistently predict subjective ratings, with a minimum of p = -0.45 for KIDS MOS. Signal-based metrics (MCD, PESQ) occasionally show negative correlations as well, which could be due to "oversmoothing" with systems predicting the average of many possible utterances achieving a better score in said metrics, while leading to worse human ratings.
19

---

Published as a conference paper at ICLR 2026
F COMPUTE
Synthesising samples for all TTS systems in this work, across datasets and languages, including failed runs and reruns, used 28.8 hours single A100 GPU. We estimate each rerun use approximately 8 compute hours for synthesis, subject to changing number of languages and TTS systems.
Running TTSDS2 is CPU-bound due the Wasserstein distance computations. Each TTSDS2 score computation took z 9.4 minutes using an Intel (R) Xeon (R) CPU E5-2620 v4 10GHz.
G LANGUAGE DISTANCE VISUALISATION
28
8
10
Figure Multidimensional scaling (MDS) distance plots between languages (left to right) for Uriel+ typological distances ii) TTSDS2 without multilingual changes iii) multilingual TTSDS2. The three closest neighbors of each language are connected via lines.
As discussed in Section 4 TTSDS2 scores, when interpreted as distances between ground truth language datasets, correlate with Uriel+ typological distances. Figure these distances visually. An interesting effect is that English is more distant from other languages than in Uriel+, likely due to much larger amount English data used even when training explicitly multilingual models.
H FILTERING OF CONTROVERSIAL CONTENT
In preliminary experiments, we find L.L.-based filtering to too resource-intensive for recur- ring benchmark, while toxic comment classification work does fails filter potentially contro- versial but not explicitly toxic content, and often is not available in multiple languages. As de- tailed in Section 4.1 therefore follows negat ive, ppolitt ialllilaa,, gender, reli religion, sexual, cont ronersivial rare word, incompleteeee.e.e race The threshholds used are [0.6, 6, 6,6, 0.6 6, 0.6, 0.6, 0,8, 0. 6, 0,9, 0. 6] and are set using spor-checking on English data. The multilingual entailment model used is lf. htt//Moottzllaurr/mmeeeeeee
entailment model. The entailment words used as
base mnli-xn17
20


Reviews:
[{"review_id": "WdrPt45TiH", "reviewer": "Reviewer_jomi", "summary": "This paper introduces TTSDS2, a framework for evaluating text-to-speech (TTS) systems. While TTS has advanced rapidly, existing evaluation metrics often fail to capture the human-level quality of recent systems (particularly in multilingual contexts). Moreover, subjective evaluations such as MOS are difficult to compare across systems due to differing evaluators and datasets. TTSDS2 addresses these issues by assessing how closely the distributions of various perceptual factors (e.g., speaker identity, prosody, intelligibility) in synthetic speech resemble those in real speech, relative to noise. Experimental results demonstrate that TTSDS2 correlates strongly with human listening tests and consistently outperforms other open-source objective metrics.", "questions": "- Why do the authors use *noise* samples as the negative anchor? Wouldn’t failed or unnatural TTS outputs serve as a more realistic negative baseline?\n- How does the inference or computation time of TTSDS2 compare to other objective evaluation metrics?\n- What is the rationale behind averaging all factor scores equally to form the final TTSDS2 score? Are all perceptual dimensions equally important for perceived quality?\n- Could the authors report the correlation of each individual factor (e.g., prosody, intelligibility) with mean opinion scores to clarify which aspects contribute most to perceptual alignment?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Comprehensive comparison against numerous evaluation metrics.\n- Evaluation across 20 recent TTS systems and 14 languages.\n- Diverse and well-structured test sets covering clean, noisy, wild, and children’s speech domains.\n- Offers a promising, scalable solution for objective TTS evaluation.\n- Shows clear and consistent alignment with subjective (human) evaluation results.", "weaknesses": "- The overall methodology closely resembles the original TTSDS, with limited conceptual novelty.\n- The paper averages multiple perceptual factor scores into a single TTSDS2 score but provides no justification for using equal weighting.\n- The selection criteria for the feature sets within each factor are not clearly explained or motivated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QmSBCL4lXJ", "reviewer": "Reviewer_LwgN", "summary": "This paper introduces TTSDS2, a factorized, distributional objective metric for evaluating modern TTS systems. It measures similarity to real speech across multiple perceptual dimensions (generic, speaker, prosody, intelligibility) using Wasserstein distance over model-derived embeddings. The authors evaluate 20 recent open-weight TTS models across four domains and show that TTSDS2 is the only metric achieving Spearman > 0.5 consistently against MOS/CMOS/SMOS. The paper also provides a multilingual benchmark (14 languages) and an automated pipeline for recurring evaluation. Overall, I believe the work is timely, well-executed, and useful.", "questions": "Are there clear failure cases where TTSDS2 disagrees with MOS/CMOS? If so, what characteristics do these samples/systems share?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The problem is well-motivated: subjective evaluation is expensive and rapidly becoming insufficient as systems approach human quality. The factorized distributional approach is intuitive and gives interpretable sub-scores.\n\n* The empirical evaluation is broad, covering 20 recent models and multiple domains (clean, noisy, wild, children).\n\n* Demonstrating consistent correlation >0.5 with human ratings across all domains is compelling.\n\n* Benchmark and pipeline release are valuable to the community and likely to be widely used.\n\n* Writing is clear and situates the work well within the literature.", "weaknesses": "* A more principled justification or ablation is warranted as the choice of feature set can appears somewhat tuned for correlation.\n* Figure 3 is not referenced anywhere in text and the caption is vague.\n* The multilingual evaluation lacks validation with human preference tests (e.g., CMOS / MUSHRA), making it difficult to verify whether TTSDS2 maintains reliability beyond English.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cO61RpmAQ6", "reviewer": "Reviewer_KrsR", "summary": "This paper introduces TTSDS2, a comprehensive distributional metric for evaluating text-to-speech (TTS) quality. TTSDS2 integrates four key quality dimensions—generic quality, speaker similarity, prosody, and intelligibility—and, for each dimension, leverages multiple embedding models to compute the Wasserstein distance between embeddings of synthesized and reference speech. To assess the robustness of the metric, the authors compile a diverse evaluation dataset encompassing read speech, noisy speech, YouTube speech, and children’s speech, and benchmark 20 TTS systems using both human (MOS) and automatic metrics. Experimental results demonstrate that TTSDS2 consistently correlates strongly with human judgments across domains. The authors also provide a continually updatable evaluation pipeline designed to mitigate data leakage and maintain a reliable benchmark over time.", "questions": "- **Motivation for Distributional Metric:**\n    \n    What is the underlying intuition or theoretical justification for why a *distributional* metric should better reflect human perception compared to point-wise measures?\n    \n- **Averaging Strategy Across Dimensions:**\n    \n    The paper averages scores across different embedding models within each dimension, and then across the four dimensions. Have the authors explored using weighted combinations or learned weights to better capture the relative importance of each dimension?\n    \n- **Speaker Similarity Performance:**\n    \n    The speaker similarity metrics consistently rank as the second-best in correlation with human judgments. Can the authors provide an explanation or analysis for why this particular dimension performs slightly worse than the others?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- **Comprehensive and Diverse Evaluation:**\n    \n    The paper introduces a test set with human annotations covering four diverse domains—read speech, noisy speech, YouTube speech, and children’s speech. Both the dataset and the evaluation are publicly released, enabling future reproducibility and comparison. The study also benchmarks a wide range of baseline TTS models and compares against multiple existing quality metrics, ensuring a thorough evaluation.\n    \n- **Strong Correlation with Human Judgments:**\n    \n    TTSDS2 demonstrates consistently high correlation with human perceptual ratings across different speech domains, confirming its robustness and reliability as an automatic TTS quality metric.\n    \n- **Continually Updatable Benchmark:**\n    \n    The proposed benchmark pipeline supports continual updates while preventing data leakage, which makes it a valuable and sustainable resource for future TTS research and model evaluation.", "weaknesses": "- **Poor Writing Quality and Structure:**\n    \n    While the main contribution of the paper is clear, the overall writing is repetitive and poorly structured, particularly in Section 1. \n    \n- **Questionable Gaussian Assumption in Metric Design:**\n    \n    TTSDS2 assumes that the speech embeddings follow a multivariate Gaussian distribution when computing the Wasserstein distance. However, this assumption may not hold in practice, and the authors provide no empirical justification for it. A more principled alternative would be to employ an optimal transport algorithm to estimate the Wasserstein distance directly from the embedding distributions.\n    \n- **Unconvincing Multilingual Evaluation:**\n    \n    The paper’s claim of multilingual validity for TTSDS2 is not sufficiently supported. To substantiate the multilingual generalization of the proposed metric, additional human evaluations across languages are necessary.\n    \n- **Potential Annotation Bias:**\n    \n    Each annotator is assigned to only one dataset, which could introduce domain-specific bias and limit cross-domain consistency in the human evaluation results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)