Rethinking MUSHRA: Addressing Modern Challenges in Text-to-Speech Evaluation
Authors: Praveen Srinivasa Varadhan
amogh gulati
Ashwin Sankar
Srija Anand
Anirudh Gupta
Anirudh Mukherjee
Shiva Kumar Marepally
Ankur Bhatia
Saloni Jaju
Suvrat Bhooshan
Mitesh M Khapra
Venue: Accepted by TMLR ()

Abstract: Despite rapid advancements in TTS models, a consistent and robust human evaluation framework is still lacking. For example, MOS tests fail to differentiate between similar models, and CMOS's pairwise comparisons are time-intensive. The MUSHRA test is a promising alternative for evaluating multiple TTS systems simultaneously, but in this work we show that its reliance on matching human reference speech unduly penalises the scores of modern TTS systems that can exceed human speech quality. More specifically, we conduct a comprehensive assessment of the MUSHRA test, focusing on its sensitivity to factors such as rater variability, listener fatigue, and reference bias. Based on our extensive evaluation involving 492 human listeners across Hindi and Tamil we identify two primary shortcomings: (i) reference-matching bias, where raters are unduly influenced by the human reference, and (ii) judgement ambiguity, arising from a lack of clear fine-grained guidelines. To address these issues, we propose two refined variants of the MUSHRA test. The first variant enables fairer ratings for synthesized samples that surpass human reference quality. The second variant reduces ambiguity, as indicated by the relatively lower variance across raters. By combining these approaches, we achieve both more reliable and more fine-grained assessments. We also release MANGO, a massive dataset of 246,000 human ratings, the first-of-its-kind collection for Indian languages, aiding in analyzing human preferences and developing automatic metrics for evaluating TTS systems.

Published in Transactions on Machine Learning Research (05/2025)
Rethinking MUSHRA: Addressing Modern Challenges in Text-to-Speech Evaluation
Praveen Srinivasa Varadhan', Amogh Gulati?, Ashwin Sankar', Srija Anand', , Anirudh Gupta², Anirudh Mukherjee', Shiva Kumar Marepally', Ankur Bhatia², Saloni Jaju?, Suvrat Bhooshan', Mitesh M. Khapra' 'AlABharat, Indian Institute of Technology Madras, Gan.A1
{csº1d201, miteshk @ese. wtm. oc.in
Reviewed on OpenReview: https: openreview. net/ forum? id= oYmRsWCQ1W
Abstract
Despite rapid advancements in TTS models, consistent and robust human evaluation framework is still lacking. For example, MOS tests fail to differentiate between similar mod- cls, and CMOS's pairwise comparisons are timo-intensive. The MUSHRA test is a promising allernative for evaluating multiple TTS systems simullaneously, but in this work we show that its reliance on matching human reference speech unduly penalises the scores of modern TTS systems that can exceed human speoch quality. More specifically, we conduct compre- hensive assessment of the MUSHRA test, focusing on its sensitivity to factors such as rater variability, listener fatigue, and reference bias. Based ing 492 human listeners across Hindi and Tamil we identify two primary shortcomings: (i) siference-macchnng bias, where raters are unduly inffuenced by the human reference, and (ii) judgement ambiguily, arising from a lack of clear fine-grained guidelines. To address these issues, we propose two refined variants of the MUSHRA test. The first variant enables fairer ratings for synthesized samples that surpass human reference quality. The second variant reduces ambiguity, as indicated by the relatively lower variance across raters. By combin- ing these approaches, we achieve both more reliable and more fine-grained assessments. We also release MANCcH a massive dataset of 246,000 human ratings, the crrst-of-its-king collec- lion for Indian languages, aiding in analyzing human preferences and developing automatic metrics for evaluating TTS systems.
extensive evaluation involv-
1 Introduction
Huraa evaluation is widely regarded as the gold standard for Texl-To-Speech (TTS) assessment; however, it lacks standardization. This issue is more realized with the rapid advancements in TTS synthesis, where numerous models claim superiority over prior systems or human speech (Li et al. 2023 Wang et al.) 2023 Tan
et al. 2024). Deciphering the true extent of improvement from one model to the next highly challenging due to inconsistent and often inadequately described subjective evaluation methodologies actoss studies.
The above problem is well studied for the Mcan Opinion Scores (MOS) test WWester et al. 2015 Finkelstein
et al. 2023 Kirkland et al. 2023 Le Maguer et al.) 2024) which has received much constructive criticism over the past. few years. Specifically, in MOS test., listeners assess each system independently, which can result in an inability accurately capture the subtle relative differences between similar systems. This poscs significant challenge in modern TTS evaluation where systems that perform equally well need to be compared against each other. To address these issues some of the recent works rely on CMOS tests [oizoou 2011). However, this test is costly and time-consuming as it involves (N) comparisons between all pairs of N systems.
The dataset is publicly available http:///wwnggiisee..c.ccttttttts

---

Published in Transactions on Machine Learning Research (05/2025)
The MUSHRA test has been gaining popularity in addressing these issues. This test scales better by enabling parallel comparison of the N systems, and addresses the limitations of MOS tests that only allow isolated evaluation. However, we show that even the MUSHRA test is not devoid of issues. To begin with, we note that the MUSHRA test was conventionally designed to assess iutuuit audio systems (ITU-R 2015). However, stale-of-the-art TTS systems (Ju et al. 2024) are not of intermediate quality and instead generate audios having quality on par or even better than human recordings. To align with these modern developments, several works adopt variants of MUSHRA (Merritt et al. 2022; Li et al. 2023 Shen et al. 2024h, which differ in implementation but the validity of these modified tests is unknown.
Given this situation, we critically assess the reliability, sensitivity, and validity of the MUSHRA tests by asking a series of research questions, such as: Is MUSHRA a reliable test, consistently yielding results comparable to other widely adopted subjective tests such as CMOS? Is the mean statistic reported in MUSHRA reliable, or is there significant variance across listeners and utterances? How sensitive is MUSHRA to implementation details? Particularly, how many listeners and utterances are required to yield statistically significant. results? Is the conventional MUSHRA reject rule appropriate when TTS outputs sometimes outperform ground-truths? How does the choice of anchor affect MUSHRA scores, and what is the optimal anchor? While some of these questions have been studied for MOS (Wester et al. 2015 assessment of MUSHRA remains lacking.
comprehensive
With the goal of seeking answers to the above questions, we collected 246,000 human ratings by conducting the MUSHRA test involving 3 systems across two languages, viz., Tamil and Hindi. Our in-depth analysis based on these ratings, reveals two primary shortcomings: (i) reference-matching bias and (ii) judgement ambiguity. To mitigate these issues, we propose two refined variants of the MUSHRA test. The first variant does not explicitly identify the human reference to the rater. Doing 80, prevents unfair penalties for well-synthesized samples that differ from the human reference, such as those with natural prosody that do not match the reference's prosody, In the second variant, raters are provided scoresheets to systematically calculate MUSHRA scores, by explicitly marking promunciation mistakes, unnatural panses, digital artifacts, word skips, liveliness, voice quality, rhythm, etc. Using the scores for these fine-grained criteria, they arrive at the final MUSHRA score. Our studies show that both these variants lead to a more reliable evaluation with the second variant also allowing for fino-grained fault isolation during evaluation. While MUSHRA-DG does require additional time to complete the tests, we believe that, given the limitations of the current MUSHRA setup, a slightly more time-intensive solution is justified to ensure the integrity and reliability of the evaluation process. We then show that combination of these two approaches that leverages their individual strongths ensures both consistency and gramularity. It allows modern TTS systems to be evaluated without being unfairly penalized for surpassing the reference naturalness or prosody. The detailed scoring for pronunciation, prosody, and other factors provides actionable insights, and helps practitioners understand precisely where their systems excel and where improvements are needed. This combination creates a more balanced and sensitive evaluation framework, offering clearer and more reliable assessment of TTS system performance.
In summary, our main contributions are:
1. A comprehensive assessment of the reliability, sensitivity, and validity of the MUSHRA test imple- mentation in evaluating modern high-quality TTS systems.
2. Identification of two primary shortcomings of the MUSHRA test: (i) Referencce-atchhnng bias and (ii) Judgement Ambiguity.
3. Proposal of two variants of MUSHRA aimed at addressing these shortcomings.
4. Large-scale empirical validation of proposed variants resulting in MANGO, a dataset of 246,000 ratings from 492 listeners across Hindi and Tamil, examining three TTS systems.
2 Related Work
Perceptual Judgments. TTS evaluation primarily relies on naturalness, which, while lacking a universally agreed-upon definition (Shirali-Sahhrza & Penn 2023 generally refers to how closely synthesized speech
2

---

Published in Transactions on Machine Learning Research (05/2025)
resembles human speech in perception. This concept aligns with standard evaluation practices and is typically
assessed through subjective listening tests such as MOS (Fańcucki 2021 Ren et al. 2021 Kim el al. 2021
CMOS (Shen et al.) 2024] Tan et al. 2024 Li et al., 2023) or MUSHRA (Lajszezak et al. 2024), where listeners rate speoch samples based on their perceived rcalism and overall fidelity.
Critiques of TTS Evaluation. Prior works mainly focused on a critique of MOS tests. Wester et al. (2015) analyze results from the Blizzard Challenge 2013 and highlight that an adequate number of listeners and utterances are needed to accurately identify significant differences. [lark et al. (2019) find that MOS tests are context-sensitive and yield different results when evaluating sentences in isolation opposed to rating whole paragraphs. MOS tests are also known to show high variance in ratings (Finkelstein et al. 2023), subject to how raters are chosen. Kirkland et al. (2023) realize the importance of reporting scale labels, increments, and instructions, and show how these variables can affect scores. A recent study (Cooper & Yamagishi] 2023) highlights the presence of range-oqualizing bias MOS tests. Chiang et al. (2023) analyze over 80 papers, noting insufficient description of evaluation details and its impact on evaluation outcomes. Similarly, Le Maguer et al. (2024) highlight the need for better evaluation protocols.
Emergence of Modern Tests. Several variants of MUSHRA have been employed to overcome known shortcomings. To evaluate the robustness of TTS trained on imperfect transcripts. Fong et al. (2019). adopt the MUSHRA test without an anchor and also provide text transcripts during evaluation. TTaylor & Richmond (2020) measure impact of morphology using a hidden natural reference, and utterances containing out-of- vocabulary words. AAggarwal et al.] (2020) extend the MUSHRA test to also measure emotional strength of the synthesised speech. Merritt et al. (2022) adopt MUSHRA for evaluating speaker and accent similarity, by including both an upper-anchor and lower-anchor along with hidden reference. Li et al. (2023) adopt variant of the MOS test, similar to MUSHRA, for testing naturalness and speaker similarity.
Learnings from Human Evaluations in NLP Freitag et al. (2021) highlighted the need for compre- hensive, standardized evaluation frameworks like Multidimensional Quality Metrics for MT, which is crucial for TTS too. Ethayarajh & Jurafsky (2022) show that the average of Likert ratings (as followed in MOS tests in TTS) can be biased estimate potentially leading to misleading rankings. [Amidei et al. (2019) discuss how insufficient descriptions can make it difficult to interpret evaluation results. Howcroft & Rieser (2021) emphasize that current evaluations are inadequate for detecting subtle distinctions between systems; problem we find recurring in TTS evaluations. Direct assessments have been popular in WMT evaluations [Barrault et al. 2020 Akhbardeh et al. 2021], however Knowles (2021) highlight several of its issues, a caution that carries over to human evaluations for TTS. They also advocate evaluating multiple systems on the same subset of documents, a practice we mirror in this work using audio samples instead.
3 MANGO: A Corpus of Human Ratings for Speech
We introduce new dataset, MANGO: MUSHRA Assessment corpus using Native listeners and Guidelines to understand human Opinions at scale. It is a first-of-its-kind collection for any Indian language, comprising 246,000 human ratings of TTS systems and ground-truth human speech in both Hindi and Tamil, making it an expensive endeavour but one that we hope will contribute meaningfully to research in speech evaluation. Given the shortcomings of Mean Opinion Score (MOS) and Comparative Mcan Opinion Score (CMOS) tests, our goal is to critically examine promising alternative the MUSHRA test. by conducting large-scale evaluation involving multiple raters, systems, and languages. To do so, we adopt the standard MUSHRA test (ITURR 2015).
Raters evaluate multiple stimuli on each page, including an explicit (mentioned) reference that serves as a benchmark for high-quality speech, along with an anchor and implicit (hidden) reference to calibrate judgments. Each stimulus is rated on a continuous scale from to 100, which is also discretized: 100-80 (Excellent), 80-60 (Good), 60-40 (Fair), 40-20 (Poor), and 20-0 (Bad). We describe our evaluation sctup below, and provide the detailed instructions provided to participants in Appendix A.T
3

---

Published in Transactions on Machine Learning Research (05/2025)
3.1 MUSHRA-Based Annotation Framework
To evaluate TTS systems, we use the standard MUSHRA test ITU-R (0155), a methodology originally devel- oped for codec evaluation and widely used in speoch and audio quality asscssment. MUSHRA particularly suited for listeners with normal hearing and experience in critical listening, as it requires evaluating multiple stimuli side by side while referencing a high-quality benchmark. Raters assess multiple stimuli per page, including explicit reference (a. high-quality speech benchmark), anchor (a lower-quality version for cal- ibration), and an implicit (hidden) reference to normalize judgments. Each stimulus is rated on a continuous 0-100 scale, discretized into five categories: 100-80 (Excellent), 80-60 (Good), 60-40 (Fair), 40-20 (Poor), and 20-0 (Bad). To ensure reliable and consistent evaluations, MUSHRA follows established guidelines according to the standard. Listeners must complete pro-test training to familiarize themselves with impairments and test signals. Stimuli should be around 10 seconds long, with a maximum of 12 seconds, to minimize listener fatigue and enhance response stability. An assessor's responses are excluded if they rate the hidden reference below 90 in more than 15% of test items. Tests must use either headphones or loudspeakers, but not both within a session, ensuring consistency across participants. When listening conditions are technically and behaviorally controlled, data from as few as 20 subjects can yield statistically reliable conclusions. These measures help maintain the rigor and reprodueibility of MUSHRA- specch evaluations, making it a robust choice for large-scale TTS assessment.
Comparison with Alternative Tests. MUSHRA offers several advantages over Mean Opinion Score (MOS) and Comparative MOS (CMOS) for speech quality assessment. MOS, despite its widespread use, has been criticized for its poor reliability and inability to distinguish between similar-sounding systems due to its single-stimulus nature and the absence of an explicit reference. While CMOS provides pairwise comparisons, it is inherently limited to evaluating only two systems at a time, making large-scale assessments both expen- sive and difficult to scale. In contrast, MUSHRA enables simultaneous comparisons across multiple systems, allowing listeners to switch between samples and directly compare quality, which enhances sensitivity to sub- tle differences that MOS often fails to capture. The continuous 0 100 scale provides finer granularity than MOS's coarse 5-point rating, allowing for more nuanced judgments. Additionally, MUSHRA is more efficient than exhaustive pairwise CMOS tests, as it allows multiple systems to be evaluated in single trial rather than requiring numerous pairwisc comparisons. Another key strength is reference- based calibration, where listeners rate systems relative to a known high-quality reference and an anchor, with the aim of reducing scoring variability. Ribeiro et al. (2015) further confirm MUSHRA's 'ss advantages, demonstrating that it more effectively distinguishes between models compared to MOS, reinforcing its suitability for evaluating modern TTS systems.
Evolution of MUSHRA. Over time, researchers have refined the MUSHRA framework to better align with the evolving demands of speech synthesis evaluation. While the standard MUSHRA test remains widely used, several variations have emerged, modifying key aspects such as reference anchoring and test structure to address specific challenges in assessing TTS quality. For instance, Tase [Tajszzzak et al. 2024 follows the MUSHRA framework but omits anchors, suggesting that explicit anchoring may not always be necessary for reliable judgments. NaturalSpeech2 (Shen et al., (2044) further questions the necessity of strict reference matching by employing a MUSHRA-
CMOS setup without an explicit reference. Additionally, Expressive- MUSHRA has been introduced to evaluate expressiveness in speech synthesis (Huybrechts et al. 2021; Varadhan et al., 2024), highlighting MUSHRA's flexibility in capturing prosodic variation and expressive muances. Beyond these structural modifications, studies have also examined MUSHRA's effectiveness and limitations. For example, Merritt et al. (2018) critique the traditional 3.5 kHz anchor, arguing that it may not be idcal for modern TTS, as distortions in neural TTS systems differ from those originally considered in codec evaluations. These adaptations reflect a growing recognition that MUSHRA, while valuable, may require refinements to remain optimally effective for contemporary TTS evaluation. Recognizing this need, we systematically explore alternative test designs and their impact evaluation reliability.
3.2 Online Annotation Platform
We enhance the webMUSHRA (Schoefferr et al. 2018)) platform to address its key limitations. Specifically, we modify fork (Pauwels et al.] 2021) and introduce session management to enable saving test progress,
4

---

Published in Transactions on Machine Learning Research (05/2025)
Table Dataset statistics of MANGO.
Gender
Age
# Participants in MUSHRA Variants
Language
# Ratings
Female
Male
18-25
25-30
30-35
35-40
40+
Original
NMR DG
DG-NMR
Hindi Tamil
127,500
163
140
18
11
113
102
118,500
154
81
82
73
36
28
16
100
97
20
20
and thereby allowing for breaks for listeners. We integrate consent forms and controls, such ensuring raters listen to all audio samples in their entirety, to ensure more reliable ratings. We also integrate an event-tracking system to analyze the time spent per page.
3.3 Synthesizing Speech Samples for Annotation
To generate samples for TTS evaluation, train TTS systems the Hindi and Tamil subsets of the IndicTTSS database (Baby et al. 2016). Each language consists of recordings from female and male speaker
(Hindi: 20.17 hours; Tamil: 20.59 hours). We train FastSpeech2 (FS2) (Ren et al. 2021 with HiFiGAN vl
(Kong et. al. 2020) and VITS (Kim et al.
2021) from scratch on the train-lest splits using hyper-parameters
suggested in a recent study (Kumar et al.
2023b).
We finetune StyleTTS2 (ST2) (Li et al. 2023 from the
LibrITTS chockpoint.
3.4 Annotation Process and Dataset Statistics
To ensure reliable evaluation, we recruited native speakers of the target languages through reputable recruit- ment agencies. These agencies played a vital role in guaranteeing participant demographics aligned with the target language of each test. Please refer to Section A.9 for details on recruitment, consent, and com- pensation. Once recruited, the annotators underwent comprchensive training process comprising multiple sessions aimed at familiarizing them with the evaluation platform, lest. interface, and evaluation criteria. As part of this process, demo calls were conducted, where participants were introduced to the rating interface and guided through a sample rating task. These sessions allowed participants to ask questions and receive clarifications ions in real time. Additionally, a structured review process was implemented, wherein participants initially rated five pilot samples. This phase allowed them to seek clarifications, provide feedback, and ensure their understanding of the evaluation criteria before proceeding to rate the 100 test samples. All instructions provided participants are included in Appendix A.6 for transparency. This approach not only improved their confidence, but also helped standardize the assessment process across all evaluators.
With the above process, collected 246,000 human ratings for TTS systems. We also collect demographic information from majority of the 492 participants, while subset of 21 individuals preferred not to disclose their details. Table shows the demographic distribution, the number of participants, and the total number of ratings across all MUSHRA tests and our proposed variants (MUSHRA-NMR, MUSHRA-DG, MUSHRA- DG-NMR) which are described in Section The table reflects responses from participants who disclosed demographic details, while a small subset preferred not to share this information.
4 Key Insights on MUSHRA
In this section, we address the research questions outlined in Section I and identify key challenges based on ratings collected in the MANGO datasct.
4.1 Is MUSHRA A Reliable Test?
In Table 2 we present the results of the MUSHRA lest among 3 systems and find VITS and ST2 score highest in Hindi and Tamil respectively. Surprisingly, all systems attain scores in the "Good" bin with MUSHRA scores between and 80, while the reference surpassos all systems with scores the "Excellent" bin. Given that state-of-the-att TTS systems are able to reach quality on par with references, one would expect much smaller gap between the reference and systems. To confirm this, we conduct the more reliable
5

---

Published in Transactions on Machine Learning Research (05/2025)
Table 2: MUSHIRA scores for Hindi and Tamil using Anchor- X and Anchor-Y, respectively, as anchors (ANC), y represents the mean, a represents the standard deviation, and the 95% confidence intervals (CT) provided.
Table 3: Mean Comparitive-Mean
Opinion-Scoree (CMOS) with 95% con- fidence intervals for Hindi & Tamil.
Hindi
Tamil
System
Hindi
Tamil
System
C1
a
C1
REF
ST2
-0.11 + 0.08
0.24 + 0.09
FS2
64.17
22.89
0.42
64.98
19.23
0.38
VITS
-0,10 + 0.07
-0.57 + 0.09
ST2
66.74
21.65
0.40
71.38
18.31
0.33
FS2
-0.66 + 0.08
-0.60 + 0.09
VITS
67.65
20.58
0.38
65.66
18.91
0.37
ANC
70.81
20.92
0.39
20.08
16.69
0.38
REF
84.18
15.49
0.29
85.22
15.98
0.31
but expensive CMOS test with 15 listeners in each language. In this test, we ask the rater to compare a given system, such as VITS, with reference audio sample. The rater evaluates both the reference and the output from system being tested without prior knowledge of which audio sample corresponds to which system, ensuring an unbiased comparison. The raters assign a single score ranging from -3 to +3 in increments of 0.5. A score of -3 indicates that System A is much worse than System B A score of +3 indicates that System is much better than System B. A score of O means that both systems are equal in quality. As seen from the scores in Table 3 CMOS indicates that the outputs synthesised by VITS and ST2 are very close in quality to the reference in Hindi and Tamil respectively, while MUSHRA scores do not reflect this at all. We hypothesize that listeners in the MUSHRA test. are subject to various biases, one of which we term the reference-matchinn bias. This bias may lead to situations where systems that perform comparably to or better than the reference are rated less favorably, as listeners tend to focus on aligning their ratings with the reference outputs while evaluating the systems. While this may have been acceplable when TTS systems lagged behind human speech quality, it is undesirable in the current scenario where modern TTS systems often exceed the reference in aspects like naturalness and prosody (Li et al., 2023 Shen et al. 2024). This suggests that the MUSHRA test, in its conventional form, may no longer be sufficient for evaluating state-of-the-art TTS systems. Instead, alternative methodologies, such as the variants we propose in Section may help ensure more fair and accurate assessments.
Further evidence questioning the role of the reference as the highest standard in MUSHRA evaluations is provided in Appendix [A.2] where we analyze CMOS-derived preference ratings and highlight cases where TTS systems are notably preferred over the human reference.
4.2 How reliable is the mean statistic in MUSHRA scores?
As mentioned carlier, cach rater rates 100 utterances. In Figure L box-plots to visualize the distri- bution of MUSHRA scores (y-axis) for each rater (x-axis) across these utterances for each system, including the reference and anchor. While we acknowledge that the figure may appear overwhelming, we believe it is crucial for conveying the comprehensive view across both raters and utterances. We make two important observations from the figure. First, many the individual box-plots (cspecially of a system) have a high vari- ance indicating that the same rater rates the system very differently across utterances. Second, looking at the means of the box-plots across different raters, we obscrve that there is a high variance in the means, indicating ambiguity in the perception of the MUSHRA labels across raters. We refer to this phenomenon as judgement ambiguity. This highlights the shortcomings of reporting mean statistics for MUSHRA scores, even when reported with confidence intervals (CT).
To delve deeper into judgment ambiguily, we examine variations between two systems. We consider an utterance where the mean scores for the samples generated by VITS and ST2 are nearly identical, but the variance across raters for each system is high. This high variance indicates significant ambiguity. Upon listening many such utterances and speaking to many raters, we hypothesize that the ambiguity likely stems from different raters focusing on different. aspects of the generated samples. For instance, some raters
6

---

Published in Transactions on Machine Learning Research (05/2025)
REF
FS2
ST2
ANC VITS
10 12 25 34 35 401
45
20
55 62- 70 75 $0 155 92 50
110
105
110
Rater ID
Figure Visualization of the MUSHRA score distributions per rater across three systems FS2, ST2, and VITS, along with the reference (REF) and anchor (ANC) for Hindi. Each boxplot represents ratings (0-100) across all test utterances for a system by one rater. While some boxplots exhibit relatively low variance, several display substantial height, indicating that certain raters assign widely varying scores to the same system across utterances. The variation in the means of the boxplot across raters suggests high level of inter-rater variance. Raters are sorted in ascending order of their mean scores for the reference.
may prioritize prosody, others voice quality, and yet others the presence of digital artifacts. We hypothesize that asking raters to highlight these subtle differences across multiple dimensions while assigning a single score can lead to ambiguity determining how much penalize or reward system's output. Hence, clear guidelines which take into account fine-grained evaluation across different aspects would help (as proposed later in Section
4.3
How sensitive MUSHRA to number of listeners and utterances?
We use the procedure outlined in [Wester et 2015) to study the effect of number of listeners and utterances interested in knowing if smaller number of listeners and utterances
MUSHRA scores. Specifically, would result in the same rankings of systems obtained using the full set of listeners and utterances. To achieve this, we randomly sample smaller subset of utterances and listeners and compute the mean system scores. We then calculate the Spearman rank correlation with the mean system scores obtained using all utterances and listeners. We repeat this process 1000 times, and compute the average over these large number of trials.
Figure 2 illustrates the average correlation of MUSHRA ratings in Hindi between subset of listeners and utterances compared to the fully-scaled test (involving all listeners and utterances). Firstly, it is evident that using minimum of 20 listeners is crucial to achieve correlations above 90%. Secondly, when employing smaller number of listeners and utterances (c.g., fewer than 40 in both cascs), increasing the number of listeners proves to be more beneficial than increasing the number of utterances. Using more than 30 listeners and 30 utterances invariably yields correlations above 95%. We notice similar trends for Tamil, but with higher correlations achieved with lesser number of listeners and utterances (Appendix B.. These results highlight the sensitivity of MUSHRA evaluations the number of listeners and utterances, emphasizing the importance of careful selection and scaling of these factors to ensure reliable and meaningful evaluations.
7

---

Published in Transactions on Machine Learning Research (05/2025)
110
09
0.
10
10
10
10
10
100
10
10
0C.
90
100
10
1.0
10
0.98
1.0
1.0
1.0
1,0
1.0
10
90
10
0.91
10
1.0
1.0
1.0
1.0
1.0
1.0
100
0.95
85
FS2
ST2
80
0.84
0.99
10
0.98
0.96
10
0.98
100
10
10
0.9
MUSHRA
80
VITS
# Listeners
70
0.87
0.97
1.0
0.99
1.0
0.95
1,0
1.0
1.0
0.85
ANC
60
0.99
0.98
1.0
REF
0.8
50
0.99
0.99
0.99
70
0.75
40
0.98
0.7
65
0.91
0.93
0.93
0.93 0.93
0.65
60
10
0.72
0.89
0.89
0.91
0.86 0.85 0.87 0.87 0.87 0.86
75
80
85
90
95
0.6
(R=113)
(R=106)
(R=80)
(R=69)
(R=56)
50 60 70 80 90 100
# Utterances
Figure 3: MUSHRA Scores in Hindi show score- variance but rank-invariance across systems when raters who rate Reference S X for more than 15% of utterances are rejected. R is the number of raters retained.
Figure 2: Rank correlation of mean scores obtained using subsets of listeners and utterances and mean scores obtained using all listeners and utterances in Hindi.
4.4 What is the impact of rejecting raters per standard MUSHRA protocol?
Traditionally, MUSHRA employs a rater rejection criterion, wherein raters scoring the hidden reference (HR) below a threshold (A) more than 15% of the time are rejected. This rejection rule stems from the inherent assumption that the HR the gold standard, which is not true in modern TTS settings where TTS systems (Li et al., 2023 Shen et al., 2024 either par with or surpass the reference. In such cases, a rater consistently scoring the HR lower might not necessarily reflect unreliability, but rather a nuanced perception of the reference's limitations compared to the evaluated systems. This is reinforced by observations from T where raters clearly prefer ST2 over reference for Tamil with a mean CMOS score of 0.24. This observation is further supported by the MUSHRA scores shown in Figure 3 The table shows that while rejecting raters based on the conventional MUSHRA criterion does not affect system rankings, it does notably shift scores. Specifically, system scores decrease with increasing X, while reference scores increase. This trend hints the reference-malchinng bias wherein ralers who give the HR high scores might be unconsciously matching system samples to the mentioned reference, rather than rating based their absolute perception of quality.
4.5 How does Anchor affect scores?
In MUSHRA tests, the anchor serves the purpose setting the expectation what a "Fair" sample sounds like. Typically, the anchor is created by minimally degrading the ground-truth by first. downsampling it to 3.5 kHz and then upsampling it to 24 klz.. We refer such an anchor Anchor-X. In Table 2 the mean scores for Anchor-X in Hindi indicate that this anchor performs significantly better than all other systems, attaining high score of 70.81. We believe these scores explained by the resampling strategy used to create the Anchor-X, which introduces some artifacts in the audio but retains similar naturalnoss to the reference, especially in lerms of prosody. Once again, this intuition indicates the lendency of raters to rate systems that match the reference with higher scores treference-matchingg bias). We conclude that using this anchor may not be idcal, as can lead to potentially "Excellent" TTS systems being unfairly rated "Good", Essentially, the raters may perceive that if one of systems (anchor, in this case) sounds very similar to the reference, there is little justification for rating other systems highly.
8

---

Published in Transactions on Machine Learning Research (05/2025)
Table 4:
MUSHRA-Exxended Table 5: Objective evaluation of TTS systems in Hindi and Tamil, mea- suring intelligibility, speech quality, and signal distortion.
scores with 95% CI for Hindi.
System
/4
0
Hindi
Tamil
System
FS2
63.12
21.30
0.93
STOI
PESQ
SI-SDR
STOI
PESQ
SI-SDR
ST2
65.15
21.76
0.95
ANC
0.964
3.39
27.53
0.995
3.65
22.29
VITS-R
68.47
19.70
0.86
ST2
0.997
3.72
24.72
0.998
3.99
28.63
VITS
68.99
19.67
0.86
REF
0.997
3.93
27.76
0.998
4.08
30.25
ANC
73.62
19.56
0.79
VITS
0.998
4.11
30.17
0.997
3.98
25.63
XTTS
73.65
18.52
0.86
FS2
0.999
4.17
28.89
0.998
4.09
26.74
REF
76.39
18.05
0.81
Next, we study the use of an alternative anchor (Anchor-Y) that we know would likely fall in the "Poor" or "Fair" category, given its construction process. Specifically, we construct Anchor-Y by degrading ST2 outputs by averaging the pitch, reducing the number of diffusion steps, slowing the audio by 1.2 times, and inducing mispronunciation via the input text, along with word skips and word repeats in 20% of the samples. To obtain average voice quality, we set the number of diffusion steps to 3 with a B = 0.8. As expected, from the Tamil MUSHRA scores in Table 2 this anchor does indeed score poorly with a mean of 20.08. Interestingly, we see that despite a very low-quality anchor, other systems are not rated very highly, and there still a huge disparity between ST2 (71.38) and the Reference (85.22). Given that high-quality anchors unfairly bias raters against other systems, while low-quality anchors seem to have no effect on the ralings for other systems, we believe there merit in conducting MUSHRA evaluations without anchors Lajszczak et al. 2024), which also saves costs by reducing human effort.
4.6 Does adding more systems affect scores?
To better understand cognitive overload in the MUSHRA test, scale up the number of systems to be rated by introducing one new competitive system XTTSv2, and repeating an existing system VITS (VITS-R) in the original Hindi MUSHRA test. We finetune XTTSv2 (CoquiAT, 2023) starting from the multilingual checkpoint with the hyper-parameters from their original implementations on the same splits described in Section 3.3 We call this test consistent, with VITS and VTTS-R receiving nearly identical scores (68.99 and 68.47, respectively), despite the randomized order. Introducing XTTS, which outperformed other systems with a score of 73.65, did not disrupt the relative ranking of the remaining systems, which remained consistent with the original MUSHRA test. Thus, there does not seem to be significant cognitive overload, as still observe consistent results. Note that we study cognitive load using n = 7 systems, but it remains to be seen how large n can be before cognitive overload starts impacting the scores. For detailed scores, please refer to Table 4
The results show that raters were highly
4.7 Do objective metrics align with human perception?
To complement our subjective evaluations, we compute three widely used objective metrics (i) PESQ
(Perceptual Evaluation Speech Quality), (ii) STOI (Short-Time Objective Intelligibility). and (iii) SI- SDR (Scale-Invariant Signal-to-Distortiin Ratio), using TudchAudio-Squin (Kumar et al. (023a).. PFSQ estimates speoch quality by assossing distortions caused by noise and compression, while STOI measures intelligibility by evaluating how well speech information enhancement, quantifies the level of distortion in a signal relative to clean reference. These metrics offer quantifiable insights, but are primarily designed and validated on English datasets, making their applicability to languages like Hindi and Tamil uncertain.
preserved. SI-SDR, commonly used in speech
Table reveals weak alignment between MUSHRA scores and both PESQ and STOL, suggesting that these metries do not fully capture percoptual speech quality. This misalignment may stem from their focus on in- telligibility and signal distortions rather than higher-level perceptual attributes such as naturalness, prosody, and timbre. In contrast, SI-SDR exhibits strong Pearson correlation of 0.76 with MUSHRA scores, indicat-
9

---

Published in Transactions on Machine Learning Research (05/2025)
ing that the degree of signal distortion plays key role in perceptual judgments, or that SI-SDR effectively captures aspects of TTS quality that influence human ratings. These findings highlight the limitations of existing objective metrics and the need for more refined evaluation methods that better align with perceptual judgments, either as complementary measures or as reliable alternatives to subjective ratings.
5 Rethinking MUSHRA
We summarize two issues identified in Section 4 First, reference-mallching bias that arises when listeners rate systems that perform at or above the level of the reference lower than deserved due to their efforts to align system outputs with the reference during evaluation. Second, judgement ambiguity that arises when listeners rate a system on a single scale using broadly defined metrics like "naturalness", leaving room for subjective interpretation of sub-criteria such as "prosody", "voice quality", "liveliness", etc. leading to high variability in ratings. Findings from our post-evaluation survey (Appendix A.5) further confirm the presence of these challenges, highlighting the influence of afference-matchinn and the inconsistencies in how raters interpret evaluation criteria. In response to this, propose two refined variants of the MUSHRA test to address the identified challenges, as described below.
The first variant,, MUSHRA-NMR (MUSHRA with No Mentioned Reference), aims to mitigate the reference- reference-matching bias observed in our analysis. MUSHRA-NMI follows all other standard protocols of the MUSHRA (TTU-R 2015) test, except for the omission of the explicitly mentioned ground- truth reference that is presented to the listener. In the absence of this explicilty mentioned reference, the listener will be able to independently assess the quality of the TTS systems without trying to match them against the reference.
MUSHRA-NMR.
The second variant, MUSHRA-DG (MUSHRA With Detailed Guidelines), introduces comprehensive guidelines to reduce the ambiguity in rating samples for naturalness. In this test, we present raters with scoresheets and a formula to arrive at MUSHRA scores systematically. Each rater was asked to mark the number of (i) mild pronunciation mistakes, (fi) severe pronunciation mistakes, (iii) unnatural pauses, speedups, slowdowns, (iv) digital artifacts, (v) sudden energy fluctuations, and (vi) word skips. Further, raters were also asked to rate more perceptual measures such as (i) liveliness, (fi) voice quality, and (iii) rhythm each of these dimensions can be found in Appendix A.6 We analytically derive a MUSHRA naturalness score for raters using an intuitive formula with weights (provided in Appendix A.6 for different dimensions listed above. These weights can be tweaked depending on the specific use-case. For example, in a TTS application designed for audiobooks, where fluidity and expressiveness are crucial for user engagement, we might assign higher weights to liveliness and rhythm.
MUSHRA-DG.
continuous scale from 0-100. The detailed guidelines provided to raters to assess across
We understand that devising scoring formula involves some subjectivity. To address this, encouraged raters to review their evaluations and adjust their fine-grained scores if they felt the overall scores from the formula did not accurately reflect the differences they perceived between system pairs. Notably, we tracked these revisions and found that they occurred in only 1.3% cases. This suggests that raters likely found the computed scores to align with their assessments. Such aligument, is reasonable, as the test design and the formula, including its specific weights and penaltics, were refined through itcrative evaluations and sustained engagement with ralers over several months. Throughout this process, adjustments the guidelines were made based on direct feedback and practical insights from raters, while also being guided by the intuition of TTS practitioners.
More interestingly, we notice that the scores derived from the MUSHRA-DG test preserve the rankings obtained from the gold-standard Comparative Mean Opinion Score (CMOS) tests (Table thus reinforcing the validity of our evaluations. Additionally, the variance MUSHRA scores calculatod using the formula is significantly lower, indicating reduced ambiguity in ratings and providing a clearer distinction between different systems' performances.
10

---

Published in Transactions on Machine Learning Research (05/2025)
Table Comparison MUSHRA scores and Proposed Variants for Hindi and Tamil languages.
MUSHRA-NMR
MUSHRA-DG
MUSHRA-DG-NMA
Language
System
95%
CI
95% CI
μ
95% CI
FS2
61.99
23.86
0.46
72.73
11.65
0.51
81.51
11.50
0.50
ST2
68.09
22.01
0.43
73.41
12.03
0.53
84.97
12.22
0.54
Ilindi
VITS
68.75
21.04
0.41
75.62
10.97
0.48
85.68
10.29
Anchor-X
71.83
19.97
0.39
80.67
13.57
0.59
88.45
7.38
0.32
Reference
76.39
18.08
0.35
90.87
9.34
0.41
88.63
7.39
0.32
Anchor-Y Y
21.94
16.74
0.38
45.00
11.47
0.35
54.66
17.91
0.46
FS2
66.77
19.12
0.35
82.36
8,06
0.32
82.87
10.48
0.35
Tamil
VITS
68.52
18.28
0.36
81.96
7.41
0.35
83.86
10.12
0.44
ST2
76.64
17.68
0.33
88.82
7.88
0.50
91.10
8.05
0.78
Reference
78.69
17.26
0.34
94.61
6.98
0.31
95.99
6.86
0.30
6
Results
We present human evaluation results of our proposed MUSHRA variants from the MANGO dataset.
6.1
Evaluations using MUSHRA-NMR
Does our proposed variant help mitigate the reference-matchhing bias?
In Table d we present the results of the MUSHRA-NMR test. We find the results to be rank-consistent with the scaled-up MUSHRA lests (Table 2h. In the case of Tamil, we observe that the best performing system (ST2) is scored much closer to the reference, clearly suggesting that the reference-matching bias has been mitigated. We observed that the score assigned to the reference itself decrcased, indicating that the raters were strict. In the case of Hindi, the gap between the best performing system and the reference has again decreased but is not as small as in the case of Tamil.
We want to re-emphasize that this expectation a reduced gap between the system and reference scores is nell-founded. Feedback from TTS practitioners, including some of the authors who are native speakers, revealed that while some of the systems performed impressively in practice, the original MUSHRA scores did not seem fully reflect their quality. This shortcoming is also clearly seen by the significant score differences between the reference and the other systems in the original MUSHRA test, whereas the CMOS scores in Table indicate a closer aligument of system's performance with the human ground-truth reference. Collectively, our findings above reinforce the merits of our proposed MUSHRA-NMR variant, which offers more reliable relative assessments compared to the MUSHRA test while retaining the advantages MUSHRA has over the CMOS test.
How sensitive is MUSHRA-NME to number of listeners and utterances?
Subjective evaluations are often resource-intensive making it desirable to minimize the number of listeners and utterances without compromising assessment quality. To explore this, we present the correlation between the scores dcrived from the MUSHRA variants using a subset listoners and the scores obtained from the complete listener set, as shown in Figure We also do a similar comparison across utterances. Our findings reveal that MUSHRA-NMR achieves a Spearman rank correlation exceeding 95% with the fully scaled-up MUSHRA test using just 20 utterances or 40 listeners. This indicates that significant reductions in both parameters possible while maintaining reliability. Interestingly, analysis indicates that enhancing the number of listeners has greater impact the accuracy of assessments compared to simply increasing the number of utterances.
6.2 Evaluations using MUSHRA-DG
Does our proposed variant help mitigate judgement ambiguity while rating? In Table G we show the effects of presenting detailed guidelines along with scoresheets to 20 participants to systematically
11

---

Published in Transactions on Machine Learning Research (05/2025)
Spearman Rank Correlation
0.95
0.98
0.9
MUSHRA
(Hindi)
MUSHRA
(Hindi)
0.85
0.96
MUSHRA-NMM
(Lindi)
MUSHRA-NME
(Iiiddi)
0.8
MUSHRA
(Tamil)
MUSHRA
(Tamil)
0.94
0.75
MUSHRA-NMR
(Tamil)
MUSHRA-NME
(Tamil)
0.7
0.92
10
20
30
60
70
80
100
110
10
20
30
40
50
90
100
Number of Raters
Number of Utterances
Figure 4: (Left) Correlation between scores from a subset of listeners and all listeners. (Right) Correlation between scores from a subset of utterances and all utterances.
arrive at MUSHRA scores. We find MUSHRA-DG scores to be rank-consistent with the CMOS tests while systems scores are much higher and closer to the "Excellent" label, as expected. More importantly, the standard deviation of scores across all systems reduced by 43% in Hindi and 53% in Tamil when compared to the original MUSHRA, indicating that our proposed variant is able to reduce the ambiguity of rating naturalness on single bar while preserving ranks.
It is important to explicitly address the apparent discrepancy in the ranking of VITS and FS2 systems in Tamil to clearly validate our claim that our proposed test also preserves rankings. While the MUSHRA-DO scores for these systems in Tamil suggest a reversed ranking between these systems, a closer inspection at the scores reveals that their perceived naturalness highly comparable. This observation is also corroborated by the CMOS scores presented in Table g which show minimal separation between the two systems in Tamil. To confirm this more appropriately, VITS. Using scale from +3 (indicating FS2 is significantly better) to -3 (indicating VITS is significantly better), 15 listeners rated the systems, and we obtained a CMOS score of 0.13. This result reinforces the notion that the two systems perform similarly.
conducted focused CMOS test that directly compared FS2 and
While system rankings in closely matched scenarios can be debated, we argue that such cases highlight the need to focus on fine-grained differences. Understanding the specific contexts in which model outperforms another provides valuable insights into system behavior and guides targeted improvements. Keeping this in mind, we subsequently discuss the effectiveness of the MUSHRA-DG test in fault isolation.
Fault Isolation. We collate the scoresheets of participants to obtain more fine-grained insights on where each model underperforms. In Figure 5a we report the error rates of instances where an attribute received a rating greater than 0 for the six objective altributes and in Figure 5b the absolute perceptual scores on scale of 100 for the remaining attributes. The granular ratings reveal the true power of this test in identifying defects in TTS outputs, especially among systems that achieved similar mean scores in the original MUSHRA. Specifically, we observe that for Hindi, deterministic system like FS2 performs well in terms of pronunciation but suffers in prosody and word-skipping. Conversely, the close difference between VITS and ST2 is better explained by noting that VITS nearly outperforms in all dimensions, except that VITS exhibits nearly twice as many sudden energy fluctuations as ST2 and performs slightly worse in terms of rhythm.
Similarly, for Tamil, as detailed in Figure 10 in Appendix A.7 the marginal differences across dimensions clarify the perceived inconsistency in the rankings of VITS and FS2 mentioned earlier, which, upon closer inspection, is not truly inconsistency but rather reflection of their comparable overall performance. We also take this opportunity to address the inflated anchor scores observed in the MUSHRA-DG test compared to the original MUSHRA scores in Tamil. One explanation for this inflation could be attributed to leniency in the scoring formula applied to subjective dimensions, resulting in higher anchor scores. However, a more compelling explanation is that the original MUSHRA test may be influenced by Range Equalizing Bias Zielinski (2016), where participants tend stretch scores across the entire scale. Consequently, low-quality anchors are often relegated to the extreme lower end of the scale, even when the perceived degradation is
12

---

Published in Transactions on Machine Learning Research (05/2025)
VITS
120
ST2
100
FS
100
REF
95
ANC
Error Rates (%)
90
60
Score
85
40
80
75
* *VITS ST2 FS REF ANC
Mild
Severe Pron.
Unnatural
Digital Artifact
Sudden Energy Variations
Word Skips
70
Pron.
Speed Up
Down
Liveliness
Voice Quality
Rhythm
(a) Objective (Hindi)
(b) Perceptual (Hindi)
Figure 5: Visualization of the six objective and three perceptual dimensions of the MUSHRA-DG test. The objective scores are represented using stacked bars, where multiple error categories are displayed cumulatively rather than as independent percentages. The subjective dimensions are represented using a scatter plot with scores ranging from 0 100.
not as severe. In contrast, fine-grained evaluations like MUSHRA-DG emphasize detailed attributes and systematic scoring, which mitigate this bias. This approach reduces the lendency to artificially stretch scores, leading to higher and arguably more accurate anchor ratings. Crucially, this inflation does not affect the relative rankings of systems the conclusions drawn from MUSHRA-DG. By minimizing variance and enhancing scoring consistency, MUSHRA-DG continues demonstrate its robustness as reliable and effective diagnostic and ranking tool for TTS evaluation.
Moreover, while the Reference scores appear higher in MUSHRA-DG, this increase is not merely case of inflation. Notably, the scores of the systems also increase, resulting in all systems being classified within the "Excellent" category. This trend aligns with the expectations set by the CMOS scores shown in Table 3 Furthermore, MUSHRA-DG brings the scores of the pe-pperforiig models closer to the Reference scores, a pattern consistent with the results from CMOS evaluations. This alignment highlights the effectiveness of MUSHRA-DG providing fine-grained assessments that capture subtle distinctions between systems while ensuring the overall scores accurately represent system performance.
Time Complexity of MUSHRA-DCE We hypothesize that the additional detail of evaluating each audio sample across multiple dimensions inevitably increases the time required for participants to complete the test. To verify this hypothesis, we visualize the average time taken across pages in Figure 0 and find that the MUSHRA-DG test indeed takes nearly twice as much time as the original MUSHRA test. However, we believe this extra time results in a much more comprehensive understanding of TTS system performance, and allows for fine-graines fault isolation, making the trade-off worthwhile.
6.3 Evaluations using MUSHRA-DG-NNN
present the results of our combined variant, wherein we provide detailed guidelines (DG) and remove the mentioned reference (NMR). We observe that the majority of system scores now align more closely with the reference ratings and predominantly fall within the "Excellent" category, as anticipated from the CMOS tests. Moreover, compared to the MUSHRA-NMR test, the variance in scores has significantly diminished, indicating a marked reduction in rating ambiguity.
In Table 6
As TTS practitioners and native speakers of the language, we would like emphasize that, despite the relative rankings being preserved in nearly all variants of the test, the combined variant. is more reliable
13

---

Published in Transactions on Machine Learning Research (05/2025)
MUSHRA (Hindi)
30
MUSHRA (Tamil)
Average Time Nonmalised
25
MUSHRA-DDE (Hindi)
MUSHRA-DG (Tamil)
20
15
10
10
20
30
50
60
70
90
100
Pages
Figure 6: MUSHRA-DG exhibits higher average time (normalized by audio durations) across pages compared to MUSHRA.
because the scores now reflect the expected proximity of the systems to the reference (also established by the CMOS scores).
7 Generalization to English
Thus far, our study has primarily focused on Hindi and Tamil, two low-resource languages from distinct language families, IryonAryan and Dravidian, spoken by combined 562 million people. This choice ensures coverage, diversity, and generalizability across language families in India. We do realize that evaluating our proposed MUSHRA variants on high-resource language like English would further assess the broader applicability our findings. Expanding the analysis requires additional resources, as human evaluations are both time and cost. intensive. Given constraints, focus our exploration on MUSHRA and the combined variant, MUSHRA-DG-NMR, drawing parallels with CMOS to analyze how refinements translate to English. We conducted these evaluations on English TTS systems trained on the LJSpecch dataset, with 30 native US English speakers rating 30 utterances each, balanced across age (18 to 60) and gender. Additionally, we performed a CMOS test,, where 15 participants compared all systems against. the reference, rating 30 utterances each. Table 7 presents the results, which closely align with our earlier findings in Hindi and Tamil, reinforcing the robustness of our proposed evaluation methodologics.
7.1 Does the Reference- Mattrhin-MMacchnng Bias exist?
A key motivation for work was to assess whether MUSHRA reliably captures the perceptual differences between TTS systems. The scores for English TTS systems in Table ? reinforce our previous findings: while the reference (REF) is the only system placed in the 'Excellent' bin, all other systems are restricted to the "Good" or "Fair" bins. However, CMOS results tell a different story, showing that ST2 and VITS are comparable to, or even preferred over, the reference. This discrepancy highlights MUSHRA's inability fully capture perceptual parity or superiority when modern TTS systems approach or excced reference quality.
These findings further confirm the presence of reference-matching bias (RMB) in English. As result, MUSHRA scores may misrepresent true perceptual differences, reinforcing the need for evaluation methods that minimize the influence of reference dependence.
7.2 Does Judgement Ambiguity Exist?
To assess the reliability of the mean statistic MUSHRA for English, we visualize the distribution of MUSHRA scores across raters and systems (see Appendix: Figure 13). These results reaffirm the key issues previously observed in Tamil and Hindi
14

---

Published in Transactions on Machine Learning Research (05/2025)
Table 7: Comparison of MUSHRA, MUSHRA-DG-NMR, and CMOS for English.
MUSHRA
MUSHRAADDG-MMR
CMOS
System
1
95% CI
μ
95% CI
μ
95% CI
ANC
41.93
3.30
56.38
3.91
FS2
56.86
3.76
63.33
3.32
-0.78
0.22
VITS
72.77
2.75
75.89
2.96
0.02
0.12
ST2
73.27
2.81
78.47
2.84
0.21
0.12
REF 82.62
1.54
79.57
2.96
0
1. High variability in individual ratings: A single rater does not always score system consistently across different utterances, leading to fluctuations in their assessments.
2. Significant perceptual differences across raters: The mean scores assigned by different raters vary widely, indicating inconsistency in how MUSHRA labels are perceived and applied.
These findings highlight the presence of judgment ambiguity, where raters interpret and use the MUSHRA scale differently, leading to inconsistencies in scoring. This further reinforces our claim that the mean statistic alone is not reliable measure of perceived quality, as it is strongly influenced by both intra-rater inconsistency and inter-rater variation.
7.3
Does our proposed variant (MUSHRA-DG-NMP) help?
Our proposed MUSHRA-DG-NMR test was designed to mitigate reference-matching bias (RMB) and judg- ment ambiguity, and the English results further confirm its effectiveness. As seen in Table 7 MUSHRA- DG-NMR brings system scores closer to the reference, suggesting that RMB is reduced. At the same time, it preserves the relative rankings of systems while shifting scores, mirroring our findings in Hindi and Tamil. This particularly important, as it indicates that MUSHRA-DG-NMR does not distort system rankings but instead provides scores that may better reflect perceptual judgments.
While score variance in English does not show significant reduction, a key strength of MUSHRA-DG- NMR lies in its ability to provide fine-grained insights beyond a single score. This is especially valuable when systems receive similar MUSHRA scores but differ in perceptual quality. By analyzing both subjective and objective dimensions, we can pinpoint the specific factors shaping listener preferences. For example, VITS exhibits nearly twice as many sudden energy variations as ST2, while ST2 has almost double the digital artifacts of VITS. Despite these trade-offs, ST2 is preferred overall due its improved rhythmic consistency, reinforcing that simple numerical score does not capture all aspects of TTS quality. For reference, visualizations of both objective and subjective dimensions are provided in Appendix[A.7 Although MUSHRA-DG-NMR requires additional time for evaluation, its ability to uncover detailed system strengths and weaknesses makes it valuable tool for TTS research and development. These fine- tinengrained insights go beyond aggregated scores, offering a deeper understanding of where synthetic speech systems excel and where refinements are needed.
Summary. We release all English ratings, expanding our annotation dataset to 255,000 subjective annota- tions. Those results confirm that MUSHRA scores are affected by referenco-matching bias, mean scores be unreliable due to rating inconsistencies, and MUSHRA-DG-NMIL effectively mitigates bias while preserv- ing system rankings. Additionally, our fine-grained analysis provides deeper insights beyond a single score, allowing for a more precise evaluation of system strengths and weaknesses. Overall, these findings further validate our conclusions across both high-resource (English) and low-resource (Hindi and Tamil) languages.
8
Future Work
While our study introduces refined MUSHRA variants enhance the evaluation of modern TTS systems, several promising directions remain unexplored. The ITU-R standard ITU-R (2015) advises rating more
15

---

Published in Transactions on Machine Learning Research (05/2025)
than 12 signals at a time, yet real-world evaluations could require comparisons across much larger set of systems for creating a leaderboard. A natural extension of our work would be to adapt MUSHRA-DG- NMR for tournament-style evaluations, allowing new systems to be seamlessly integrated while maintaining a dynamic leaderboard. Another compelling avenue is the development of objective metrics that can either complement subjective evaluations or, certain cases, serve as viable replacements, offering a more holistic perspective on TTS quality while improving efficiency. Additionally, refining MUSHRA-DG to make it more accessible for large-scale studies could further extend its impact, without sacrificing the fine- seue-grained insights it provides into promunciation, prosody, and other key speech attributes. Beyond this, exploring cultural and demographic biases within the MUSHRA framework could illuminate potential disparities in how different listener groups perceive synthesized speech, leading to fairer and more representative evaluations. Further- more, validating our refined MUSHRA variants in practical deployment settings will be crucial to ensure their seamless integration into real-world use cases, fostering broader adoption and iterative refinements basod on real-world foedback. These directions build upon our proposed MUSHRA variants and pave the way for more rigorous, equitable, and insightful assessments of TTS systems.
9 Conclusion
Our comprehensive study reveals significant shortcomings in the current use of the MUSHRA test for eval- nating modern high-quality TTS systems. Through an extensive analysis involving 246,000 human ratings for Hindi and Tamil, we identified two primary issucs: reforenco-matching bias and judgment ambiguity. To address these issues, we propose two refined variants of the MUSHRA Lest: MUSHRA-NMR, which omits explicit identification of the human reference, and MUSHRA-DG, which uses detailed guidelines to calculate MUSHRA scores systematically. Our findings indicate that both variants lead to more reliable evaluations, with MUSHRA-DG offering the additional benefit of fine-grained fault isolation during assessment. Through this work, we also release MANGO, a large human rating dataset, to further support research in this area.
References
Vatsal Aggarwal, Marius Cotescu, Nishant Prateek, Jaime Lorenzo- and Roberto ChiratC Using vacs and normalizing flows for ono-shot text-to-speoch synthesis of expressive speech. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020, pp. 6179-6183, IEEE, 2020. doi: 10.100//CAASSP40776.... 200009997772222000033377.7 URL bttps://doi.. org/ 10. www.110/CAASSPP077...2 2020 99036678
Farhad Akhbardeh, Arkady Arkhangorodsky, Magdalena Biesialska, Ondrej Bojar, Rajen Chatterjee, Vishrav Chaudhary, Marta R. Costa-jussà, Cristina España-Bonet, Angela Fan, Christian Federmann, Markus Freitag, Yvelte Graham, Roman Grundkiewicz, Barry Haddow, Leonie Harter, Kenneth Heafield, Christopher Homan, Matthias Huck, Kwabena Amponsah- Kaakyire, Jungo Kasai, Daniel Khashabi, Kevin Knight, Tom Koemi, Philipp Kochn, Nicholas Lourie, Christof Monz, Makoto Morishita, Masaaki Nagata, Ajay Nagesh, Toshiaki Nakazawa, Matteo Negri, Santanu Pal, Allahsera Auguste Tapo, Marco Turchi, Valentin Vydrin, and Marcos Zampieri. Findings of the 2021 conference on machine translation (WMT21). In Loie Barrault, Ondrej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussá, Christian Fe- dermann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno- Philipp Koehn, Tom Koemi, André Martins, Makoto Morishita, and Christof Monz (eds.), Proceedings of the Sixth Conference on Machine Translation, WMT@EMNLP 2021, Online Event, November 10-11, 2021, pp. 1-88. Association for Com- putational Linguistics, 2021. URL
hwapchhoooogy.oggooooooo org/20211 2021 1.1
Jacopo Amidei, Paul Piwek, and Alistair Willis. The use of rating and likert scales in natural language gen- eration human evaluation tasks: A review and some recommendations. In Kees van Deemter, Chenghua Lin, and Hiroya Takamura (ods.), Proceedings of the 12th International Conference on Natural Language Generation, INLG 2019, Tokyo, Japan, October 29 November 1, 2019, pp. 397-402. Association for Com- putational Linguistics, 2019. doi: 10.18653, /1//3//1//19-- V1/W19-8648.
16

---

Published in Transactions on Machine Learning Research (05/2025)
Arun Baby, Anju Lecla Thomas, NL Nishanthi, TTS Consortium, et al. Resources for indian languages. In Proceedings of Text, Speech and Dialogue, 2016.
Loic Barrault, Magdalena Biesialska, Ondrej Bojar, Marta R. Costa-jusså, Christian Federmann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Joanis, Tom Kocmi, Philipp Koehn, Chi-kiu Lo, Nikola Ljubesic, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Santanu Pal, Matt Post, and Marcos Zampieri. Findings of the 2020 conference on machine translation (WMT20). In Loic Barrault, Ondrej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussä, Christian Federmann, Mark Fishel, Alexander Fraser, Yvette Graham, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno- Yepes, Philipp Kochn, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, and Matteo Negri (eds.), Proceedings of the Fifth Conference on Machine Translation, WMT@EMNLP 2020, Online, November 19-20, 2020, pp. 1-55. Association for Computational Linguistics, 2020. URL https: httsss//ccaatthlogg. org/2020. wmt-1.11
Chn Chiang, Wei-Ping Huang, and Hung yi Lec. Why we should report the details in subjective evaluation of tts more rigorously, 2023.
Rob Clark, Hanna Silén, Tom Kenter, and Ralph Leith. Evaluating long-form text-to-speech: Comparing the ratings of sentences and paragraphs. CoRR, abs/1909.00e 2019. URL http://arxiv.org/ abs/ 1909.03965
Erica Cooper and Junichi Yamagishi. Investigating range-equalizing bias in mean opinion score ratings of synthesized speech. arXiv preprint arXiv:2305.10600 2023.
CoquiAI. Xtts v2. https://github htttps///gihaab.mmm/oqppaaallll com/coqui-ai/Trs 2023.
Kawin Ethayarajh and Dan Jurafsky. The authenticity gap in human evaluation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 6056- 6070. Association for Computational Linguisties, 2022. doi: A111663/V1/2022EMMTPPPP http:///doi..0011..188111 1ttp:/000.6653/11//000 emnlp-main.44 4066
URL
Lev Finkelstein, Joshua Camp, and Rob Clark. Importance of human factors in text-to-speech evaluations. In 12th Speech Synthesis Workshop (SSW) 2023, 2023.
Jason Fong, Pilar Oplustil Gallegos, Zack Hodari, and Simon King. Investigating the robustness of sequence-to--sequeeee texl-lo-speech models to imperfeclly-transcrbbedd training data. In Gernot Ku- bin and Zdravko Kacic (eds.), Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019, pp. 1546-1550. ISCA, 2019. doi: 2019-1824
200.SSEEE....211112222 2019- 1824. URL httph:tt///ooo /ttp::////oooor/110/// 10.
Markus Freitag, George F. Foster, David Grangier, Viresh Ratnakar, Qiuun Tan, and Wolfgang Macherey, Experts, errors, and context: A large-scale study of human evaluation for machine translation. Trans. Assoc. Comput. Linguistics, 9:1460- 1474, 2021. doi: 10.1162/TACTA A) 00437. URL. httpp:://doi.
org/19 10. 11l2/taacccc.c.. a 00437
David M. Howcroft and Verena Rieser. What happens if you treat ordinal ratings as interval data? human evaluations in NLP are even more mower-powered than you think. In Maric-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, EMNIPP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 8932 8939. Association for Computational Linguistics, 2021. doi: 10. /1112222.11.122PP222
URL /tttp:://doi.rrg// httpp://booo.oogg.....e6////.... emnle- main. 703
Goeric Huybrochts, Thomas Merritt, Giulia Comini, Bartck Perz, Raahil Shah, and Jaime Lorenzo-Trucba. Low-resource expressive text-to-speech using data augmentation. In ICASSP 2021-2021 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6593 6597. IEEE, 2021.
ITU-R. Method for the subjective www.bjjcctiveesssssseeettt assessment of intermediate quality level of audio systems. https://wwx 2015.
tbu.int/din httppp/ppeeeeeeeeeeee/////a//asa htt.aa/ah.a.aneeee///////aaaaaaa
17

---

Published in Transactions on Machine Learning Research (05/2025)
Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, Zhizheng Wu, Tao Qin, Xiang-Yara Li, Wei Ye, Shikun Zhang, Jiang Bian, Lei He, Jinyu Li, and Sheng Zhao. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models, 2024.
Jaehyeon Kim, Jungil Kong, and Juhee Son, Conditional variational autoencoder with adversarial learning for end-to-end text-to-spoech. In Marina Meila and Tong Zhang (ods.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedlings of Machine Learning Research, pp. 553005544 PMLR, 18 24 Jul 2021. URL https: stttp:://proceedingss mlr. www.rr.eessssvie https////1///mmmmmmm .html
Ambika Kirkland, Shivam Mehta, Harm Lameris, Gustav Eje Henter, Eva Szekely, and Joakim Gustafson. Stuck in the MOS pit: A critical analysis of MOS test methodology in TTS evaluation. In Proc. 12th ISCA Speech Synthesis Workshop (SSW2023), pp. 41-47, 2023. doi: 10.21437 2023-7.
Rebecca Knowles. On the stability of system rankings at WMT. In Loic Barrault, Ondrej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jusså, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvelle Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno- Yepes, Philipp Koehn, Tom Koemi, André Martins, Makoto Morishita, and Christof Monz (eds.), Proceedings of the Sizth Conference on Machine Translation, WMT@EMNLP 2021, Online Event, November 10-11, 2021, pp. 464-477. Association for Computational Linguistics, 2021. URL https: //aclanthology.org org. 20211. amt-1.56
Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis, 2020.
Anurag Kumar, Ke Tan, Zhaoheng Ni, Pranay Manocha, Xiaohui Zhang, Ethan Henderson, and Buye Xu. Torchaudio squim: Reference-less speech quality and intelligibilityy measures in torchaudio. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023a.
Gokul Karthik Kumar, Praveen S V, Pratyush Kumar, Mitesh M. Khapra, and Karthik Nandakumar. Towards building text-to-speech systems for the next billion users. In ICASSP 2023 -2033 IEEE In- ternational Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, 2023b. doi: AAA.:///CAASSPP933.....00 10096669..
Mateusz Lajszezak, Guillermo Cámbara, Yang Li, Fatih Beyhan, Arent van Korlaar, Fan Yang, Arnand Joly, Álvaro Martin-Cortinas, Ammar Abbas, Adam Michalski, Alexis Moinet, Sri Karlapati, Ewa Muszynska, Haohan Guo, Bartosz Putrycz, Soledad López Gambino, Kayeon Yoo, Elena Sokolova, and Thomas Drugman. BASE TTS: lessons from building a billion-parameter text-to-speech model on 100k hours of data. CoRR, abs/2402.08002 2024. doi: 10.48550/ARXI/ Att.:/RXXXV.22220090900 URL
https://doi. htt.////..oo/o////// 10. 48550/ arXiv. 2402.08093.
Adrian Lańcucki. Fastpitch: Parallel text-to-speech with pitch prediction. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 6588 6592. IEFE, 2021.
Sébastien Le Maguer, Simon King, and Naomi Harte. The limits of the mean opinion score for speech synthesis evaluation. Computer Speech and Language, 84:101577, 2024. ISSN 0885-2308. doi: https: //doi.org//1..116//... w1w.://./.......222.10..000 $08852308230009877
URL
Yinghao Aaron Li, Cong Han, Vinay S Raghavan, Gavin Mischler, and Nima Mesgarani. StyleTTS 2: Towards human-level text-to-speech through style diffusion and adversarial training with large speech language models. In Thirty-seventhh Conference on Neural Information Processing Systems, 2023. URL
https://openreview.. htt.://wwww...eeeooooouraamdddap
Philipos C. Loizou. Speech Quality Assessment, pp. 623 654. Springer Berlin Heidelberg, Berlin, Heidelberg, 2011. ISBN 378-3-644-11551-88 978-3-642-19551-8 doi: 10.1007 09707/977-340 6-6---8-------1111 19551-8 23. URL hntpps///oo: http:/:///oo.oo//...111
18

---

Published in Transactions on Machine Learning Research (05/2025)
Thomas Merritt, Bartosz Putryez, Adam Nadolski, Tianjun Ye, Daniel Korzekwa, Wiktor Dolecki, Thomas Drugman, Viacheslav Klimkov, Alexis Moinet, Andrew Breen, et al. Comprehensive evaluation of statisti- cal speech waveform synthesis. In 2018 IEEE Spoken Language Technology Workshop (SLT), pp. 325 331.
IEEE, 2018.
Thomas Merritt, Abdelhamid Ezzerg, Piotr Bilinski, Magdalena Proszowska, Kamil Pokora, Roberto Barra- Chicote, and Daniel Korzekwa. Text-free non-parallel many-lo-many voice conversion using normalising flows. CoRR, abs/2203.,r 00009, 2022. doi: 10.48550/AA0 ARXIV 2203.08009. URL https://doi. org/10.
28650/arXiv 2203. 2003.08009
J. Pauwels, S. Dixon, and J. D. Reiss. A front end for adaptive online listening tests. Centre for Digital Music, Queen Mary University of London, 2021.
Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie- Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.neg! 2021. URL https::/opeereviiee
net/forrmmiddpiiPPXottta
Manuel Sam Ribeiro, Junichi Yamagishi, and Robert AJ Clark. A perceptual investigation of wavelet-based decomposition of ft for text-to-speech synthesis. In INTERSPEECH 2015 16th Annual Conference of the International Speech Communication Association, pp. 1586 1590. International Speech Communication Association, 2015.
M. Schocffter, S. Frees, S. Goetze, and J. Habets. webmushra- comprchensive framework for web-basod listening tests. volume 6, pp. 8, 2018. doi: 10.5334/jors.163
Kai Shen, Zeqian Ju, Xu Tan, Eric Liu, Yichong Leng, Lei He, Tao Qin, sheng zhao, and Jiang Bian. Naturalspeech 2: Latent diffuusion models are natural and zero-shot speech and singing synthesizers. The Twelfth International Conference on Learning Representations, 2024. URL https://openrevies. ket/forum/idd-RcGaaW....
Sajad Shirali-Shahree and Gerald Penn. Better replacement for tts naturalness evaluation. In 18th Speech Synthesis Workshop (SSW) 2023, 2023.
Xu Tan, Jiawei Chen, Haohe Liu, Jian Cong, Chen Zhang, Yanqing Liu, Xi Wang, Yichong Leng, Yuanhao Vi, Lei He, Sheng Zhao, Tao Qin, Frank Soong, and Yan Liu. Naturalspeech: End-to-end text-to-speech synthesis with human-level quality. IEEE Transactions on Pullern Analysis and Machine Intelligence, pp. 1-12, 2024. doi: 10.1109 (00009/PAMM22000 2024.33322...
Jason Taylor and Korin Richmond. Enhancing sequence-to-sseqencc lext-to-speech with morphology. In Helen Meng, Bo Xu, and Thomas Fang Zheng (eds.), Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020, pp. 1738-1742. ISCA, 2020. doi:
2040-1547. URL https://doi org/10.
21437/Inersspeech. 2020-1547
Praveen Srinivasa Varadhan, Ashwin Sankar, Giri Raju, and Milesh M. Khapra. Rasa: Building Expressive Speech Synthesis Systems for Indian Languages in Low-resource Settings. In Proc. ANTERSPEECH 2024, 2024.
Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. Neural codec language models are zero- shot text to speech synthesizers. CoRR, 0ts//200..02111 2023. doi: A44350/AAXIVV 2301.02111. URL https://doi.com ort./1.. 48550/8rX5i...... 23011.vii.00....... 2301. 02111
Mirjam Wester, Cassia -ottiinii,,,,,ao, and Gustav Eje Henter. Are we using enough listeners? no! - an empirically-suppor critique of interspeech 2014 TTS evaluations. In INTERSPEECH 2015, 16th Annual Conference of the International Speech Commanicaation Association, Dresden, Germany, September 6-10, 2015, pp. 3476-3445 3480. ISCA, 2015. doi: 1122447/INTERRPPESCHHHO00. URL https:///ott org//...
21437/ 21437/Interspeech. 2twp:rr5ppeec...116699999 2015-689
19

---

Published in Transactions on Machine Learning Research (05/2025)
Slawomir Zielinski. On some biases encountered in modern audio quality listening tests (part 2): Selected graphical examples and discussion. Journal of the Audio Engineering Sociely, 64(1/2):55-74) 74, 2016.
20

---

Published in Transactions on Machine Learning Research (05/2025)
A
Appendix
A.1 Instructions For MUSHRA
Instructions Participants for MUSHRA Evaluations of Text-to-Speece Systems
Hease keep n mind that you can odjuet your ratings os you listen to different samples. Please regulor fotigue.
Thank partlppotin poting speech evalustion study the quclity of Text-to- Sext-to-Speech (TT5) sjttems. Please follow the instructions given below corefally
every 50 minutes stroin
Overview
Evaluction Criteta After istening to each sample, rote the quality based on its
noturalness and overal quolity Conwider foctors such
avoluction, you istan different samples produceed
systems.
Naturalness: simlor does the oudio sample sound
speech? rhuthm, stress?
Your tosk is to rose these sampres es based on specific citeeia using the MUSHRA (Multiale Stimul with Hidden Reference and Anchor) mathocology
intelligbiity Ity is the speech clear and easy to un understond?
Prosedy;
aporopriate
Evaluction Procedure
Comparative Assessment Compare each sample with the others the page. frisre that your rotings ref true relative rankings of the systems based on your perception. Your evolustions should capture the differences in quality acourately possible.
Setup:
Please good quality headshones speokers to you nuences the oudio samples.
Finoliiing Your Rottings
Find quiet space minimise Istroction during the evaluction.
Once you have rated all somples for a poge, you may move the next
page:
a consistent playback device throughout the eva uction to maintoih
Ensure that you are satisfied with your ratings before submitting, as they recorded.
unformity in issening conditions.
2. Roting Scole:
you have ang questions or need assistance curing the evoluction, please feel free ask.
6. You will use a scole from to 100 to rate the quality of each oudio sample.
rotings comnepand the following conegories
100-80 Excellent
60 4D. Fair
40 900200 Poor
5. Listening and Rottig::
Procedure: rattnng page
1. Listen to the menticned reference carefull to understand high quality.
Then, lster to each
You can listen samp muttplle
11. Ensure you listen to each oudio somple its entirety with without internuptions. them bosse
After lstening to sample, quality noturalness and overall quality
Figure 7: Guidelines sent to participants taking the MUSHRA test. They were given live demo of the rating page and walked through the guideline sheet.
A.2 CMOS Evidence of Limitations of Reference Matching
The assumption that the human reference always represents the highest quality standard in MUSHRA evaluations may not. hold in practice, particularly as modern TTS systems continue to improve. Since MUSHRA inherently relies on reference matching, this approach may unintentionally penalize systems that perform on par with or even surpass the reference. If raters subconsciously anchor their judgments to the provided human reference, systems that differ from it, even in a perceptually superior way, may receive lower scores.
To investigate this, we analyze CMOS-derived preference ratings, where CMOS > indicates preference for the TTS system over the reference, CMOS < 0 indicates preference for the reference, and CMOS 0 suggests no preference. As shown in Table g StyleTTS2 is preferred over the reference in 54% of cases in Tamil and 43.8% in English, demonstrating that model outputs can surpass human recordings in perceptual quality. Even VITS is preferred 37.2% of times against. the reference in English, further challenging the assumption that the human reference should always be treated as the gold standard.
These findings provide direct evidence that MUSHRA'S 's reliance on reference matching may skew evaluations when systems outperform the reference. This motivates to explore MUSHRA-NMR, variant. where
21

---

Published in Transactions on Machine Learning Research (05/2025)
explicit reference is provided, allowing for fairer assessments when TTS models reach exceed human quality. While inspired by multiple listening test methodologies, our goal remains to refine MUSHRA to better accommodate the evolving capabilities of TTS models while preserving its core strengths.
Table 8: CMOS-Derived Preference percentages for TTS Systems in English, Hindi, and Tamil.
Language
System
Reference (%)
Equal (%)
System (%)
ST2
39.1
17.1
43.8
English
VTTS
38.7
24.1
37.2
FS2
68.3
13.5
18.2
ST2
42.8
18.0
39.2
Hindi
VITS
43.9
18.2
37.8
FS2
59.9
11.8
28.3
ST2
41.0
5.0
54.0
Tamil
VITS
63.6
4.8
31.5
FS2
62.9
3.5
33.5
A.3 Visualizing MUSHRA Distributions
In Section 4 discussed the distribution of MUSHRA scores across raters for Hindi using Figure [ Similarly, in Figure 12 and Figure 13 wC visualize the MUSHRA scores rater across the systems for Tamil and English, respectively. Figure 14 Figure 15 visualizes the MUSHRA scores for each utterance, averaged across raters, for Hindi and Tamil respectively.
A.4 Sensitivity of MUSHRA in Tamil
100
10
20
10
10
15
10
1.0
10
10
10
(a)
80
10
10
0.4
0.6
10
0.9
70
10
20
1.
357
0:5
0.99
1.0
10
10
10
Listeners
60
039
1.0
0.85
50
0:34
0.9 556
0.9
0.8
40 0:42
04
344
0:77
0.67
1.0
0.4
098
0.0
0.75
30
359
0.99
0.59
0.00
0.7
10
0.99 0.98
009
559 9:95 0.95 0.8
0.94
0.99
007
0.6
In Figure we present the rank correlation of MUSHRA scores in Tamil, comparing subset of listeners and utterances to the scores obtained using all listeners and utterances. Similar trends are observed as in the case of Hindi.
# Utterances
Figure 8: Spearman rank correlation of MUSHRA scores in Tamil
A.5 Post-Evaluation Survey
To better understand how raters made their judgments during the MUSHRA lest,, we conducted structured post-evaluation survey with 89 participants after the main evaluation. This survey provided direct insights into participant decision-making, particularly regarding reference-matching bias, judgment ambiguity, rating confusion, and cognitive load. The results, sunmarized in Figure reinforce key concerns about strict reference dependence in MUSHRA-based evaluations.
A majority of participants (55%) preferred the system that was closer to the reference, confirming that referenco-matching plays strong role in decision-making However, this bias was not absolute-188 pre- ferred system that deviated from the reference, and 11% rated solely based on overall quality, demonstrating
22

---

Published in Transactions on Machine Learning Research (05/2025)
Question
Hypothesis Factor
Option
Please listen the 3 audios below completely
Reference-Matten Matching
System is better than System A because System is closer to the Reference
System is better than System B because System is closer to the Reference
Reference audio
System
3. System
System is better than System A in general.
Kindly mark your preference.
System is better than System B in general.
7
Can 't find the difference.
7
Please listen to the audios below completely
Judgement Ambiguity
would rate System better than System B.
56
Reference audio
2. System A
System B
Compare System and System against with the reference. System made pronunciation mistake at 2 words but had excellent voice quality. System B had average digital voice quality but showed excellent pronunciation with mistakes. Which option would you choose?
would rate System better than System A.
33
would rate System A equal to System B.
11
How often did you come across confusing situations above?
Confounders
Only times did face difficulty in assigning scores different systems.
56
Always clear how to rate the different systems.
31
Many times faced difficulty in assigning scores different systems due to ambiguity in scoring guidelines.
7
Every page difficulty in assigning scores to different systems.
Please listen to the 3 reference audios below completely
Label Ambiguity [Referencel
Excellent (100 80)
42
Reference
Good (80 60)
38
Reference 2
3. Reference 3
Fair (60 -40)
19
On average, felt the reference audio samples listened were
Poor (40 20)
How would you below sample generated System A?
Label Ambiguity [Systeml
Good (60 80)
45
Excellent (80 -100)
System
Fair (40 -60)
12
Poor (20 40)
6
Bad (0 20)
scale 5, how difficult was the Difficulty MUSHRA test?
Moderate
Easy
34
Difficult
12
Very easy
Very difficult.
3
Figure 9: Summary of participant responses from the post-evaluation survey conducted after the MUSHRA test.
23

---

Published in Transactions on Machine Learning Research (05/2025)
that strict reference dependence may not always be ideal. Furthermore, responses highlighted judgment am- biguity: while 56% prioritized voice quality over promunciation errors, 33% favored better pronunciation over voice quality, showing that raters face trade-offs that are not always resolved by aligning with the reference.
Despite the provided instructions, 56% of participants reported experiencing rating confusson at least a few times, confirming that label ambiguity exists the MUSHRA framework. Notably, 58% of raters did not rate the reference as 'Excellent," with 19% rating it Fair and 1% as Poor, further challenging the assumption that the reference always represents the highest quality standard. Additionally, 22% of participants found the MUSHRA test difficult very difficult, suggesting that cognitive load may impact rating consistency,
These findings support our argument that while MUSHRA remains a valuable evaluation tool, strict reference-malching and scoring assumptions may not always be ideal. The observed ambiguity, cognitive challenges, and frequent encounters with confusing rating situations, reinforce the need for alternative eval- uation sctups, such as MUSHRA-NMR, which relaxes reference dependence while maintaining the benefits of comparative evaluation. The survey results directly informed our analysis, ensuring that our interpreta- tions were grounded in participant responses rather than speculation. For additional details, the full survey findings are referenced in Figure g
A.6 Detailed Guidelines for MUSHRA-DG
We present the complete guidelines shown raters in Figure 16 We derive a formula that takes into account several factors: mild pronunciation mistakes (M,), severe pronunciation mistakes (SP), unnatural speedup slowdown (US), liveliness (L), voice quality (VQ), rhythm (R), digital artifacts (DA), sudden energy fluctuations (SFF), and word skips (WS). The MUSHRA score is calculated by averaging the perceptual measures and then penalizing for various mistakes and artifacts. Specifically, we penalize every word skip by deducting 25 points, every severe promunciation mistake by deducting 10 points, and every mild prommciation mistake by deducting points. Likewise, all other non-perceptual measures are penalized by points. The MUSHRA score (5 ) for a system is given by,
L L+VQ +
SM
min [M,, 15) x 5 min(SP, 10 US x 5- DAX5
WS 25 SEF x 5
We believe this formulation ensures systematic approach to scoring, accounting for both percept qualities and penalizing observable errors effectively.
A.7 Fault Isolation in English and Tamil
Similar to Figure we visualize the gramular ratings across the objective and perceptual dimensions for English in Figure and, Tamil in Figure 10
A.8 Limitations
Our study focuses on human evaluations for Hindi and Tamil, representing a major Indo-Aryan and Dravidian language, respectively. However, we did not extend our analysis to English, widely spoken and diverse language. This limitation is due to the scope of our current research and resource constraints. We also wish to state that while the demographics table summarizes the age and gender of participants, details for 21 out of the 492 participants are missing as raters opted out from providing this information. This constitutes a minor fraction and we believe does not significantly affect the analysis. While this work focuses on identifying two key issues - reference- matching bias and judgment ambiguity - several other biases may also exist TTS evaluations, which warrant further investigation. A more broader analysis could provide more comprehensive insights into the applicability and robustness of our proposed MUSHRA variants across diverse linguistic contexts and practical applications.
24

---

Published in Transactions on Machine Learning Research (05/2025)
ST2
120
VITS
100
ANC
95
100
FS2
90
REF
Error Rates (%)
80
85
80
Score
75
40
70
20
65
60
ST2 VITS ANC FS2 REF
Mild Pron.
Severe Pron.
Unnatural
Digital Artifact
Sudden Energy Variations
Word Skips
Speed Up
55
Down
Liveliness
Voice Quality
Rhythm
(a) Objective (Tamil)
(b) Perceptual (Tamil)
Figure 10: Visualization of the 6 objective and 3 perceptual dimensions of the MUSHRA-DG test.
120
ANC
100
FS2
REF
90
ST2
Error Rates (%)
VITS
80
Score
70
60
ANC # FS2 REF ST2 VITS
Mild Pron.
Unmatural Speed Up
Digital Artifect
Sudden Energy Variations
Word Skips
50
Pron.
DDown
Voice Quality
Rhythm
Liveliness
(a) Objective (English)
(b) Perceptual (English)
Figure 11: Visualization the objective and 3 perceptual dimensions of the MUSHRA-DG test.
A.9 Ethical Considerations
We prioritized ethical conduct throughout our research. All human listeners involved in the study provided informed consent before participating in the evaluation, recruited through professional data annotation agencies. These agencies verified participant language proficiency for task relevance. We established an education criterion of completing high school ensure participants' ability to accurately annotate audio content. Participants were compensated fairly for their time and expertise, following industry standards. They were also fully informed about the study nature, procedures, and their right to withdraw at any point without consequence. We ensured that study adhered to ethical guidelines by obtaining approval from the Institutional Ethics Committee, which reviewed our methodology and confirmed compliance with ethical standards.
We strived for inclusivity and bias mitigation. Participants came from diverse demographic backgrounds, and we recruited only native speakers capture the subtle linguistic and cultural nuances cach language. To minimize rater burden and bias in the new MUSHRA test variations, we prioritized sver-frienlliess and transparency in the design, providing clear guidelines.
25

---

Published in Transactions on Machine Learning Research (05/2025)
We release the evaluations dataset under CC-BY-4.0 license after careful consideration of privacy and ethical use. Identifiable information about the participants was anonymized to protect their privacy. We encourage the use of this dataset for advancing TTS evaluation metries, emphasizing that it should be used responsibly and ethically, adhering to principles of transparency and fairness. Finally, we acknowledge that our study locuses on Hindi and Tamil, and we recognize the importance of extending such evaluations other lan- guages, including English, to generalize our findings. Future research should continue to explore these ethical dimensions, ensuring that the development and evaluation of TTS systems are conducted with respect for the diversity and rights of all participants involved.
REF
FS2
ST2
ANC VITS
15
75 75
Rater ID
Figure 12: Visualization of the MUSHRA score distributions per rater across three systems- FS2, ST2, and VITS, along with the reference (REF) and anchor (ANC) for Tamil. Each boxplot represents ratings (0-100) across all test ulterances for a system by one rater. The substantial heights of some boxplots indicate significant variance in the scores of that rater. The variation in boxplot means across raters suggests a high level of inter-rater variance
26

---

Published in Transactions on Machine Learning Research (05/2025)
REF
FS2
STZ
VITS
ANC
Rater ID
Figure 13: Visualization of the MUSHRA score distributions per rater across three systems- FS2, ST2, and VITS, along with the reference (REF) and anchor (ANC) for English. Each boxplot represents ratings (0-100) across all test utterances for system by one rater. The substantial heights of some boxplots indicate significant. variance in the scores of that rater. The variation in boxplot means across raters suggests a high level of inter-rater variance
27

---

Published in Transactions on Machine Learning Research (05/2025)
REF
FS2
ST2
ANC VITS
Utterance ID
Figure 14: Visualization of the MUSHRA score distributions utterance across three systems- FS2, ST2, and VITS, along with the reference (REF) and anchor (ANC) for Hindi. The X-axis represents each of the 100 utterances. Each boxplot represents ratings (0-100) across raters for a system for given utterance. The substantial heights of some boxplots indicate siguificant variance in the scores given by different raters for single utterance.
REF
FS2
ST2
VITS
ANC
Utterance ID
Figure 15: Visualization of the MUSHRA score distributions per utterance across three systems FS2, ST2, and VITS, along with the reference (REF) and anchor (ANC) for Tamil.
28

---

Published in Transactions on Machine Learning Research (05/2025)
Criteria
To Mark
Mark number of mild pronunciation errors. If no errors, mark 0 here.
Mild Pronunciation
mild pronunciation error where any character, for example an is; palf-pronounce and not fully clear.
Mark number of severe promunciation errors. If errors, mark here.
Severe Pronunciation
severe pronunciation error is where any character such as "r" or is skipped/ mis- pronouucdd..
Unnatural Pauses, speedup slowdown
Mark number of places where there was unnatural sunses//ppeedu//llowdddde in audio.
Mark 100 human-like
Mark semi-expressive/ semi-enthusiasticc/ semi:iisiiiittits-iiiiiiiee semi-lively
Liveliness
Mark 70 if robotic/monotonic
You may adjust scores in-between based on opinion.
Mark 100 perfect human like voice quality
Mark 85 if slight digitalness voice
Voice Quality/Clariinn
Mark 60-70 high
robotic voice
You may adjust scores in ne-between based on opinion
Mark 100 human-like
Mark 85 if slightly fast/slow
Rhythm
Mark 60 if too fast/slow
You may adjust scores in-between based on opinion
Mark number digital artifacts heard audio. artifacts, mark here.
Digital Artifacts
digital artifact could be a "click" sound, "pop" sound, digital vibration in pauses, etc.
Mark number of regions in which the energy, rhythm, pitch of the speech suddenly irregularly change.
Sudden
Energy
Fluctuations
Mark here if no such changes noticed.
Word Skips
Mark the number of words the model has skipped. If no skips, mark 0 here.
Figure 16: Guidelines presented to raters across multiple evaluation criteria in the MUSHRA-DG Test.
29


Reviews:
[{"review_id": "RcZsHhuvmq", "reviewer": "Reviewer_e65S", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Key strengths:\n- this is a large scale study conducted across 2 languages from 2 different language families and a number of modern TTS architectures\n- it explores several variants of the standard MUSHRA test\n- it measure several aspects of MUSHRA (reliability, sensitivity, anchor)\n\nKey weaknesses:\n- lack of a dedicated section providing a good overview of MUSHRA, work done on improving this listening protocol and comparison to other kinds of listening test; some of this information is present in the paper but it is scattered making for a very confusing presentation\n- I struggle to understand why reference matching is presented as a shortcoming of MUSHRA. This is exactly what MUSHRA appears to be measuring. Changing that aspect leads to a different test that combines features of multiple listening tests\n- lack of clarity on the number of participants and ratings provided by each participant; one part of the paper says that 226,000 rating were collected; another part says that each participant provided 100 ratings; this suggests that there were 2260 participants; however, table 1 (left) says that the total number of ratings was 246,000 and the total number of participants is 471 and table 1 (right) suggests yet another number of participants. \n- subjective statements regarding participant on-boarding: \"were clearly explained\", \"any doubts were addressed\"\n- no exit interview with participants were performed to get insights into their decision making, instead such insights were speculated\n- naturalness, quality and other ambiguous concepts not clearly defined\n- imprecise statements about Table 1: only some boxes reflect high variance, a substantial number of boxes has low variance\n- Anchor-Y TTS model was never introduced\n- speculative statements regarding true opinions based on 1.3% ratings revised; the lack of incentive to revise puts that assertion to question; only exit interview could have corroborated that to some extent\n- informal language like prowess\n- error rates higher than 100% in Figure 5 need to be explained", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This article conducts a large-scale MUSHRA listening test to gain an understanding of its features and limitations and examines two variants that better match authors expectations of an ideal listening test.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the following critical required changes (copied from above with small changes):\n\n- lack of a dedicated section providing a good overview of MUSHRA, work done on improving this listening protocol and comparison to other kinds of listening test; some of this information is present in the paper but it is scattered making for a very confusing presentation\n- I struggle to understand why reference matching is presented as a shortcoming of MUSHRA. This is exactly what MUSHRA appears to be measuring. Changing that aspect leads to a different test that combines features of multiple listening tests\n- lack of clarity on the number of participants and ratings provided by each participant; one part of the paper says that 226,000 rating were collected; another part says that each participant provided 100 ratings; this suggests that there were 2260 participants; however, table 1 (left) says that the total number of ratings was 246,000 and the total number of participants is 471 and table 1 (right) suggests yet another number of participants. \n- subjective statements regarding participant on-boarding: \"were clearly explained\", \"any doubts were addressed\"\n- clarify why no exit interview with participants were performed to get insights into their decision making, why instead such insights were speculated\n- naturalness, quality and other ambiguous concepts not clearly defined\n- imprecise statements about Table 1: only some boxes reflect high variance, a substantial number of boxes has low variance\n- Anchor-Y TTS model was never introduced\n- address speculative statements regarding true opinions based on 1.3% ratings revised; the lack of incentive to revise puts that assertion to question; only exit interview could have corroborated that to some extent\n- informal language like prowess\n- error rates higher than 100% in Figure 5 need to be explained", "broader_impact_concerns": "I can think of none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CTaxA3SVQi", "reviewer": "Reviewer_1BMu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. **Comprehensive Analysis of MUSHRA:**\n   - The submission offers an in-depth assessment of the MUSHRA test, addressing its limitations with empirical evidence derived from an extensive dataset (246,000 human ratings).\n   - The authors critically evaluate key challenges such as reference-matching bias and judgment ambiguity, which are highly relevant to modern TTS evaluation.\n\n2. **Innovative Variants Proposed:**\n   - The introduction of **MUSHRA-NMR** and **MUSHRA-DG** is novel and well-motivated:\n     - **MUSHRA-NMR** mitigates biases by removing explicit references.\n     - **MUSHRA-DG** introduces fine-grained evaluation criteria, providing a more systematic and diagnostic approach to rating.\n\n3. **Robust Dataset Contribution:**\n   - The **MANGO dataset** significantly contributes to the community, representing diverse linguistic contexts (Hindi and Tamil) and offering a resource for benchmarking and studying human evaluations in TTS.\n\n4. **Empirical Validation:**\n   - The submission provides thorough empirical support for the proposed methods, demonstrating improved reliability, reduced variance, and enhanced diagnostic insights through fault isolation.\n   - Results are well-illustrated using statistical correlations, variance analyses, and comparative studies with CMOS scores.\n\n5. **Practical Relevance:**\n   - The work addresses challenges faced by practitioners evaluating state-of-the-art TTS systems, aligning evaluation metrics with the realities of modern speech synthesis quality.\n\n6. **Clear and Structured Presentation:**\n   - The paper is well-structured, with clear motivation, methodology, and results. The inclusion of detailed guidelines for raters and a formula-based scoring approach in **MUSHRA-DG** adds practical value.\n\n### Weaknesses\n\n1. **Limited Language Scope:**\n   - The focus on Hindi and Tamil is valuable but narrows the generalizability of the findings. Including results for widely-used languages like English could strengthen the study’s global applicability.\n\n2. **Time Complexity of MUSHRA-DG:**\n   - While the added granularity in **MUSHRA-DG** is a strength, the increased time required for evaluations may pose scalability concerns, particularly for large-scale studies.\n\n3. **Subjectivity in Scoring Formula:**\n   - The weights and penalties in the **MUSHRA-DG** scoring formula are subject to interpretation and may not align perfectly with all TTS use cases. More justification or adaptability of these parameters could improve its applicability.\n\n4. **Anchor Design Limitations:**\n   - Choosing anchors (Anchor-X and Anchor-Y) introduces variability that might affect system scores. The study could benefit from exploring alternative anchor design strategies or justifying their current selection in more detail.\n\n5. **Potential Overhead in Combined Variant:**\n   - The **MUSHRA-DG-NMR** variant combines the strengths of both approaches but might also inherit their weaknesses, such as increased time complexity and potential subjectivity.\n\n6. **Bias Considerations:**\n   - While the work addresses reference-matching bias and judgment ambiguity, other potential biases (e.g., cultural or gender-related influences on rating) remain unexplored.\n\n### Suggestions for Improvement\n\n1. Extend the study to include a broader range of languages, particularly high-resource languages like English, to validate findings across diverse linguistic contexts.\n2. Provide a deeper discussion on the scalability of **MUSHRA-DG**, particularly in large-scale evaluation setups.\n3. To enhance flexibility, Consider offering adaptive or task-specific scoring formulas in **MUSHRA-DG**.\n4. Explore and evaluate additional anchor design methodologies to ensure evaluation robustness and consistency.\n5. Investigate other potential biases in human evaluations, such as demographic or cultural factors, to further enhance fairness and reliability.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission introduces significant advancements in evaluating Text-to-Speech (TTS) systems through a critical assessment and refinement of the MUSHRA (Multiple Stimuli with Hidden Reference and Anchor) test, a cornerstone in TTS evaluation. Below is a summary of the primary contributions:\n\n#### 1. **Comprehensive Evaluation of MUSHRA**\n   - Conducted a detailed analysis of the MUSHRA test using 246,000 human ratings across Hindi and Tamil, revealing critical shortcomings in the current setup:\n     - **Reference-Matching Bias:** Listeners tend to rate systems lower if they deviate from the explicitly mentioned reference, even when surpassing the reference in quality.\n     - **Judgment Ambiguity:** Ratings vary due to the lack of fine-grained evaluation criteria and subjective interpretations.\n\n#### 2. **Proposed Variants of MUSHRA**\n   - Introduced two refined methodologies to address the limitations:\n     - **MUSHRA-NMR (No Mentioned Reference):** Eliminates explicit mention of the human reference to mitigate biases, enabling a fairer comparison of systems.\n     - **MUSHRA-DG (Detailed Guidelines):** Provides systematic guidelines and a scoring formula to reduce ambiguity, introducing fine-grained criteria like pronunciation errors, rhythm, and liveliness.\n\n#### 3. **Creation of the MANGO Dataset**\n   - Released a novel dataset, **MANGO**, comprising 246,000 ratings for TTS evaluations, representing a diverse set of human preferences and linguistic nuances for Indian languages. This dataset enables further exploration of TTS evaluation methods and automatic metrics.\n\n#### 4. **Empirical Validation and Insights**\n   - Demonstrated that the proposed variants enhance reliability and granularity:\n     - **MUSHRA-NMR** reduces the reference bias, yielding more accurate rankings.\n     - **MUSHRA-DG** ensures lower variance in ratings and offers fault isolation capabilities.\n   - Highlighted the sensitivity of MUSHRA to factors like listener count, utterance diversity, and anchor selection.\n\n#### 5. **Actionable Framework for Future TTS Evaluations**\n   - Proposed a combined variant, **MUSHRA-DG-NMR**, which synthesizes the strengths of both variants to provide a robust evaluation framework, ensuring fairness, reliability, and diagnostic utility.\n\n#### 6. **Broad Impact**\n   - Established the importance of adapting evaluation methodologies to account for advancements in TTS systems that can surpass human benchmarks, setting a precedent for fair and comprehensive assessments in AI and speech synthesis research.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Requested Changes\n\n#### **Critical Changes**\n1. **Generalizability Across Languages:**\n   - Expand the analysis to include additional languages, particularly high-resource languages such as English, to validate the findings across diverse linguistic contexts. This is essential for ensuring the broader applicability of the proposed methodologies.\n\n2. **Anchor Design Justification:**\n   - Provide a more detailed rationale for the choice of anchors (Anchor-X and Anchor-Y) and discuss their potential impact on system scores. Additionally, consider exploring alternative anchor design strategies to mitigate any unintended biases.\n\n3. **Scalability Discussion for MUSHRA-DG:**\n   - Include a discussion on the time complexity of **MUSHRA-DG** and **MUSHRA-DG-NMR** in large-scale evaluation settings. Offer potential solutions to mitigate the increased evaluation time, such as parallelization or adaptive guidelines.\n\n4. **Subjectivity in Scoring Formula:**\n   - Justify the specific weights and penalties used in the **MUSHRA-DG** scoring formula or provide a framework for task-specific adaptations to improve its versatility.\n\n#### **Suggested Enhancements**\n1. **Bias Analysis:**\n   - Investigate additional potential biases (e.g., cultural or demographic influences) in human evaluations to strengthen the study's fairness and reliability claims.\n\n2. **Visualization Improvements:**\n   - Enhance the clarity of the visualizations (e.g., box plots and fault isolation charts) by adding more descriptive legends and captions to improve interpretability for readers unfamiliar with the dataset.\n\n3. **Dataset Accessibility:**\n   - Provide more details about the **MANGO dataset**, including example ratings or summaries of specific entries, to illustrate its value for researchers better.\n\n4. **Practical Applicability:**\n   - Include case studies or examples of how the proposed variants (e.g., **MUSHRA-DG** and **MUSHRA-NMR**) have successfully applied in real-world TTS evaluations.\n\n5. **Comparative Analysis with Other Tests:**\n   - Deepen the comparison of the proposed variants with other evaluation methodologies, such as MOS and CMOS, to highlight relative strengths and limitations in various scenarios.\n\n6. **Future Research Directions:**\n   - Add a dedicated section on potential future research, such as adapting **MUSHRA-DG** for real-time applications or incorporating automatic metrics to complement human evaluations.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8prlVqLLV3", "reviewer": "Reviewer_y228", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper presents thorough experiments and analysis that contribute to a deeper understanding of the MUSHRA test. The two limitations highlighted by the authors appear to be relevant and well-founded.\n- The two solutions proposed by the authors seem to be effective. By removing the explicit reference, the gap between the best-performing TTS system and the hidden reference has been reduced. Besides, the use of more detailed scoresheets can help guide the evaluators in their assessments and decrease the variance in the evaluation scores. These detailed scoresheets also offer practitioners a better understanding of TTS system performance from different aspects (pronunciation mistakes, prosody, liveness, etc.)\n\nWeaknesses:\n- As the authors mentioned in Section 6.2, the MUSHRA test with detailed scoresheet is more time-consuming. Moreover, the formula used to compute the final score based on evaluations of different aspects (specified in appendix A.4) appears somewhat subjective and requires more rigorous justification.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the effectiveness and robustness of the MUSHRA subjective evaluation test for assessing Text-to-Speech (TTS) systems.\n\nSpecifically, the authors conduct experiments using two Indian language datasets: Hindi and Tamil, and evaluate three TTS systems: FastSpeech2, VITS and StyleTTS2. The MUSHRA tests are conducted with 492 local language listeners. Through these experimental investigations, the authors identify two key limitations of the MUSHRA tests: 1) evaluators tend to align their ratings with the explicit reference provided, and 2) using a single score for evaluation introduces ambiguity in assessing audio quality.\n\nTo address these two limitations, this paper propose two variants of MUSHRA: 1) MUSHRA with no explicitly mentioned reference, and 2) MUSHRA with detailed guidelines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The authors could also report objective audio quality evaluation metrics, such as SDR, PESQ, and ESTOI, for the tested TTS systems as a point of reference.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work examines and enhances the MUSHRA test—a perceptual audio quality metric originally developed for evaluating audio compression. Through a series of experiments, the authors highlight two major limitations of the standard MUSHRA protocol: (1) evaluators often bias their ratings toward the explicitly provided reference, and (2) the use of a single rating score can lead to ambiguity in quality assessment. To mitigate these issues, the authors introduce two alternative MUSHRA configurations: one without an explicit reference and another with more detailed rating guidelines. Additionally, they present the MANGO dataset, designed for TTS evaluation. 

All reviewers acknowledged the contribution and strengths of the paper. The authors have also thoroughly addressed the concerns raised during the reviewers during the authors-reviewers discussion period. Therefore, I recommend acceptance. 

I also encourage the authors to incorporate all clarifications and additional results provided during the rebuttal into the final version, as these would significantly enhance the manuscript.
Decision: Accept (poster)