Research task (AI & ML):
How do you evaluate modern (up to 2025) Text-to-speech (TTS) models, including advancements in prosody, emotion, and multilingual capabilities. Explore emerging tasks such as voice cloning, zero-shot synthesis, and cross-lingual voice transfer.

Evaluation rubric:
criterion: The response ensures all acronyms are expanded on it during their first appearance and provides clear definitions for all specialized terms and mathematical symbols upon first use, accompanied by explanatory prose or examples (e.g., MOS as Mean Opinion Score, F0 as Fundamental Frequency, EER as Equal Error Rate, MCD as Mel-Cepstral Distortion).
weight: 4
axis: Communication Quality

criterion: The response lists at least three publicly available datasets used for TTS evaluation (e.g., (1) LJSpeech, (2) Multilingual LibriSpeech, (3) VCTK (Voice Cloning Toolkit), (4) Libri-TTS/ LibriTTS-R).
weight: 4
axis: Implicit Criteria

criterion: The response is logically structured, progressing from foundational metrics to advanced capabilities and emergent tasks (e.g., starting with foundational concepts such as Mean Opinion Score (MOS), Comparative Mean Opinion Score (CMOS), then exploring the frontiers of evaluation of prosody and emotion, and finally the advanced applications such as voice cloning, cross-lingual transfer, and zero-shot synthesis).
weight: 4
axis: Communication Quality

criterion: The response explains the "one-to-many" mapping problem for prosody (i.e., the same sentence can be spoken with many valid variations of prosody).
weight: 3
axis: Implicit Criteria

criterion: The response explains the "subjectivity bottleneck," where human listening tests are the most accurate but are also slow, expensive, and hard to repeat consistently.
weight: 3
axis: Implicit Criteria

criterion: The response discusses how to check for "emotional consistency" after editing speech for emotion, mentioning specific tools like EmoCorrector.
weight: 3
axis: Explicit Criteria

criterion: The response claims BLEU(Bilingual Evaluation Understudy) or ROUGE(Recall-Oriented Understudy for Gisting Evaluation) are standard metrics for evaluating the acoustic quality of TTS output.
weight: -4
axis: Implicit Criteria

criterion: The response explains why, regarding intelligibility and naturalness, one can be perfect and the other one fails (e.g., Naturalness is much harder than intelligibility because a perfectly intelligible audio, that has a low word error rate, can sound entirely robotic and unnatural. There is more subjectivity in evaluating what is natural than in evaluating what is intelligible).
weight: 3
axis: Implicit Criteria

criterion: The response explains, for multilingual TTS, the methodological "chicken-and-egg" problem where (Automatic Speech Recognition) ASR-based metrics like WER (Word Error Rate) are dependent on the quality of the ASR model itself.
weight: 3
axis: Explicit Criteria

criterion: The response uses clear tables to compare different options, such as models or evaluation metrics, showing their main pros and cons.
weight: 2
axis: Communication Quality

criterion: The response describes a special metric for evaluating code-switching (mixing languages) (e.g., PIER (Point-of-Interest Error Rate), a variant of WER that focuses only on specific words of interest, (2) T-index, that leverages machine translation systems to capture properties of code-switched words in relation to the participating language pair, (3) Intonation Units (IUs) that are prosodic speech segments, (4)  Fidelity to the Original Audio, Accuracy, and Latency (FAL) which aims to overcome the limitations of traditional metrics used to assess ASR (Automatic Speech Recognition) systems).
weight: 3
axis: Implicit Criteria

criterion: The response explains tests like CMOS (Comparative Mean Opinion Score) or MUSHRA (Multiple Stimuli with Hidden Reference and Anchor) that are better than a basic MOS (Mean Opinion Score) test for comparing very good TTS systems. (e.g., Comparative Mean Opinion Score (CMOS) builds on Mean Opinion Score (MOS) by letting the listener judge on a pair of samples side-by-side. This is important because it is hard to absolutely judge the Mean Opinion Score for systems that already have very high scores.)
weight: 2
axis: Implicit Criteria

criterion: The response claims that the MCD (Mel-Cepstral Distortion) metric is good for measuring how natural a voice sounds.
weight: -3
axis: Explicit Criteria

criterion: The response explains, for voice cloning, the trade-off between sounding like the target person (similarity) and sounding good (quality).
weight: 3
axis: Explicit Criteria

criterion: The response explains modern metrics like TTSDS (Text-to-Speech Distribution Score) that work better by comparing groups of audio samples instead of just one-to-one.
weight: 3
axis: Implicit Criteria

criterion: The response describes using AI models as "judges" (like in EmergentTTS-Eval) because they give scores and also explain their reasoning like humans.
weight: 3
axis: Implicit Criteria

criterion: The response explains that the main challenge of using a voice in another language is removing the original speaker's accent.
weight: 2
axis: Implicit Criteria

criterion: The response explains that evaluation, for zero-shot TTS, means checking quality and voice similarity using only a few seconds of audio from a new speaker.
weight: 5
axis: Explicit Criteria

criterion: The response explains in detail the value of "living benchmarks" that update regularly to keep tests fair and prevent models from overfitting to them.
weight: 3
axis: Implicit Criteria

criterion: The response presents automated benchmarks as perfect, ignoring their known flaws, like bias or the inability to catch every error.
weight: -3
axis: Implicit Criteria

criterion: The response points out that evaluating long audio (like audiobooks) is a major unsolved problem.
weight: 2
axis: Implicit Criteria

criterion: The response discusses how to evaluate "controllability", how well the TTS model follows specific commands for style or emotion.
weight: 2
axis: Implicit Criteria

criterion: The response explains that there must be an evaluation that is specific to the application (e.g., a screen reader needs clarity).
weight: 3
axis: Synthesis of Information

criterion: The response suggests a smart hybrid evaluation plan, such as using computer tests daily, bigger automated tests for milestones, and human listeners for final checks.
weight: 3
axis: Synthesis of Information

criterion: The response discusses practical issues, like evaluating a model's speed, cost, and memory usage, especially for devices like phones.
weight: 3
axis: Synthesis of Information

criterion: The response notes that there is currently no good, standard metric for measuring a "foreign accent" in TTS.
weight: 2
axis: Implicit Criteria

criterion: The response explains the need to evaluate the training data itself for things like sound diversity before building a model.
weight: 2
axis: Implicit Criteria

criterion: The response explains how to evaluate models for algorithmic bias, checking if they work poorly for certain accents or groups of people.
weight: 3
axis: Synthesis of Information

criterion: The response discusses evaluating the long-term user experience, such as listening fatigue or how much a user trusts the voice.
weight: 3
axis: Synthesis of Information

criterion: The response explains the need to check for "semantic faithfulness", making sure the model reads numbers and facts correctly, even if it sounds natural.
weight: 3
axis: Implicit Criteria

criterion: The response explains how to evaluate a voice's "persona" to see if it fits a specific brand.
weight: 2
axis: Implicit Criteria

criterion: The response treats TTS evaluation as only a technical problem, ignoring ethics and societal harm (e.g., Voice cloning used for deepfake audio, fraudulent activities such as impersonation and defamation).
weight: -3
axis: Implicit Criteria

criterion: The response discusses how to measure speaker identity preservation across languages (e.g., Speaker Encoder Cosine Similarity (SECS), Similarity Mean Opinion Score (SMOS)).
weight: 3
axis: Implicit Criteria

criterion: The response discusses how to quantify accent transfer versus removal (e.g., Accent Classifer, or SpeechCodeVAE model that decouples and reconstructs speech signals through three disentangled representations: speech content tokens, speaker timbre embeddings, and prosodic features, Zero-shot voice transfer).
weight: 3
axis: Implicit Criteria