Preferences of a Voice-First Nation: Large-Scale Pairwise Evaluation and Preference Analysis for TTS in Indian Languages
Authors: name: Srija Anand, name: Ashwin Sankar, name: Ishvinder Sethi, name: Aaditya Pareek, name: Kartik Rajput, name: Gaurav Yadav, name: Nikhil Narasimhan, name: Adish Pandya, name: Deepon Halder, name: Mohammed Safi Ur Rahman Khan, name: Praveen S V, name: Shobhit Banga, name: Mitesh M Khapra
Organization: AI4Bharat

Crowdsourced pairwise evaluation has emerged as a scalable approach for assessing foundation models. However, applying it to Text to Speech(TTS) introduces high variance due to linguistic diversity and multidimensional nature of speech perception. We present a controlled multidimensional pairwise evaluation framework for multilingual TTS that combines linguistic control with perceptually grounded annotation. Using 5K+ native and code-mixed sentences across 10 Indic languages, we evaluate 7 state-of-the-art TTS systems and collect over 120K pairwise comparisons from over 1900 native raters. In addition to overall preference, raters provide judgments across 6 perceptual dimensions: intelligibility, expressiveness, voice quality, liveliness, noise, and hallucinations. Using Bradley-Terry modeling, we construct a multilingual leaderboard, interpret human preference using SHAP analysis and analyze leaderboard reliability alongside model strengths and trade-offs across perceptual dimensions.