Towards Building Large Scale Datasets and State-of-the-Art Automatic Speech Translation Systems for 13 Indian Languages
Authors: Ashwin Sankar, Sparsh Jain, Nikhil Narasimhan, Devilal Choudhary, Dhairya Suman, Mohammed Safi Ur Rahman Khan, Anoop Kunchukuttan, Mitesh M Khapra, Raj Dabre
Accepted at: acl25

Towards Building Large Scale Datasets and State-of-the-Art Automatic Speech Translation Systems for 14 Indian Languages

Ashwin Sankar$^{1,2*}$ Sparsh Jain$^1$ Nikhil Narasimhan$^1$ Devilal Choudhary$^{4\dagger}$
Dhairya Suman$^{3\ddagger}$ Mohammed Safi Ur Rahman Khan$^{1,2}$
Anoop Kunchukuttan$^{1,5}$ Mitesh M Khapra$^{1,2}$ Raj Dabre$^{1,2,6,7\ddagger\&\S}$
$^1$Nilekani Centre at AI4Bharat $^2$Indian Institute of Technology, Madras
$^3$Indian Institute of Technology, Delhi $^4$Delhi Technological University
$^5$Microsoft $^7$Indian Institute of Technology, Bombay
$huggingface.co/BhasaAnuvaad$ $github.com/BhasaAnuvaad$

Abstract
Speech translation for Indian languages remains a challenging task due to the scarcity of large-scale, publicly available datasets that capture the linguistic diversity and domain coverage essential for real-world applications. Existing datasets cover a fraction of Indian languages and lack the breadth needed to train robust models that generalize beyond curated benchmarks. To bridge this gap, we introduce BHASAANUVAAD, the largest speech translation dataset for Indian languages, spanning over 44 thousand hours of audio and 17 million aligned text segments across 14 Indian languages and English. Our dataset is built through a threefold methodology: (a) aggregating high-quality existing sources, (b) large-scale web crawling to ensure linguistic and domain diversity, and (c) creating synthetic data to model real-world speech disfluencies. Leveraging BHASAANUVAAD, we train INDIC-SEAMLESS, a state-of-the-art speech translation model for Indian languages that performs better than existing models. Our experiments demonstrate improvements in the translation quality, setting a new standard for Indian language speech translation. We will release all the code, data and model weights in the open-source, with permissive licenses to promote accessibility and collaboration.

1 Introduction
Automatic Speech Translation (AST) has become crucial to break language barriers and enable communication across languages and cultures. Traditionally, speech translation systems have relied on cascaded architectures, where Automatic Speech Recognition (ASR) is followed by Machine Translation (MT). However, recent advances have led to more integrated end-to-end (E2E) models (Babu et al., 2021; Pratap et al., 2024; Radford et al., 2023; Communication et al., 2023) that directly translate speech from one language into text in another. Additionally, Audio-LLM-based systems have emerged (Wu et al., 2023; Chu et al., 2023; Fathullah et al., 2023; Gaido et al., 2024) further advancing the field. Despite these advancements, progress has been concentrated on English and other European languages, while low- and mid-resource languages, including Indian languages, have remained largely underrepresented. The primary challenge is the lack of training data, which leads to suboptimal models.

India’s linguistic diversity, with 22 officially recognized languages and numerous dialects, presents significant challenges for speech translation, including diverse phonetic systems (Mujadia and Sharma, 2023), frequent code-switching (Shankar et al., 2024), and syntactic variations. However, the primary bottleneck remains the scarcity of large-scale, high-quality datasets, aggravated by the limited web presence of Indian languages. To bridge this gap, we introduce BHASAANUVAAD, a large-scale speech translation dataset comprising of over 44K hours of speech translation data covering 14 Indian languages alongside English, making it the most comprehensive speech translation resource for the region to date. The included languages - Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Sindhi, Tamil, Telugu, and Urdu - span both medium and low resource languages. Our dataset construction follows a three-pronged approach to ensure high-quality and diverse data collection: (1) Aggregating High-Quality Existing Sources: We incorporate publicly available speech-text pairs from prior works, refining them with improved alignment techniques to enhance usability for speech translation tasks. (2) Large-Scale Web Crawling: To expand linguistic and domain coverage, we employ an automated web-mining pipeline that
we call ST-ALIGNER (Figure 1) to extract speech-text pairs from diverse online sources, including government portals, educational materials, spiritual discourses, and multilingual podcasts. This enables us to capture naturally occurring speech across varied contexts, improving real-world applicability. (3) Synthetic Data Generation: To address gaps in speech diversity, we leverage the INDICVOICES ASR dataset and use LLAMA-3.1-405B-INSTRUCT for synthetic translations into 13 languages, while generating English speech using a TTS model (Sankar et al., 2025).

We also build BHASAANUVAAD-TEST by randomly sampling 20 minutes of data for each language pair from BHASAANUVAAD. Additionally, we introduce INDICSEAMLESS, a finetuned SEAMLESSM4T-V2-LARGE model on BHASAANUVAAD that surpasses existing AST setups - both cascaded and end-to-end - on established benchmark. We observe an overall average improvement of 10 chrft++ scores over previous approaches on BHASAANUVAAD-TEST demonstrating the efficacy of BHASAANUVAAD.

In summary, our contributions are: (i) the release of BHASAANUVAAD, the largest Indic-language speech translation corpus, comprising over 44 thousand hours of data across 14 Indian languages and English, (ii) a speech-translation data mining pipeline, ST-ALIGNER (iii) an evaluation of popular AST baselines on Conneau et al. (2022) and BHASAANUVAAD-TEST, a test split from BHASAANUVAAD , and (iv) the release of state-of-the-art speech translation model for 14 Indian languages. All code and datasets developed as part of this work will be made publicly available to support future research in AST for Indic languages.

2 Related Work

Cascaded and End-to-End AST. Cascaded Automatic Speech Translation (AST) has been explored extensively in early research (Di Gangi et al., 2019; Cheng et al., 2019; Bahar et al., 2020), connecting Automatic Speech Recognition (ASR) and Machine Translation (MT) systems in sequence. These systems suffer from cascaded error propagation (Sperber and Paulik, 2020; Beck et al.,
2019), loss of prosodic information (Bentivogli et al., 2021), and increased computational complexity (Lee et al., 2022). In contrast, end-to-end (E2E) models can avoid this (Weiss et al., 2017; Inaguma et al., 2021), but suffer from lack of good quality data (Sethiya and Maurya, 2023). With the rise of modern Large Language Models (LLMs), recent works have started exploring the feasibility of using Audio-based LLMs (or Speech LLMs), which combine Speech Foundation models (SFMs) and LLMs (Fathullah et al., 2023; Wu et al., 2023; Huang et al., 2023; Gaido et al., 2024).

Datasets for AST. The collection of speech-translation datasets for training end-to-end models has proven to be a persistent challenge. While efforts like SpeechMatrix (Duquenne et al., 2022) and SeamlessAlign (Communication et al., 2023) attempt to mine data from the web, the resulting datasets are often noisy. Most research has focused on curating data using comparable corpora from domains like Education (Duríšková et al., 2024; Song et al., 2019), TED talks (Salesky et al., 2021), Parliament Debates (Koehn, 2005), among others. Additionally, with the rise of generative models, recent works have also started exploring the generation of large amounts of synthetic speech translation data in a flexible and cost-effective manner (Ye et al., 2023), (Bamfo Odoom et al., 2024). Although the use of synthetic data has been widely studied in text-based machine translation (Sennrich et al., 2016; Gala et al., 2023), its application in Speech translation, particularly for low-resource languages like Indic languages, remains underexplored, a gap which we fill.

Indian Languages AST. Unlike English, the shortage of parallel datasets has hindered the progress in Indian Languages (Mujadia and Sharma, 2023; Mhaskar et al., 2023). Multilingual speech translation models like mSLAM (Bapna et al., 2022), WHISPER (Radford et al., 2023), SEAMLESS (Communication et al., 2023) have made strides in addressing this gap, but their effectiveness in low-resource and real-world spontaneous speech settings remains largely unexplored. Furthermore, speech-based benchmarks for Indian languages are still underdeveloped (Sethiya and Maurya, 2023), with FLEURS (Conneau et al., 2022) being the primary option. In this paper, we push the boundaries of AST for Indian languages with the help of the large-scale dataset we construct.

3 BHASAANUVAAD

BHASAANUVAAD is the largest spoken translation dataset for any Indian language, totaling approximately 44,400 hours of speech and text data. Its construction involved a multi-step approach, as illustrated in Figure 1, combining (i) Aggregating existing datasets, (ii) Mining parallel speech and text data from comparable sources, and (iii) Synthetically generated speech/text parallel data. A language wise composition of this dataset is illustrated in Figure 2. A detailed breakdown of language and dataset-wise statistics is provided in Appendix A in Tables A1 & A2.

3.1 Aggregating Existing Datasets

We begin by aggregating several existing datasets in Indian Languages, which form the foundation of BHASAANUVAAD. These include:

Indic-TEDST: The Indic-TEDST (Sethiya et al., 2024) dataset includes the audio, transcripts, and translations of popular TED Talks in nine widely spoken Indian languages - Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Punjabi, Tamil, and Telugu - totaling approximately 203 hours of English-to-Indic data.

FLEURS: The FLEURS-train dataset (Conneau et al., 2022) is a multilingual speech corpus covering over 100 languages for training automatic speech recognition (ASR) and speech translation (AST) models. FLEURS was previously the largest source of data, covering 14 Indian languages, making it a valuable resource.

CVSS: The CVSS (Jia et al., 2022) corpus is a multilingual speech-to-speech translation dataset derived from CoVoST 2 (Wang et al., 2020), which is sampled from the Common Voice dataset. Spanning 21 languages, it includes only one Indian language, Tamil, with approximately 3 hours of data.

Khan Academy Corpus: The Khan Academy corpus (Duríšková et al., 2024) provides data for the education domain in 29 languages. It includes 3 Indian languages, with 105 hours of audio.

SeamlessAlign: We recreate the SeamlessAlign data introduced by Communication et al. (2023) to get around 7500 hours of data for 5 Indic languages.

3.2 Mining from comparable sources

We explore the extraction of parallel speech-text data from various online sources containing high-quality content across multiple domains. While
many sources provide transcripts in different languages, these are often misaligned with the corresponding audio. To address this issue, we develop an automated pipeline that aligns transcripts with their corresponding source audios, to produce high-quality parallel speech-text data.

3.2.1 ST-Aligner
Our pipeline consists of four stages: normalization, punctuation restoration, audio-transcript alignment, and parallel text alignment.

1. Normalization: To ensure compatibility, all audio files are first converted to a mono channel and resampled at 16 kHz, as ASR models operate on 16 kHz audio. The corresponding transcripts are then cleaned by removing commonly found noise like extraneous symbols, extra spaces, and line breaks. We also replace the terminal punctuation marks with a non-standard character to aid sentence segmentation during alignment.

2. Punctuation Restoration: Many subtitles and transcripts lack proper punctuation, which can hinder accurate alignment. We conducted a small experiment with indic-punct (Gupta et al., 2022), a punctuation restoration model, and found that it did not perform well in spoken contexts. Given this limitation, we opt for the LLAMA-3.1-405B-INSTRUCT model, which demonstrated superior performance in restoring punctuation and effectively structuring the text for alignment (Figure A1).

3. Audio-Transcript Alignment: The cleaned transcripts and audio are processed using the Nemo Forced Aligner (NFA)\footnote{https://github.com/NVIDIA/NeMo/tree/main/tools/nemo_forced_aligner}, which generates token-level, word-level, and segment-level timestamps based on CTC-based ASR models. Given the linguistic diversity of our dataset, we utilize INDIC-COMFORMER ASR models for Indian languages and Nvidia FastConformer\footnote{https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_en_fastconformer_hybrid_large_pc} for English, ensuring that the alignment is optimized for the phonetic and acoustic characteristics of each language.

Long-form audio presents additional challenges, as ASR models and forced aligners struggle with excessive sequence lengths, often resulting in degraded alignment accuracy. To address this, we employ Silero Voice Activity Detection (Silero VAD) (Team, 2024), a lightweight yet highly effective neural model that detects speech boundaries and segments long recordings into smaller, more manageable chunks. This pre-segmentation step not only improves computational efficiency but also enhances alignment precision by reducing errors that arise from excessive length mismatches between audio and transcript.

4. Parallel Text Alignment: Once audio segments are paired with transcripts, we align them with their corresponding target language text. We generate sentence embeddings for both the source and target texts using SONAR (Duquenne et al., 2023). Next,

![Distribution of dataset hours in log scale for En-Indic (left) and Indic-En (right) language pairs across three data sources: Previous Datasets (blue), Curated & Mined (green), and Synthetic (red).](page_186_186_1077_495.png)

Figure 2: Distribution of dataset hours in log scale for En-Indic (left) and Indic-En (right) language pairs across three data sources: Previous Datasets (blue), Curated & Mined (green), and Synthetic (red).
we compute cosine similarity scores between these embeddings to identify the most likely parallel sentence pairs.

3.2.2 Sources of Mining Corpora

We apply the pipeline described in Section 3.2.1 on the following sources to extract translation pairs.

Spoken Tutorial: Spoken tutorial, is an initiative under the National Mission on Education under the Ministry of Human Resource Development, Government of India to enable the translation of various vocational educational content in different languages. This dataset covers Indic-En translations with 499.04 hours of educational tutorials in 12 Indian languages.

UGCE Resources: The University Grants Commission translates a wide range of university-level courses into multiple Indian languages. Originally delivered in English (En audio), these courses have their transcripts translated into 8 additional languages, resulting in over 430 hours of aligned audio-text data for each language pair.

VaaniPedia: Vaanipedia is a comprehensive collection of religious and spiritual audio-text content in 14 Indian languages, comprising over 121 hours of aligned audio shared across each language pair.

Mann ki Baat: Mann Ki Baat is a monthly radio broadcast by the Prime Minister of India that addresses various national issues. Each episode is transcribed and manually translated into 13 Indian languages. These translations are also manually recorded, resulting in over 435 hours of speech data across all language pairs, covering both Indic-to-English and English-to-Indic directions. Notably, Mann Ki Baat is the only source in this dataset that includes Manipuri, which is provided in Bengali script (rather than the Meitei script). While tables and figures do not explicitly highlight it, we are releasing the Manipuri data in Bengali script as sourced from Mann Ki Baat.

WordProject: This dataset consists of Bible audiobooks sourced from the Word Project website in 12 Indian languages, totaling approximately 1,612 hours of speech data across both Indic-En and En-Indic directions.

NPTEL: The National Programme on Technology Enhanced Learning (NPTEL) is an Indian e-learning platform for university-level science, technology, engineering, and mathematics (STEM) subjects. It is the largest e-repository in the world, offering courses in engineering, basic sciences, and selected humanities and management subjects\footnote{https://en.wikipedia.org/wiki/National\_Programme\_on\_Technology\_Enhanced\_Learning}. These courses are primarily recorded in English, but they are transcribed and re-recorded in multiple languages manually. This is the largest component of our dataset, contributing over 22,500 hours of spoken content across 10 Indian languages. It covers a wide range of STEM subjects and includes both En-XX and XX-En translations, making it one of the most significant sources in our collection.

While these datasets are high quality, they are not diverse with majority of the content being from Educational domain. Furthermore, the audio styles they cover lack the diversity necessary for models to adapt effectively to real-world, where background noise, spontaneous speech, and varying accents are common. To overcome these limitations, we augment our dataset by using synthetic data generation as described below.

3.3 Synthetic Data Generation

To ensure that we have audio diversity across different domains, demographics, and languages, we use INDICVOICES (Javed et al., 2024) which is the largest collection of natural and spontaneous speech for Indian languages containing over 4,000 hours of transcribed speech. Its extensive language diversity makes it well-suited for training models capable of performing effectively in real-world scenarios.

Synthetic Translations: The INDICVOICES dataset consists of spontaneous speech, which includes disfluencies, repetitions, and informal constructions that pose challenges for direct text-based translation. Since INDICVOICES also provides normalized transcripts, where such artifacts are already addressed, we use these instead of verbatim transcripts for translation. This ensures cleaner and more structured inputs while retaining the essence of conversational speech.

To generate high-quality English translations, we first apply punctuation restoration and translate the unsegmented version of the transcripts using LLAMA-3.1-405B-INSTRUCT (Figure A1). By pairing the original Indic audio with these translated transcripts, we construct XX-En pairs suitable for speech translation tasks. We then utilize ST-ALIGNER to align the translations at the segment level, preserving synchronization between the audio and textual representations.
Figure 3: Distribution of SONAR mining scores for XX → En sentence pairs from the SeamlessAlign (left) and BHASAANUVAAD (right) datasets. BHASAANUVAAD consistently exhibits a higher concentration of high-quality alignments, with mining scores skewed toward the upper end of the scale. In contrast, SeamlessAlign distributions are centered around lower mining scores, indicating overall lower alignment quality. This comparison underscores the improved precision of alignments in BHASAANUVAAD.

Unlike specialized translation models such as INDICTRANS2 (Gala et al., 2023), which are optimized for formal text and often struggle with colloquial or spontaneous speech, LLAMA-3.1-405B-INSTRUCT offers a steerable alternative. Its instruction-following capability enables more flexible adaptation to informal and conversational registers through prompting with examples and contextual cues.

To validate this, we conducted a small-scale internal human evaluation on a subset of the INDICVOICES dataset. LLAMA-3.1-405B-INSTRUCT consistently outperformed INDICTRANS2, particularly in preserving the nuances and fluency of informal spoken language. This makes it well-suited for real-world applications, where such variability is common. In a broader multilingual evaluation, we assessed the translation quality of LLAMA-3.1-405B-INSTRUCT across 1,000 documents per language. The model achieved an average human-rated score of 8 out of 10, indicating adequacy for our use case.

Synthetic Speech Generation: To overcome the time and cost limitations of manual voice recordings, we augment our dataset synthetic speech generation. We begin with verbatim transcripts in INDICVOICES, which preserve natural speech phenomena such as disfluencies, false starts, repetitions, and pauses. These are translated into colloquial English using the LLAMA-3.1-405B-INSTRUCT model, producing outputs that reflect spontaneous conversational patterns, including hesitations and filler words (Figure A3).

To synthesize English speech with prosodic and pronunciation features characteristic of Indian-accented speakers, we use INDIC PARLER TTS (Sankar et al., 2025), a state-of-the-art text-to-speech model fine-tuned for Indian languages. The model offers fine-grained prosodic control, enabling realistic variation in rhythm, intonation, and articulation.

By pairing the synthesized English speech with the normalized Indic transcripts from INDICVOICES, we construct a high-quality En-XX dataset. This resource enhances both linguistic diversity and domain robustness, supporting more effective training and evaluation for real-world speech translation applications.

3.4 Quality Control

To ensure the reliability of the aligned speech-text pairs, we adopt a two-fold evaluation strategy; one for audio-text alignment quality and the other for translation quality.

Alignment Quality: We quantify the accuracy of audio-transcript alignment by computing Levenshtein distance-based similarity scores between the cleaned reference transcripts and ASR-predicted outputs, following the methodology of Bhogale et al. (2022). This metric enables us to identify and filter out poorly aligned segments, thus maintaining high fidelity in timestamped pairs that are critical for training downstream speech and translation models.

Translation Quality: To evaluate the quality of the aligned parallel text, we compute cosine similarity scores between sentence embeddings of the source transcript and its corresponding translation.
These embeddings are obtained using the SONAR model (Duquenne et al., 2023), which provides robust cross-lingual sentence representations. *While the dataset is not filtered based on these metrics in the public release, we include both the alignment and translation quality scores alongside the data. This allows users to apply their own thresholds or filtering strategies depending on the specific needs of their applications.*

**Comparison with Prior Web-Mined Corpora:** Figure 3 compares SONAR-based mining scores for XX → En sentence pairs from BHASAANUVAAD and the SeamlessAlign corpus. SeamlessAlign shows a concentration of low scores, indicating noisy or weak alignments, whereas BHASAANUVAAD yields a markedly higher proportion of high-quality pairs, with scores skewed toward the upper end of the distribution. Language-wise alignment score distributions are shown in Figures A4 and A5, with comprehensive plots for all languages provided in Figures A6, A7, and A8. These results underscore the effectiveness of our mining pipeline in producing cleaner and more semantically aligned sentence pairs.

4 **Speech Translation Models**

In this study, we primarily consider the Unified (or end-to-end) and the Cascaded Speech translation paradigms across various models. While recent advancements in Audio-LLMs (or SLLMs) introduce new paradigms by integrating Speech Foundation Models with LLMs for Automatic Speech Translation (AST), we do not include them in this study due to the lack of support for Indian languages.

4.1 **Our Model**

**SEAMLESSM4T (SD\(_{BA}\))** We fine-tune the SEAMLESSM4T-v2-LARGE model on a filtered subset of BHASAANUVAAD (SD\(_{BA}\): IndicSeamless), using a high-quality filtered subset to ensure effective adaptation to Indian languages. Training is conducted on 8 × A100 GPUs for 430,000 steps with an effective per-device batch size of 64. We employ the Adam 8-bit optimizer with a learning rate of \(1 \times 10^{-5}\) and 1,000 warmup steps. To mitigate overfitting, we apply early stopping with a patience of 10.

**4.1.1 Data**

We construct the seed dataset by retaining only those audio-text pairs that meet the following quality thresholds: a cosine similarity-based mining score \( \sigma \geq 0.6 \), and a Levenshtein distance-based alignment score \( \tau \geq 0.8 \), following Bhogale et al. (2022) for the alignment criterion, from the newly collected sources. These thresholds are chosen to ensure that only highly reliable and accurately aligned data contribute to model fine-tuning.

From the filtered data, we construct the evaluation set—BHASAANUVAAD-TEST (Table A3)—by randomly sampling approximately 20 minutes of speech per language for each translation direction (En → XX and XX → En). This setup provides a balanced test set across all covered languages, while preserving diversity in speaker accents, linguistic complexity, and domain coverage. The remaining filtered data, excluding the test portion, is used for training the model.

4.2 **Baselines**

**Unified Systems:** In this paradigm, a single end-to-end speech translation model is used. The unified AST models considered in our study are as follows:
**SEAMLESSM4T (SD)** (Communication et al., 2023) is a foundational multilingual and multitask model supporting speech-to-speech (S2S), speech-to-text (S2T), text-to-speech (T2S), text-to-text (T2T) translation, and automatic speech recognition (ASR) for up to 100 languages. For our experiments, we use the SEAMLESS-M4T-v2-LARGE MODEL.
**AZURE** is a closed-source speech translation system developed by Microsoft, providing cloud-based automatic speech recognition (ASR) and speech-to-text (S2T) translation services. While details about its architecture and training data remain proprietary, Azure’s speech translation API supports multiple languages and is widely used in commercial applications. For our study, we evaluate its performance against other baselines and SD\(_{BA}\).

**Cascaded Systems:** In this paradigm, separately trained Automatic Speech Recognition (ASR) and Machine Translation (MT) models are combined in a pipeline for performing Automatic Speech Translation (AST). The different ASR and NMT models used in our experiments are listed below.
**SEAMLESSM4T(SC)** (Communication et al., 2023) where we integrate the ASR and NMT capabilities of the SEAMLESSM4T model in a sequential pipeline.
**WHISPER (W)** (Radford et al., 2023) is a multilingual model trained for automatic speech recognition (ASR) and speech translation (AST). For
<table>
  <tr>
    <th rowspan="2">lang</th>
    <th colspan="4">Direct</th>
    <th colspan="4">Cascaded</th>
  </tr>
  <tr>
    <th colspan="2">SD</th>
    <th colspan="2">SD<sub>BA</sub></th>
    <th colspan="2">SC</th>
    <th colspan="2">W + IT2</th>
    <th colspan="2">SC + IT2</th>
  </tr>
  <tr>
    <th></th>
    <th>FL</th>
    <th>BA</th>
    <th>FL</th>
    <th>BA</th>
    <th>FL</th>
    <th>BA</th>
    <th>FL</th>
    <th>BA</th>
    <th>FL</th>
    <th>BA</th>
  </tr>
  <tr>
    <td>asm</td>
    <td>37.90</td>
    <td>32.60</td>
    <td>39.30</td>
    <td>42.20</td>
    <td>37.90</td>
    <td>36.40</td>
    <td>36.20</td>
    <td>35.30</td>
    <td>40.80</td>
    <td>43.40</td>
    <td>40.50</td>
    <td>40.00</td>
  </tr>
  <tr>
    <td>ben</td>
    <td>47.50</td>
    <td>31.00</td>
    <td>48.90</td>
    <td>43.10</td>
    <td>47.00</td>
    <td>36.30</td>
    <td>46.40</td>
    <td>36.90</td>
    <td>51.00</td>
    <td>42.50</td>
    <td>50.40</td>
    <td>38.80</td>
  </tr>
  <tr>
    <td>guj</td>
    <td>48.90</td>
    <td>33.80</td>
    <td>50.10</td>
    <td>46.00</td>
    <td>48.60</td>
    <td>38.20</td>
    <td>48.40</td>
    <td>38.50</td>
    <td>52.10</td>
    <td>45.80</td>
    <td>51.70</td>
    <td>40.70</td>
  </tr>
  <tr>
    <td>hin</td>
    <td>54.20</td>
    <td>25.00</td>
    <td>55.10</td>
    <td>51.90</td>
    <td>53.10</td>
    <td>45.40</td>
    <td>54.20</td>
    <td>45.60</td>
    <td>56.00</td>
    <td>49.90</td>
    <td>55.70</td>
    <td>44.80</td>
  </tr>
  <tr>
    <td>kan</td>
    <td>48.90</td>
    <td>38.00</td>
    <td>49.90</td>
    <td>50.30</td>
    <td>48.90</td>
    <td>41.40</td>
    <td>47.80</td>
    <td>43.40</td>
    <td>52.10</td>
    <td>50.60</td>
    <td>51.40</td>
    <td>45.60</td>
  </tr>
  <tr>
    <td>mal</td>
    <td>49.00</td>
    <td>39.40</td>
    <td>51.20</td>
    <td>50.80</td>
    <td>50.30</td>
    <td>43.80</td>
    <td>47.90</td>
    <td>44.20</td>
    <td>54.70</td>
    <td>49.60</td>
    <td>53.90</td>
    <td>47.60</td>
  </tr>
  <tr>
    <td>mar</td>
    <td>44.10</td>
    <td>29.30</td>
    <td>45.70</td>
    <td>43.20</td>
    <td>43.30</td>
    <td>35.50</td>
    <td>42.30</td>
    <td>36.70</td>
    <td>47.50</td>
    <td>43.40</td>
    <td>47.20</td>
    <td>39.60</td>
  </tr>
  <tr>
    <td>npi</td>
    <td>50.00</td>
    <td>28.00</td>
    <td>50.40</td>
    <td>44.40</td>
    <td>46.80</td>
    <td>26.70</td>
    <td>47.20</td>
    <td>35.60</td>
    <td>53.70</td>
    <td>42.90</td>
    <td>53.10</td>
    <td>36.30</td>
  </tr>
  <tr>
    <td>ory</td>
    <td>46.20</td>
    <td>31.90</td>
    <td>48.50</td>
    <td>40.00</td>
    <td>45.70</td>
    <td>29.70</td>
    <td>43.40</td>
    <td>35.60</td>
    <td>46.80</td>
    <td>37.30</td>
    <td>46.50</td>
    <td>35.20</td>
  </tr>
  <tr>
    <td>pan</td>
    <td>50.00</td>
    <td>27.00</td>
    <td>51.40</td>
    <td>34.40</td>
    <td>48.10</td>
    <td>28.50</td>
    <td>48.80</td>
    <td>28.40</td>
    <td>51.50</td>
    <td>34.40</td>
    <td>50.90</td>
    <td>29.80</td>
  </tr>
  <tr>
    <td>snd</td>
    <td>47.10</td>
    <td>—</td>
    <td>48.90</td>
    <td>—</td>
    <td>—</td>
    <td>—</td>
    <td>45.90</td>
    <td>—</td>
    <td>42.30</td>
    <td>—</td>
    <td>42.00</td>
    <td>—</td>
  </tr>
  <tr>
    <td>tam</td>
    <td>50.60</td>
    <td>33.80</td>
    <td>52.30</td>
    <td>40.50</td>
    <td>50.50</td>
    <td>42.80</td>
    <td>49.50</td>
    <td>41.20</td>
    <td>53.30</td>
    <td>48.80</td>
    <td>53.00</td>
    <td>44.80</td>
  </tr>
  <tr>
    <td>tel</td>
    <td>52.00</td>
    <td>33.20</td>
    <td>53.50</td>
    <td>47.80</td>
    <td>49.90</td>
    <td>36.60</td>
    <td>51.30</td>
    <td>40.10</td>
    <td>55.30</td>
    <td>46.20</td>
    <td>54.50</td>
    <td>42.20</td>
  </tr>
  <tr>
    <td>urd</td>
    <td>47.20</td>
    <td>41.10</td>
    <td>47.30</td>
    <td>49.70</td>
    <td>45.80</td>
    <td>41.10</td>
    <td>46.60</td>
    <td>45.50</td>
    <td>48.80</td>
    <td>49.60</td>
    <td>48.80</td>
    <td>46.10</td>
  </tr>
  <tr>
    <td>avg.</td>
    <td>48.10</td>
    <td>33.20</td>
    <td>49.46</td>
    <td>44.75</td>
    <td>47.38</td>
    <td>37.11</td>
    <td>46.85</td>
    <td>39.33</td>
    <td>50.42</td>
    <td>44.84</td>
    <td>49.97</td>
    <td>40.90</td>
  </tr>
</table>

Table 1: chrF++ (↑) scores for different direct and cascaded speech translation models evaluated on the FLEURS-TEST (FL; green) and BHASAANUVAAAD-TEST (BA; yellow) datasets in En → XX direction. The colors represent a heatmap, where darker shades indicating better translation performance.

our experiments, we use the WHISPER-LARGE-V3 model which has been trained on 1 million hours of weakly labeled and 4 million hours of pseudo-labeled audio.

INDIC-CONFORMER (IC)\footnote{En-Indic sacreBLEU ChrF++ signature:nrefs:1|case:mixed|eff:yes|nc:6|nw:2|space:no|version:2.3.1} (Bhogale et al., 2025) which is a suite of ASR models based on the Conformer architecture, models supporting all 22 scheduled Indian languages.

INDICTRANS2 (IT2) \footnote{https://github.com/urduhack/urduhack} (Gala et al., 2023) which is an open-source transformer-based multilingual NMT model supporting translations across all 22 scheduled Indic languages. For our experiments, we use the INDICTRANS2-1B model.

In our experiments, we combine the INDICTRANS2-1B model and the SEAMLESS-M4T V2 LARGE translation model with various ASR models to perform automatic speech translation (AST). Specifically, the translation capabilities of INDICTRANS2-1B and SEAMLESS-M4T V2 LARGE (T2TT) are integrated with ASR outputs from systems such as WHISPER, and INDIC-CONFORMER.

\subsection*{4.3 Metrics}

In this work, we adopt chrF++ (Popović, 2017) as our primary metric, which offers strong alignment with human judgments (Sai B et al., 2023) particularly Indian languages. We compute chrF++ scores using sacreBLEU (Post, 2018); for Indic-En\textsuperscript{5} evaluation, we use the standard mteval-v13a tokenizer, while for En-Indic, we apply Indic-specific tokenizers from IndicNLP (Kunchukuttan, 2020) and Urduhack\footnote{https://github.com/urduhack/urduhack} before scoring, ensuring linguistically informed segmentation. We also report additional metrics such as BLEU (Papineni et al., 2002) and XCOMET (Guerreiro et al., 2024) scores in Appendix Tables A4 & A5 and A6 & A7 respectively for both the directions.

\section*{5 Results}

We evaluate the translation performance of various models in both En →XX and XX→ En directions using chrF++ scores on the FLEURS-TEST and BHASAANUVAAAD-TEST (Table A3) datasets. Our analysis focuses on the comparative effectiveness of direct and cascaded models, highlighting the impact of fine-tuning and architectural choices.

\subsection*{5.1 Direct vs. Cascaded Systems}

A comparison between direct and cascaded approaches on the FLEURS-TEST set reveals that their performance remains closely matched across most languages. In En →XX translation, the average chrF++ score difference between the two paradigms is marginal, suggesting that direct speech translation methods are increasingly closing the performance gap with cascaded pipelines. To analyze this further, we compare the strongest
<table>
  <tr>
    <th rowspan="2">lang</th>
    <th colspan="4">Direct</th>
    <th colspan="4">Cascaded</th>
  </tr>
  <tr>
    <th colspan="2">SD</th>
    <th colspan="2">SD<sub>BA</sub></th>
    <th colspan="2">SC</th>
    <th colspan="2">IC + IT2</th>
    <th colspan="2">SC + IT2</th>
  </tr>
  <tr>
    <th></th>
    <th>FL</th>
    <th>BA</th>
    <th>FL</th>
    <th>BA</th>
    <th>FL</th>
    <th>BA</th>
    <th>FL</th>
    <th>BA</th>
    <th>FL</th>
    <th>BA</th>
  </tr>
  <tr><td>asm</td><td>44.00</td><td>46.90</td><td>49.70</td><td>53.80</td><td>38.20</td><td>41.20</td><td>46.30</td><td>49.40</td><td>47.20</td><td>51.50</td><td>47.10</td><td>50.70</td></tr>
  <tr><td>ben</td><td>48.50</td><td>45.20</td><td>55.00</td><td>57.40</td><td>46.10</td><td>40.50</td><td>51.20</td><td>47.20</td><td>52.50</td><td>49.50</td><td>53.30</td><td>48.80</td></tr>
  <tr><td>guj</td><td>52.80</td><td>51.20</td><td>60.20</td><td>60.00</td><td>45.40</td><td>32.70</td><td>55.90</td><td>54.20</td><td>57.10</td><td>54.50</td><td>57.50</td><td>54.40</td></tr>
  <tr><td>hin</td><td>50.60</td><td>48.60</td><td>59.10</td><td>59.40</td><td>51.40</td><td>47.50</td><td>53.20</td><td>51.20</td><td>56.50</td><td>55.80</td><td>56.50</td><td>53.70</td></tr>
  <tr><td>kan</td><td>47.60</td><td>47.60</td><td>54.80</td><td>56.90</td><td>47.10</td><td>38.10</td><td>50.10</td><td>50.80</td><td>52.00</td><td>52.50</td><td>52.00</td><td>52.20</td></tr>
  <tr><td>mal</td><td>48.40</td><td>43.50</td><td>55.90</td><td>54.00</td><td>46.00</td><td>35.60</td><td>52.00</td><td>46.30</td><td>54.20</td><td>49.50</td><td>54.10</td><td>48.70</td></tr>
  <tr><td>mar</td><td>48.40</td><td>46.20</td><td>55.30</td><td>56.20</td><td>43.80</td><td>37.70</td><td>52.20</td><td>50.10</td><td>54.30</td><td>51.10</td><td>53.50</td><td>50.20</td></tr>
  <tr><td>npi</td><td>50.60</td><td>35.70</td><td>56.60</td><td>51.90</td><td>50.10</td><td>31.20</td><td>54.30</td><td>38.70</td><td>55.30</td><td>46.50</td><td>56.40</td><td>40.90</td></tr>
  <tr><td>ory</td><td>48.60</td><td>50.10</td><td>54.90</td><td>57.70</td><td>48.20</td><td>44.00</td><td>52.80</td><td>53.50</td><td>55.10</td><td>52.10</td><td>54.20</td><td>52.90</td></tr>
  <tr><td>pan</td><td>50.30</td><td>50.00</td><td>56.40</td><td>57.60</td><td>44.00</td><td>44.70</td><td>53.00</td><td>55.00</td><td>55.10</td><td>53.60</td><td>54.70</td><td>55.40</td></tr>
  <tr><td>snd</td><td>31.10</td><td>—</td><td>43.40</td><td>—</td><td>—</td><td>—</td><td>36.20</td><td>—</td><td>39.30</td><td>—</td><td>38.50</td><td>—</td></tr>
  <tr><td>tam</td><td>44.80</td><td>41.60</td><td>51.00</td><td>52.20</td><td>45.40</td><td>37.30</td><td>48.10</td><td>44.30</td><td>50.70</td><td>47.30</td><td>50.30</td><td>46.80</td></tr>
  <tr><td>tel</td><td>47.50</td><td>44.50</td><td>54.30</td><td>55.00</td><td>48.30</td><td>39.40</td><td>51.50</td><td>47.50</td><td>55.60</td><td>51.20</td><td>54.40</td><td>50.10</td></tr>
  <tr><td>urd</td><td>47.60</td><td>48.90</td><td>54.40</td><td>61.10</td><td>47.60</td><td>42.80</td><td>50.30</td><td>51.20</td><td>51.30</td><td>52.70</td><td>53.10</td><td>53.00</td></tr>
  <tr><td>avg.</td><td>47.20</td><td>46.15</td><td>54.36</td><td>56.40</td><td>46.28</td><td>39.44</td><td>50.51</td><td>49.18</td><td>52.59</td><td>51.37</td><td>52.54</td><td>50.60</td></tr>
</table>

Table 2: chrF++ (†) scores for different direct and cascaded speech translation models evaluated on the FLEURS-TEST (FL; green) and BHASAANUVAAD-TEST (BA; yellow) datasets in \( X \rightarrow En \) direction. The colors represent a heatmap, where darker shades indicating better translation performance.

direct model, SD<sub>BA</sub>, with the best-performing cascaded setups: W + IT2 for En→XX and IC + IT2 for XX→En.

For En → XX (Table 1), SD<sub>BA</sub> and W + IT2 achieve comparable results across most languages, underscoring the efficacy of both direct speech translation and cascaded modeling. However, an exception arises in Tamil in BHASAANUVAAD-TEST, where W + IT2 significantly outperforms SD<sub>BA</sub>. This deviation warrants further investigation into language-specific factors that may influence direct model performance.

In the XX → En direction, SD<sub>BA</sub> surpasses IC + IT2 across multiple languages, demonstrating strong generalization capabilities. Notably, on the BHASAANUVAAD-TEST set, SD<sub>BA</sub> achieves significantly higher ChrF++ scores for Nepali, Bengali, and Urdu, reinforcing its ability to handle diverse linguistic structures effectively. These results suggest that fine-tuned direct models can be competitive or even superior to cascaded systems, where cascading errors from ASR and MT modules compound and degrade translation quality.

5.2 Performance Gains from Fine-Tuning

Fine-tuning on the BhasaAnuvaad dataset (BA) leads to consistent improvements across both translation directions (En→XX and XX→En). SD<sub>BA</sub> achieves substantial gains over SD, particularly on the BA test set, where it outperforms the baseline by at least 10 points on average.

On the FLEURS-TEST set, SD<sub>BA</sub> consistently outperforms SD across all languages, achieving an average improvement of over 7 points. These results highlight the effectiveness of domain-specific fine-tuning in enhancing direct speech translation performance, particularly in real-world, low-resource conditions.

6 Conclusion

In this work, we present BHASAANUVAAD, the largest publicly available Indian language speech translation dataset, comprising 44,000+ hours of speech and 17 million aligned text segments across 14 Indian languages and English. Our dataset is built using a threefold approach: (i) aggregating high-quality sources, (ii) large-scale web crawling, and (iii) synthetic augmentation to capture real-world disfluencies. Using BHASAANUVAAD, we train INDICSEAMLESS, a state-of-the-art speech translation model for Indian languages. Our evaluations show that fine-tuned direct speech translation models (SD<sub>BA</sub>) match or surpass strong cascaded systems, particularly in low-resource settings where cascading errors degrade performance. By releasing all data, models, and code under open-source licenses, we aim to advance research in Indian language speech translation.

7 Limitations

Despite the scale and diversity of BHASAANUVAAD, several limitations remain, addressing
which will be essential to making speech translation systems for Indian languages more robust, reliable, and truly representative of real-world usage.

First, while our dataset integrates a range of sources, spontaneous speech remains underrepresented. Real-world conversational speech often includes disfluencies, hesitations, code-switching, and speaker variability that are not fully captured in our current dataset.

Second, domain coverage remains a challenge. While BHASAANUVAAD aggregates speech from multiple sources including educational content, government archives, and podcasts certain critical domains such as informal dialogues, social media discourse, medical interactions, and legal proceedings remain underrepresented.

Third, benchmarking Indian language AST remains an open challenge. Most prior evaluations rely on FLEURS, which consists of acted-out, read speech rather than spontaneous speech. While useful for measuring system performance under controlled conditions, FLEURS fails to reflect the complexities of natural, conversational speech, where hesitations, overlaps, and real-world noise significantly impact translation quality. Our experiments indicate that models achieving strong results on FLEURS do not necessarily maintain similar performance on more challenging, real-world speech scenarios.

8 Ethics

The code and datasets created in this work will be made available under permissible licenses. Generative AI systems were only used for assistance purely with the language of the paper, eg: paraphrasing, spell-check, polishing the author’s original content and for writing boiler plate code.

9 Acknowledgements

We gratefully acknowledge the generous support and funding provided by Digital India Bhashini, the Centre for Development of Advanced Computing (C-DAC) Pune, Yotta, EkStep Foundation, and Nilekani Philanthropies. We also thank Pranjal Agadh Chitale for his insightful discussions and valuable feedback throughout the development of this work.

References

Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli. 2021. Xls-r: Self-supervised cross-lingual speech representation learning at scale. arXiv preprint arXiv: 2111.09296.

Parnia Bahar, Patrick Wilken, Tamer Alkhouri, Andreas Gut, Pavel Golik, Evgeny Matusov, and Christian Herold. 2020. Start-before-end and end-to-end: Neural speech translation by AppTek and RWTH Aachen University. In Proceedings of the 17th International Conference on Spoken Language Translation, pages 44–54, Online. Association for Computational Linguistics.

Bismarck Bamfo Odoom, Nathaniel Robinson, Elijah Rippeth, Luis Tavarez-Arce, Kenton Murray, Matthew Wiesner, Paul McNamee, Philipp Koehn, and Kevin Duh. 2024. Can synthetic speech improve end-to-end conversational speech translation? In Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pages 167–177, Chicago, USA. Association for Machine Translation in the Americas.

Ankur Bapna, Colin Cherry, Yu Zhang, Ye Jia, Melvin Johnson, Yong Cheng, Simran Khanuja, Jason Riesa, and Alexis Conneau. 2022. mslam: Massively multilingual joint pre-training for speech and text. arXiv preprint arXiv: 2202.01374.

Daniel Beck, Trevor Cohn, and Gholamreza Haffari. 2019. Neural speech translation using lattice transformations and graph networks. In Proceedings of the Thirteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-13), pages 26–31, Hong Kong. Association for Computational Linguistics.

Luisa Bentivogli, Mauro Cettolo, Marco Gaido, Alina Karakanta, Alberto Martinelli, Matteo Negri, and Marco Turchi. 2021. Cascade versus direct speech translation: Do the differences still make a difference? arXiv preprint arXiv:2106.01045.

Kaushal Santosh Bhogale, Deovrat Mehendale, Tahir Javed, Devbrat Anuragi, Sakshi Joshi, Sai Sundaresan, Aparna Ananthanarayanan, Sharmistha Dey, Sathish Kumar Reddy G, Anusha Srinivasan, Abhigyan Raman, Pratyush Kumar, and Mitesh M. Khapra. 2025. Towards bringing parity in pretraining datasets for low-resource indian languages. In ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5.

Kaushal Santosh Bhogale, Abhigyan Raman, Tahir Javed, Sumanth Doddapaneni, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh M. Khapra. 2022. Effectiveness of mining audio and text pairs from public data for improving asr systems for low-resource languages. arXiv preprint.
Qiao Cheng, Meiyuan Fan, Yaqian Han, Jin Huang, and Yitao Duan. 2019. Breaking the data barrier: Towards robust speech translation via adversarial stability training. In Proceedings of the 16th International Conference on Spoken Language Translation, Hong Kong. Association for Computational Linguistics.

Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv: 2311.07919.

Seamless Communication, Loïc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, Christopher Klaiber, Pengwei Li, Daniel Licht, Jean Maillard, Alice Rakotoarison, Kaushik Ram Sadagopan, Guillaume Wenzek, Ethan Ye, Bapi Akula, Peng-Jen Chen, Naji El Hachem, Brian Ellis, Gabriel Mejia Gonzalez, Justin Haaheim, Prangthip Hansanti, Russ Howes, Bernie Huang, Min-Jae Hwang, Hirofumi Inaguma, Somya Jain, Elahe Kalbasi, Amanda Kallet, Ilia Kulikov, Janice Lam, Daniel Li, Xutai Ma, Ruslan Mavlyutov, Benjamin Pelequin, Mohamed Ramadan, Abinesh Ramakrishnan, Anna Sun, Kevin Tran, Tuan Tran, Igor Tufanov, Vish Vogeti, Carleigh Wood, Yilin Yang, Bokai Yu, Pierre Andrews, Can Balioglu, Marta R. Costa-jussà, Onur Cebeli, Mahbubul Elbayad, Cynthia Gao, Francisco Guzmán, Justine Kao, Ann Lee, Alexandre Mourachko, Juan Pino, Sravya Popuri, Christophe Ropers, Safiyah Saleem, Holger Schwenk, Padam Tomaselso, Changhan Wang, Jeff Wang, and Skyler Wang. 2023. Seamless4t: Massively multilingual & multimodal machine translation. arXiv preprint arXiv: 2308.11596.

Alexis Conneau, Min Ma, Simran Khanduja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. 2022. FLEURS: few-shot learning evaluation of universal representations of speech. In IEEE Spoken Language Technology Workshop, SLT 2022, Doha, Qatar, January 9-12, 2023, pages 798–805. IEEE.

Matti Di Gangi, Robert Enyedi, Alessandra Brusadin, and Marcello Federico. 2019. Robust neural machine translation for clean and noisy speech transcripts. In Proceedings of the 16th International Conference on Spoken Language Translation, Hong Kong. Association for Computational Linguistics.

Paul-Ambroise Duquenne, Hongyu Gong, Ning Dong, Jingfei Du, Ann Lee, Vedanuj Goswani, Changhan Wang, Juan Pino, Benoît Sagot, and Holger Schwenk. 2022. Speechmatrix: A large-scale mined corpus of multilingual speech-to-speech translations. ARXIV.

Paul-Ambroise Duquenne, Holger Schwenk, and Benoît Sagot. 2023. SONAR: sentence-level multimodal and language-agnostic representations. CoRR, abs/2308.11466.

Dominika Ďurišková, Daniela Jurášová, Matúš Žilinec, Eduard Šubert, and Ondřej Bojar. 2024. Khan academy corpus: A multilingual corpus of khan academy lectures. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 9743–9752, Torino, Italia. ELRA and ICCL.

Yassir Fatullaha, Chunyang Wu, Egor Lakomkin, Ke Li, Junteng Jia, Yuan Shangguan, Jay Mahadeokar, Ozlem Kalinli, Christian Fuegen, and Mike Seltzer. 2023. Audiochatllama: Towards general-purpose speech abilities for llms. arXiv preprint arXiv: 2311.06753.

Marco Gaido, Sara Papi, Matteo Negri, and Luisa Ben-tivogli. 2024. Speech translation with speech foundation models and large language models: What is there and what is missing? Preprint, arXiv:2402.12025.

Jay P. Gala, Pranjal A. Chitale, AK Raghavan, Varun Gunma, Sumantha Doddapaneni, M. AswathKumar, J. Nawale, Anupama Sujatha, Ratish Puduppully, Vivek Raghavan, Pratyush Kumar, Mitesh M. Khapra, Raj Dabre, and Anoop Kunchukuttan. 2023. Indictrans2: Towards high-quality and accessible machine translation models for all 22 scheduled indian languages. Trans. Mach. Learn. Res.

Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André F. T. Martins. 2024. xcomet: Transparent machine translation evaluation through fine-grained error detection. Transactions of the Association for Computational Linguistics, 12:979–995.

Anirudh Gupta, Neeraj Chhimwal, Ankur Dhuriya, Rishabh Gaur, Priyanshi Shah, Harveen Singh Chadha, and Vivek Raghavan. 2022. indic-punct: An automatic punctuation restoration and inverse text normalization framework for indic languages. Preprint, arXiv:2203.16825.

Zhichao Huang, Rong Ye, Tom Ko, Qianqian Dong, Shanbo Cheng, Mingxuan Wang, and Hang Li. 2023. Speech translation with large language models: An industrial practice. arXiv preprint arXiv: 2312.13585.

Hirofumi Inaguma, Yosuke Higuchi, Kevin Duh, Tatuya Kawahara, and Shinji Watanabe. 2021. Orthros: non-autoregressive end-to-end speech translation with dual-decoder. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7503–7507.

Tahir Javed, Janki Nawale, Eldho George, Sakshi Joshi, Kaushal Bhogale, Deovrat Mehendale, Ishvinder Sethi, Aparna Ananthanarayanan, Hafsa Faquih, Pratiti Palit, Sneha Ravishankar, Saranya Sukumaran, Tripura Panchagnula, Sunjay Murali, Kunal Gandhi, Ambujavalli R., Manickam M., C Vaijayanthi, Krishnan Karunganii, Pratyush Kumar, and Mitesh Khapra. 2024. IndicVoices: Towards building an inclusive multilingual speech dataset for Indian languages. In Findings of the Association for Computational Linguistics ACL 2024, pages 10740–10782,
Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.

Ye Jia, Michelle Tadmor Ramanovich, Quan Wang, and Heiga Zen. 2022. CVSS corpus and massively multilingual speech-to-speech translation. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6691–6703, Marseille, France. European Language Resources Association.

Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Papers, pages 79–86, Phuket, Thailand.

Anoop Kunchukuttan. 2020. The IndicNLP Library. https://github.com/anoopkunchukuttan/indic_nlp_library/blob/master/docs/indicnlp.pdf.

Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Sravya Popuri, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, Juan Pino, and Wei-Ning Hsu. 2022. Direct speech-to-speech translation with discrete units. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3327–3339, Dublin, Ireland. Association for Computational Linguistics.

Shivam Mhaskar, Vineet Bhat, Akshay Batheja, Sourabh Deoghare, Paramveer Choudhary, and Pushpak Bhattacharyya. 2023. Vakta-setu: A speech-to-speech machine translation service in select indic languages. arXiv preprint arXiv: 2305.12518.

Vandan Mujadia and Dipti Sharma. 2023. Towards speech to speech machine translation focusing on Indian languages. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 161–168, Dubrovnik, Croatia. Association for Computational Linguistics.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.

Maja Popović. 2017. chrf++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, pages 612–618. Association for Computational Linguistics.

Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Belgium, Brussels. Association for Computational Linguistics.

Vineel Pratap, Andros Tjandra, Bowen Shi, Padem Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, et al. 2024. Scaling speech technology to 1,000+ languages. Journal of Machine Learning Research, 25(97):1–52.

Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 28492–28518. PMLR.

Ananya Sai B, Tanay Dixit, Vignesh Nagarajan, Anoop Kunchukuttan, Pratyush Kumar, Mitesh M. Khapra, and Raj Dabre. 2023. IndicMT eval: A dataset to meta-evaluate machine translation metrics for Indian languages. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14210–14228, Toronto, Canada. Association for Computational Linguistics.

Elizabeth Salesky, Matthew Wiesner, Jacob Bremerman, R. Cattoni, Matteo Negri, Marco Turchi, Douglas W. Oard, and Matt Post. 2021. The multilingual tedx corpus for speech recognition and translation. INTERSPEECH.

Ashwin Sankar, Yoach Lacombe, Sherry Thomas, Praveen Srinivasa Varadhan, Sanchit Gandhi, and Mitesh M Khapra. 2025. Rasimalai: Resources for adaptive speech modeling in indian languages with accents and intonations. In Interspeech 2025.

Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86–96, Berlin, Germany. Association for Computational Linguistics.

Nivedita Sethiya and Chandresh Kumar Maurya. 2023. End-to-end speech-to-text translation: A survey. arXiv preprint arXiv: 2312.01053.

Nivedita Sethiya, Saanvi Nair, and Chandresh Maurya. 2024. Indic-TEDST: Datasets and baselines for low-resource speech to text translation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 9019–9024, Torino, Italia. ELRA and ICCL.

Bhavani Shankar, Preethi Jyothi, and Pushpak Bhattacharyya. 2024. Costa: Code-switched speech translation using aligned speech-text interleaving. arXiv preprint arXiv: 2406.10993.

Haiyue Song, Raj Dabre, Atsushi Fujita, and S. Kurohashi. 2019. Coursera corpus mining and multistage fine-tuning for improving lectures translation. International Conference on Language Resources and Evaluation.

Matthias Sperber and Matthias Paulik. 2020. Speech translation and the end-to-end promise: Taking stock
of where we are. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7409–7421, Online. Association for Computational Linguistics.

Silero Team. 2024. Silero vad: pre-trained enterprise-grade voice activity detector (vad), number detector and language classifier. https://github.com/snakers4/silero-vad.

Changhan Wang, Anne Wu, and Juan Pino. 2020. Covost 2 and massively multilingual speech-to-text translation. Preprint, arXiv:2007.10310.

Ron J. Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and Zhifeng Chen. 2017. Sequence-to-sequence models can directly translate foreign speech. In Interspeech 2017, pages 2625–2629.

Jian Wu, Yashesh Gaur, Zhuo Chen, Long Zhou, Yilun Zhu, Tianrui Wang, Jinyu Li, Shujie Liu, Bo Ren, Linquan Liu, and Yu Wu. 2023. On decoder-only architecture for speech-to-text and large language model integration. Automatic Speech Recognition & Understanding.

Rong Ye, Chengqi Zhao, Tom Ko, Chutong Meng, Tao Wang, Mingxuan Wang, and Jun Cao. 2023. Gigast: A 10,000-hour pseudo speech translation corpus. Preprint, arXiv:2204.03939.

Appendix

A Data Statistics

This section outlines the datasets used for training and evaluation, covering diverse Indic languages and both En-Indic and Indic-En directions. We begin with the test set composition, followed by training data statistics.

A.1 BHASAANUVAAD-TEST
The BHASAANUVAAD-TEST set (Table A3) contains 26.22 hours of En-Indic and 28.18 hours of Indic-En speech translation data, covering 13 Indian languages. Most languages have balanced data in both directions, with minor variations. Notably, mni and npi have smaller amounts of test data, with mni lacking Indic-En coverage. This test set provides a reliable benchmark for evaluating spoken translation performance across diverse Indic languages.

A.2 Training Data
En-Indic: The dataset combines 202.9 hours (91K utterances) of existing En-Indic speech data, 3453.8 hours (2619.3K utterances) of mined data, and 2694.1 hours (269K utterances) of synthetic data. Mined sources like NPTEL contribute the largest share with 15940.9 hours and 6344.7K utterances, while synthetic data from IndicVoices helps balance low-resource languages. This diverse collection ensures comprehensive coverage for spoken translation across multiple Indic languages.

Indic-En The dataset includes 108 hours (90K utterances) of existing Indic-En speech data, 1076.7 hours (308.6K utterances) of mined data, and 3988.2 hours (288K utterances) of synthetic data. Among the mined sources, NPTEL contributes the largest portion with 6626.5 hours and 2603.3K utterances. The synthetic IndicVoices dataset further strengthens the collection with 3988.2 hours and 288K utterances, ensuring better representation across low-resource Indic languages. This combination provides a balanced and diverse dataset for advancing Indic-English spoken translation.

B Prompt Design for Punctuation Restoration

To address the lack of punctuation in automatic speech recognition (ASR) outputs and enhance downstream readability, we design a structured prompt for punctuation restoration using LLAMA-3.1-405B-INSTRUCT. As shown in Figure A1, the prompt guides the model to insert only punctuation marks into a given raw text while strictly preserving the original word order and structure.

The prompt emphasizes four key principles:

1. Accuracy: Punctuation must conform to the grammatical rules of the target language.

2. Readability: Insert appropriate marks (e.g., commas, periods, question marks) to enhance sentence clarity.

3. Consistency: Follow the punctuation style observed in reference texts or previous examples.

4. Structure Preservation: Do not add, delete, or reorder words—only punctuation should be inserted.

To support multilingual usage, we provide language-specific reference metadata, including the expected sentence terminator. The model is instructed to return output in a clean JSON format, making it suitable for direct integration into data processing pipelines.
<table>
  <tr>
    <th rowspan="2">lang</th>
    <th colspan="6">Existing</th>
    <th colspan="6">Mined</th>
    <th colspan="4">Synthetic</th>
  </tr>
  <tr>
    <th>Indic-TedST</th><th>Fleurs</th><th>VaaniPedia</th><th>WordProject</th><th>UGCE Resources</th><th>Mann Ki Baat</th><th>NPTEL</th><th>IndicVoices</th>
  </tr>
  <tr>
    <th># hours</th><th># uttr.</th><th># hours</th><th># uttr.</th><th># hours</th><th># uttr.</th><th># hours</th><th># uttr.</th><th># hours</th><th># uttr.</th><th># hours</th><th># uttr.</th><th># hours</th><th># uttr.</th>
  </tr>
  <tr>
    <td>asm</td><td>-</td><td>-</td><td>12.0</td><td>1.5</td><td>3.4</td><td>1.6</td><td>-</td><td>-</td><td>-</td><td>-</td><td>28.3</td><td>15.7</td><td>10.6</td><td>5.3</td><td>348.9</td><td>23.5</td>
  </tr>
  <tr>
    <td>ben</td><td>13.1</td><td>6.5</td><td>12.0</td><td>1.5</td><td>121.0</td><td>52.5</td><td>50.7</td><td>23.9</td><td>431.7</td><td>329.6</td><td>38.3</td><td>21.4</td><td>1516.6</td><td>618.8</td><td>300.5</td><td>30.3</td>
  </tr>
  <tr>
    <td>guj</td><td>17.2</td><td>1.5</td><td>12.0</td><td>1.5</td><td>121.0</td><td>52.5</td><td>50.7</td><td>23.9</td><td>431.7</td><td>329.4</td><td>39.1</td><td>22.0</td><td>1977.9</td><td>784.9</td><td>16.7</td><td>2.8</td>
  </tr>
  <tr>
    <td>hin</td><td>100.4</td><td>52.5</td><td>12.0</td><td>1.5</td><td>121.0</td><td>52.5</td><td>50.7</td><td>23.9</td><td>431.7</td><td>329.2</td><td>37.7</td><td>21.3</td><td>3021.6</td><td>1176.8</td><td>206.6</td><td>20.4</td>
  </tr>
  <tr>
    <td>kan</td><td>3.7</td><td>1.9</td><td>12.0</td><td>1.5</td><td>23.2</td><td>9.9</td><td>50.7</td><td>23.9</td><td>431.7</td><td>328.6</td><td>36.9</td><td>20.8</td><td>1601.4</td><td>638.0</td><td>175.6</td><td>11.7</td>
  </tr>
  <tr>
    <td>mal</td><td>9.2</td><td>3.2</td><td>12.0</td><td>1.5</td><td>23.5</td><td>52.5</td><td>50.7</td><td>23.9</td><td>431.7</td><td>329.6</td><td>38.3</td><td>21.3</td><td>1729.1</td><td>699.3</td><td>278.1</td><td>22.9</td>
  </tr>
  <tr>
    <td>mar</td><td>29.0</td><td>12.7</td><td>12.0</td><td>1.5</td><td>50.5</td><td>24.4</td><td>50.7</td><td>23.9</td><td>431.7</td><td>322.4</td><td>39.0</td><td>22.0</td><td>1601.7</td><td>626.3</td><td>138.0</td><td>16.2</td>
  </tr>
  <tr>
    <td>api</td><td>-</td><td>-</td><td>12.0</td><td>1.5</td><td>121.0</td><td>52.5</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>316.3</td>
  </tr>
  <tr>
    <td>ory</td><td>-</td><td>-</td><td>12.0</td><td>1.5</td><td>79.0</td><td>34.0</td><td>49.2</td><td>23.1</td><td>-</td><td>-</td><td>36.8</td><td>20.5</td><td>0.8</td><td>0.3</td><td>161.1</td><td>19.1</td>
  </tr>
  <tr>
    <td>pan</td><td>1.2</td><td>0.4</td><td>12.0</td><td>1.5</td><td>0.5</td><td>0.3</td><td>50.7</td><td>23.9</td><td>-</td><td>-</td><td>37.0</td><td>20.5</td><td>9.6</td><td>5.8</td><td>194.1</td><td>18.9</td>
  </tr>
  <tr>
    <td>snd</td><td>-</td><td>-</td><td>12.0</td><td>1.5</td><td>121.0</td><td>52.5</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>tam</td><td>20.1</td><td>10.1</td><td>12.0</td><td>1.5</td><td>121.0</td><td>52.5</td><td>30.5</td><td>14.1</td><td>431.7</td><td>327.9</td><td>37.4</td><td>20.9</td><td>2631.9</td><td>1048.1</td><td>378.6</td><td>33.8</td>
  </tr>
  <tr>
    <td>tel</td><td>9.1</td><td>2.2</td><td>12.0</td><td>1.5</td><td>121.0</td><td>52.5</td><td>50.7</td><td>23.9</td><td>431.7</td><td>322.6</td><td>37.8</td><td>21.4</td><td>1839.9</td><td>741.2</td><td>314.3</td><td>21.6</td>
  </tr>
  <tr>
    <td>urd</td><td>-</td><td>-</td><td>12.0</td><td>1.5</td><td>2.3</td><td>1.1</td><td>50.7</td><td>23.9</td><td>-</td><td>-</td><td>29.0</td><td>16.0</td><td>-</td><td>-</td><td>186.6</td><td>19.1</td>
  </tr>
  <tr>
    <th>Total</th><th>202.9</th><th>91.0</th><th>168.0</th><th>21</th><th>891.1</th><th>439.0</th><th>536.2</th><th>252.3</th><th>3453.8</th><th>2619.3</th><th>435.7</th><th>244.0</th><th>15940.9</th><th>6344.7</th><th>2694.1</th><th>269.0</th>
  </tr>
</table>

Table A1: Overview of En-Indic spoken translation dataset across various Indic languages. The table details the hours of audio and the number of utterances collected from multiple sources. Number of utterances are in thousands (K)

<table>
  <tr>
    <th rowspan="2">lang</th>
    <th colspan="6">Existing</th>
    <th colspan="6">Mined</th>
    <th colspan="4">Synthetic</th>
  </tr>
  <tr>
    <th>CVSS & Khan Academy</th><th>SeamlessAlign</th><th>Fleurs</th><th>WordProject</th><th>Spokentutorial</th><th>Mann Ki Baat</th><th>NPTEL</th><th>IndicVoices</th>
  </tr>
  <tr>
    <th># hours</th><th># uttr.</th><th># hours</th><th># uttr.</th><th># hours</th><th># uttr.</th><th># hours</th><th># uttr.</th><th># hours</th><th># uttr.</th><th># hours</th><th># uttr.</th><th># hours</th><th># uttr.</th>
  </tr>
  <tr>
    <td>asm</td><td>-</td><td>-</td><td>12.0</td><td>1.5</td><td>-</td><td>-</td><td>36.9</td><td>23.2</td><td>23.0</td><td>13.9</td><td>9.3</td><td>5.7</td><td>443.6</td><td>26.0</td>
  </tr>
  <tr>
    <td>ben</td><td>-</td><td>-</td><td>12.0</td><td>1.5</td><td>117.6</td><td>28.9</td><td>49.9</td><td>31.1</td><td>38.5</td><td>20.7</td><td>388.4</td><td>149.1</td><td>450.6</td><td>33.2</td>
  </tr>
  <tr>
    <td>guj</td><td>12.0</td><td>10.0</td><td>-</td><td>-</td><td>12.0</td><td>1.5</td><td>82.1</td><td>29.0</td><td>63.6</td><td>37.6</td><td>37.4</td><td>21.3</td><td>795.0</td><td>300.9</td><td>30.2</td><td>3.0</td>
  </tr>
  <tr>
    <td>hin</td><td>-</td><td>2355.4</td><td>1021.5</td><td>12.0</td><td>1.5</td><td>88.5</td><td>29.5</td><td>63.6</td><td>37.9</td><td>40.9</td><td>21.3</td><td>1433.9</td><td>477.5</td><td>282.9</td><td>21.6</td>
  </tr>
  <tr>
    <td>kan</td><td>-</td><td>174.2</td><td>68.6</td><td>12.0</td><td>1.5</td><td>108.7</td><td>29.5</td><td>25.0</td><td>15.4</td><td>38.5</td><td>21.1</td><td>569.9</td><td>219.3</td><td>199.5</td><td>12.2</td>
  </tr>
  <tr>
    <td>mal</td><td>-</td><td>-</td><td>12.0</td><td>1.5</td><td>134.4</td><td>55.5</td><td>24.4</td><td>15.4</td><td>35.6</td><td>24.1</td><td>721.0</td><td>320.2</td><td>378.4</td><td>24.9</td>
  </tr>
  <tr>
    <td>mar</td><td>-</td><td>-</td><td>12.0</td><td>1.5</td><td>98.8</td><td>29.2</td><td>68.0</td><td>41.3</td><td>42.1</td><td>22.7</td><td>738.7</td><td>289.7</td><td>207.7</td><td>17.5</td>
  </tr>
  <tr>
    <td>api</td><td>-</td><td>-</td><td>12.0</td><td>1.5</td><td>-</td><td>-</td><td>58.8</td><td>35.4</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>340.0</td>
  </tr>
  <tr>
    <td>ory</td><td>-</td><td>-</td><td>12.0</td><td>1.5</td><td>97.1</td><td>28.5</td><td>6.5</td><td>3.5</td><td>34.0</td><td>20.0</td><td>-</td><td>-</td><td>-</td><td>-</td><td>219.6</td><td>20.0</td>
  </tr>
  <tr>
    <td>pan</td><td>-</td><td>-</td><td>12.0</td><td>1.5</td><td>113.2</td><td>1.1</td><td>19.8</td><td>11.4</td><td>78.2</td><td>8.2</td><td>-</td><td>-</td><td>-</td><td>-</td><td>231.4</td><td>19.9</td>
  </tr>
  <tr>
    <td>snd</td><td>-</td><td>-</td><td>12.0</td><td>1.5</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>tam</td><td>96.0</td><td>80.0</td><td>1224.2</td><td>478.8</td><td>12.0</td><td>1.5</td><td>51.5</td><td>18.9</td><td>69.1</td><td>39.9</td><td>41.2</td><td>19.9</td><td>1231.8</td><td>538.8</td><td>454.5</td><td>35.5</td>
  </tr>
  <tr>
    <td>tel</td><td>-</td><td>873.3</td><td>329.5</td><td>12.0</td><td>1.5</td><td>90.4</td><td>29.2</td><td>12.5</td><td>6.5</td><td>35.3</td><td>26.3</td><td>738.4</td><td>302.1</td><td>377.5</td><td>22.8</td>
  </tr>
  <tr>
    <td>urd</td><td>-</td><td>2837.1</td><td>1107.7</td><td>12.0</td><td>1.5</td><td>94.3</td><td>29.4</td><td>-</td><td>-</td><td>11.9</td><td>7.0</td><td>-</td><td>-</td><td>-</td><td>-</td><td>239.2</td><td>17.4</td>
  </tr>
  <tr>
    <th>Total</th><th>108.0</th><th>90.0</th><th>7462.3</th><th>3006.0</th><th>156.0</th><th>21</th><th>1076.7</th><th>308.6</th><th>499.0</th><th>298.7</th><th>456.4</th><th>226.5</th><th>6626.5</th><th>2603.3</th><th>3988.2</th><th>288.0</th>
  </tr>
</table>

Table A2: Overview of Indic-En spoken translation dataset across various Indic languages. The table details the hours of audio and the number of utterances collected from multiple sources. Number of utterances are in thousands (K)
<table>
  <tr>
    <th>lang</th>
    <th>En-XX</th>
    <th>XX-En</th>
  </tr>
  <tr>
    <td>asm</td>
    <td>1.51</td>
    <td>2.01</td>
  </tr>
  <tr>
    <td>ben</td>
    <td>2.51</td>
    <td>2.50</td>
  </tr>
  <tr>
    <td>guj</td>
    <td>2.51</td>
    <td>2.51</td>
  </tr>
  <tr>
    <td>hin</td>
    <td>2.50</td>
    <td>2.50</td>
  </tr>
  <tr>
    <td>kan</td>
    <td>2.51</td>
    <td>2.50</td>
  </tr>
  <tr>
    <td>mal</td>
    <td>2.13</td>
    <td>2.50</td>
  </tr>
  <tr>
    <td>mar</td>
    <td>2.51</td>
    <td>2.51</td>
  </tr>
  <tr>
    <td>npi</td>
    <td>0.50</td>
    <td>0.64</td>
  </tr>
  <tr>
    <td>ory</td>
    <td>1.50</td>
    <td>2.01</td>
  </tr>
  <tr>
    <td>pan</td>
    <td>1.02</td>
    <td>1.98</td>
  </tr>
  <tr>
    <td>tam</td>
    <td>2.51</td>
    <td>2.51</td>
  </tr>
  <tr>
    <td>tel</td>
    <td>2.51</td>
    <td>2.51</td>
  </tr>
  <tr>
    <td>urd</td>
    <td>1.50</td>
    <td>1.50</td>
  </tr>
  <tr>
    <td><i>tot.</i></td>
    <td>26.22</td>
    <td>28.18</td>
  </tr>
</table>

Table A3: Statistics of data obtained across each language in both Indic-En and En-Indic directions for BHASAANUVAAD-TEST set. For each language and in both directions, we show the number of hours.

This prompt is especially beneficial for restoring structure in spontaneous speech transcripts, facilitating improved performance in downstream tasks such as machine translation, summarization, and language modeling where well-punctuated input is essential.

C Prompts Used for Synthetic Data Generation:

Prompt for Normal Translation (Figure A2) is structured to generate neutral, grammatically correct translations faithful to the original content. It emphasizes three key principles: (1) accuracy, ensuring no information is lost, (2) natural tone, and (3) meaningful restructuring, which encourages fluent and idiomatic English rather than literal renderings. This prompt is used to simulate conventional text translation, making the output suitable for benchmark comparison with traditional MT systems.

In contrast, Prompt for Colloquial Translation (Figure A3) is tailored for generating synthetic translations that mimic spontaneous, conversational speech. It guides the model to incorporate disfluencies, fillers, contractions, and informal vocabulary, reflecting the characteristics of real-world spoken language. The prompt explicitly encourages natural phrasing and flexibility in sentence structure while preserving the core meaning. This helps in constructing datasets that better model speech patterns and are particularly valuable for training or evaluating speech-to-speech translation systems in informal contexts.

D Analysis of Mining Score Distributions

A histogram depicting the distribution of mining scores for each language pair in the dataset is illustrated in Figure A4, Figure A5, Figure A6, Figure A7, and Figure A8. These visualizations highlight a significant challenge: the SONAR model’s performance is considerably constrained when applied to low-resource languages.

For instance, the Assamese-English language pair (Figure A8a) exhibits a distribution of mining scores that is heavily skewed towards lower values, with a distinct peak frequency around a mere 0.2. Similarly, the English-Sindhi pair, as depicted in Figure A7e, also demonstrates this limitation, with its most prominent frequency peak for mining scores occurring around 0.2 to 0.3, despite some presence of higher scores. This performance with Sindhi, a low-resource language, strongly suggests that the SONAR model struggles to generate

Prompt for Punctuation Restoration

You are a expert in inserting punctuation.
Please help in adding punctuation to the following text, while strictly preserving the original words and structure.

Enhance readability by inserting only punctuation marks. Do not modify, add, or remove any words.

Follow these guidelines:\n\n
1. **Accuracy:** Ensure punctuation is applied correctly based on the language's grammatical rules.

2. **Readability:** Improve sentence clarity by inserting appropriate punctuation marks (commas, periods, question marks, etc.).

3. **Consistency:** Follow the punctuation style observed in the provided reference text.

4. **Preservation of Structure:** Do not alter word order or introduce new elements—only punctuation should be adjusted.

Reference Information:
- Language of the text: {lang}
- Sentence terminator for {lang}: {terminator}

Output Format:
Provide only the punctuated text in JSON format with the structure:

json
{"punctuated_text": "Your punctuated text here" }

Figure A1: Prompt For Punctuation Restoration
Prompt for Normal Translation

You are machine translation model.
Please help in translating the following English text to tgt_lang.

"Translate the following text from English to tgt_lang.

Follow these guidelines:\n\n"

"1. **Accuracy:** Translate the text as accurately as possible, ensuring no information is missed.\n"

"2. **Tone and Naturalness:** Maintain the original tone of the text and ensure the translation sounds natural in English.\n"

"3. **Meaningful Restructuring:** Do not translate word-for-word. Restructure sentences as needed to ensure clarity and fluency in English.\n"

"Give only the translation and nothing else. Give the output as a JSON with the structure "

Figure A2: Prompt For Normal Translation

Prompt for Colloquial Translation

You are colloquial machine translation model. Please help in translating the following English text to (tgt_lang).
I want to generate some synthetic speech data. For this, I will give you a verbatim text in (tgt_lang), and your job is to colloquially translate it to English while making it sound natural, as if it were spoken in casual conversation.

"The translation should maintain the informal tone and spoken quirks of the original Hindi, using contractions, fillers, or casual."

"phrasing that a native English speaker might use. However, please ensure that the meaning stays accurate to the original text.\n\n"

"Avoid making the English sound overly formal or stilted. The translated text should feel spontaneous and conversational, like real speech. "

"Just give the text in English and nothing else. Below are some additional suggestions: \n\n"

"1. Use contractions (e.g., 'I'm' instead of 'I am', 'you're' instead of 'you are') to make the text sound more conversational. \n"

"2. Include natural fillers (e.g., 'you know', 'like', 'well', 'I mean') where appropriate, to reflect spoken language.\n"

"3. Use colloquial vocabulary. Avoid formal or academic words; opt for simpler, everyday language.\n"

"4. Retain casual expressions that might exist in the original text, replacing them with equivalent English expressions (e.g., 'Yaar' can be translated as 'buddy' or 'man').\n"

"5. Rearrange sentence structure if needed to reflect how a native English speaker might say it naturally.\n"

"6. Ensure smooth flow between sentences, with transitions that mimic spontaneous speech (e.g., 'So', 'Anyway', 'By the way')."

Figure A3: Prompt For Colloquial Translation

consistently effective cross-lingual embeddings for under-represented languages.

In contrast, the model demonstrates commendable efficacy with higher-resource Indian languages. For language pairs like Bengali-English (Figure A8b), Gujarati-English (Figure A8c), Malayalam-English (Figure A8d), and Marathi-English (Figure A8e), the mining scores predominantly cluster at higher values, typically peaking between 0.6 and 0.8. This indicates the SONAR model’s capability to capture semantic similarities effectively when ample linguistic resources are available.

These divergent outcomes underscore a critical need: current multilingual embedding models like SONAR are not robust enough for low-resource languages. To address this, two major steps are imperative: (1) the adoption of language-specific thresholds when filtering mined data, rather than applying uniform cutoffs across all language pairs, and (2) the development of improved multilingual embedding models that can provide more accurate and reliable cross-lingual representations, particularly for under-represented languages like Assamese and Sindhi. Without such targeted improvements, data mining efforts risk perpetuating the under-representation and degraded performance of these languages in downstream tasks.
<table>
  <tr>
    <th rowspan="2">lang</th>
    <th colspan="6">Direct</th>
    <th colspan="6">Cascaded</th>
  </tr>
  <tr>
    <th colspan="2">SD</th>
    <th colspan="2">SD<sub>BA</sub></th>
    <th colspan="2">AZURE</th>
    <th colspan="2">SC</th>
    <th colspan="2">IC + IT2</th>
    <th colspan="2">SC + IT2</th>
  </tr>
  <tr>
    <th>FL</th><th>BA</th><th>FL</th><th>BA</th><th>FL</th><th>BA</th><th>FL</th><th>BA</th><th>FL</th><th>BA</th><th>FL</th><th>BA</th>
  </tr>
  <tr><td>asm</td><td>0.77</td><td>0.76</td><td>0.75</td><td>0.83</td><td>0.59</td><td>0.70</td><td>0.58</td><td>0.60</td><td>0.54</td><td>0.64</td><td>0.58</td><td>0.78</td></tr>
  <tr><td>ben</td><td>0.86</td><td>0.82</td><td>0.83</td><td>0.88</td><td>0.74</td><td>0.68</td><td>0.80</td><td>0.78</td><td>0.77</td><td>0.75</td><td>0.80</td><td>0.82</td></tr>
  <tr><td>guj</td><td>0.86</td><td>0.84</td><td>0.85</td><td>0.88</td><td>0.66</td><td>0.59</td><td>0.88</td><td>0.85</td><td>0.87</td><td>0.83</td><td>0.88</td><td>0.86</td></tr>
  <tr><td>hin</td><td>0.87</td><td>0.85</td><td>0.85</td><td>0.88</td><td>0.80</td><td>0.78</td><td>0.73</td><td>0.77</td><td>0.71</td><td>0.70</td><td>0.73</td><td>0.85</td></tr>
  <tr><td>kan</td><td>0.86</td><td>0.84</td><td>0.84</td><td>0.89</td><td>0.72</td><td>0.69</td><td>0.77</td><td>0.77</td><td>0.76</td><td>0.77</td><td>0.77</td><td>0.85</td></tr>
  <tr><td>mal</td><td>0.86</td><td>0.80</td><td>0.84</td><td>0.87</td><td>0.71</td><td>0.66</td><td>0.86</td><td>0.77</td><td>0.86</td><td>0.80</td><td>0.87</td><td>0.81</td></tr>
  <tr><td>mar</td><td>0.82</td><td>0.82</td><td>0.80</td><td>0.87</td><td>0.66</td><td>0.69</td><td>0.67</td><td>0.66</td><td>0.65</td><td>0.66</td><td>0.67</td><td>0.84</td></tr>
  <tr><td>npi</td><td>0.83</td><td>0.65</td><td>0.81</td><td>0.83</td><td>0.66</td><td>0.55</td><td>0.75</td><td>0.61</td><td>0.71</td><td>0.67</td><td>0.75</td><td>0.67</td></tr>
  <tr><td>ory</td><td>0.82</td><td>0.85</td><td>0.82</td><td>0.89</td><td>0.73</td><td>0.75</td><td>0.80</td><td>0.78</td><td>0.81</td><td>0.81</td><td>0.80</td><td>0.87</td></tr>
  <tr><td>pan</td><td>0.79</td><td>0.71</td><td>0.80</td><td>0.79</td><td>0.59</td><td>0.68</td><td>0.77</td><td>0.72</td><td>0.78</td><td>0.74</td><td>0.77</td><td>0.72</td></tr>
  <tr><td>snd</td><td>0.43</td><td>-</td><td>0.60</td><td>-</td><td>-</td><td>-</td><td>0.39</td><td>-</td><td>0.21</td><td>-</td><td>0.39</td><td>-</td></tr>
  <tr><td>tam</td><td>0.81</td><td>0.74</td><td>0.78</td><td>0.85</td><td>0.71</td><td>0.67</td><td>0.69</td><td>0.67</td><td>0.69</td><td>0.68</td><td>0.69</td><td>0.77</td></tr>
  <tr><td>tel</td><td>0.83</td><td>0.77</td><td>0.80</td><td>0.86</td><td>0.70</td><td>0.65</td><td>0.73</td><td>0.71</td><td>0.73</td><td>0.71</td><td>0.73</td><td>0.81</td></tr>
  <tr><td>urd</td><td>0.82</td><td>0.82</td><td>0.82</td><td>0.86</td><td>0.70</td><td>0.67</td><td>0.73</td><td>0.77</td><td>0.67</td><td>0.70</td><td>0.73</td><td>0.83</td></tr>
  <tr><th>total</th><th>0.80</th><th>0.79</th><th>0.80</th><th>0.86</th><th>0.69</th><th>0.67</th><th>0.73</th><th>0.73</th><th>0.70</th><th>0.73</th><th>0.73</th><th>0.81</th></tr>
</table>

Table A4: COMET (↑) scores for different direct and cascaded speech translation models evaluated on the FLEURS-TEST (FL; green) and BHASAANUVAAD-TEST (BA; yellow) datasets in \( X \times \rightarrow En \) direction. The colors represent a heatmap, where darker shades indicating better translation performance.

<table>
  <tr>
    <th rowspan="2">lang</th>
    <th colspan="6">Direct</th>
    <th colspan="6">Cascaded</th>
  </tr>
  <tr>
    <th colspan="2">SD</th>
    <th colspan="2">SD<sub>BA</sub></th>
    <th colspan="2">AZURE</th>
    <th colspan="2">SC</th>
    <th colspan="2">W + IT2</th>
    <th colspan="2">SC + IT2</th>
  </tr>
  <tr>
    <th>FL</th><th>BA</th><th>FL</th><th>BA</th><th>FL</th><th>BA</th><th>FL</th><th>BA</th><th>FL</th><th>BA</th><th>FL</th><th>BA</th>
  </tr>
  <tr><td>asm</td><td>0.59</td><td>0.49</td><td>0.56</td><td>0.62</td><td>0.49</td><td>0.50</td><td>0.42</td><td>0.31</td><td>0.42</td><td>0.35</td><td>0.42</td><td>0.56</td></tr>
  <tr><td>ben</td><td>0.77</td><td>0.59</td><td>0.74</td><td>0.73</td><td>0.69</td><td>0.61</td><td>0.77</td><td>0.46</td><td>0.78</td><td>0.50</td><td>0.77</td><td>0.67</td></tr>
  <tr><td>guj</td><td>0.86</td><td>0.65</td><td>0.86</td><td>0.79</td><td>0.80</td><td>0.65</td><td>0.79</td><td>0.46</td><td>0.80</td><td>0.52</td><td>0.79</td><td>0.71</td></tr>
  <tr><td>hin</td><td>0.74</td><td>0.67</td><td>0.74</td><td>0.75</td><td>0.64</td><td>0.65</td><td>0.86</td><td>0.64</td><td>0.85</td><td>0.68</td><td>0.86</td><td>0.71</td></tr>
  <tr><td>kan</td><td>0.72</td><td>0.57</td><td>0.72</td><td>0.72</td><td>0.66</td><td>0.60</td><td>0.70</td><td>0.45</td><td>0.69</td><td>0.50</td><td>0.70</td><td>0.64</td></tr>
  <tr><td>mal</td><td>0.85</td><td>0.77</td><td>0.84</td><td>0.84</td><td>0.79</td><td>0.76</td><td>0.82</td><td>0.63</td><td>0.82</td><td>0.64</td><td>0.82</td><td>0.81</td></tr>
  <tr><td>mar</td><td>0.66</td><td>0.48</td><td>0.64</td><td>0.62</td><td>0.60</td><td>0.51</td><td>0.72</td><td>0.42</td><td>0.71</td><td>0.46</td><td>0.72</td><td>0.58</td></tr>
  <tr><td>npi</td><td>0.83</td><td>0.48</td><td>0.84</td><td>0.71</td><td>0.73</td><td>0.51</td><td>0.79</td><td>0.32</td><td>0.8</td><td>0.38</td><td>0.79</td><td>0.59</td></tr>
  <tr><td>ory</td><td>0.83</td><td>0.58</td><td>0.80</td><td>0.73</td><td>0.74</td><td>0.56</td><td>0.66</td><td>0.33</td><td>0.67</td><td>0.37</td><td>0.66</td><td>0.66</td></tr>
  <tr><td>pan</td><td>0.82</td><td>0.49</td><td>0.82</td><td>0.57</td><td>0.76</td><td>0.64</td><td>0.74</td><td>0.29</td><td>0.74</td><td>0.39</td><td>0.74</td><td>0.50</td></tr>
  <tr><td>snd</td><td>0.69</td><td>-</td><td>0.72</td><td>-</td><td>-</td><td>-</td><td>0.65</td><td>-</td><td>0.66</td><td>0.32</td><td>0.65</td><td>-</td></tr>
  <tr><td>tam</td><td>0.68</td><td>0.55</td><td>0.67</td><td>0.67</td><td>0.62</td><td>0.59</td><td>0.71</td><td>0.46</td><td>0.72</td><td>0.50</td><td>0.71</td><td>0.63</td></tr>
  <tr><td>tel</td><td>0.75</td><td>0.56</td><td>0.74</td><td>0.71</td><td>0.71</td><td>0.62</td><td>0.73</td><td>0.45</td><td>0.74</td><td>0.49</td><td>0.73</td><td>0.62</td></tr>
  <tr><td>urd</td><td>0.73</td><td>0.60</td><td>0.73</td><td>0.72</td><td>0.66</td><td>0.59</td><td>0.78</td><td>0.49</td><td>0.78</td><td>0.54</td><td>0.78</td><td>0.65</td></tr>
  <tr><th>total</th><th>0.75</th><th>0.58</th><th>0.74</th><th>0.71</th><th>0.68</th><th>0.60</th><th>0.72</th><th>0.44</th><th>0.73</th><th>0.47</th><th>0.72</th><th>0.64</th></tr>
</table>

Table A5: COMET (↑) scores for different direct and cascaded speech translation models evaluated on the FLEURS-TEST (FL; green) and BHASAANUVAAD-TEST (BA; yellow) datasets in \( En \rightarrow XX \) direction. The colors represent a heatmap, where darker shades indicating better translation performance.
<table>
  <tr>
    <th rowspan="2">lang</th>
    <th colspan="4">Direct</th>
    <th colspan="4">Cascaded</th>
  </tr>
  <tr>
    <th colspan="2">SD</th>
    <th colspan="2">SD<sub>BA</sub></th>
    <th colspan="2">AZURE</th>
    <th colspan="2">SC</th>
    <th colspan="2">W + IT2</th>
    <th colspan="2">SC + IT2</th>
  </tr>
  <tr>
    <th>FL</th><th>BA</th><th>FL</th><th>BA</th><th>FL</th><th>BA</th><th>FL</th><th>BA</th><th>FL</th><th>BA</th><th>FL</th><th>BA</th>
  </tr>
  <tr>
    <td>asm</td><td>8.20</td><td>8.70</td><td>9.50</td><td>15.50</td><td>8.20</td><td>12.40</td><td>7.70</td><td>9.10</td><td>9.40</td><td>14.60</td><td>9.20</td><td>9.10</td>
  </tr>
  <tr>
    <td>ben</td><td>14.80</td><td>8.30</td><td>17.20</td><td>16.30</td><td>15.80</td><td>11.90</td><td>14.20</td><td>10.90</td><td>18.00</td><td>14.80</td><td>17.60</td><td>10.90</td>
  </tr>
  <tr>
    <td>guj</td><td>16.80</td><td>11.80</td><td>19.50</td><td>21.00</td><td>16.90</td><td>14.70</td><td>16.40</td><td>14.70</td><td>18.90</td><td>20.00</td><td>18.50</td><td>14.70</td>
  </tr>
  <tr>
    <td>hin</td><td>28.70</td><td>11.50</td><td>31.60</td><td>29.50</td><td>29.40</td><td>25.20</td><td>28.70</td><td>23.30</td><td>30.30</td><td>27.50</td><td>30.30</td><td>23.30</td>
  </tr>
  <tr>
    <td>kan</td><td>11.90</td><td>9.60</td><td>14.80</td><td>18.10</td><td>13.90</td><td>12.50</td><td>11.40</td><td>11.90</td><td>15.00</td><td>16.50</td><td>14.50</td><td>11.90</td>
  </tr>
  <tr>
    <td>mal</td><td>9.50</td><td>8.40</td><td>13.00</td><td>17.80</td><td>12.50</td><td>13.50</td><td>9.60</td><td>11.30</td><td>14.20</td><td>16.10</td><td>13.00</td><td>11.30</td>
  </tr>
  <tr>
    <td>mar</td><td>11.10</td><td>7.60</td><td>12.30</td><td>15.50</td><td>10.70</td><td>10.80</td><td>10.40</td><td>10.90</td><td>13.30</td><td>14.60</td><td>12.80</td><td>10.90</td>
  </tr>
  <tr>
    <td>npi</td><td>14.10</td><td>6.90</td><td>16.00</td><td>15.00</td><td>12.70</td><td>5.80</td><td>13.00</td><td>9.70</td><td>17.3</td><td>13.30</td><td>16.90</td><td>9.70</td>
  </tr>
  <tr>
    <td>ory</td><td>13.30</td><td>7.50</td><td>15.30</td><td>12.50</td><td>14.30</td><td>6.10</td><td>11.40</td><td>8.80</td><td>12.20</td><td>9.30</td><td>11.90</td><td>8.80</td>
  </tr>
  <tr>
    <td>pan</td><td>24.00</td><td>11.60</td><td>26.10</td><td>17.30</td><td>22.90</td><td>10.20</td><td>22.80</td><td>12.60</td><td>25.70</td><td>16.20</td><td>25.20</td><td>12.60</td>
  </tr>
  <tr>
    <td>snd</td><td>20.50</td><td>—</td><td>23.60</td><td>—</td><td>—</td><td>19.80</td><td>—</td><td>16.80</td><td>—</td><td>16.20</td><td>—</td><td>—</td>
  </tr>
  <tr>
    <td>tam</td><td>18.30</td><td>6.30</td><td>14.90</td><td>16.60</td><td>12.90</td><td>13.80</td><td>11.00</td><td>9.80</td><td>13.60</td><td>16.60</td><td>13.30</td><td>9.80</td>
  </tr>
  <tr>
    <td>tel</td><td>15.80</td><td>7.60</td><td>18.70</td><td>17.00</td><td>15.20</td><td>10.60</td><td>15.30</td><td>9.80</td><td>18.80</td><td>15.40</td><td>17.90</td><td>9.80</td>
  </tr>
  <tr>
    <td>urd</td><td>21.80</td><td>19.70</td><td>23.00</td><td>24.90</td><td>22.10</td><td>20.50</td><td>21.20</td><td>22.50</td><td>23.60</td><td>27.30</td><td>23.90</td><td>22.50</td>
  </tr>
  <tr>
    <th colspan="2">total</th><th colspan="2">16.34</th><th colspan="2">9.65</th><th colspan="2">18.25</th><th colspan="2">15.96</th><th colspan="2">12.92</th><th colspan="2">15.21</th><th colspan="2">12.72</th><th colspan="2">17.65</th><th colspan="2">17.09</th><th colspan="2">17.23</th><th colspan="2">12.72</th>
  </tr>
</table>

Table A6: BLEU (↑) scores for different direct and cascaded speech translation models evaluated on the FLEURS-TEST (FL; green) and BHASAANUVAAAD-TEST (BA; yellow) datasets in En → XX direction. The colors represent a heatmap, where darker shades indicating better translation performance.

<table>
  <tr>
    <th rowspan="2">lang</th>
    <th colspan="4">Direct</th>
    <th colspan="4">Cascaded</th>
  </tr>
  <tr>
    <th colspan="2">SD</th>
    <th colspan="2">SD<sub>BA</sub></th>
    <th colspan="2">AZURE</th>
    <th colspan="2">SC</th>
    <th colspan="2">W + IT2</th>
    <th colspan="2">SC + IT2</th>
  </tr>
  <tr>
    <th>FL</th><th>BA</th><th>FL</th><th>BA</th><th>FL</th><th>BA</th><th>FL</th><th>BA</th><th>FL</th><th>BA</th><th>FL</th><th>BA</th>
  </tr>
  <tr>
    <td>asm</td><td>14.70</td><td>21.40</td><td>22.40</td><td>29.50</td><td>12.80</td><td>16.70</td><td>17.00</td><td>23.60</td><td>17.90</td><td>26.40</td><td>16.90</td><td>23.60</td>
  </tr>
  <tr>
    <td>ben</td><td>18.40</td><td>21.80</td><td>27.40</td><td>35.20</td><td>19.40</td><td>18.30</td><td>21.00</td><td>29.50</td><td>22.20</td><td>29.80</td><td>22.00</td><td>23.30</td>
  </tr>
  <tr>
    <td>guj</td><td>22.80</td><td>27.20</td><td>33.50</td><td>37.70</td><td>19.40</td><td>14.20</td><td>21.60</td><td>25.50</td><td>27.70</td><td>34.30</td><td>26.70</td><td>29.50</td>
  </tr>
  <tr>
    <td>hin</td><td>20.30</td><td>25.40</td><td>33.40</td><td>37.50</td><td>23.40</td><td>24.70</td><td>23.70</td><td>27.80</td><td>26.70</td><td>35.20</td><td>26.90</td><td>27.80</td>
  </tr>
  <tr>
    <td>kan</td><td>17.30</td><td>21.10</td><td>27.60</td><td>32.70</td><td>17.50</td><td>15.40</td><td>19.80</td><td>25.70</td><td>21.80</td><td>31.30</td><td>20.50</td><td>25.70</td>
  </tr>
  <tr>
    <td>mal</td><td>18.20</td><td>18.80</td><td>28.70</td><td>31.20</td><td>18.70</td><td>12.50</td><td>22.10</td><td>21.50</td><td>24.60</td><td>26.80</td><td>23.80</td><td>21.50</td>
  </tr>
  <tr>
    <td>mar</td><td>18.30</td><td>21.20</td><td>28.30</td><td>33.40</td><td>17.80</td><td>14.10</td><td>21.80</td><td>24.70</td><td>24.50</td><td>29.90</td><td>22.00</td><td>24.70</td>
  </tr>
  <tr>
    <td>npi</td><td>21.40</td><td>14.80</td><td>30.50</td><td>29.90</td><td>20.20</td><td>10.90</td><td>24.90</td><td>16.20</td><td>26.50</td><td>22.20</td><td>27.00</td><td>16.20</td>
  </tr>
  <tr>
    <td>ory</td><td>18.30</td><td>25.20</td><td>27.20</td><td>34.70</td><td>19.90</td><td>19.90</td><td>23.20</td><td>29.70</td><td>25.90</td><td>33.40</td><td>24.00</td><td>29.70</td>
  </tr>
  <tr>
    <td>pan</td><td>20.30</td><td>24.10</td><td>30.30</td><td>29.20</td><td>15.60</td><td>21.60</td><td>23.70</td><td>28.80</td><td>25.70</td><td>24.90</td><td>25.70</td><td>28.80</td>
  </tr>
  <tr>
    <td>snd</td><td>6.30</td><td>—</td><td>15.30</td><td>—</td><td>—</td><td>8.00</td><td>—</td><td>12.70</td><td>—</td><td>10.30</td><td>—</td><td>—</td>
  </tr>
  <tr>
    <td>tam</td><td>15.50</td><td>17.80</td><td>24.20</td><td>29.50</td><td>17.50</td><td>15.50</td><td>18.10</td><td>21.00</td><td>20.90</td><td>26.00</td><td>19.70</td><td>21.00</td>
  </tr>
  <tr>
    <td>tel</td><td>18.10</td><td>19.90</td><td>27.40</td><td>31.30</td><td>19.60</td><td>15.80</td><td>22.80</td><td>22.90</td><td>26.90</td><td>29.10</td><td>25.10</td><td>22.90</td>
  </tr>
  <tr>
    <td>urd</td><td>17.50</td><td>27.00</td><td>27.30</td><td>41.00</td><td>18.50</td><td>21.70</td><td>20.70</td><td>29.30</td><td>22.50</td><td>35.60</td><td>23.80</td><td>29.30</td>
  </tr>
  <tr>
    <th colspan="2">total</th><th colspan="2">17.67</th><th colspan="2">21.98</th><th colspan="2">27.39</th><th colspan="2">33.29</th><th colspan="2">18.48</th><th colspan="2">16.90</th><th colspan="2">20.88</th><th colspan="2">24.92</th><th colspan="2">23.32</th><th colspan="2">29.61</th><th colspan="2">22.46</th><th colspan="2">24.92</th>
  </tr>
</table>

Table A7: BLEU (↑) scores for different direct and cascaded speech translation models evaluated on the FLEURS-TEST (FL; green) and BHASAANUVAAAD-TEST (BA; yellow) datasets in XX → En direction. The colors represent a heatmap, where darker shades indicating better translation performance.
(a) Seamless Align - Hindi
(b) BHASAANUVAAD- Hindi
(c) Seamless Align - Kannada
(d) BHASAANUVAAD- Kannada
(e) Seamless Align - Tamil
(f) BHASAANUVAAD- Tamil
(g) Seamless Align - Telugu
(h) BHASAANUVAAD- Telugu

Figure A4: Distribution of SONAR mining scores for XX → En sentence pairs from the SeamlessAlign (left) and BHASAANUVAAD (right) datasets across four languages: Hindi, Kannada, Tamil, and Telugu. BHASAANUVAAD consistently exhibits a higher concentration of high-quality alignments, with mining scores skewed toward the upper end of the scale. In contrast, SeamlessAlign distributions are centered around lower mining scores, indicating overall lower alignment quality. This comparison underscores the improved precision of alignments in BHASAANUVAAD .
(a) Seamless Align - Urdu
(b) BHASAANUVAAD- Urdu

Figure A5: Distribution of SONAR mining scores for XX → En sentence pairs from the SeamlessAlign (left) and BHASAANUVAAD (right) datasets across four languages: Hindi, Kannada, Tamil, and Telugu. BHASAANUVAAD consistently exhibits a higher concentration of high-quality alignments, with mining scores skewed toward the upper end of the scale. In contrast, SeamlessAlign distributions are centered around lower mining scores, indicating overall lower alignment quality.

(a) English - Assamese
(b) English - Bengali
(c) English - Gujarati
(d) English - Hindi
(e) English - Kannada
(f) English - Malayalam

Figure A6: Histogram of SONAR mining scores for En → XX language pairs in the BHASAANUVAAD dataset. The distributions indicate a strong skew toward high-quality alignments (scores above 0.6) across most language pairs, reflecting the effectiveness of the mining and filtering pipeline used in BHASAANUVAAD.
Figure A7: Histogram of SONAR mining scores for En → XX language pairs in the BHASAANUVAAD dataset. The distributions indicate a strong skew toward high-quality alignments (scores above 0.6) across most language pairs, reflecting the effectiveness of the mining and filtering pipeline used in BHASAANUVAAD (contd).
Figure A8: Histogram of SONAR mining scores for XX → Any language pairs in the BHASAANUVAAD dataset. The distributions indicate a strong skew toward high-quality alignments (scores above 0.6) across most language pairs, reflecting the effectiveness of the mining and filtering pipeline used in BHASAANUVAAD.

Metareview score: 2.5
Metareview: This paper presents a large dataset for Speech machine translation for 13 Indian languages. Additionally, the authors present results for a speech translation model with the created dataset. 
One reviewer mentioned the limited scope of the study, which only included Indian languages. However, when most research only focuses on English, a study that included 13 languages from two language families (Dravidian and Indo-European), I believe that the focus on Indian languages solely is not a limitation.
Several comments include the lack of details about the dataset collected. Comments include a lack of statistical details (which are mentioned to some extent in the Appendix), inconsistencies about the number of languages mentioned and a lack of details on how the datasets have been collected. The reason why one language from VaaniPedia is unclear.
Several reviewers also mention the lack of details and quality control on the force alignement part of the creation of the dataset.
The evaluation of the MT system only includes the ChRF++ score, which is insufficient. 
It is also not clear to me if the results provided on Machine translation outperform results in the existing literature
Summary of reasons to publish: - Very Large dataset on 13 (14?) Indian languages for Speech Machine Translation purposes
- Large Translation model