GenAI4ELab/papercli-papers huggingface dataset:
Improving child speech recognition with augmented child-like speech
Authors: Yuanyuan Zhang; Zhengjun Yue; Tanvina Patel; Odette Scharenborg
Venue: Interspeech 2024 (main)

Abstract: State-of-the-art ASRs show suboptimal performance for child speech. The scarcity of child speech limits the development of child speech recognition (CSR). Therefore, we studied child-to-child voice conversion (VC) from existing child speakers in the dataset and additional (new) child speakers via monolingual and cross-lingual (Dutch-to-German) VC, respectively. The results showed that cross-lingual child-to-child VC significantly improved child ASR performance. Experiments on the impact of the quantity of child-to-child cross-lingual VC-generated data on fine-tuning (FT) ASR models gave the best results with two-fold augmentation for our FT-Conformer model and FT-Whisper model which reduced WERs with â¼3% absolute compared to the baseline, and with six-fold augmentation for the model trained from scratch, which improved by an absolute 3.6% WER. Moreover, using a small amount of âhigh-qualityâ VC-generated data achieved similar results to those of our best-FT models.
Mixed Children/Adult/Childrenized Fine-Tuning for Childrenâs ASR: How to Reduce Age Mismatch and Speaking Style Mismatch
Authors: Thomas Graave; Zhengyang Li; Timo Lohrenz; Tim Fingscheidt
Venue: Interspeech 2024 (main)

Abstract: Todayâs end-to-end (E2E) ASR models achieve strong performance when applied to adult speech, but deteriorate on childrenâs speech. Most E2E ASR models are pre-trained on adult speech, which introduces an age mismatch that can be addressed by finetuning on child data. However, due to limited availability of child datasets, fine-tuning on childrenâs speech may introduce new domain shifts such as speaking style mismatch. In this work, we explore mixed fine-tuning on partially matched data, namely read adult speech and spontaneous childrenâs speech, to improve the performance of E2E ASR on read childrenâs speech. We isolate the individual impact of age mismatch and speaking style mismatch and investigate the use of childrenization of read adult speech. Our proposed method reduces the WER by up to 5% absolute (21% relative) compared to the pre-trained E2E ASR and by roughly 3% absolute (15% relative) compared to individual fine-tuning on partially matched datasets.
Exploring Speech Foundation Models for Speaker Diarization in Child-Adult Dyadic Interactions
Authors: Anfeng Xu; Kevin Huang; Tiantian Feng; Lue Shen; Helen Tager-Flusberg; Shrikanth Narayanan
Venue: Interspeech 2024 (main)

Abstract: Speech foundation models, trained on vast datasets, have opened unique opportunities in addressing challenging low-resource speech understanding, such as child speech. In this work, we explore the capabilities of speech foundation models on child-adult speaker diarization. We show that exemplary foundation models can achieve 39.5% and 62.3% relative reductions in Diarization Error Rate and Speaker Confusion Rate, respectively, compared to previous speaker diarization methods. In addition, we benchmark and evaluate the speaker diarization results of the speech foundation models with varying the input audio window size, speaker demographics, and training data ratio. Our results highlight promising pathways for understanding and adopting speech foundation models to facilitate child speech understanding.
IIITH Ucchar e-Sudharak: an automatic English pronunciation corrector for school-going children with a teacher in the loop
Authors: Meenakshi Sirigiraju; Arjun Rajasekar; Abhishikth Meejuri; Chiranjeevi Yarra
Venue: Interspeech 2024 (main)

Abstract: The language acquisition skills are predominant during childhood. Due to the lack of resources, a gap often arises in the language communication abilities of students. To bridge this gap, it is necessary to consider automatic methods for improving the children language skills. Our IIITH Ucchar e-Sudharak web based tool, deployed in a real environment, helps school-going children to practice English reading skills with a teacher in the loop. The choice of English as the target language is strategic, given its status as a global lingua franca. Within the tool, children are provided with class and chapter-specific English sentences for practice tailored by their teachers. During practice sessions, the tool provides real-time feedback in the form of sentence and word-level scores, which are computed by comparing against teacher (expert) audio recordings. These scores offer insights into word-level pronunciation correctness and sentence-level quality by measuring fluency. Moreover, the tool provides a key feature where teachers can assign specific sentences to individual students and assess their performance. This functionality empowers teachers to personalize their instruction, catering to the specific needs of each student. As far as we know, this is a one-of-a-kind tool specifically designed to meet the needs of school-going children.
Speech enabled visual acuity test
Authors: Boon Peng Yap; Kok Liang Tan; Zhenghao Li; Rong Tong
Venue: Interspeech 2024 (main)

Abstract: This paper introduces a novel automatic visual acuity test system employing automatic speech recognition and image recognition. The system provides users a private and self-paced eye assessment experience. During such visual acuity test, the system communicates with user via speech interaction: the system prompts user to speak out the letters shown and recognizes userâs speech input to assess userâs eye sights. The system is also able to automatically detect userâs posture and notify user when incorrect posture is detected. When the testing process is completed, the system will analyze the result automatically and record the details in database.
A ChatGPT-based oral Q&A practice system for first-time student participants in international conferences
Authors: Mayuko Aiba; Daisuke Saito; Nobuaki Minematsu
Venue: Interspeech 2024 (main)

Abstract: A ChatGPT-based system was developed to enhance studentsâ oral Q&A skills at international conferences. By uploading their original papers to ChatGPT in the system, it generates oral questions to the students, who can practice Q&A. We examined six configurations for question generation: 1) whether reference papers should also be explicitly uploaded to the system, and 2) whether or not the questions should be generated for each keyword or each section in sequence. Surprisingly, all the students and teachers who participated in the experiment chose the simplest configuration â no explicit upload of reference papers and question generation from the entire scope of the paper. This finding led us to combine the simply-configured ChatGPT with a speech interface to create an interactive practice environment. Our system orally poses questions to a student about their papers, who responds verbally. After using this environment, all the participants strongly affirmed its effectiveness of practicing Q&A to attend future international conferences.
Visual scene display application for augmentative and alternative communication
Authors: Karthik Venkat Sridaran; Raja Praveen; Reuben T Varghese; Ajish K Abraham; Shankar R; Winnie Rachel Cherian
Venue: Interspeech 2024 (main)

Abstract: Augmentative and Alternative Communication (AAC) refers to various strategies and tools used to enhance communication for individuals with limited speech or language abilities. Visual Scene Displays (VSDs) have emerged as a promising approach within AAC, leveraging visual cues and contextualized scenes to support communication and language development. This show and tell presentation describes the development of a Visual Scene Display application, implemented as a webpage. The application offers a user-friendly interface that allows individuals to create and customize their own scenes by uploading images or capturing real-life scenes using the device's camera. The system incorporates a robust symbol library, including pictograms, icons, and audio to support comprehension and expression. The development process involved iterative design, usability testing, and feedback from AAC professionals, individuals with communication challenges, and their caregivers. This collaborative approach ensured that the VSD system addressed the diverse needs and abilities of its target users. Preliminary evaluations of the VSD system have shown promising results, indicating its potential to enhance communication and language development among individuals with limited speech or language abilities.
CALL system using pitch-accent feature representations reflecting listenersâ subjective adequacy
Authors: Ikuyo Masuda-Katsuse; Ayako Shirose
Venue: Interspeech 2024 (main)

Abstract: This paper presents a CALL system that implements a method for automatically and quantitatively evaluating the pitch accents of spoken words of Japanese learners in terms of how well they are accepted by native speakers.
The speech motor chaining web app for speech motor learning
Authors: Jonathan L Preston; Nina R Benway; Nathan Prestopnik; Nathan Preston
Venue: Interspeech 2024 (main)

Abstract: Systematic review has indicated that existing computerized speech learning apps show unclear potential to invoke speech change, highlighting the need to incorporate tested theoretical principles in app design. As speech is a motor skill, this paper illustrates how the principles of motor learning underlie the Speech Motor Chaining speech therapy web app. Features such as game design, telepractice, and speech analysis are discussed.
Visualization for improving foreign language pronunciation
Authors: Charlotte Yoder; Karrie Karahalios; Mark Hasegawa-Johnson; Shreyansh Agrawal
Venue: Interspeech 2024 (main)

Abstract: Speaking coherently in a second language requires knowledge and execution of pronunciation. Often, pronunciation training is seldom emphasized in language learning. However, explicitly teaching pronunciation can increase confidence and attitudes towards pronunciation in learners. We present a visual feedback system for vowels using a speaker calibrated vowel chart. The demo includes vowel chart calibration, an interactive tutorial for learning how to read a vowel chart, and a practice page for three vowel minimal pairs.
CaptainA self-study mobile app for practising speaking: task completion assessment and feedback with generative AI
Authors: Nhan Phan; Anna von Zansen; Maria Kautonen; TamÃ¡s GrÃ³sz; Mikko Kurimo
Venue: Interspeech 2024 (main)

Abstract: We introduce the CaptainA mobile app, designed to meet the needs of second language (L2) learners engaged in self-study of Finnish, with potential applicability to other languages. Our app can provide automatic speaking assessment (ASA) of task completion in picture-based tasks, along with grading explanations and corrective feedback. It can also automatically generate pictures for visual tasks, providing users with unlimited practice opportunities. The mobile app is based on our framework that combines visual natural language generation
(NLG), automatic speech recognition (ASR), and prompting large language model (LLM) for low-resource language. Our goal is to promote the development of next-generation speech-based computer-assisted language learning (CALL) systems capable of providing automatic scoring with feedback for learners, even when minimal speech data of L2 learners is available. While the mobile app demonstration is designed for Finnish, the app can also be tested in English.
Speech transcription from South Tyrolean Dialect to Standard German with Whisper
Authors: Luca Ducceschi; Greta H. Franzini
Venue: Interspeech 2025 (main)

Abstract: This study presents the first fine-tuned Whisper model for the automatic translation of South Tyrolean dialectal speech into Standard German text. To address an unmet need for subtitling and translation, we introduce a small corpus of manually annotated and synthetic speech data compiled for this task. Through fine-tuning and hyperparameter optimisation, our model achieves a BLEU score of 86.18 significantly outperforming baseline error rates. Our findings highlight Whisper's effectiveness in handling dialectal speech, contributing to low-resource language research. The model is already being used in a heritage collaboration for large-scale translation of audiovisual archival material and is also being considered for application in news broadcasting and tourism promotion. Future directions include expanding the training data and extending hyperparameter optimisation to improve the model's performance and generalisation across South Tyrolean dialectal variations.
Length Aware Speech Translation for Video Dubbing
Authors: Aswin Shanmugam Subramanian; Harveen Chadha; Vikas Joshi; Shubham Bansal; Jian Xue; Rupeshkumar Mehta; Jinyu Li
Venue: Interspeech 2025 (main)

Abstract: In video dubbing, aligning translated audio with the source audio is a significant challenge. Our focus is on achieving this efficiently, tailored for real-time, on-device video dubbing scenarios. We developed a phoneme-based end-to-end length-sensitive speech translation (LSST) model, which generates translations of varying lengths - short, normal, and long - using predefined tags. Additionally, we introduced length-aware beam search (LABS), an efficient approach to generate translations of different lengths in a single decoding pass. This approach maintained comparable BLEU scores compared to a baseline without length awareness while significantly enhancing synchronization quality between source and target audio, achieving a mean opinion score (MOS) gain of 0.34 for Spanish and 0.65 for Korean, respectively.
ArticulateX: End-to-End Monolingual Speech Translation in Articulator Space
Authors: Vishal Kumar; Vinayak Abrol
Venue: Interspeech 2025 (main)

Abstract: We present ArticulateX, the first non-autoregressive direct speech-to-speech translation (S2ST) model that operates through an articulatory latent space, offering an efficient alternative to existing cascaded models. It consists of a direct speech-to-articulator encoder, a latent articulator-to-MelSpectrogram mapper, and a vocoder for high-fidelity speech synthesis. By leveraging articulatory representations, which are inherently language-agnostic, our model effectively captures speech dynamics, preserving speaker identity, prosody and expressiveness across languages. Unlike prior autoregressive models, ArticulateX eliminates the need for intermediate text, discrete units and/or complex self-supervised objectives, enabling faster inference, stable training, and improved translation quality. We demonstrate the efficacy of the proposed model in fr-en and de-en speech-to-speech translation on the CVSS dataset, achieving BLEU scores better or comparable to existing models.
CMSP-ST: Cross-modal Mixup with Speech Purification for End-to-End Speech Translation
Authors: Jiale Ou; Hongying Zan
Venue: Interspeech 2025 (main)

Abstract: End-to-end speech translation (E2E ST) aims to directly convert speech in a source language into text in a target language, and its performance is constrained by the inherent modality gap. Existing methods attempt to align speech and text representations to perform cross-modal mixup at the token level, which overlooks the impact of redundant speech information. In this paper, we propose cross-modal mixup with speech purification for speech translation (CMSP-ST) to address this issue. Specifically, we remove the non-content features from speech through orthogonal projection and extract the purified speech features for cross-modal mixup. Additionally, we employ adversarial training under the Soft Alignment (S-Align) to relax the alignment granularity and improve robustness. Experimental results on the MuST-C En-De, CoVoST-2 Fr-En, and CoVoST-2 De-En benchmarks demonstrate that CMSP-ST effectively improves the speech translation performance of existing cross-modal mixup methods.
End-to-End Speech Translation Guided by Robust Translation Capability of Large Language Model
Authors: Yosuke Higuchi; Tetsuji Ogawa; Tetsunori Kobayashi
Venue: Interspeech 2025 (main)

Abstract: We present an end-to-end speech translation (ST) model that uses a large language model (LLM) to guide the translation process. Recent advances in LLMs have shown strong contextual understanding and robustness to noisy text, making them beneficial for mitigating automatic speech recognition (ASR) errors. Building on these strengths, we develop an LLM-driven ST model within an encoder-decoder framework, with the encoder handling an auxiliary ASR task and the decoder incorporating an LLM at its front end. Here, the encoder generates an ASR hypothesis that cues the LLM to perform machine translation. The LLM output is then fed into the decoder to yield the final translation. This two-pass design capitalizes on the LLM's robust and accurate translation capabilities, while enabling end-to-end optimization tailored to specific ST tasks. Experimental results on various ST tasks reveal significant performance gains with our LLM integration, and extensive analyses further validate our approach.
Empowering Large Language Models for End-to-End Speech Translation Leveraging Synthetic Data
Authors: Yu Pu; Xiaoqian Liu; Guangyu Zhang; Zheng Yan; Wei-Qiang Zhang; Xie Chen
Venue: Interspeech 2025 (main)

Abstract: Speech-to-speech translation (S2ST) is a key technology for seamless cross-lingual communication. Traditional cascaded systems, which involve speech recognition, text translation, and speech synthesis, are prone to error propagation and latency. In this work, we present SLAM-TR, an end-to-end speech translation model which directly map input speech to output speech, eliminating the need for intermediate text representations. By fine-tuning from the large language model Qwen2-0.5B, SLAM-TR achieves superior performance over the cascaded baseline and state-of-the-art open-source models with minimal training time. Additionally, SLAM-TR demonstrates strong generalization, achieving an ASR-BLEU score of 8.20 on the FLEURS benchmark, outperforming both cascaded and open-source systems. In addition, addressing the challenge of limited natural speech translation data, we propose SynStard-1000, a 1,000-hour synthetic speech translation dataset.
Speech-to-Text Translation with Phoneme-Augmented CoT: Enhancing Cross-Lingual Transfer in Low-Resource Scenarios
Authors: Gerard I. GÃ¡llego; Oriol Pareras; MartÃ­ Cortada Garcia; Lucas Takanori; Javier Hernando
Venue: Interspeech 2025 (main)

Abstract: We propose a Speech-to-Text Translation (S2TT) approach that integrates phoneme representations into a Chain-of-Thought (CoT) framework to improve translation in low-resource and zero-resource settings. By introducing phoneme recognition as an intermediate step, we enhance cross-lingual transfer, enabling translation even for languages with no labeled speech data. Our system builds on a multilingual LLM, which we extend to process speech and phonemes. Training follows a curriculum learning strategy that progressively introduces more complex tasks. Experiments on multilingual S2TT benchmarks show that phoneme-augmented CoT improves translation quality in low-resource conditions and enables zero-resource translation, while slightly impacting high-resource performance. Despite this trade-off, our findings demonstrate that phoneme-based CoT is a promising step toward making S2TT more accessible across diverse languages.
Scheduled Interleaved Speech-Text Training for Speech-to-Speech Translation with LLMs
Authors: Hayato Futami; Emiru Tsunoo; Yosuke Kashiwagi; Yuki Ito; Hassan Shahmohammadi; Siddhant Arora; Shinji Watanabe
Venue: Interspeech 2025 (main)

Abstract: Speech-to-speech translation (S2ST) has been advanced with large language models (LLMs), which are fine-tuned on discrete speech units. In such approaches, modality adaptation from text to speech has been an issue. LLMs are trained on text-only data, which presents challenges to adapt them to speech modality with limited speech-to-speech data. To address the training difficulty, we propose scheduled interleaved speech-text training in this study. We use interleaved speech-text units instead of speech units during training, where aligned text tokens are interleaved at the word level. We gradually decrease the ratio of text as training progresses, to facilitate progressive modality adaptation from text to speech. We conduct experimental evaluations by fine-tuning LLaMA3.2-1B for S2ST on the CVSS dataset. We show that the proposed method consistently improves the translation performances, especially for languages with limited training data.
End-to-End Speech Translation for Low-Resource Languages Using Weakly Labeled Data
Authors: Aishwarya Pothula; Bhavana Akkiraju; Srihari Bandarupalli; Charan D; Santosh Kesiraju; Anil Kumar Vuppala
Venue: Interspeech 2025 (main)

Abstract: The scarcity of high-quality annotated data presents a significant challenge in developing effective end-to-end speech-to-text translation (ST) systems, particularly for low-resource languages. This paper explores the hypothesis that weakly labeled data can be used to build ST models for low-resource language pairs. We constructed speech-to-text translation datasets with the help of bitext mining using state-of-the-art sentence encoders. We mined the multilingual Shrutilipi corpus to build Shrutilipi-anuvaad, a dataset comprising ST data for language pairs Bengali-Hindi, Malayalam-Hindi, Odia-Hindi, and Telugu-Hindi. We created multiple versions of training data with varying degrees of quality and quantity to investigate the effect of quality versus quantity of weakly labeled data on ST model performance. Results demonstrate that ST systems can be built using weakly labeled data, with performance comparable to massive multi-modal multilingual baselines such as SONAR and SeamlessM4T.
Self-Improvement for Audio Large Language Model using Unlabeled Speech
Authors: Shaowen Wang; Xinyuan Chen; Yao Xu
Venue: Interspeech 2025 (main)

Abstract: Recent audio LLMs have emerged rapidly, demonstrating strong generalization across various speech tasks. However, given the inherent complexity of speech signals, these models inevitably suffer from performance degradation in specific target domains. To address this, we focus on enhancing audio LLMs in target domains without any labeled data. We propose a self-improvement method called SI-SDA, leveraging the information embedded in large-model decoding to evaluate the quality of generated pseudo labels and then perform domain adaptation based on reinforcement learning optimization. Experimental results show that our method consistently and significantly improves audio LLM performance, outperforming existing baselines in WER and BLEU across multiple public datasets of automatic speech recognition (ASR), spoken question-answering (SQA), and speech-to-text translation (S2TT). Furthermore, our approach exhibits high data efficiency, underscoring its potential for real-world deployment.
Optimized Real-time Speech Enhancement with Deep SSMs on Raw Audio
Authors: Yan Ru Pei; Ritik Shrivastava; Fnu Sidharth
Venue: Interspeech 2025 (main)

Abstract: We present aTENNuate, a simple deep state-space autoencoder configured for efficient online raw speech enhancement in an end-to-end fashion. The network's performance is primarily evaluated on raw speech denoising, with additional assessments on tasks such as super-resolution and de-quantization. We benchmark aTENNuate on the VoiceBank + DEMAND and the Microsoft DNS1 synthetic test sets. The network outperforms previous real-time denoising models in terms of PESQ score, parameter count, MACs, and latency. Even as a raw waveform processing model, the model maintains high fidelity to the clean signal with minimal audible artifacts. In addition, the model remains performant even when the noisy input is compressed down to 4000Hz and 4 bits, suggesting general speech enhancement capabilities in low-resource environments.