GenAI4ELab/papercli-papers huggingface dataset:
Can Multimodal Foundation Models Help Analyze Child-Inclusive Autism Diagnostic Videos?
Authors: Aditya Kommineni; Digbalay Bose; Tiantian Feng; So Hyun Kim; Helen Tager-Flusberg; Somer Bishop; Catherine Lord; Sudarsana Kadiri; Shrikanth Narayanan
Venue: Interspeech 2025 (main)

Abstract: Multimodal foundation models have paved the way for a paradigm shift in long video understanding. The extent to which these models can help analyze verbal and non-verbal behaviors in the context of human interactions is underexplored, particularly in the challenging settings of clinical diagnosis and treatment. We investigate the use of foundation models across speech, video, and text modalities to analyze child-focused interactions in the context of autism diagnosis. We evaluate model performance in two related tasks, i.e. activity understanding and atypical behavior detection. We further propose a unified methodology for merging information from audio and video streams by leveraging large language models as reasoning agents. Our experiments reveal that, while models perform relatively well for coarse-grained tasks such as activity recognition and over-activity identification, they fail to generalize to fine-grained tasks such as anxiety detection and activity segmentation.
A Cascaded Multimodal Framework for Automatic Social Communication Severity Assessment in Children with Autism Spectrum Disorder
Authors: Jihyun Mun; Sunhee Kim; Minhwa Chung
Venue: Interspeech 2025 (main)

Abstract: Autism Spectrum Disorder (ASD) is a neurodevelopmental condition characterized by deficits in social communication, affecting both language use and speech patterns. Since assessment relies on behavioral observations rather than standardized medical tests, developing an objective evaluation method is essential. Recognizing that ASD impacts both language and speech production, this study proposes a cascaded multimodal framework for ASD severity assessment. The framework processes raw audio, generates transcriptions via automatic speech recognition, and extracts linguistic and acoustic features using speech-language foundation models. Given the atypical suprasegmental and segmental speech characteristics in ASD, two speech foundation models are employed. A co-attention mechanism then integrates these representations to estimate severity. Achieving a Spearmanâs correlation of 0.5629 with human ratings, the proposed approach offers a scalable, fully automated ASD assessment tool.
Accessible Real-time Eye-gaze Tracking for Neurocognitive Health Assessment: A Multimodal Web-based Approach
Authors: Daniel Tisdale; Jackson Liscombe; David Paulter; Michael Neumann; Vikram Ramanarayanan
Venue: Interspeech 2025 (main)

Abstract: We introduce a novel integration of real-time predictive eye-gaze tracking models into a scalable cloud-based multimodal dialogue system tailored for remote health assessments. A virtual human guide interacted with 10 participants with Mild Cognitive Impairment (MCI) and 29 healthy controls during a crowdsourced pilot study. She engaged participants in an approximately 10-minute interview with 11 interactive eye-gaze tasks, ranging from simple free gaze exploration to more specialized directed gaze tasks. We found that metrics of eye-gaze dynamics and reaction times extracted from these tasks, combined with an Adaboost classifier effectively differentiates MCI participants from healthy adults, with an average accuracy of 0.94. Furthermore, over 80% of participants reported high engagement and liked the user experience with the platform, across demographics and internet connections, demonstrating its robustness and scalability for real world use.
Multimodal Biomarkers for Schizophrenia: Towards Individual Symptom Severity Estimation
Authors: Gowtham Premananth; Philip Resnik; Sonia Bansal; Deanna L. Kelly; Carol Espy-Wilson
Venue: Interspeech 2025 (main)

Abstract: Studies on schizophrenia assessments using deep learning typically treat it as a classification task to detect the presence or absence of the disorder, oversimplifying the condition and reducing its clinical applicability. This traditional approach overlooks the complexity of schizophrenia, limiting its practical value in healthcare settings. This study shifts the focus to individual symptom severity estimation using a multimodal approach that integrates speech, video, and text inputs. We develop unimodal models for each modality and a multimodal framework to improve accuracy and robustness. By capturing a more detailed symptom profile, this approach can help in enhancing diagnostic precision and supports personalized treatment, offering a scalable and objective tool for mental health assessment.
Fact-Controlled Diagnosis of Hallucinations in Medical Text Summarization
Authors: Suhas BN; Han-Chin Shing; Lei Xu; Mitch Strong; Jon Burnsky; Jessica Ofor; Jordan R. Mason; Susan Chen; Sundararajan Srinivasan; Chaitanya Shivade; Jack Moriarty; Joseph Paul Cohen
Venue: Interspeech 2025 (main)

Abstract: Hallucinations in large language models (LLMs) during summarization of patient-clinician dialogues pose significant risks to patient care and clinical decision-making. However, the phenomenon remains understudied in the clinical domain, with uncertainty surrounding the applicability of general-domain hallucination detectors. The rarity and randomness of hallucinations further complicate their investigation. In this paper, we conduct an evaluation of hallucination detection methods in the medical domain, and construct two datasets for the purpose: A fact-controlled Leave-N-out dataset - generated by systematically removing facts from source dialogues to induce hallucinated content in summaries; and a natural hallucination dataset - arising organically during LLM-based medical summarization. We show that general-domain detectors struggle to detect clinical hallucinations, and that performance on fact-controlled hallucinations does not reliably predict effectiveness on natural hallucinations. We then develop fact-based approaches that count hallucinations, offering explainability not available with existing methods. Notably, our LLM-based detectors, which we developed using fact-controlled hallucinations, generalize well to detecting real-world clinical hallucinations. This research contributes a suite of specialized metrics supported by expertannotated datasets to advance faithful clinical summarization systems.
Evaluating Automatic Speech Recognition Pipelines for Mandarin-English Bilingual Child Language Assessment in Telehealth
Authors: Hongchen Wu; Yao Du; Zirong Li; Yixin Gu; Disha Thotappala Jayaprakash; Li Sheng
Venue: Interspeech 2025 (main)

Abstract: Bilingualism is rising worldwide, yet bilingual child assessments face major challenges. A shortage of bilingual clinicians and the labor-intensive nature of speech data annotation often cause misdiagnoses, delaying care and research. Using a Mandarin-English adult-child speech dataset (53 telehealth sessions), we explore how speech models can automate the annotation of clinical data involving multi-languages, multi-speakers, children's speech, and code-switching utterances. Findings indicated that simple pre-processing improves automatic speech recognition (ASR) accuracy. Specifically, integrating speaker diarization with OpenAIâs Whisper medium model reduces word error rates to 35% for child speech and 30% for code-switching, rivaling fine-tuned transformer models. As the first ASR pipeline evaluation for a Mandarin-English clinical dataset, our study highlights model limitations, establishes a benchmark for bilingual speech technology, and improves clinical services.
Enhancing Lyrics Transcription on Music Mixtures with Consistency Loss
Authors: Jiawen Huang; Felipe Sousa; Emir Demirel; Emmanouil Benetos; Igor Gadelha
Venue: Interspeech 2025 (main)

Abstract: Automatic Lyrics Transcription (ALT) aims to recognize lyrics from singing voices, similar to Automatic Speech Recognition (ASR) for spoken language, but faces added complexity due to domain-specific properties of the singing voice. While foundation ASR models show robustness in various speech tasks, their performance degrades on singing voice, especially in the presence of musical accompaniment. This work focuses on this performance gap and explores Low-Rank Adaptation (LoRA) for ALT, investigating both single-domain and dual-domain fine-tuning strategies. We propose using a consistency loss to better align vocal and mixture encoder representations, improving transcription on mixture without relying on singing voice separation. Our results show that while naÃ¯ve dual-domain fine-tuning underperforms, structured training with consistency loss yields modest but consistent gains, demonstrating the potential of adapting ASR foundation models for music.
Tonality-Based Accompaniment-Guided Automatic Singing Evaluation
Authors: Pei-Chin Hsieh; Yih-Liang Shen; Ngoc-Son Tran; Tai-Shih Chi
Venue: Interspeech 2025 (main)

Abstract: Automatic singing evaluation is highly desirable in industrial performance evaluation scenarios, including education and entertainment, given the high cost of human judges. Current singing evaluation systems assess a singer's performance by comparing reference vocals to the singing track. Unfortunately, reference vocals are not always available. Moreover, the similarity measure may not provide a reliable evaluation due to the inherently variable nature of singing performances. This paper proposes a tonality-based method for song-independent automatic singing evaluation. Experimental results show the proposed method outperforms other non-intrusive evaluation algorithms.
Investigating the Reasonable Effectiveness of Speaker Pre-Trained Models and their Synergistic Power for SingMOS Prediction
Authors: Orchid Chetia Phukan; Girish; Mohd Mujtaba Akhtar; Swarup Ranjan Behera; Pailla Balakrishna Reddy; Arun Balaji Buduru; Rajesh Sharma
Venue: Interspeech 2025 (main)

Abstract: In this study, we focus on Singing Voice Mean Opinion Score (SingMOS) prediction. Previous research have shown the performance benefit with the use of state-of-the-art (SOTA) pre-trained models (PTMs). However, they havenât explored speaker recognition speech PTMs (SPTMs) such as x-vector, ECAPA and we hypothesize that it will be the most effective for SingMOS prediction. We believe that due to their speaker recognition pretraining, it equips them to capture fine-grained vocal features (e.g., pitch, tone, intensity) from synthesized singing voices in a much more better way than other PTMs. Our experiments with SOTA PTMs including SPTMs and music PTMs validates the hypothesis. Additionally, we introduce a novel fusion framework, BATCH that uses Bhattacharya Distance for fusion of PTMs. Through BATCH with the fusion of speaker recognition SPTMs, we report the topmost performance comparison to all the individual PTMs and baseline fusion techniques as well as setting SOTA.
Focal Modulation Network: A Novel Solution for Polyphonic Music Instrument Recognition without Attention and Aggregation Strategy
Authors: Lekshmi C R; Rajeev Rajan
Venue: Interspeech 2025 (main)

Abstract: This study introduces a novel framework for recognizing multiple predominant instruments in polyphonic music using Focal Modulation Networks (FMN). Unlike traditional methods that rely on sliding window analysis and aggregation strategy, FMN leverages a focal modulation module for hierarchical contextualization and gated aggregation, leading to improved accuracy and interpretability. Trained on single-instrument data, the model effectively recognizes multiple instruments in unseen polyphonic compositions. Extensive experiments show our proposed FMN outperforms existing transformer architectures, including Vision Transformers (ViT), Swin Transformers (SwinT), and Compact Convolutional Transformers (CCT), with improved accuracy and interpretability, achieving state-of-the-art results. The proposed scheme reports micro and macro F1 scores of 0.66 and 0.58, representing 10% and 16% improvements over the state-of-the-art Han's model, respectively.
A Joint Network for Singing Melody Extraction from Polyphonic Music with Attention Aggregation and Self-Consistency Training
Authors: Jiabo Jing; Ying Hu; Hao Huang; Liang He; Zhijian Ou
Venue: Interspeech 2025 (main)

Abstract: Singing melody extraction (SME) is an important task in music information retrieval (MIR). In this paper, we propose a separate spectrum-based SME model and a joint network that combines the pre-trained and spectrum-based models. In the joint network, we design an attention aggregation module (AAM) consisting of cross-attention (CA) and adaptive decision fusion (ADF) to effectively fuse the intermediate features from two models. Furthermore, we introduce a self-consistency training strategy, which utilizes hard and soft labels to supervise two separate models to better obtain the SME task-relevant information. Experimental results show that our proposed method, Joint Network, outperforms six compared state-of-the-art methods, achieving overall accuracy (OA) scores of 91.6%, 92.5%, and 78.9% on the ADC 2004, MIREX 05, and MEDLEY DB datasets, respectively. Visualized results show that the Joint Network can reduce the octave and melody detection errors.
Position also matters! Separating Same Instruments in String Quartet using Timbral and Positional Cues
Authors: Yuetonghui Xu; Yiwen Wang; Xihong Wu; Xiaobing Li; Feng Yu
Venue: Interspeech 2025 (main)

Abstract: String quartets inherently face challenges such as fundamental frequency overlap, similar timbres, and complex acoustic environments, making existing source separation methods relying solely on timbral characteristics less effective. This paper proposes a novel approach that enhances separation performance by encoding timbral and spatial positional cues to guide a neural network for source separation. To evaluate our method, we created a dataset of string quartet recordings with realistic reverberation with two different seating arrangements. Experiments demonstrate that our approach outperforms current state-of-the-art (SOTA) methods in source separation, achieving superior performance even in seating arrangements different from training, thus highlighting the importance of our timbral and positional encoding in improving separation quality and generalizability.
WhisperMSS: A Two-Stage Framework for Mandarin Singing Transcription and Segmentation Using Pretrained Models
Authors: Ruoxuan Liang; Xiangjian Zeng; Zhen Liu; Qingqiang Wu; RuiChen Zhang; Le Ren
Venue: Interspeech 2025 (main)

Abstract: This paper addresses the challenges of Mandarin singing transcription and segmentation by proposing a two-stage framework based on Whisper. Our research makes three key contributions: First,enhancing transcription accuracy via WhisperMLT, which incorporates a Chinese-specific text embedding layer, a CTC branch atop the encoder, and a Transformer-based contextual network; Second, optimizing CTC posterior probabilities through syllable-aligned pseudo-labeling, which generates one-hot frame-level labels from timestamp-annotated datasets;Finally, achieving precise segmentation with CTC-Vseg, which implements silence label insertion, constrained state transitions, and dynamic programming-based path optimization.Experiments demonstrate superior performance in Mandarin singing segmentation, offering novel solutions for audio processing tasks.
Low Complex IIR Adaptive Hear-Through Ambient Filtering for Overcoming Practical Constraints in Earbuds
Authors: Rishabh Gupta; MLNS Karthik; Yughendaran P
Venue: Interspeech 2025 (main)

Abstract: The materials used in the headphones and earbuds typically attenuate sound passively above frequency of 500-700 Hz. Hear-Through (HT) techniques can compensate for the passive attenuation characteristics of listening devices to achieve a listening experience similar to the open ear for enhancing situational awareness of the users. Most studies have proposed usage of fixed-filter or adaptive finite impulse response (FIR) HT techniques, which can lead to mismatch between the modelled and desired open ear response for different users and fittings or lead to higher computations and larger processing delays. This paper proposes a virtual sensing model with an infinite impulse response (IIR) based adaptive equalization method to model the open ear response at userâs eardrum. The proposed method reduces the computational complexity by 34%, while maintaining stability and providing superior HT performance upto 10 dB compared to existing methods in dynamic acoustic scenarios.
Sub-band based Adaptive IIR Algorithm with Biquad Filter Stability Constraints for Feedforward Hear-Through Equalization
Authors: Rishabh Gupta; MLNS Karthik; Chelamkuri Omsrinath
Venue: Interspeech 2025 (main)

Abstract: Hear through (HT) filter is designed to provide a listening experience similar to open ear listening. The key challenge in the design of the HT filter is to ensure low processing delay to closely match the open ear response. Existing adaptive HT approaches such as adaptive finite impulse response (FIR) and neural network-based approaches can lead to higher processing delays. This paper proposes a sub-band based infinite impulse response (IIR) adaptive feedforward equalization method to improve the HT and convergence performance for sound sources incident from different directions. To ensure IIR filter stability, we propose inclusion of stability constraint based on least mean square fourth criterion. The proposed method improves performance over existing methods up to 13 dB for simulated outdoor and indoor scenarios, while maintaining stability and similar complexity as existing adaptive IIR and FIR techniques in simulated complex indoor and outdoor real-world acoustic environments.
Discrete Audio Representations for Automated Audio Captioning
Authors: Jingguang Tian; Haoqin Sun; Xinhui Hu; Xinkang Xu
Venue: Interspeech 2025 (main)

Abstract: Discrete audio representations, termed audio tokens, are broadly categorized into semantic and acoustic tokens, typically generated through unsupervised tokenization of continuous audio representations. However, their applicability to automated audio captioning (AAC) remains underexplored. This paper systematically investigates the viability of audio token-driven models for AAC through comparative analyses of various tokenization methods. Our findings reveal that audio tokenization leads to performance degradation in AAC models compared to those that directly utilize continuous audio representations. To address this issue, we introduce a supervised audio tokenizer trained with an audio tagging objective. Unlike unsupervised tokenizers, which lack explicit semantic understanding, the proposed tokenizer effectively captures audio event information. Experiments conducted on the Clotho dataset demonstrate that the proposed audio tokens outperform conventional audio tokens in the AAC task.
CLAP-ART: Automated Audio Captioning with Semantic-rich Audio Representation Tokenizer
Authors: Daiki Takeuchi; Binh Thien Nguyen; Masahiro Yasuda; Yasunori Ohishi; Daisuke Niizumi; Noboru Harada
Venue: Interspeech 2025 (main)

Abstract: Automated Audio Captioning (AAC) aims to describe the semantic contexts of general sounds, including acoustic events and scenes, by leveraging effective acoustic features. To enhance performance, an AAC method, EnCLAP, employed discrete tokens from EnCodec as an effective input for fine-tuning a language model BART. However, EnCodec is designed to reconstruct waveforms rather than capture the semantic contexts of general sounds, which AAC should describe. To address this issue, we propose CLAP-ART, an AAC method that utilizes "semantic-rich and discrete" tokens as input. CLAP-ART computes semantic-rich discrete tokens from pre-trained audio representations through vector quantization. We experimentally confirmed that CLAP-ART outperforms baseline EnCLAP on two AAC benchmarks, indicating that semantic-rich discrete tokens derived from semantically rich AR are beneficial for AAC.
Temp4Cap: Temporally-aligned Automated Audio Captioning
Authors: Ho-Young Choi; Jae-Heung Cho; Pil Moo Byun; Won-Gook Choi; Joon-Hyuk Chang
Venue: Interspeech 2025 (main)

Abstract: Automated audio captioning (AAC) is a crucial task in machine perception within the audio domain. AAC struggles to interpret and incorporate temporal relationships of sound events in captions. However, existing studies often fail to capture the temporal relationship, leading to incorrect captions. Some recent studies leverage sound event detection models to extract temporal relationships but remain limited by their dependence on independent pre-trained models. In this study, we propose Temp4Cap, a novel AAC framework that directly trains temporal alignment via contrastive learning, using the âtemporal captionâ generated by a large language model. To capture temporal relationships, we apply a temporal negative sampling strategy, which includes event- and order-level shuffle and random substitution when generating negative samples during contrastive learning. Experimental results on Clotho and AudioCaps show that Temp4Cap significantly improves both captioning and temporal metrics.
Optimizing CLAP Reward with LLM Feedback for Semantically Aligned and Diverse Automated Audio Captioning
Authors: Seyun Ahn; Pil Moo Byun; Won-Gook Choi; Joon-Hyuk Chang
Venue: Interspeech 2025 (main)

Abstract: Deep learning-based automated audio captioning (AAC) systems describe audio well, yet they often overfit to reference styles. To address this, reinforcement learning (RL) techniques have been adopted to directly optimize evaluation metrics, but these methods often suffer from word repetition and contextual distortion. Embedding-based rewards, such as those derived from contrastive language-audio pretraining (CLAP), may bias the model toward specific words or phrases that human evaluators find unnatural. In this paper, we propose a novel reward system that combines a CLAP-based reward with a repetition penalty (CRRP) and a large language model (LLM) evaluator. CRRP computes rewards using CLAP similarity, applies a repetition penalty and reward clipping to stabilize training, and uses LLM feedback to enhance naturalness. Our method shows outstanding performance in semantic evaluations and both human and AI-based assessments.
Bridging Audio and Vision: Zero-Shot Audiovisual Segmentation by Connecting Pretrained Models
Authors: Seung-jae Lee; Paul Hongsuck Seo
Venue: Interspeech 2025 (main)

Abstract: Audiovisual segmentation (AVS) aims to identify visual regions corresponding to sound sources, playing a vital role in video understanding, surveillance, and human-computer interaction. Traditional AVS methods depend on large-scale pixel-level annotations, which are costly and time-consuming to obtain. To address this, we propose a novel zero-shot AVS framework that eliminates task-specific training by leveraging multiple pretrained models. Our approach integrates audio, vision, and text representations to bridge modality gaps, enabling precise sound source segmentation without AVS-specific annotations. We systematically explore different strategies for connecting pretrained models and evaluate their efficacy across multiple datasets. Experimental results demonstrate that our framework achieves state-of-the-art zero-shot AVS performance, highlighting the effectiveness of multimodal model integration for fine-grained audiovisual segmentation1.