GenAI4ELab/papercli-papers huggingface dataset:
A Simple-Yet-Effective Data Augmentation Method for Speaker Identification in Novels
Authors: Wenjie Zhong; Jason Naradowsky; Yusuke Miyao
Venue: Interspeech 2025 (main)

Abstract: Speaker identification in novels is crucial for speech synthesis systems to assign appropriate voices in audiobook production. It attributes a speaker to an utterance through context analysis. Traditional approaches heavily rely on human-annotated datasets, which are costly and scarce, limiting model performance. To overcome this, we propose a simple-yet-effective data augmentation method using large language models (LLMs) to generate synthetic dialogues and post-process the dialogues into augmented training instances. Our experiments show that this method achieves a state-of-the-art accuracy of 82.6%, surpassing the previous baseline by 2.4%. Performance gains are especially notable in the Implicit (hard) category, where our method exceeds the previous baseline by 3.5%. Our analysis suggests that it enhances the ability to capture long-term dependencies and there is a mutually reinforce effect between the Implicit and Anaphoric (middle) categories.
IDIR: Identifying and Distilling Informative Relations for Speaker Verification
Authors: Chong-Xin Gan; Zhe Li; Zezhong Jin; Zilong Huang; Man-Wai Mak; Kong Aik Lee
Venue: Interspeech 2025 (main)

Abstract: Traditional feature-based knowledge distillation aligns the studentâs features with the teacherâs features. However, these one-to-one alignments overlook the structural relations between speakers in a mini-batch. Also, the large capacity gap between the two networks causes significant discrepancies between their features. To address these limitations, we propose distilling the inter- and intra-speaker relations. Instead of mimicking all pairwise relations between the student's and teacher's feature vectors, we propose Identifying and Distilling Informative Relations (IDIR), enabling the student network to acquire speakers' relationships from the teacher. Moreover, a margin is added to the similarity scores of the informative pairs, further reducing intra-speaker variances and increasing inter-speaker separations. Evaluations with a simple x-vector student network demonstrate the method's superb performance across three test sets, showcasing its merits and effectiveness.
Analysis of ABC Frontend Audio Systems for the NIST-SRE24
Authors: Sara Barahona; Anna Silnova; Ladislav MoÅ¡ner; Junyi Peng; OldÅich Plchot; Johan Rohdin; Lin Zhang; Jiangyu Han; Petr Palka; Federico Landini; LukÃ¡Å¡ Burget; Themos Stafylakis; Sandro Cumani; Dominik BoboÅ¡; Miroslav HlavaÄek; Martin Kodovsky; TomaÅ¡ PavliÄek
Venue: Interspeech 2025 (main)

Abstract: We present a comprehensive analysis of the embedding extractors (frontends) developed by the ABC team for the audio track of NIST SRE 2024. We follow the two scenarios imposed by NIST: using only a provided set of telephone recordings for training (fixed) or adding publicly available data (open condition). Under these constraints, we develop the best possible speaker embedding extractors for the pre-dominant conversational telephone speech (CTS) domain. We explored architectures based on ResNet with different pooling mechanisms, recently introduced ReDimNet architecture, as well as a system based on the XLS-R model, which represents the family of large pre-trained self-supervised models. In open condition, we train on VoxBlink2 dataset, containing 110 thousand speakers across multiple languages. We observed a good performance and robustness of VoxBlink-trained models, and our experiments show practical recipes for developing state-of-the-art frontends for speaker recognition.
Identifying Primary Stress Across Related Languages and Dialects with Transformer-based Speech Encoder Models
Authors: Nikola LjubeÅ¡iÄ; Ivan Porupski; Peter Rupnik
Venue: Interspeech 2025 (main)

Abstract: Automating primary stress identification has been an active research field due to the role of stress in encoding meaning and aiding speech comprehension. Previous studies relied mainly on traditional acoustic features and English datasets. In this paper we investigate the approach of fine-tuning a pre-trained transformer model with an audio frame classification head. Our experiments use a new Croatian training dataset, with test sets in Croatian, Serbian, the Chakavian dialect, and Slovenian. By comparing an SVM classifier using traditional acoustic features with the fine-tuned speech transformer, we demonstrate the transformer's superiority across the board, achieving near-perfect results for Croatian and Serbian, with a 10-point performance drop for more distant Chakavian and Slovenian. Finally, we show that only a few hundred multi-syllabic training words suffice for strong performance. We release our datasets and model under permissive licenses.
SupraDoRAL: Automatic Word Prominence Detection Using Suprasegmental Dependencies of Representations with Acoustic and Linguistic Context
Authors: Jhansi Mallela; Upendra Vishwanath Y. S.; Sankara Bharadwaj Rangavajjala; Bhaskar Bhatt; Chiranjeevi Yarra
Venue: Interspeech 2025 (main)

Abstract: Word-level prominence plays a crucial role in spoken language understanding, as it helps to interpret the speakerâs intent. However, automatic word-level prominence detection remains underexplored, particularly in non-native speech, where prominence patterns are more variable due to native language influence. While prominence is determined by acoustic features such as energy, duration, and pitch, capturing statistics on these acoustics at the word level provides only a global representation, missing finer suprasegmental variations. In this study, we consider syllables as the suprasegmental unit and propose a methodology that jointly models syllable prominence variations and their contribution to word prominence using sequential neural networks. This enables learning word-level prominence representations with minimal labeled data. Our method outperforms an unsupervised n-gram-based baseline by 24.01% and a supervised SVM by 5.73%, demonstrating its effectiveness over both approaches.
LombardTokenizer: Disentanglement and Control of Vocal Effort in a Neural Speech Codec
Authors: Maxime Jacquelin; MaÃ«va Garnier; Laurent Girin; RÃ©my Vincent; Olivier Perrotin
Venue: Interspeech 2025 (main)

Abstract: Disentangling distinct types of information in speech representations is crucial for improving speech synthesis and voice conversion systems. In this work, we introduce LombardTokenizer, a neural speech codec able to separate features related to vocal effort from other acoustic (and semantic) information. This model is built on SpeechTokenizer, a model proposed in the literature based on multi-stage quantisation, which focused on isolating semantic content in its first quantisation layer. We show that the level of vocal effort can be effectively captured in the second quantisation layer by conditioning the quantisation layer with neural encoders trained to represent vocal effort. Experimental results demonstrate that the proposed method significantly outperforms existing methods in speech conversion between neutral and Lombard speech, while maintaining excellent speech synthesis quality, offering improved control over vocal effort and naturalness of synthesised speech.
Robust Personal Voice Activity Detection for Mitigating Domain Mismatch and False Acceptance Scenarios
Authors: Yuke Lin; Jun Chen; Wenjie Li; Longshuai Xiao; Chao Weng
Venue: Interspeech 2025 (main)

Abstract: Personal Voice Activity Detection (pVAD), which leverages pre-enrolled speaker information to identify the presence of a specific speaker, has been widely adopted in mobile devices. However, domain mismatches between enrolled and test data are common in real-world scenarios, resulting in significant performance degradation. Additionally, existing pVAD models primarily optimize detection performance for a target speaker but often fail to address the challenge of false acceptance, especially when interfering speakers share similar voice characteristics. To address these limitations, we propose a novel backbone integrated with an auxiliary decoder and utilize an embedding-updating method during the inference phase to enhance performance under domain mismatch conditions. Furthermore, we introduce an on-the-fly hard-sample data simulation strategy, which has been shown to significantly reduce false acceptance rates, as demonstrated by our experimental results.
Adaptive Knowledge Distillation for Device-Directed Speech Detection
Authors: Hyung-gun Chi; Florian Pesce; Wonil Chang; Oggi Rudovic; Arturo Argueta; Stefan Braun; Vineet Garg; Ahmed Hussen Abdelaziz
Venue: Interspeech 2025 (main)

Abstract: Device-directed speech detection (DDSD) is a binary classification task that separates the userâs queries to a voice assistant (VA) from background speech or side conversations. This is important for achieving naturalistic user experience. To this end, we propose knowledge distillation (KD) to enhance DDSD accuracy while ensuring efficient deployment. Specifically, we introduce a novel adaptive KD method that transfers knowledge from general representations of an ASR large pre-trained acoustic encoder (teacher). We apply task-specific adapters, on top of the (frozen) teacher encoder, trained jointly with the student model on DDSD. We demonstrate that the proposed adaptive KD outperforms the student model without distillation in the keyword and keyword-free (follow-up) invocations, with an improvement of +26% and +19% in terms of Equal Error Rate, respectively. We also show that this approach generalizes across the transformer and conformer-based model architectures.
Flexible VAD-PVAD Transition: A Detachable PVAD Module for Dynamic Encoder RNN VAD
Authors: En-Lun Yu; Chien-Chun Wang; Jeih-Weih Hung; Shih-Chieh Huang; Berlin Chen
Venue: Interspeech 2025 (main)

Abstract: In this paper, we propose Flexible Dynamic Encoder RNN (FDE-RNN), an innovative model capable of seamlessly switching between VAD and PVAD without incurring redundant resource consumption. In static PVAD modeling, performing VAD typically requires either merging categories or omitting speaker embeddings, often resulting in excessively large models that are impractical for VAD tasks. In contrast, FDERNN efficiently adapts by removing the personalization module when functioning as VAD, significantly reducing resource demands. Furthermore, on PVAD tasks, FDE-RNN leverages dynamic neural networks with a gating-based skipping mechanism, enabling it to bypass redundant computations during non-speech segments, further optimizing computational efficiency. Extensive experiments demonstrate that FDE-RNN outperforms all other prior arts on both PVAD and VAD tasks in terms of overall performance. Notably, when functioning as a VAD, FDE-RNN merely utilizes 30% of the parameters required by the competitive models, underscoring its remarkable efficiency and scalability.
Speaker Conditioning of Voice Activity Detection via Implicit Separation
Authors: Matthew Maciejewski
Venue: Interspeech 2025 (main)

Abstract: Within the domain of multi-talker recordings, many speech technologies rely on an initial segmentation step of finding when each person was talking. One common approach to this task is Target-Speaker Voice Activity Detection (TS-VAD), in which a model is supplied with a representation corresponding to a particular speaker and then identifies the temporal regions when that person was talking. As in many cases, the increased complexity of this task over regular Voice Activity Detection (VAD) imposes constraints on the data used to train such a model. In this work, we explore conversion of a pre-trained VAD model into a TS-VAD model via an implicitly-trained separation front endâdecoupling the need for speaker-discriminative training data from the basic speech/non-speech data used in training VAD modelsâwhich can lead to improvements in model robustness and speech recall in the domains present only in the training data of the VAD.
ASDA: Audio Spectrogram Differential Attention Mechanism for Self-Supervised Representation Learning
Authors: Junyu Wang; Tianrui Wang; Meng Ge; Longbiao Wang; Jianwu Dang
Venue: Interspeech 2025 (main)

Abstract: In recent advancements in audio self-supervised representation learning, the standard Transformer architecture has emerged as the predominant approach, yet its attention mechanism often allocates a portion of attention weights to irrelevant information, potentially impairing the modelâs discriminative ability. To address this, we introduce a differential attention mechanism, which effectively mitigates ineffective attention allocation through the integration of dual-softmax operations and appropriately tuned differential coefficients. Experimental results demonstrate that our ASDA model achieves state-of-the-art (SOTA) performance across multiple benchmarks, including audio classification (49.0% mAP on AS-2M, 41.5% mAP on AS20K), keyword spotting (98.3% accuracy on SPC-2), and environmental sound classification (96.1% accuracy on ESC-50). These results highlight ASDA's effectiveness in audio tasks, paving the way for broader applications.
DuRep: Dual-Mode Speech Representation Learning via ASR-Aware Distillation
Authors: Prabash Reddy Male; Swayambhu Nath Ray; Harish Arsikere; Akshat Jaiswal; Prakhar Swarup; Prantik Sen; Debmalya Chakrabarty; K V Vijay Girish; Nikhil Bhave; Frederick Weber; Sambuddha Bhattacharya; Sri Garimella
Venue: Interspeech 2025 (main)

Abstract: Recent advancements in speech encoders have drawn attention due to their integration with Large Language Models for various speech tasks. While most research has focused on either causal or full-context speech encoders, there's limited exploration to effectively handle both streaming and non-streaming applications, while achieving state-of-the-art performance. We introduce DuRep, a Dual-mode Speech Representation learning setup, which enables a single speech encoder to function efficiently in both offline and online modes without additional parameters or mode-specific adjustments, across downstream tasks. DuRep-200M, our 200M parameter dual-mode encoder, achieves 12% and 11.6% improvements in streaming and non-streaming modes, over baseline encoders on Multilingual ASR. Scaling this approach to 2B parameters, DuRep-2B sets new performance benchmarks across ASR and non-ASR tasks. Our analysis reveals interesting trade-offs between acoustic and semantic information across encoder layers.
Joint Estimation and Inference for Data Integration Problems based on Multiple Multi-layered Gaussian Graphical Models
Authors: Subhabrata Majumdar; George Michailidis
Venue: JMLR 2022 (main)

Abstract: The rapid development of high-throughput technologies has enabled the generation of data from biological or disease processes that span multiple layers, like genomic, proteomic or metabolomic data, and further pertain to multiple sources, like disease subtypes or experimental conditions. In this work, we propose a general statistical framework based on Gaussian graphical models for horizontal (i.e. across conditions or subtypes) and vertical (i.e. across different layers containing data on molecular compartments) integration of information in such datasets. We start with decomposing the multi-layer problem into a series of two-layer problems. For each two-layer problem, we model the outcomes at a node in the lower layer as dependent on those of other nodes in that layer, as well as all nodes in the upper layer. We use a combination of neighborhood selection and group-penalized regression to obtain sparse estimates of all model parameters. Following this, we develop a debiasing technique and asymptotic distributions of inter-layer directed edge weights that utilize already computed neighborhood selection coefficients for nodes in the upper layer. Subsequently, we establish global and simultaneous testing procedures for these edge weights. Performance of the proposed methodology is evaluated on synthetic and real data.
Debiased  Distributed Learning for Sparse Partial Linear Models in  High Dimensions
Authors: Shaogao Lv; Heng Lian
Venue: JMLR 2022 (main)

Abstract: Although various distributed machine learning schemes have been proposed recently for purely linear models and fully nonparametric models, little attention has been paid to distributed optimization for semi-parametric models with multiple structures (e.g. sparsity, linearity and nonlinearity).  To address these issues, the current paper proposes a new communication-efficient distributed learning algorithm for sparse partially linear models with an increasing number of features. The proposed method is based on the classical divide and conquer strategy for handling big data and the computation on each subsample consists of  a debiased estimation of the doubly regularized least squares approach. With the proposed method, we theoretically prove that our global parametric estimator can achieve the optimal parametric rate in our semi-parametric model given an appropriate partition on the total data. Specifically, the choice of data partition  relies on the underlying smoothness of the nonparametric component, and it is adaptive to the sparsity parameter. Finally, some simulated experiments are carried out to illustrate the empirical performances of our debiased technique under the distributed setting.
Recovering shared structure from multiple networks with unknown edge distributions
Authors: Keith Levin; Asad Lodhia; Elizaveta Levina
Venue: JMLR 2022 (main)

Abstract: In increasingly many settings, data sets consist of multiple samples from a population of networks, with vertices aligned across networks; for example, brain connectivity networks in neuroscience. We consider the setting where the observed networks have a shared expectation, but may differ in the noise structure on their edges. Our approach exploits the shared mean structure to denoise edge-level measurements of the observed networks and estimate the underlying population-level parameters. We also explore the extent to which edge-level errors influence estimation and downstream inference. In the process, we establish a finite-sample concentration inequality for the low-rank eigenvalue truncation of a random weighted adjacency matrix, which may be of independent interest. The proposed approach is illustrated on synthetic networks and on data from an fMRI study of schizophrenia.
Exploiting locality in high-dimensional Factorial hidden Markov models
Authors: Lorenzo Rimella; Nick Whiteley
Venue: JMLR 2022 (main)

Abstract: We propose algorithms for approximate filtering and smoothing in high-dimensional Factorial hidden Markov models. The approximation involves discarding, in a principled way, likelihood factors according to a notion of locality in a factor graph associated with the emission distribution. This allows the exponential-in-dimension cost of exact filtering and smoothing to be avoided. We prove that the approximation accuracy, measured in a local total variation norm, is "dimension-free" in the sense that as the overall dimension of the model increases the error bounds we derive do not necessarily degrade. A key step in the analysis is to quantify the error introduced by localizing the likelihood function in a Bayes' rule update. The factorial structure of the likelihood function which we exploit arises naturally when data have known spatial or network structure. We demonstrate the new algorithms on synthetic examples and a London Underground passenger flow problem, where the factor graph is effectively given by the train network.
Empirical Risk Minimization under Random Censorship
Authors: Guillaume Ausset; Stephan ClÃ©menÃ§on; FranÃ§ois Portier
Venue: JMLR 2022 (main)

Abstract: We consider the classic supervised learning problem where a continuous non-negative random label $Y$ (e.g. a random duration) is to be predicted based upon observing a random vector $X$ valued in $\mathbb{R}^d$ with $d\geq 1$ by means of a regression rule with minimum least square error. In various applications, ranging from industrial quality control to public health through credit risk analysis for instance, training observations can be right censored, meaning that, rather than on independent copies of $(X,Y)$, statistical learning relies on a collection of $n\geq 1$ independent realizations of the triplet $(X, \; \min\{Y,\; C\},\; \delta)$, where $C$ is a nonnegative random variable with unknown distribution, modelling censoring and $\delta=\mathbb{I}\{Y\leq C\}$ indicates whether the duration is right censored or not. As ignoring censoring in the risk computation may clearly lead to a severe underestimation of the target duration and jeopardize prediction, we consider a plug-in estimate of the true risk based on a Kaplan-Meier estimator of the conditional survival function of the censoring $C$ given $X$, referred to as Beran risk, in order to perform empirical risk minimization. It is established, under mild conditions, that the learning rate of minimizers of this biased/weighted empirical risk functional is of order $O_{\mathbb{P}}(\sqrt{\log(n)/n})$ when ignoring model bias issues inherent to plug-in estimation, as can be attained in absence of censoring. Beyond theoretical results, numerical experiments are presented in order to illustrate the relevance of the approach developed.
XAI Beyond Classification: Interpretable Neural Clustering
Authors: Xi Peng; Yunfan Li; Ivor W. Tsang; Hongyuan Zhu; Jiancheng Lv; Joey Tianyi Zhou
Venue: JMLR 2022 (main)

Abstract: In this paper, we study two challenging problems in explainable AI (XAI) and data clustering. The first is how to directly design a neural network with inherent interpretability, rather than giving post-hoc explanations of a black-box model. The second is implementing discrete $k$-means with a differentiable neural network that embraces the advantages of parallel computing, online clustering, and clustering-favorable representation learning. To address these two challenges, we design a novel neural network, which is a differentiable reformulation of the vanilla $k$-means, called inTerpretable nEuraL cLustering (TELL). Our contributions are threefold. First, to the best of our knowledge, most existing XAI works focus on supervised learning paradigms. This work is one of the few XAI studies on unsupervised learning, in particular, data clustering. Second, TELL is an interpretable, or the so-called intrinsically explainable and transparent model. In contrast, most existing XAI studies resort to various means for understanding a black-box model with post-hoc explanations. Third, from the view of data clustering, TELL possesses many properties highly desired by $k$-means, including but not limited to online clustering, plug-and-play module, parallel computing, and provable convergence. Extensive experiments show that our method achieves superior performance comparing with 14 clustering approaches on three challenging data sets. The source code could be accessed at www.pengxi.me.