GenAI4ELab/papercli-papers huggingface dataset:
The Second DISPLACE Challenge: DIarization of SPeaker and LAnguage in Conversational Environments
Authors: Shareef Babu Kalluri; Prachi Singh; Pratik Roy Chowdhuri; Apoorva Kulkarni; Shikha Baghel; Pradyoth Hegde; Swapnil Sontakke; Deepak K T; S.R. Mahadeva Prasanna; Deepu Vijayasenan; Sriram Ganapathy
Venue: Interspeech 2024 (main)

Abstract: The DIarization of SPeaker and LAnguage in Conversational Environments (DISPLACE) 2024 challenge is the second in the series of DISPLACE challenges, which involves tasks of speaker diarization (SD) and language diarization (LD) on a challenging multilingual conversational speech dataset.  In the DISPLACE 2024 challenge, we also introduced the task of automatic speech recognition (ASR) on this dataset. The dataset containing 158 hours of speech, consisting of both supervised and unsupervised mono-channel far-field recordings, was released for LD and SD tracks. Further, 12 hours of close-field mono-channel recordings were provided for the ASR track conducted on 5 Indian languages. The details of the dataset, baseline systems and the leader board results are highlighted in this paper.  We have also compared our baseline models and the team's performances on evaluation data of DISPLACE-2023 to emphasize the advancements made in this second version of the challenge.
TalTech-IRIT-LIS Speaker and Language Diarization Systems for DISPLACE 2024
Authors: Joonas Kalda; Tanel Alumae; Martin Lebourdais; HervÃ© Bredin; SÃ©verin Baroudi; Ricard Marxer
Venue: Interspeech 2024 (main)

Abstract: This paper describes the submissions of team TalTech-IRIT-LIS to the DISPLACE 2024 challenge. Our team participated in the speaker diarization and language diarization tracks of the challenge. In the speaker diarization track, our best submission was an ensemble of systems based on the pyannote.audio speaker diarization pipeline utilizing powerset training and our recently proposed PixIT method that performs joint diarization and speech separation. We improve upon PixIT by using the separation outputs for speaker embedding extraction. Our ensemble achieved a diarization error rate of 27.1% on the evaluation dataset. In the language diarization track, we fine-tuned a pre-trained Wav2Vec2-BERT language embedding model on in-domain data, and clustered short segments using AHC and VBx, based on similarity scores from LDA/PLDA. This led to a language diarization error rate of 27.6% on the evaluation data. Both results were ranked first in their respective challenge tracks.
Exploring Energy-Based Models for Out-of-Distribution Detection in Dialect Identification
Authors: Yaqian Hao; Chenguang Hu; Yingying Gao; Shilei Zhang; Junlan Feng
Venue: Interspeech 2024 (main)

Abstract: The diverse nature of dialects presents challenges for models trained on specific linguistic patterns, rendering them susceptible to errors when confronted with unseen or out-of-distribution ï¼OODï¼ data. This study introduces a novel margin-enhanced joint energy model (MEJEM) tailored specifically for OOD detection in dialects. By integrating a generative model and the energy margin loss, our approach aims to enhance the robustness of dialect identification systems. Furthermore, we explore two OOD scores for OOD dialect detection, and our findings conclusively demonstrate that the energy score outperforms the softmax score. Leveraging Sharpness-Aware Minimization to optimize the training process of the joint model, we enhance model generalization by minimizing both loss and sharpness. Experiments conducted on dialect identification tasks validate the efficacy of Energy-Based Models  and provide valuable insights into their performance.
Exploring Spoken Language Identification Strategies for Automatic Transcription of Multilingual Broadcast and Institutional Speech
Authors: Martina Valente; Fabio Brugnara; Giovanni Morrone; Enrico Zovato; Leonardo Badino
Venue: Interspeech 2024 (main)

Abstract: This paper addresses spoken language identification (SLI) and speech recognition of multilingual broadcast and institutional speech, real application scenarios that have been rarely addressed in the SLI literature. Observing that in these domains language changes are mostly associated with speaker changes, we propose a cascaded system consisting of speaker diarization and language identification and compare it with more traditional language identification and language diarization systems. Results show that the proposed system often achieves lower language classification and language diarization error rates (up to 10% relative language diarization error reduction and 60% relative language confusion reduction) and leads to lower WERs on multilingual test sets (more than 8% relative WER reduction), while at the same time does not negatively affect speech recognition on monolingual audio (with an absolute WER increase between 0.1% and 0.7% w.r.t. monolingual ASR).
AG-LSEC: Audio Grounded Lexical Speaker Error Correction
Authors: Rohit Paturi; Xiang Li; Sundararajan Srinivasan
Venue: Interspeech 2024 (main)

Abstract: Speaker Diarization (SD) systems are typically audio-based and operate independently of the ASR system in traditional speech transcription pipelines and can have speaker errors due to SD and/or ASR reconciliation, especially around speaker turns and regions of speech overlap. To reduce these errors, a Lexical Speaker Error Correction (LSEC), in which an external language model provides lexical information to correct the speaker errors, was recently proposed. Though the approach achieves good Word Diarization error rate (WDER) improvements, it does not use any additional acoustic information and is prone to miscorrections. In this paper, we propose to enhance and acoustically ground the LSEC system with speaker scores directly derived from the existing SD pipeline. This approach achieves significant relative WDER reductions in the range of 25-40% over the audio-based SD, ASR system and beats the LSEC system by 15-25% relative on RT03-CTS, Callhome American English and Fisher datasets.
Speaker Change Detection with Weighted-sum Knowledge Distillation based on Self-supervised Pre-trained Models
Authors: Hang Su; Yuxiang Kong; Lichun Fan; Peng Gao; Yujun Wang; Zhiyong Wu
Venue: Interspeech 2024 (main)

Abstract: Speaker Change Detection (SCD) is an essential problem in speech processing and has various applications in many fields. The self-supervised models have shown impressive performance on many downstream tasks in the pre-training and fine-tuning paradigm. However, it has limitations to apply a fine-tuned self-supervised pre-trained model to frame-level SCD task in real industry because it typically requires a smaller model that consumes fewer computational resources. To tackle this issue, we propose using Knowledge Distillation (KD) to leverage the capabilities of the self-supervised model. First, a basic KD method based on the pre-trained model is proposed. Then, a weighted-sum KD method is proposed to selectively extract information from the pre-trained model. Experimental results demonstrate the effectiveness of the basic KD method as well as a further improvement for the weighted-sum KD method. The proposed method is more suitable for industrial applications compared with fine-tuning.
SOMSRED: Sequential Output Modeling for Joint Multi-talker Overlapped Speech Recognition and Speaker Diarization
Authors: Naoki Makishima; Naotaka Kawata; Mana Ihori; Tomohiro Tanaka; Shota Orihashi; Atsushi Ando; Ryo Masumura
Venue: Interspeech 2024 (main)

Abstract: This paper proposes SOMSRED, which jointly models the multi-talker automatic speech recognition (ASR) and speaker diarization (SD) for fully overlapped speech of unknown speakers. The conventional method that jointly estimates ASR and SD requires non-overlapping speech and a  separate clustering-based SD component for accurately identifying speakers. However, the speech is often overlapped, which deteriorates speaker identification performance, and the separate model makes the whole system sub-optimal. To address this problem, our idea is to build a sequential output model that outputs transcriptions, timestamps, and newly introduced speaker identifiers recursively from overlapped speech. Since speaker identifier do not fully represent the speaker characteristics of unknown speakers, SOMSRED utilizes the intermediate feature as speaker embeddings. Experimental results show the efficacy of the proposed method in speaker recognition, SD, and multi-talker ASR.
Song Data Cleansing for End-to-End Neural Singer Diarization Using Neural Analysis and Synthesis Framework
Authors: Hokuto Munakata; Ryo Terashima; Yusuke Fujita
Venue: Interspeech 2024 (main)

Abstract: We propose a data cleansing method that utilizes a neural analysis and synthesis (NANSY++) framework to train an end-to-end neural diarization model (EEND) for singer diarization. Our proposed model converts song data with choral singing commonly contained in popular music and unsuitable for generating a simulated dataset to the solo singing data. This cleansing is based on NANSY++, which is a framework trained to reconstruct an input non-overlapped audio signal. We exploit the pre-trained NANSY++ to convert choral singing into clean, non-overlapped audio. This cleansing process mitigates the mislabeling of choral singing to solo singing and helps the effective training of EEND models even when the majority of available song data contains choral singing sections. We experimentally evaluated the EEND model trained with a dataset using our proposed method using annotated popular duet songs. As a result, our proposed method improved 14.8 points in diarization error rate.
DiffATR: Diffusion-based Generative Modeling for Audio-Text Retrieval
Authors: Yifei Xin; Xuxin Cheng; Zhihong Zhu; Xusheng Yang; Yuexian Zou
Venue: Interspeech 2024 (main)

Abstract: Existing audio-text retrieval (ATR) methods are essentially discriminative models that aim to maximize the conditional likelihood, represented as p(candidates|query). Nevertheless, this methodology fails to consider the intrinsic data distribution p(query), leading to difficulties in discerning out-of-distribution data. In this work, we attempt to tackle this constraint through a generative perspective and model the relationship between audio and text as their joint probability p(candidates, query). To this end, we present a diffusion-based ATR framework (DiffATR), which models ATR as an iterative procedure that progressively generates joint distribution from noise. Throughout its training phase, DiffATR is optimized from both generative and discriminative viewpoints: the generator is refined through a generation loss, while the feature extractor benefits from a contrastive loss, thus combining the merits of both methodologies. Experiments on the AudioCaps and Clotho datasets with superior performances, verify the effectiveness of our approach. Notably, without any alterations, our DiffATR consistently exhibits strong performance in out-of-domain retrieval settings.
Bridging Language Gaps in Audio-Text Retrieval
Authors: Zhiyong Yan; Heinrich Dinkel; Yongqing Wang; Jizhong Liu; Junbo Zhang; Yujun Wang; Bin Wang
Venue: Interspeech 2024 (main)

Abstract: Audio-text retrieval is a challenging task, requiring the search for an audio clip or a text caption within a database.  The predominant focus of existing research on English descriptions poses a limitation on the applicability of such models, given the abundance of non-English content in real-world data. To address these linguistic disparities, we propose a language enhancement (LE), using a multilingual text encoder (SONAR) to encode the text data with language-specific information. Additionally, we optimize the audio encoder through the application of consistent ensemble distillation (CED), enhancing support for variable-length audio-text retrieval. Our methodology excels in English audio-text retrieval, demonstrating state-of-the-art (SOTA) performance on commonly used datasets such as AudioCaps and Clotho. Simultaneously, the approach exhibits proficiency in retrieving content in seven other languages with only 10\% of additional language-enhanced training data, yielding promising results.
Domain Adaptation for Contrastive Audio-Language Models
Authors: Soham Deshmukh; Rita Singh; Bhiksha Raj
Venue: Interspeech 2024 (main)

Abstract: Audio-Language Models (ALM) aim to be general-purpose audio models by providing zero-shot capabilities at test time. The zero-shot performance of ALM improves by using suitable text prompts for each domain. The text prompts are usually hand-crafted through an ad-hoc process and lead to a drop in ALM generalization and out-of-distribution performance. Existing approaches to improve domain performance, like few-shot learning or fine-tuning, require access to annotated data and iterations of training. Therefore, we propose a test-time domain adaptation method for ALMs that does not require access to annotations. Our method learns a domain vector by enforcing consistency across augmented views of the testing audio. We extensively evaluate our approach on 12 downstream tasks across domains. With just one example, our domain adaptation method leads to 3.2% (max 8.4%) average zero-shot performance improvement. After adaptation, the model still retains the generalization property of ALMs.
tinyCLAP: Distilling Constrastive Language-Audio Pretrained Models
Authors: Francesco Paissan; Elisabetta Farella
Venue: Interspeech 2024 (main)

Abstract: Contrastive Language-Audio Pretraining (CLAP) became of crucial importance in the field of audio and speech processing. Its employment ranges from sound event detection to text-to-audio generation. However, one of the main limitations is the considerable amount of data required in the training process and the overall computational complexity during inference. This paper investigates how we can reduce the complexity of contrastive language-audio pre-trained models, yielding an efficient model that we call tinyCLAP. We derive an unimodal distillation loss from first principles and explore how the dimensionality of the shared, multimodal latent space can be reduced via pruning. tinyCLAP uses only 6% of the original Microsoft CLAP parameters with a minimal reduction (less than 5%) in zero-shot classification performance across the three sound event detection datasets on which it was tested.
BTS: Bridging Text and Sound Modalities for Metadata-Aided Respiratory Sound Classification
Authors: June-Woo Kim; Miika Toikkanen; Yera Choi; Seoung-Eun Moon; Ho-Young Jung
Venue: Interspeech 2024 (main)

Abstract: Respiratory sound classification (RSC) is challenging due to varied acoustic signatures, primarily influenced by patient demographics and recording environments. To address this issue, we introduce a text-audio multimodal model that utilizes metadata of respiratory sounds, which provides useful complementary information for RSC. Specifically, we fine-tune a pretrained text-audio multimodal model using free-text descriptions derived from the sound samples' metadata which includes the gender and age of patients, type of recording devices, and recording location on the patient's body. Our method achieves state-of-the-art performance on the ICBHI dataset, surpassing the previous best result by a notable margin of 1.17%. This result validates the effectiveness of leveraging metadata and respiratory sound samples in enhancing RSC performance. Additionally, we investigate the model performance in the case where metadata is partially unavailable, which may occur in real-world clinical setting.
Enhanced Feature Learning with Normalized Knowledge Distillation for Audio Tagging
Authors: Yuwu Tang; Ziang Ma; Haitao Zhang
Venue: Interspeech 2024 (main)

Abstract: Pre-trained transformer-based models have been the mainstream of audio tagging. Transformer-based models bring high performance at the cost of huge model size and slow inference speed, while pre-training methods heavily rely on large-scale data and vast computing resources. We argue that a more lightweight CNN-based backbone with customized feature learning can achieve the comparable performance as transformers. Thus an efficient audio tagging framework is proposed to capture more abundant feature information with several enhanced feature learning blocks. We further employ the method of knowledge distillation (KD) and propose a normalized KD loss calculation with adaptive temperature coefficients according to the samples distribution. Extensive experiments demonstrate that our method obtains the state-of-the-art results with a lightweight CNN-based model.
RaD-Net 2: A causal two-stage repairing and denoising speech enhancement network with knowledge distillation and complex axial self-attention
Authors: Mingshuai Liu; Zhuangqi Chen; Xiaopeng Yan; Yuanjun Lv; Xianjun Xia; Chuanzeng Huang; Yijian Xiao; Lei Xie
Venue: Interspeech 2024 (main)

Abstract: In real-time speech communication systems, speech signals are often degraded by multiple distortions. Recently, a two-stage Repair-and-Denoising network (RaD-Net) was proposed with superior speech quality improvement in the ICASSP 2024 Speech Signal Improvement (SSI) Challenge. However, failure to use future information and constraint receptive field of convolution layers limit the system's performance. To mitigate these problems, we extend RaD-Net to its upgraded version, RaD-Net 2. Specifically, a causality-based knowledge distillation is introduced in the first stage to use future information in a causal way. We use the non-causal repairing network as the teacher to improve the performance of the causal repairing network. In addition, in the second stage, complex axial self-attention is applied in the denoising network's complex feature encoder/decoder. Experimental results on the ICASSP 2024 SSI Challenge blind test set show that RaD-Net 2 brings 0.10 OVRL DNSMOS improvement compared to RaD-Net.
DNN-based monaural speech enhancement using alternate analysis windows for phase and magnitude modification
Authors: Xi Liu; John H.L. Hansen
Venue: Interspeech 2024 (main)

Abstract: In recent decades, considerable research has been devoted to speech enhancement leveraging the short-term Fourier transform (STFT) analysis. As speech processing technology evolves, the significance of phase information in enhancing speech intelligibility becomes more noticeable. Typically, the Hanning window has been widely employed as analysis window in STFT. In this study, we propose the Chebyshev window for phase analysis, and the Hanning window for magnitude analysis. Next, we introduce a novel cepstral domain enhancement approach designed to robustly reinforce the harmonic structure of speech. The performance of our model is evaluated using the DNS challenge test set as well as the naturalistic APOLLO Fearless Steps evaluation set. Experimental results demonstrate that the Chebyshev-based phase solution outperforms the Hanning option for in phase-aware speech enhancement. Furthermore, the incorporation of quefrency emphasis proves effective in enhancing overall speech quality.
Improved Remixing Process for Domain Adaptation-Based Speech Enhancement by Mitigating Data Imbalance in Signal-to-Noise Ratio
Authors: Li Li; Shogo Seki
Venue: Interspeech 2024 (main)

Abstract: RemixIT and Remixed2Remixed are domain adaptation-based speech enhancement (DASE) methods that use a teacher model trained under full supervision to generate pseudo-paired data by remixing the outputs of the teacher model. The student model for enhancing real-world recorded signals is trained using pseudo-paired data without the ground truth. Because noisy signals are recorded in natural environments, the dataset inevitably suffers from data imbalance in some acoustic properties, leading to subpar performance for the underrepresented data. The signal-to-noise ratio (SNR), inherently balanced in supervised learning, is a prime example. In this paper, we provide empirical evidence that the SNR of pseudo-data has a significant impact on model performance using the dataset of the CHiME-7 UDASE task, highlighting the importance of a balanced SNR in DASE. Furthermore, we propose adopting curriculum learning to encompass a broad range of SNRs to boost the performance of underrepresented data.
Neural Network Augmented Kalman Filter for Robust Acoustic Howling Suppression
Authors: Yixuan Zhang; Hao Zhang; Meng Yu; Dong Yu
Venue: Interspeech 2024 (main)

Abstract: Acoustic howling suppression (AHS) is a critical challenge in audio communication systems. In this paper, we propose a novel approach that leverages the power of neural networks (NN) to enhance the performance of traditional Kalman filter algorithms for AHS. Specifically, our method involves the integration of NN modules into the Kalman filter, enabling refining reference signal, a key factor in effective adaptive filtering, and estimating covariance metrics for the filter which are crucial for adaptability in dynamic conditions, thereby obtaining improved AHS performance. As a result, the proposed method achieves improved AHS performance compared to both standalone NN and Kalman filter methods. Experimental evaluations validate the effectiveness of our approach.
Improving Speech Enhancement by Integrating Inter-Channel and Band Features with Dual-branch Conformer
Authors: Jizhen Li; Xinmeng Xu; Weiping Tu; Yuhong Yang; Rong Zhu
Venue: Interspeech 2024 (main)

Abstract: Recent speech enhancement methods based on convolutional neural networks (CNNs) and transformer have been demonstrated to efficaciously capture time-frequency (T-F) information on spectrogram. However, the correlation of each channels of speech features is failed to explore. Theoretically, each channel map of speech features obtained by different convolution kernels contains information with different scales demonstrating strong correlations. To fill this gap, we propose a novel dual-branch architecture named channel-aware dualbranch conformer (CADB-Conformer), which effectively explores the long range time and frequency correlations among different channels, respectively, to extract channel relation aware time-frequency information. Ablation studies conducted on DNS-Challenge 2020 dataset demonstrate the importance of channel feature leveraging while showing the significance of channel relation aware T-F information for speech enhancement. Extensive experiments also show that the proposed model achieves superior performance than recent methods with an attractive computational costs.
An Exploration of Length Generalization in Transformer-Based Speech Enhancement
Authors: Qiquan Zhang; Hongxu Zhu; Xinyuan Qian; Eliathamby Ambikairajah; Haizhou Li
Venue: Interspeech 2024 (main)

Abstract: The use of Transformer architectures has facilitated remarkable progress in speech enhancement. Training Transformers using substantially long speech utterances is often infeasible as self-attention suffers from quadratic complexity. It is a critical and unexplored challenge for a Transformer-based speech enhancement model to learn from short speech utterances and generalize to longer ones. In this paper, we conduct comprehensive experiments to explore the length generalization problem in speech enhancement with Transformer. Our findings first establish that position embedding provides an effective instrument to alleviate the impact of utterance length on Transformer-based speech enhancement. Specifically, we explore four different position embedding schemes to enable length generalization. The results confirm the superiority of relative position embeddings (RPEs) over absolute PE (APEs) in length generalization.