GenAI4ELab/papercli-papers huggingface dataset:
Vision Transformer Segmentation for Visual Bird Sound Denoising
Authors: Sahil Kumar; Jialu Li; Youshan Zhang
Venue: Interspeech 2024 (main)

Abstract: Audio denoising, especially in the context of bird sounds, remains a challenging task due to persistent residual noise. Traditional and deep learning methods often struggle with artificial or low-frequency noise. In this work, we propose ViTVS, a novel approach that leverages the power of the vision transformer (ViT) architecture. ViTVS adeptly combines segmentation techniques to disentangle clean audio from complex signal mixtures. Our key contributions encompass the development of ViTVS, introducing comprehensive, long-range, and multi-scale representations. These contributions directly tackle the limitations inherent in conventional approaches. Extensive experiments demonstrate that ViTVS outperforms state-of-the-art methods, positioning it as a benchmark solution for real-world bird sound denoising applications. Source code is available at: https://github.com/aiai-4/ViVTS.
DB3V: A Dialect Dominated Dataset of Bird Vocalisation for Cross-corpus Bird Species Recognition
Authors: Xin Jing; Luyang Zhang; Jiangjian Xie; Alexander Gebhard; Alice Baird; BjÃ¶rn Schuller
Venue: Interspeech 2024 (main)

Abstract: In ornithology, bird species are known to have varieditâs widely acknowledged that bird species display diverse dialects in their calls across different regions. Consequently, computational methods to identify bird species onsolely through their calls face critsignificalnt challenges. There is growing interest in understanding the impact of species-specific dialects on the effectiveness of bird species recognition methods. Despite potential mitigation through the expansion of dialect datasets, the absence of publicly available testing data currently impedes robust benchmarking efforts. This paper presents the Dialect Dominated Dataset of Bird Vocalisation (D3BV), the first cross-corpus dataset that focuses on dialects in bird vocalisations. The D3BV comprises more than 25 hours of audio recordings from 10 bird species distributed across three distinct regions in the contiguous United States (CONUS). In addition to presenting the dataset, we conduct analyses and establish baseline models for cross-corpus bird recognition. The data and code are publicly available online: https://zenodo.org/records/11544734
Investigating self-supervised speech models' ability to classify animal vocalizations: The case of gibbon's vocal signatures
Authors: Jules Cauzinille; BenoÃ®t Favre; Ricard Marxer; Dena Clink; Abdul Hamid Ahmad; Arnaud Rey
Venue: Interspeech 2024 (main)

Abstract: With the advent of pre-trained self-supervised learning (SSL) models, speech processing research is showing increasing interest towards disentanglement and explainability. Amongst other methods, probing speech classifiers has emerged as a promising approach to gain new insights into SSL models out-of-domain performances. We explore knowledge transfer capabilities of pre-trained speech models with vocalizations from the closest living relatives of humans: non-human primates. We focus on classifying the identity of northern grey gibbons (Hylobates funereus) from their calls with probing and layer-wise analysis of state-of-the-art SSL speech models compared to pre-trained bird species classifiers and audio taggers. By testing the reliance of said models on background noise and timewise information, as well as performance variations across layers, we propose a new understanding of the mechanisms underlying speech models efficacy as bioacoustic tools.
Study Selectively: An Adaptive Knowledge Distillation based on a Voting Network for Heart Sound Classification
Authors: Xihang Qiu; Lixian Zhu; Zikai Song; Zeyu Chen; Haojie Zhang; Kun Qian; Ye Zhang; Bin Hu; Yoshiharu Yamamoto; BjÃ¶rn W. Schuller
Venue: Interspeech 2024 (main)

Abstract: Phonocardiogram classification methods using deep neural networks have been widely applied to the early detection of cardiovascular diseases recently. Despite their excellent recognition rate, the sizeable computational complexity limits their further development. Nowadays, knowledge distillation (KD) is an established paradigm for model compression. While current research on multi-teacher KD has shown potential to impart more comprehensive knowledge to the student than single-teacher KD, this approach is not suitable for all scenarios. This paper proposes a novel KD strategy to realise an adaptive multi-teacher instruction mechanism. We design a teacher selection strategy called voting network to tell the contribution of different teachers on each distillation points, so that the student can choose the useful information and renounce the redundant one. An evaluation demonstrates that our method reaches excellent accuracy (92.8%) while maintaining a low computational complexity (0.7M).
SimuSOE: A Simulated Snoring Dataset for Obstructive Sleep Apnea-Hypopnea Syndrome Evaluation during Wakefulness
Authors: Jie Lin; Xiuping Yang; Li Xiao; Xinhong Li; Weiyan Yi; Yuhong Yang; Weiping Tu; Xiong Chen
Venue: Interspeech 2024 (main)

Abstract: Obstructive Sleep Apnea-Hypopnea Syndrome (OSAHS) is a prevalent chronic breathing disorder caused by upper airway obstruction. Previous studies advanced OSAHS evaluation through machine learning-based systems trained on sleep snoring or speech signal datasets. However, constructing datasets for training a precise and rapid OSAHS evaluation system poses a challenge, since 1) it is time-consuming to collect sleep snores and 2) the speech signal is limited in reflecting upper airway obstruction. In this paper, we propose a new snoring dataset for OSAHS evaluation, named SimuSOE, in which a novel and time-effective snoring collection method is introduced for tackling the above problems. In particular, we adopt simulated snoring which is a type of snore intentionally emitted by patients to replace natural snoring. Experimental results indicate that the simulated snoring signal during wakefulness can serve as an effective feature in OSAHS preliminary screening.
Multi-mic Echo Cancellation Coalesced with Beamforming for Real World Adverse Acoustic Conditions
Authors: Premanand Nayak; Kamini Sabu; M. Ali Basha Shaik
Venue: Interspeech 2024 (main)

Abstract: Robust acoustic echo cancellation (AEC) is essential for voice enabled smart devices. Multi-channel signals are used in AEC along with beamformer (BF) for better residual echo suppression (RES). In this work, we introduce a deep neural network (DNN) based novel unified framework for multi-microphone AEC (MMAEC) and RES under adverse signal-to-echo (SER) conditions. We propose the use of deep-MVDR which uses deep steering vector and deep power spectral density (Deep PSD) estimator to conceptually implement minimum variance distortionless beamformer. We also introduce additional novelty in our framework by jointly training the MMAEC and deep-MVDR modules. Both of these methods give consistent significant improvement in ERLE which is further enriched by the incorporation of playback reconstruction loss. Our system outperforms competitive baselines while being robust in adverse real-world conditions such as very low input SER, dominant far-end sources, and moving near-end speech sources.
Interference Aware Training Target for DNN based joint Acoustic Echo Cancellation and Noise Suppression
Authors: Vahid Khanagha; Dimitris Koutsaidis; Kaustubh Kalgaonkar; Sriram Srinivasan
Venue: Interspeech 2024 (main)

Abstract: Despite remarkable performance of Deep Learning based Acoustic Echo Cancellation (AEC) systems, effective handling of double-talk scenarios remains a challenge. During double-talk the speech signal from the far-end talker overlaps with the target near-end speech and results in degraded performance in form of near-end speech deletions or audible echo residuals shadowing the voice of target speaker.  This paper introduces an approach to reduce the shadowing effect through altering the ground truth used for model training such that the model can effectively clean up spectral components where the interference is stronger than the target speech. The alteration is accomplished leveraging the availability of interference signal during training data generation by masking spectral components of the ground truth where target speech is significantly weaker than the interference. Large scale subjective evaluation trials show that human listeners prefer the outputs generated by the new approach.
Low Complexity Echo Delay Estimator Based on Binarized Feature Matching
Authors: Yi Gao; Xiang Su
Venue: Interspeech 2024 (main)

Abstract: Echo delay estimation (EDE) serves as a preprocessing component within an acoustic echo canceller (AEC). Despite some progress over the past few decades, there is a dearth of literature on efficient algorithms. This paper introduces a binarized feature-matching (BFM) framework, encompassing a set of feature extraction methods, which are compared with traditional methods such as the GCC-Phat-based method, the adaptive-filter-based method, and popular methods published in WebRTC projects, in terms of both complexity and performance. The computational loads of the BFM methods are significantly lower, and a hybrid BFM method further enhances performance in terms of convergence speed and robustness. This method, characterized by its low complexity, benefits both traditional and NN-based AEC.
MSA-DPCRN: A Multi-Scale Asymmetric Dual-Path Convolution Recurrent Network with Attentional Feature Fusion for Acoustic Echo Cancellation
Authors: Ye Ni; Cong Pang; Chengwei Huang; Cairong Zou
Venue: Interspeech 2024 (main)

Abstract: Echo cancellation plays a crucial role in modern speech applications. Numerous deep-learning models have been developed for the echo cancellation task and achieved great progress by incorporating additional features; however, the majority of these models overlook the characteristics of different features and simply merge them along the channel dimension. In this paper, we proposed a multi-scale asymmetric dual-path convolution recurrent network (MSA-DPCRN) consisting of two asymmetric encoding paths to extract spectrum and relevant features from the input reference and microphone signals. Moreover, we propose a frequency-wise attentional feature fusion (AFF) method to fuse the two features while maintaining the original dynamic range. The experiments validate the effectiveness of each component in MSA-DPCRN and indicate that our model outperforms the AEC challenge baseline in terms of the Echo-MOS metrics.
Efficient Joint Bemforming and Acoustic Echo Cancellation Structure for Conference Call Scenarios
Authors: Ofer Schwartz; Sharon Gannot
Venue: Interspeech 2024 (main)

Abstract: We propose an efficient scheme for combining beamformer (BF) and acoustic echo cancellation (AEC). We focus on conference call scenarios characterized by stationary background noise and multiple speakers who alternate frequently. Furthermore, aiming at low-resource devices, a common strategy is to apply a single AEC at the output of the BF rather than applying multiple AECs to each microphone signal. The main drawback of such a structure is the frequent change of the echo path due to the BF adaptation. To circumvent this problem, it is proposed to apply a single-channel pre-filter to the far-end signal, encompassing the BF weights and the relative acoustic responses between the reference microphone and all other microphones w.r.t. the echo loudspeaker. As a result, the AEC block becomes indifferent to the changes in the BF weights. The proposed scheme is evaluated using real recordings and was found advantageous over standard combined AEC-BF schemes in terms of speed of convergence.
SDAEC: Signal Decoupling for Advancing Acoustic Echo Cancellation
Authors: Fei Zhao; Jinjiang Liu; Xueliang Zhang
Venue: Interspeech 2024 (main)

Abstract: In deep learning-based acoustic echo cancellation methods, neural networks implicitly learn echo paths to cancel echoes. However, under low signal-to-echo ratio conditions, the substantial energy discrepancy between the microphone signal and the reference signal impedes the network's ability, resulting in poor performance. In this study, we propose a Signal Decoupling-based monaural Acoustic Echo Cancellation method called SDAEC. Specifically, we model the energy of the reference signal and the microphone signal to obtain an energy scaling factor. The reference signal is then multiplied by this energy scaling factor before being fed into the subsequent echo cancellation network. This approach reduces the difficulty of the subsequent echo cancellation step, thereby improving the overall cancellation performance. Experimental results demonstrate that the proposed method enhances the performance of multiple baseline models.
Spatial Voice Conversion: Voice Conversion Preserving Spatial Information and Non-target Signals
Authors: Kentaro Seki; Shinnosuke Takamichi; Norihiro Takamune; Yuki Saito; Kanami Imamura; Hiroshi Saruwatari
Venue: Interspeech 2024 (main)

Abstract: This paper proposes a new task called spatial voice conversion, which aims to convert a target voice while preserving spatial information and non-target signals. Traditional voice conversion methods focus on single-channel waveforms, ignoring the stereo listening experience inherent in human hearing. Our baseline approach addresses this gap by integrating blind source separation (BSS), voice conversion (VC), and spatial mixing to handle multi-channel waveforms. Through experimental evaluations, we organize and identify the key challenges inherent in this task, such as maintaining audio quality and accurately preserving spatial information. Our results highlight the fundamental difficulties in balancing these aspects, providing a benchmark for future research in spatial voice conversion. The proposed method's code is publicly available to encourage further exploration in this domain.
Neural Codec Language Models for Disentangled and Textless Voice Conversion
Authors: Alan Baade; Puyuan Peng; David Harwath
Venue: Interspeech 2024 (main)

Abstract: We introduce a method for textless any-to-any voice conversion based on the recent progress in speech synthesis driven by neural codec language models. To disentangle the speaker and linguistic information, we adapt a speaker normalizing procedure for discrete semantic units, and then generate with an autoregressive language model for greatly improved diversity. We further improve the similarity of the output audio to the target speaker's voice by leveraging classifier free guidance. We evaluate our techniques against current text to speech synthesis and voice conversion systems and compare the effectiveness of different neural codec language model pipelines. We demonstrate state-of-the-art results in accent disentanglement and speaker similarity for voice conversion with significantly less compute than existing codec language models such as VALL-E.
Fine-Grained and Interpretable Neural Speech Editing
Authors: Max Morrison; Cameron Churchwell; Nathan Pruyne; Bryan Pardo
Venue: Interspeech 2024 (main)

Abstract: Fine-grained editing of speech attributes - such as prosody (i.e., the pitch, loudness, and phoneme durations), pronunciation, speaker identity, and formants - is useful for fine-tuning and fixing imperfections in human and AI-generated speech recordings for creation of podcasts, film dialogue, and video game dialogue. Existing speech synthesis systems use representations that entangle two or more of these attributes, prohibiting their use in fine-grained, disentangled editing. In this paper, we demonstrate the first disentangled and interpretable representation of speech with comparable subjective and objective vocoding reconstruction accuracy to Mel spectrograms. Our interpretable representation, combined with our proposed data augmentation method, enables training an existing neural vocoder to perform fast, accurate, and high-quality editing of pitch, duration, volume, timbral correlates of volume, pronunciation, speaker identity, and spectral balance.
FastVoiceGrad: One-step Diffusion-Based Voice Conversion with Adversarial Conditional Diffusion Distillation
Authors: Takuhiro Kaneko; Hirokazu Kameoka; Kou Tanaka; Yuto Kondo
Venue: Interspeech 2024 (main)

Abstract: Diffusion-based voice conversion (VC) techniques such as VoiceGrad have attracted interest because of their high VC performance in terms of speech quality and speaker similarity. However, a notable limitation is the slow inference caused by the multi-step reverse diffusion. Therefore, we propose FastVoiceGrad, a novel one-step diffusion-based VC that reduces the number of iterations from dozens to one while inheriting the high VC performance of the multi-step diffusion-based VC. We obtain the model using adversarial conditional diffusion distillation (ACDD), leveraging the ability of generative adversarial networks and diffusion models while reconsidering the initial states in sampling. Evaluations of one-shot any-to-any VC demonstrate that FastVoiceGrad achieves VC performance superior to or comparable to that of previous multi-step diffusion-based VC while enhancing the inference speed.
DualVC 3: Leveraging Language Model Generated Pseudo Context for End-to-end Low Latency Streaming Voice Conversion
Authors: Ziqian Ning; Shuai Wang; Pengcheng Zhu; Zhichao Wang; Jixun Yao; Lei Xie; Mengxiao Bi
Venue: Interspeech 2024 (main)

Abstract: Streaming voice conversion has gained popularity for its applicability in real-time applications. The recently proposed DualVC 2 has successfully achieved robust and high-quality streaming voice conversion in approximately 180ms.  However, DualVC 2 is based on the recognition-synthesis framework, with multi-level cascade models that cannot be jointly optimized, and faces severe performance drops with small chunks caused by the ASR encoder. To address these issues, we propose an end-to-end model DualVC 3. It incorporates K-means clustered SSL features to guide the training of the content encoder and adopts an optional language model for pseudo-content generation to improve the conversion quality. Experimental results demonstrate that DualVC 3 achieves comparable performance to DualVC 2 in both subjective and objective metrics, with a latency of only 50 ms. We have made our audio samples publicly available.
Towards Realistic Emotional Voice Conversion using Controllable Emotional Intensity
Authors: Tianhua Qi; Shiyan Wang; Cheng Lu; Yan Zhao; Yuan Zong; Wenming Zheng
Venue: Interspeech 2024 (main)

Abstract: Realistic emotional voice conversion (EVC) aims to enhance emotional diversity of converted audios, making the synthesized voices more authentic and natural. To this end, we propose Emotional Intensity-aware Network (EINet), dynamically adjusting intonation and rhythm by incorporating controllable emotional intensity. To better capture nuances in emotional intensity, we go beyond mere distance measurements among acoustic features. Instead, an emotion evaluator is utilized to precisely quantify speakerâs emotional state. By employing an intensity mapper, intensity pseudo-labels are obtained to bridge the gap between emotional speech intensity modeling and run-time conversion. To ensure high speech quality while retaining controllability, an emotion renderer is used for combining linguistic features smoothly with manipulated emotional features at frame level. Furthermore, we employ a duration predictor to facilitate adaptive prediction of rhythm changes condition on specifying intensity value. Experimental results show EINetâs superior performance in naturalness and diversity of emotional expression compared to state-of-the-art EVC methods.
InterBiasing: Boost Unseen Word Recognition through Biasing Intermediate Predictions
Authors: Yu Nakagome; Michael Hentschel
Venue: Interspeech 2024 (main)

Abstract: Despite recent advances in end-to-end speech recognition methods, their output is biased to the training dataâs vocabulary, resulting in inaccurate recognition of unknown terms or proper nouns. To improve the recognition accuracy for a given set of such terms, we propose an adaptation parameter-free approach based on Self-conditioned CTC. Our method improves the recognition accuracy of misrecognized target keywords by substituting their intermediate CTC predictions with corrected labels, which are then passed on to the subsequent layers. First, we create pairs of correct labels and recognition error instances for a keyword list using Text-to-Speech and a recognition model. We use these pairs to replace intermediate prediction errors by the labels. Conditioning the subsequent layers of the encoder on the labels, it is possible to acoustically evaluate the target keywords. Experiments conducted in Japanese demonstrated that our method successfully improved the F1 score for unknown words.
SEQ-former: A context-enhanced and efficient automatic speech recognition framework
Authors: Qinglin Meng; Min Liu; Kaixun Huang; Kun Wei; Lei Xie; Zongfeng Quan; Weihong Deng; Quan Lu; Ning Jiang; Guoqing Zhao
Venue: Interspeech 2024 (main)

Abstract: Contextual information is crucial for automatic speech recognition (ASR). Effective utilization of contextual information can improve the accuracy of ASR systems. To improve the model's ability to capture this information, we propose a novel ASR framework called SEQ-former, emphasizing simplicity, efficiency, and quickness. We incorporate a Prediction Decoder Network and a Shared Prediction Decoder Network to enhance contextual capabilities. To further increase efficiency, we use intermediate CTC and CTC Spike Reduce Methods to guide attention masks and reduce redundant peaks. Our approach demonstrates state-of-the-art performance on the AiShell-1 dataset, improves decoding efficiency, and delivers competitive results on LibriSpeech. Additionally, it optimizes 6.3% over 11,000 hours of private data compared to Efficient Conformer.
How Much Context Does My Attention-Based ASR System Need?
Authors: Robert Flynn; Anton Ragni
Venue: Interspeech 2024 (main)

Abstract: For the task of speech recognition, the use of more than 30 seconds of acoustic context during training is uncommon and under-investigated in literature. In this work, we conduct an empirical study on the effect of scaling the sequence length used to train/evaluate (dense-attention-based) acoustic models on speech recognition performance. For these experiments, a dataset of roughly 100,000 pseudo-labelled Spotify podcasts is used, with context lengths of 5 seconds to 1 hour being explored. Zero-shot evaluations are presented on the long-format datasets: Earnings-22, Tedlium and Rev16. Results demonstrate a benefit from training with up to 21.8 minutes of acoustic context, showing up to a 14.5% relative improvement from a baseline trained with 10 seconds of context. We find that the model's width/depth, positional encoding scheme and number of attention heads impact its ability to use longer contexts.
Rich speech signal: exploring and exploiting end-to-end automatic speech recognizersâ ability to model hesitation phenomena
Authors: Vincenzo Norman Vitale; Loredana Schettino; Francesco Cutugno
Venue: Interspeech 2024 (main)

Abstract: Modern automatic speech recognition systems can achieve remarkable performances. However, they usually neglect speech characteristic phenomena such as fillers () or segmental prolongations (the) which are still only considered as disrupting objects to be detected and removed, despite their acknowledged regularity and procedural value. This study investigates the ability of state-of-the-art systems based on end-to-end models (E2E-ASRs) to model distinctive features of hesitation phenomena. Two types of pre-trained systems with the same Conformer-based encoding architecture but different decoders are evaluated: the Connectionist Temporal Classification (CTC) decoder and a Transducer decoder. E2E-ASRs ability to model the acoustic information tied to such phenomena can be exploited rather than disregarded as a noise source, which would not only improve transcription and support linguistic annotation processes, but also deepen our understanding of the systemsâ working.