GenAI4ELab/papercli-papers huggingface dataset:
Can Synthetic Audio From Generative Foundation Models Assist Audio Recognition and Speech Modeling?
Authors: Tiantian Feng; Dimitrios Dimitriadis; Shrikanth S. Narayanan
Venue: Interspeech 2024 (main)

Abstract: Recent advances in foundation models have enabled audio-generative models that produce high-fidelity sounds associated with music, events, and human actions. Despite the success achieved in modern audio-generative models, the conventional approach to assessing the quality of the audio generation relies heavily on distance metrics like Frechet Audio Distance. In contrast, we aim to evaluate the quality of audio generation by examining the effectiveness of using them as training data. Specifically, we conduct studies to explore the use of synthetic audio for audio recognition. Moreover, we investigate whether synthetic audio can serve as a resource for data augmentation in speech-related modeling. Our comprehensive experiments demonstrate the potential of using synthetic audio for audio recognition and speech-related modeling. Our code is available at https://github.com/usc-sail/SynthAudio.
Scaling up masked audio encoder learning for general audio classification
Authors: Heinrich Dinkel; Zhiyong Yan; Yongqing Wang; Junbo Zhang; Yujun Wang; Bin Wang
Venue: Interspeech 2024 (main)

Abstract: Despite progress in audio classification, a generalization gap remains between speech and other sound domains, such as environmental sounds and music. Models trained for speech tasks often fail to perform well on environmental or musical audio tasks, and vice versa. While self-supervised (SSL) audio representations offer an alternative, there has been limited exploration of scaling both model and dataset sizes for SSL-based general audio classification. We introduce Dasheng, a simple SSL audio encoder, based on the efficient masked autoencoder framework.  Trained with 1.2 billion parameters on 272,356 hours of diverse audio, Dasheng obtains significant performance gains on the HEAR benchmark. It outperforms previous works on CREMA-D, LibriCount, Speech Commands, VoxLingua, and competes well in music and environment classification. Dasheng features inherently contain rich speech, music, and environmental information, as shown in nearest-neighbor classification experiments.
Audio Mamba: Selective State Spaces for Self-Supervised Audio Representations
Authors: Sarthak Yadav; Zheng-Hua Tan
Venue: Interspeech 2024 (main)

Abstract: Despite its widespread adoption as the prominent neural architecture,  the Transformer has spurred several independent lines of work to address its limitations. One such approach is selective state space models, which have demonstrated promising results for language modelling. However, their feasibility for learning self-supervised, general-purpose audio representations is yet to be investigated. This work proposes Audio Mamba, a selective state space model for learning general-purpose audio representations from randomly masked spectrogram patches through self-supervision. Empirical results on ten diverse audio recognition downstream tasks show that the proposed models, pretrained on the AudioSet dataset, consistently outperform comparable self-supervised audio spectrogram transformer (SSAST) baselines by a considerable margin and demonstrate better performance in dataset size, sequence length and model size comparisons.
MAT-SED: A Masked Audio Transformer with Masked-Reconstruction Based Pre-training for Sound Event Detection
Authors: Pengfei Cai; Yan Song; Kang Li; Haoyu Song; Ian McLoughlin
Venue: Interspeech 2024 (main)

Abstract: Sound event detection (SED) methods that leverage a large pre-trained Transformer encoder network have shown promising performance in recent DCASE challenges. However, they still rely on an RNN-based context network to model temporal dependencies, largely due to the scarcity of labeled data. In this work, we propose a pure Transformer-based SED model with masked-reconstruction based pre-training, termed MAT-SED. Specifically, a Transformer with relative positional encoding is first designed as the context network, pre-trained by the masked-reconstruction task on all available target data in a self-supervised way. Both the encoder and the context network are jointly fine-tuned in a semi-supervised manner. Furthermore, a global-local feature fusion strategy is proposed to enhance the localization capability. Evaluation of MAT-SED on DCASE2023 task4 surpasses state-of-the-art performance, achieving 0.587/0.896 PSDS1/PSDS2 respectively.
Sound Event Bounding Boxes
Authors: Janek Ebbers; FranÃ§ois G. Germain; Gordon Wichern; Jonathan Le Roux
Venue: Interspeech 2024 (main)

Abstract: Sound event detection is the task of recognizing sounds and determining their extent (onset/offset times) within an audio clip. Existing systems commonly predict sound presence posteriors in short time frames. Then, thresholding produces binary frame-level presence decisions, with the extent of individual events determined by merging presence in consecutive frames. In this paper, we show that frame-level thresholding deteriorates event extent prediction by coupling it with the systemâs sound presence confidence. We propose to decouple the prediction of event extent and confidence by introducing sound event bounding boxes (SEBBs), which format each sound event prediction as a combination of a class type, extent, and overall confidence. We also propose a change-detection-based algorithm to convert frame-level posteriors into SEBBs. We find the algorithm significantly improves the performance of DCASE 2023 Challenge systems, boosting the state of the art from .644 to .686 PSDS1.
Low-Complexity Acoustic Scene Classification Using Parallel Attention-Convolution Network
Authors: Yanxiong Li; Jiaxin Tan; Guoqing Chen; Jialong Li; Yongjie Si; Qianhua He
Venue: Interspeech 2024 (main)

Abstract: This work is an improved system that we submitted to task 1 of DCASE2023 challenge. We propose a method of low-complexity acoustic scene classification by a parallel attention-convolution network which consists of four modules, including pre-processing, fusion, global and local contextual information extraction. The proposed network is computationally efficient to capture global and local contextual information from each audio clip. In addition, we integrate other techniques into our method, such as knowledge distillation, data augmentation, and adaptive residual normalization. When evaluated on the official dataset of DCASE2023 challenge, our method obtains the highest accuracy of 56.10% with parameter number of 5.21 kilo and multiply-accumulate operations of 1.44 million. It exceeds the top two systems of DCASE2023 challenge in accuracy and complexity, and obtains state-of-the-art result. Code is at: https://github.com/Jessytan/Low-complexity-ASC.
Towards Explainable Monaural Speaker Separation with Auditory-based Training
Authors: Hassan Taherian; Vahid Ahmadi Kalkhorani; Ashutosh Pandey; Daniel Wong; Buye Xu; DeLiang Wang
Venue: Interspeech 2024 (main)

Abstract: Permutation ambiguity is a major challenge in training monaural talker-independent speaker separation. While permutation invariant training (PIT) is a widely used technique, it functions as a `black box', providing little insight into which auditory cues lead to successful training. We introduce a new approach to speaker separation by leveraging differences in pitch and onset, which are both prominent cues for auditory scene analysis. We propose pitch-based and onset-based training to resolve permutation ambiguity, assigning speakers by their pitch frequencies and onset times, respectively. This approach offers a more explainable training strategy than PIT. We also propose a hybrid criterion combining these cues to improve separation performance in challenging conditions such as same-gender speakers or close utterance onsets. Evaluation results show that pitch and onset criteria each perform competitively to PIT and the hybrid criterion surpasses PIT in separating two-speaker mixtures.
Does the Lombard Effect Matter in Speech Separation? Introducing the Lombard-GRID-2mix Dataset
Authors: Iva Ewert; Marvin Borsdorf; Haizhou Li; Tanja Schultz
Venue: Interspeech 2024 (main)

Abstract: Inspired by the human ability of selective listening, speech separation aims to equip machines with the capability to disentangle cocktail party soundscapes into the individual sound sources. Recently, neural network based algorithms have been studied to work reliably under various conditions. However, to the best of our knowledge, a change in the speaking style has not yet been studied. The Lombard effect, a reflexive change in speaking style triggered by noisy environments, is a typical behavior in everyday conversational situations. In this work, we introduce a new first of its kind dataset, called Lombard-GRID-2mix, to study speech separation for two-speaker mixtures on normal speech and Lombard speech. In a comprehensive study, we show that speech separation systems can be equipped to work for both normal speech and Lombard speech. We apply a carefully designed finetuning method to enable the system to work even if noise is present in the Lombard speech for different SNR ratios.
PARIS: Pseudo-AutoRegressIve Siamese Training for Online Speech Separation
Authors: Zexu Pan; Gordon Wichern; FranÃ§ois G. Germain; Kohei Saijo; Jonathan Le Roux
Venue: Interspeech 2024 (main)

Abstract: While offline speech separation models have made significant advances, the streaming regime remains less explored and is typically limited to causal modifications of existing offline networks. This study focuses on empowering a streaming speech separation model with autoregressive capability, in which the current step separation is conditioned on separated samples from past steps. To do so, we introduce pseudo-autoregressive Siamese (PARIS) training: with only two forward passes through a Siamese-style network for each batch, PARIS avoids the training-inference mismatch in teacher forcing and the need for numerous autoregressive steps during training. The proposed PARIS training improves the recent online SkiM model by 1.5 dB in SI-SNR on the WSJ0-2mix dataset, with minimal change to the network architecture and inference time.
OR-TSE: An Overlap-Robust Speaker Encoder for Target Speech Extraction
Authors: Yiru Zhang; Linyu Yao; Qun Yang
Venue: Interspeech 2024 (main)

Abstract: Mainstream Target Speech Extraction (TSE) systems extract target speech from a mixture using pre-enrolled reference speech. The extraction performance heavily depends on the quality of the reference speech. However, the speech signal of the same speaker may vary under different conditions, leading to a decrease in extraction performance, particularly in speech overlap. Therefore, we propose an overlap robust speaker encoder for TSE to obtain stable speaker embeddings even when using signals with overlapping interference. Our approach combines attentive statistics pooling with contrastive learning to make the model focus on the feature of main speaker while disregarding interfering information. Based on our proposed speaker encoder, we introduce a TSE framework, which derive speaker embeddings from non-overlapping regions of mixture input. The experiments shows that our speaker encoder improves the performance of TSE in different conditions of reference speech.
Multimodal Representation Loss Between Timed Text and Audio for Regularized Speech Separation
Authors: Tsun-An Hsieh; Heeyoul Choi; Minje Kim
Venue: Interspeech 2024 (main)

Abstract: Recent studies highlight the potential of textual modalities in conditioning the speech separation model's inference process. However, regularization-based methods remain underexplored despite their advantages of not requiring auxiliary text data during the test time. To address this gap, we introduce a timed text-based regularization (TTR) method that uses language model-derived semantics to improve speech separation models. Our approach involves two steps. We begin with two pretrained audio and language models, WavLM and BERT, respectively. Then, a Transformer-based audio summarizer is learned to align the audio and word embeddings and to minimize their gap. The summarizer Transformer, incorporated as a regularizer, promotes the separated sources' alignment with the semantics from the timed text. Experimental results show that the proposed TTR method consistently improves the various objective metrics of the separation results over the unregularized baselines.
SA-WavLM: Speaker-Aware Self-Supervised Pre-training for Mixture Speech
Authors: Jingru Lin; Meng Ge; Junyi Ao; Liqun Deng; Haizhou Li
Venue: Interspeech 2024 (main)

Abstract: It was shown that pre-trained models with self-supervised learning (SSL) techniques are effective in various downstream speech tasks. However, most such models are trained on single-speaker speech data, limiting their effectiveness in mixture speech. This motivates us to explore pre-training on mixture speech. This work presents SA-WavLM, a novel pre-trained model for mixture speech. Specifically, SA-WavLM follows an "extract-merge-predict" pipeline in which the representations of each speaker in the input mixture are first extracted individually and then merged before the final prediction. In this pipeline, SA-WavLM performs speaker-informed extractions with the consideration of the interactions between different speakers. Furthermore, a speaker shuffling strategy is proposed to enhance the robustness towards the speaker absence. Experiments show that SA-WavLM either matches or improves upon the state-of-the-art pre-trained models.
TSE-PI: Target Sound Extraction under Reverberant Environments with Pitch Information
Authors: Yiwen Wang; Xihong Wu
Venue: Interspeech 2024 (main)

Abstract: Target sound extraction (TSE) separates the target sound from the mixture signals based on provided clues. However, the performance of existing models significantly degrades under reverberant conditions. Inspired by auditory scene analysis (ASA), this work proposes a TSE model provided with pitch information named TSE-PI. Conditional pitch extraction is achieved through the Feature-wise Linearly Modulated layer with the sound-class label. A modified Waveformer model combined with pitch information, employing a learnable Gammatone filterbank in place of the convolutional encoder, is used for target sound extraction. The inclusion of pitch information is aimed at improving the model's performance. The experimental results on the FSD50K dataset illustrate 2.4 dB improvements of target sound extraction under reverberant environments when incorporating pitch information and Gammatone filterbank.
Enhanced Reverberation as Supervision for Unsupervised Speech Separation
Authors: Kohei Saijo; Gordon Wichern; FranÃ§ois G. Germain; Zexu Pan; Jonathan Le Roux
Venue: Interspeech 2024 (main)

Abstract: Reverberation as supervision (RAS) is a framework that allows for training monaural speech separation models from multi-channel mixtures in an unsupervised manner.  In RAS, models are trained so that sources predicted from a mixture at an input channel can be mapped to reconstruct a mixture at a target channel.  However, stable unsupervised training has so far only been achieved in over-determined source-channel conditions, leaving the key determined case unsolved. This work proposes enhanced RAS (ERAS) for solving this problem. Through qualitative analysis, we found that stable training can be achieved by leveraging the loss term to alleviate the frequency-permutation problem. Separation performance is also boosted by adding a novel loss term where separated signals mapped back to their own input mixture are used as pseudo-targets for the signals separated from other channels and mapped to the same channel. Experimental results demonstrate high stability and performance of ERAS.
Deep Echo Path Modeling for Acoustic Echo Cancellation
Authors: Fei Zhao; Chenggang Zhang; Shulin He; Jinjiang Liu; Xueliang Zhang
Venue: Interspeech 2024 (main)

Abstract: Acoustic echo cancellation (AEC) is a key audio processing technology that removes echoes from microphone inputs to enable natural-sounding full-duplex communication. In recent years, deep learning has shown great potential for advancing AEC. However, deep learning methods face challenges in generalizing to complex environments, especially unseen conditions not represented in training. In this paper, we propose a deep learning-based method to predict the echo path in the time-frequency domain. Specifically, we first estimate the echo path under single-talk scenario without near-end signal and then utilize these predicted echo paths as auxiliary labels to train the model on double-talk scenario with near-end signal. Experimental results show that our method outperforms the strong baselines and exhibits good generalization capabilities for unseen acoustic scenarios. By estimating the echo path using deep learning, this work advances AEC performance in the presence of complex conditions.
Graph Attention Based Multi-Channel U-Net for Speech Dereverberation With Ad-Hoc Microphone Arrays
Authors: Hongmei Guo; Yijiang Chen; Xiao-Lei Zhang; Xuelong Li
Venue: Interspeech 2024 (main)

Abstract: Speech dereverberation with ad-hoc microphone arrays seems not studied sufficiently, particularly in the scenario where the reverberation time is large. In this paper, we propose a novel multi-channel U-Net model for speech dereverberation with ad-hoc microphone arrays, where an attention module is integrated into the model in an end-to-end training manner to do channel selection and fusion. Specifically, we first train a single-channel U-Net model. Then, we replicate the U-Net model to each channel. Finally, we train the attention module for aggregating the information of the channels, where the parameters of the U-Net model are fixed at this stage. To our knowledge, this is the first work that U-Net was used for dereverberation with ad-hoc microphone arrays. We studied two attention mechanism, which are the self-attention and graph-attention; moreover, we integrated the attention module into either the bottleneck layer or the output layer of the multi-channel U-Net, which results in four implementations. Experimental results demonstrate that the proposed method achieves the state-of-the-art performance, and the attention module is very important in channel selection and fusion for improving the performance against long reverberation time.
Speech dereverberation constrained on room impulse response characteristics
Authors: Louis Bahrman; Mathieu Fontaine; Jonathan Le Roux; GaÃ«l Richard
Venue: Interspeech 2024 (main)

Abstract: Single-channel speech dereverberation aims at extracting a dry speech signal from a recording affected by the acoustic reflections in a room. However, most current deep learning-based approaches for speech dereverberation are not interpretable for room acoustics, and can be considered as black-box systems in that regard. In this work, we address this problem by regularizing the training loss using a novel physical coherence loss which encourages the room impulse response (RIR) induced by the dereverberated output of the model to match the acoustic properties of the room in which the signal was recorded. Our investigation demonstrates the preservation of the original dereverberated signal alongside the provision of a more physically coherent RIR.
DeWinder: Single-Channel Wind Noise Reduction using Ultrasound Sensing
Authors: Kuang Yuan; Shuo Han; Swarun Kumar; Bhiksha Raj
Venue: Interspeech 2024 (main)

Abstract: The quality of audio recordings in outdoor environments is often degraded by the presence of wind. Mitigating the impact of wind noise on the perceptual quality of single-channel speech remains a significant challenge due to its non-stationary characteristics. Prior work in noise suppression treats wind noise as a general background noise without explicit modeling of its characteristics. In this paper, we leverage ultrasound as an auxiliary modality to explicitly sense the airflow and characterize the wind noise. We propose a multi-modal deep-learning framework to fuse the ultrasonic Doppler features and speech signals for wind noise reduction. Our results show that DeWinder can significantly improve the noise reduction capabilities of state-of-the-art speech enhancement models.
ANIMAL-CLEAN â A Deep Denoising Toolkit for Animal-Independent Signal Enhancement
Authors: Alexander Barnhill; Elmar Noeth; Andreas Maier; Christian Bergler
Venue: Interspeech 2024 (main)

Abstract: Signal enhancement in bioacoustics can be of vital importance due to the fact that recordings are largely done in noise-heavy environments, in which anthrophonic, geophonic, or biophonic disturbances are myriad and can impede downstream analysis. Existing audio denoising techniques largely focus on human speech, whereas non-human animal vocalizations may not be compatible with frequency or temporal assumptions present in those methods. This work introduces ANIMALâCLEAN, an animal-independent, Noise2Noise-based deep denoising toolkit utilizing a combination of signals from the target of interest as well as existing datasets used for human speech enhancement. This toolkit enables users to develop deep learning models capable of denoising bioacoustic signals with a wide range of frequencies and signal lengths, which, when used as a preprocessing step, improve results for downstream signal segmentation and classification, without the need for noise-free bioacoustic recordings.
Elucidating Clock-drift Using Real-world Audios In Wireless Mode For Time-offset Insensitive End-to-End Asynchronous Acoustic Echo Cancellation
Authors: Premanand Nayak; M. Ali Basha Shaik
Venue: Interspeech 2024 (main)

Abstract: External playback and microphone array devices connected over wireless channels lack a common clock reference. This leads to non-linear asynchronous clock-drift effects with respect to effective echo path. Thereby, causing time-accelerated exacerbation when conventional filter based echo cancellers are used. We delineate clock-drift associated problems quantitatively by utilizing real-world audio streams recorded using various devices in wireless mode. We revisit and compare in situ conventional signal processing and deep neural network methods. We also introduce a novel end-to-end approach for asynchronous acoustic echo cancellation using Convolutional Recurrent Neural Networks and demonstrate state-of-the-art echo suppression performance without the need of time-resynchronization in the real-world observed non-linear clock-drift conditions.
QMixCAT: Unsupervised Speech Enhancement Using Quality-guided Signal Mixing and Competitive Alternating Model Training
Authors: Shilin Wang; Haixin Guan; Yanhua Long
Venue: Interspeech 2024 (main)

Abstract: Most deep learning-based speech enhancement (SE) models are supervised, requiring pairs of mixture and clean speech for training. This poses great challenges for real-world SE applications. Addressing this limitation is very crucial. In this paper, we introduce QMixCAT, an innovative unsupervised SE approach that enables unsupervised mixtures to be trained in a supervised manner within a teacher-student framework. Specifically, we propose a quality-guided signal mixing (QMix) approach to generate noisy-mixture-based supervised training data. Then, the model is trained using these data in a teacher-student framework, iteratively incorporating the QMix process during each epoch. In addition, a competitive alternating model training (CAT) mechanism is further proposed to enhance the quality of both teacher and student models. Experimental results demonstrate that QMixCAT significantly outperforms the strong baselines across multiple evaluation metrics.