GenAI4ELab/papercli-papers huggingface dataset:
Deeply Supervised Curriculum Learning for Deep Neural Network-based Sound Source Localization
Authors: Min-Sang Baek; Joon-Young Yang; Joon-Hyuk Chang
Venue: Interspeech 2023 (main)

Abstract: Deep neural network (DNN) has made impressive progress in sound source localization (SSL) tasks with the hard n-hot labels that represent specific directions-of-arrivals (DOAs). However, recent study suggested soft DOA labels, considering the correlations between targets and nearby DOAs. In this study, to effectively train a DNN using soft labels, we propose deeply supervised curriculum learning (DSCL) by adopting the two techniques for the DNN, deep supervision (DS) and curriculum learning (CL). We train a DNN to solve SSL problems progressing from easier to harder, expecting the DNN would gradually reduce the angular region of the target DOAs. It is gained by various resolution soft targets for the different DNN layers to deeply supervise the DNN, while increasing the angular selectivity of the targets from the early to late stages of training by CL. Proposed method was verified on datasets with multi-speakers, and exceeded the hard-label methods with great improvements.
Multi-channel separation of dynamic speech and sound events
Authors: Takuya Fujimura; Robin Scheibler
Venue: Interspeech 2023 (main)

Abstract: We propose a multi-channel separation method for moving sound sources. We build upon a recent beamformer for a moving speaker using attention-based tracking. This method uses an attention mechanism to compute the time-varying spatial statistics which enables tracking the moving source. While this prior work aimed to extract a single target source, we simultaneously estimate multiple sources. Our main technical contribution is to introduce attention-based tracking into the iterative source steering algorithm for independent vector analysis (IVA), enabling joint estimation of multiple sources. We experimentally show that the proposed method greatly improves the separation performance for moving speakers, including an absolute reduction of 27.2% in word error rate compared to time-invariant IVA. In addition, we demonstrate that the proposed method is effective as a pre-processing for sound event detection, showing an improvement in F1 scores of up to 4.7% in real recordings.
Rethinking the Visual Cues in Audio-Visual Speaker Extraction
Authors: Junjie Li; Meng Ge; Zexu Pan; Rui Cao; Longbiao Wang; Jianwu Dang; Shiliang Zhang
Venue: Interspeech 2023 (main)

Abstract: The Audio-Visual Speaker Extraction (AVSE) algorithm employs parallel video recording to leverage two visual cues, namely speaker identity and synchronization, to enhance performance compared to audio-only algorithms. However, the visual front-end in AVSE is often derived from a pre-trained model or end-to-end trained, making it unclear which visual cue contributes more to the speaker extraction performance. This raises the question of how to better utilize visual cues. To address this issue, we propose two training strategies that decouple the learning of the two visual cues. Our experimental results demonstrate that both visual cues are useful, with the synchronization cue having a higher impact. We introduce a more explainable model, the Decoupled Audio-Visual Speaker Extraction (DAVSE) model, which leverages both visual cues.
Using Semi-supervised Learning for Monaural Time-domain Speech Separation with a Self-supervised Learning-based SI-SNR Estimator
Authors: Shaoxiang Dang; Tetsuya Matsumoto; Yoshinori Takeuchi; Hiroaki Kudo
Venue: Interspeech 2023 (main)

Abstract: Speech separation aims to decompose mixed speeches into independent signals. Prior research on monaural timedomain speech separation has made great progress in supervised manners. Almost all of these works are trained on simulated mixed speech signals since obtaining ground truth for real-world mixed signals is problematic. To this end, we propose a novel semi-supervised learning method for speech separation (SSLM-SS), which leverages mixed speeches without ground truth. In particular, for this type of data, we further put forward a non-intrusive separated speech quality prediction network (SSQP-Net) based on self-supervised learning. According to the results, the linear correlation coefficient between the predicted results of SSQP-Net and the ground truth achieves 0.9. Moreover, the performance of SSLM-SS equipped with SSQP-Net exhibits an improvement of 0.2 dB and 1.1 dB compared to the mixture invariant training (MixIT) in the conditions of involving 10% and 50% labeled data respectively, and rivals supervised learning.
Investigation of Training Mute-Expressive End-to-End Speech Separation Networks for an Unknown Number of Speakers
Authors: Younggwan Kim; Hyungjun Lim; Kiho Yeom; Eunjoo Seo; Hoodong Lee; Stanley Jungkyu Choi; Honglak Lee
Venue: Interspeech 2023 (main)

Abstract: In speech separation, there have been a limited number of prior works for an unknown number of speakers in a speech mixture. To address this situation, one simple solution is to constitute the sufficient number of output channels greater than or equal to the expected number of speakers and ignore invalid outputs containing meaningless signals when the number of speakers is less than the output channels. To detect such invalid outputs, it is an ideal scenario for the meaningless signals to be muted. In this paper, we investigate several training methods by which separation models can mute the invalid outputs. We first introduce an on-the-fly data mixing scheme adding small random noises to the speech mixtures. As a training criterion, we analyze why the well-known scale-invariant signal-to-noise ratio is not suitable for muting the invalid outputs because of its power amplification problem and also explain why we use the signal-to-noise ratio criterion to avoid the problem.
SR-SRP: Super-Resolution based SRP-PHAT for Sound Source Localization and Tracking
Authors: Jae-Heung Cho; Joon-Hyuk Chang
Venue: Interspeech 2023 (main)

Abstract: Sound source localization and tracking have been extensively studied. Recently, there has been considerable interest in highly reverberant scenarios and steered response power with phase transform (SRP-PHAT) based models have shown a good performance. However, these models still have limitations because the SRP-PHAT algorithm cannot represent the direction of the source in such adverse environments. In this paper, we propose a novel structure combining a super-resolution model and a single sound source localization model that allows to improve direction estimation performance. The proposed method generates a robust power map that accurately represents the direction of the source, even in poor scenarios. Furthermore, the proposed structure has a lower computational cost because it uses a low-resolution map. Experimental results on simulation-based and real-world data show that the proposed method outperforms the state-of-the-art model, Cross3D.
Dual-Memory Multi-Modal Learning for Continual Spoken Keyword Spotting with Confidence Selection and Diversity Enhancement
Authors: Zhao Yang; Dianwen Ng; Xizhe Li; Chong Zhang; Rui Jiang; Wei Xi; Yukun Ma; Chongjia Ni; Jizhong Zhao; Bin Ma; Eng Siong Chng
Venue: Interspeech 2023 (main)

Abstract: Enabling continual learning (CL) from an ever-changing environment is highly valuable, but it poses significant challenges for spoken keyword spotting (KWS), which simultaneously deals with both variability in acoustic characteristics of speech signals and catastrophic forgetting issues. In this paper, we propose a novel framework for replay-based CL in KWS that uses a Dual-Memory Multi-Modal (DM3) structure to enhance generalizability and robustness. Our approach leverages short-term and long-term models to learn near-term and long-term knowledge in an adaptive manner with a dual-memory structure, while also exploiting the consistency of multiple speech perturbations to improve the robustness with a multi-modal structure. Additionally, we introduce a class-balanced selection strategy that uses confidence scores to sort training samples. Experiments demonstrate the effectiveness of our method over competitive baselines in class incremental learning and domain incremental learning KWS settings.
FN-SSL: Full-Band and Narrow-Band Fusion for Sound Source Localization
Authors: Yabo Wang; Bing Yang; Xiaofei Li
Venue: Interspeech 2023 (main)

Abstract: Extracting direct-path spatial features is critical for sound source localization in adverse acoustic environments. This paper proposes a full-band and narrow-band fusion network for estimating direct-path inter-channel phase difference (DP-IPD) from microphone signals. The alternating full-band and narrow-band layers are responsible for learning the full-band correlation and narrow-band extraction of DP-IPD, respectively. Experiments show that the proposed network noticeably outperforms other advanced methods on both simulated and real-world data.
A Neural State-Space Modeling Approach to Efficient Speech Separation
Authors: Chen Chen; Chao-Han Huck Yang; Kai Li; Yuchen Hu; Pin-Jui Ku; Eng Siong Chng
Venue: Interspeech 2023 (main)

Abstract: In this work, we introduce S4M, a new efficient speech separation framework based on neural state-space models (SSM). Motivated by linear time-invariant systems for sequence modeling, our SSM-based approach can efficiently model input signals into a format of linear ordinary differential equations (ODEs) for representation learning. To extend the SSM technique into speech separation tasks, we first decompose the input mixture into multi-scale representations with different resolutions. This mechanism enables S4M to learn globally coherent separation and reconstruction. The experimental results show that S4M performs comparably to other separation backbones in terms of SI-SDRi, while having a much lower model complexity with significantly fewer trainable parameters. In addition, our S4M-tiny model (1.8M parameters) even surpasses attention-based Sepformer (26.0M parameters) in noisy conditions with only 9.2% of multiply-accumulate operation (MACs).
Locate and Beamform: Two-dimensional Locating All-neural Beamformer for Multi-channel Speech Separation
Authors: Yanjie Fu; Meng Ge; Honglong Wang; Nan Li; Haoran Yin; Longbiao Wang; Gaoyan Zhang; Jianwu Dang; Chengyun Deng; Fei Wang
Venue: Interspeech 2023 (main)

Abstract: Recently, stunning improvements on multi-channel speech separation have been achieved by neural beamformers when direction information is available. However, most of them neglect to utilize speaker's 2-dimensional (2D) location cues contained in mixture signal, which limits the performance when two sources come from close directions. In this paper, we propose an end-to-end beamforming network for 2D location guided speech separation merely given mixture signal. It first estimates discriminable direction and 2D location cues, which imply directions the sources come from in multi views of microphones and their 2D coordinates. These cues are then integrated into location-aware neural beamformer, thus allowing accurate reconstruction of two sources' speech signals. Experiments show that our proposed model not only achieves a comprehensive decent improvement compared to baseline systems, but avoids inferior performance on spatial overlapping cases.
Monaural Speech Separation Method Based on Recurrent Attention with Parallel Branches
Authors: Xue Yang; Changchun Bao; Xu Zhang; Xianhong Chen
Venue: Interspeech 2023 (main)

Abstract: In many speech separation methods, the contextual information contained in the feature sequence is mainly modeled by recurrent layer and/or self-attention mechanism. However, how to combine these two powerful approaches more effectively needs to be explored. In this paper, a recurrent attention with parallel branches is proposed to first fully exploit the contextual information contained in the time-frequency (T-F) features. Then, this information is further modeled by the recurrent modules in a conventional manner. Specifically, the proposed recurrent attention with parallel branches uses two attention modules stacked sequentially. Each attention module has two parallel branches of self-attention to model dependencies along two axes and one convolutional layer for feature fusion. Thus, the contextual information contained in the T-F features can be fully exploited and further modeled by the recurrent modules. Experimental results showed the effectiveness of our proposed method.
Ontology-aware Learning and Evaluation for Audio Tagging
Authors: Haohe Liu; Qiuqiang Kong; Xubo Liu; Xinhao Mei; Wenwu Wang; Mark D. Plumbley
Venue: Interspeech 2023 (main)

Abstract: This study defines a new evaluation metric for audio tagging tasks to alleviate the limitation of the mean average precision (mAP) metric. The mAP metric treats different kinds of sound as independent classes without considering their relations. The proposed metric, ontology-aware mean average precision (OmAP), addresses the weaknesses of mAP by utilizing additional ontology during evaluation. Specifically, we reweight the false positive events in the model prediction based on the AudioSet ontology graph distance to the target classes. The OmAP also provides insights into model performance by evaluating different coarse-grained levels in the ontology graph. We conduct a human assessment and show that OmAP is more consistent with human perception than mAP. We also propose an ontology-based loss function (OBCE) that reweights binary cross entropy (BCE) loss based on the ontology distance. Our experiment shows that OBCE can improve both mAP and OmAP metrics on the AudioSet tagging task.
Multi-Dataset Co-Training with Sharpness-Aware Optimization for Audio Anti-spoofing
Authors: Hye-jin Shim; Jee-weon Jung; Tomi Kinnunen
Venue: Interspeech 2023 (main)

Abstract: Audio anti-spoofing for automatic speaker verification aims to safeguard users' identities from spoofing attacks. Although state-of-the-art spoofing countermeasure(CM) models perform well on specific datasets, they lack generalization when evaluated with different datasets. To address this limitation, previous studies have explored large pre-trained models, which require significant resources and time. We aim to develop a compact but well-generalizing CM model that can compete with large pre-trained models. Our approach involves multi-dataset co-training and sharpness-aware minimization, which has not been investigated in this domain. Extensive experiments reveal that proposed method yield competitive results across various datasets while utilizing 4,000 times less parameters than the large pre-trained models.
Reducing the Prior Mismatch of Stochastic Differential Equations for Diffusion-based Speech Enhancement
Authors: Bunlong Lay; Simon Welker; Julius Richter; Timo Gerkmann
Venue: Interspeech 2023 (main)

Abstract: Recently, score-based generative models have been successfully employed for the task of speech enhancement. A stochastic differential equation is used to model the iterative forward process, where at each step environmental noise and white Gaussian noise are added to the clean speech signal. While in limit the mean of the forward process ends at the noisy mixture, in practice it stops earlier and thus only at an approximation of the noisy mixture. This results in a discrepancy between the terminating distribution of the forward process and the prior used for solving the reverse process at inference. In this paper, we address this discrepancy and propose a forward process based on a Brownian bridge. We show that such a process leads to a reduction of the mismatch compared to previous diffusion processes. More importantly, we show that our approach improves in objective metrics over the baseline process with only half of the iteration steps and having one hyperparameter less to tune.
Complex-valued neural networks for voice anti-spoofing
Authors: Nicolas M. MÃ¼ller; Philip Sperl; Konstantin BÃ¶ttinger
Venue: Interspeech 2023 (main)

Abstract: Current anti-spoofing and audio deepfake detection systems use either magnitude spectrogram-based features (such as CQT or Melspectrograms) or raw audio processed through convolution or sinc-layers. Both methods have drawbacks: magnitude spectrograms discard phase information, which affects audio naturalness, and raw-feature-based models cannot use traditional explainable AI methods. This paper proposes a new approach that combines the benefits of both methods by using complex-valued neural networks to process the complex-valued, CQT frequency-domain representation of the input audio. This method retains phase information and allows for explainable AI methods. Results show that this approach outperforms previous methods on the "In-the-Wild" anti-spoofing dataset and enables interpretation of the results through explainable AI. Ablation studies confirm that the model has learned to use phase information to detect voice spoofing.
DeepVQE: Real Time Deep Voice Quality Enhancement for Joint Acoustic Echo Cancellation, Noise Suppression and Dereverberation
Authors: Nicolae Catalin Ristea; Evgenii Indenbom; Ando Saabas; Tanel PÃ¤rnamaa; Jegor Guzhvin; Ross Cutler
Venue: Interspeech 2023 (main)

Abstract: Acoustic echo cancellation (AEC), noise suppression (NS) and dereverberation (DR) are an integral part of modern full-duplex communication systems. As the demand for teleconferencing systems increases, addressing these tasks is required for an effective and efficient online meeting experience. Most prior research proposes solutions for these tasks separately, combining them with digital signal processing (DSP) based components, resulting in complex pipelines that are often impractical to deploy in real-world applications. This paper proposes a real-time cross-attention deep model, named DeepVQE, based on residual convolutional neural networks (CNNs) and recurrent neural networks (RNNs) to simultaneously address AEC, NS, and DR. DeepVQE achieves state-of-the-art performance on non-personalized tracks from the 2023 AEC and 2023 DNS Challenge test sets. Moreover, the model runs in real-time and has been successfully tested for the Microsoft Teams platform.
Diffiner: A Versatile Diffusion-based Generative Refiner for Speech Enhancement
Authors: Ryosuke Sawata; Naoki Murata; Yuhta Takida; Toshimitsu Uesaka; Takashi Shibuya; Shusuke Takahashi; Yuki Mitsufuji
Venue: Interspeech 2023 (main)

Abstract: Although deep neural network (DNN)-based speech enhancement (SE) methods outperform the previous non-DNN-based ones, they often degrade the perceptual quality of generated outputs. To tackle this problem, we introduce a DNN-based generative refiner, Diffiner, aiming to improve perceptual speech quality pre-processed by an SE method. We train a diffusion based generative model by utilizing a dataset consisting of clean speech only. Then, our refiner effectively mixes clean parts newly generated via denoising diffusion restoration into the degraded and distorted parts caused by a preceding SE method, resulting in refined speech. Once our refiner is trained on a set of clean speech, it can be applied to various SE methods without additional training specialized for each SE module. Therefore, our refiner can be a versatile post-processing module w.r.t. SE methods and has high potential in terms of modularity. Experimental results show that our method improved perceptual speech quality regardless of the preceding SE methods used. Our code is available at https://github.com/sony/diffiner.
HD-DEMUCS: General Speech Restoration with Heterogeneous Decoders
Authors: Doyeon Kim; Soo-Whan Chung; Hyewon Han; Youna Ji; Hong-Goo Kang
Venue: Interspeech 2023 (main)

Abstract: This paper introduces an end-to-end neural speech restoration model, HD-DEMUCS, demonstrating efficacy across multiple distortion environments. Unlike conventional approaches that employ cascading frameworks to remove undesirable noise first and then restore missing signal components, our model performs these tasks in parallel using two heterogeneous decoder networks. Based on the U-Net style encoder-decoder framework, we attach an additional decoder so that each decoder network performs noise suppression or restoration separately. We carefully design each decoder architecture to operate appropriately depending on its objectives. Additionally, we improve performance by leveraging a learnable weighting factor, aggregating the two decoder output waveforms. Experimental results with objective metrics across various environments clearly demonstrate the effectiveness of our approach over a single decoder or multi-stage systems for general speech restoration task.
MP-SENet: A Speech Enhancement Model with Parallel Denoising of Magnitude and Phase Spectra
Authors: Ye-Xin Lu; Yang Ai; Zhen-Hua Ling
Venue: Interspeech 2023 (main)

Abstract: This paper proposes MP-SENet, a novel Speech Enhancement Network which directly denoises Magnitude and Phase spectra in parallel. The proposed MP-SENet adopts a codec architecture in which the encoder and decoder are bridged by convolution-augmented transformers. The encoder aims to encode time-frequency representations from the input noisy magnitude and phase spectra. The decoder is composed of parallel magnitude mask decoder and phase decoder, directly recovering clean magnitude spectra and clean-wrapped phase spectra by incorporating learnable sigmoid activation and parallel phase estimation architecture, respectively. Multi-level losses defined on magnitude spectra, phase spectra, short-time complex spectra, and time-domain waveforms are used to train the MP-SENet model jointly. Experimental results show that our proposed MP-SENet achieves a PESQ of 3.50 on the public VoiceBank+DEMAND dataset and outperforms existing advanced speech enhancement methods.
TridentSE: Guiding Speech Enhancement with 32 Global Tokens
Authors: Dacheng Yin; Zhiyuan Zhao; Chuanxin Tang; Zhiwei Xiong; Chong Luo
Venue: Interspeech 2023 (main)

Abstract: This paper presents TridentSE, a new and innovative architecture for speech enhancement that efficiently combines local details and global information. The architecture uses time-frequency bin level representation for capturing detailed information and a small number of global tokens for processing global information. It employs cross attention modules to transfer information between the local and global representation, and separates the global tokens into two groups to process inter- and intra-frame information. A metric discriminator is utilized to increase perceptual quality and achieve improved performance compared to previous speech enhancement methods. With lower computational cost, TridentSE achieved a PESQ of 3.47 on the VoiceBank+DEMAND dataset and a PESQ of 3.44 on the DNS no-reverb test set, outperforming most previous methods. Visualization shows that the global tokens demonstrate diverse and interpretable global patterns.
Detection of Cross-Dataset Fake Audio Based on Prosodic and Pronunciation Features
Authors: Chenglong Wang; Jiangyan Yi; Jianhua Tao; Chu Yuan Zhang; Shuai Zhang; Xun Chen
Venue: Interspeech 2023 (main)

Abstract: Existing fake audio detection systems perform well in in-domain testing, but still face many challenges in out-of-domain testing. This is due to the mismatch between the training and test data, as well as the poor generalizability of features extracted from limited views. To address this, we propose multi-view features for fake audio detection, which aim to capture more generalized features from prosodic, pronunciation, and wav2vec dimensions. Specifically, the phoneme duration features are extracted from a pre-trained model based on a large amount of speech data. For the pronunciation features, a Conformer-based phoneme recognition model is first trained, keeping the acoustic encoder part as a deeply embedded feature extractor. FFurthermore, the prosodic and pronunciation features are fused with wav2vec features based on an attention mechanism to improve the generalization of fake audio detection models. Results show that the proposed method achieves significant performance gains in several cross-dataset experiments.