Query: speech separation

title: ADL MVDR: All Deep Learning MVDR Beamformer for Target Speech Separation
abstract: Speech separation algorithms are often used to separate the target speech from other interfering sources. However, purely neural network based speech separation systems often cause nonlinear distortion that is harmful for automatic speech recognition (ASR) systems. The conventional mask based minimum variance distortionless response (MVDR) beamformer can be used to minimize the distortion, but comes with high level of residual noise. Furthermore, the matrix operations (e.g. matrix inversion) involved in the conventional MVDR solution are sometimes numerically unstable when jointly trained with neural networks. In this paper, we propose a novel all deep learning MVDR framework, where the matrix inversion and eigenvalue decomposition are replaced by two recurrent neural networks (RNNs) to resolve both issues at the same time. The proposed method can greatly reduce the residual noise while keeping the target speech undistorted by leveraging on the RNN predicted frame wise beamforming weights. The system is evaluated on a Mandarin audio visual corpus and compared against several state of the art (SOTA) speech separation systems. Experimental results demonstrate the superiority of the proposed method across several objective metrics and ASR accuracy.

title: Continuous Speech Separation with Conformer
abstract: Continuous speech separation was recently proposed to deal with the overlapped speech in natural conversations. While it was shown to significantly improve the speech recognition performance for multichannel conversation transcription, its effectiveness has yet to be proven for a single channel recording scenario. This paper examines the use of Conformer architecture in lieu of recurrent neural networks for the separation model. Conformer allows the separation model to efficiently capture both local and global context information, which is helpful for speech separation. Experimental results using the LibriCSS dataset show that the Conformer separation model achieves the state of the art results for both single channel and multi channel settings. Results for real meeting recordings are also presented, showing significant performance gains in both word error rate (WER) and speaker attributed WER.

title: Dual Path Transformer Network: Direct Context Aware Modeling for End to End Monaural Speech Separation
abstract: The dominant speech separation models are based on complex recurrent or convolution neural network that model speech sequences indirectly conditioning on context, such as passing information through many intermediate states in recurrent neural network, leading to suboptimal separation performance. In this paper, we propose a dual path transformer network (DPTNet) for end to end speech separation, which introduces direct context awareness in the modeling for speech sequences. By introduces a improved transformer, elements in speech sequences can interact directly, which enables DPTNet can model for the speech sequences with direct context awareness. The improved transformer in our approach learns the order information of the speech sequences without positional encodings by incorporating a recurrent neural network into the original transformer. In addition, the structure of dual paths makes our model efficient for extremely long speech sequence modeling. Extensive experiments on benchmark datasets show that our approach outperforms the current state of the arts (20.6 dB SDR on the public WSj0 2mix data corpus)

title: LibriMix: An Open Source Dataset for Generalizable Speech Separation
abstract: In recent years, wsj0 2mix has become the reference dataset for single channel speech separation. Most deep learning based speech separation models today are benchmarked on it. However, recent studies have shown important performance drops when models trained on wsj0 2mix are evaluated on other, similar datasets. To address this generalization issue, we created LibriMix, an open source alternative to wsj0 2mix, and to its noisy extension, WHAM! Based on LibriSpeech, LibriMix consists of two or three speaker mixtures combined with ambient noise samples from WHAM! Using Conv TasNet, we achieve competitive performance on all LibriMix versions. In order to fairly evaluate across datasets, we introduce a third test set based on VCTK for speech and WHAM! for noise. Our experiments show that the generalization error is smaller for models trained with LibriMix than with WHAM! in both clean and noisy conditions. Aiming towards evaluation in more realistic, conversation like scenarios, we also release a sparsely overlapping version of LibriMix's test set.

title: FaceFilter: Audio visual speech separation using still images
abstract: The objective of this paper is to separate a target speaker's speech from a mixture of two speakers using a deep audio visual speech separation network. Unlike previous works that used lip movement on video clips or pre enrolled speaker information as an auxiliary conditional feature, we use a single face image of the target speaker. In this task, the conditional feature is obtained from facial appearance in cross modal biometric task, where audio and visual identity representations are shared in latent space. Learnt identities from facial images enforce the network to isolate matched speakers and extract the voices from mixed speech. It solves the permutation problem caused by swapped channel outputs, frequently occurred in speech separation tasks. The proposed method is far more practical than video based speech separation since user profile images are readily available on many platforms. Also, unlike speaker aware separation methods, it is applicable on separation with unseen speakers who have never been enrolled before. We show strong qualitative and quantitative results on challenging real world examples.

title: Neural Spatio Temporal Beamformer for Target Speech Separation
abstract: Purely neural network (NN) based speech separation and enhancement methods, although can achieve good objective scores, inevitably cause nonlinear speech distortions that are harmful for the automatic speech recognition (ASR) On the other hand, the minimum variance distortionless response (MVDR) beamformer with NN predicted masks, although can significantly reduce speech distortions, has limited noise reduction capability. In this paper, we propose a multi tap MVDR beamformer with complex valued masks for speech separation and enhancement. Compared to the state of the art NN mask based MVDR beamformer, the multi tap MVDR beamformer exploits the inter frame correlation in addition to the inter microphone correlation that is already utilized in prior arts. Further improvements include the replacement of the real valued masks with the complex valued masks and the joint training of the complex mask NN. The evaluation on our multi modal multi channel target speech separation and enhancement platform demonstrates that our proposed multi tap MVDR beamformer improves both the ASR accuracy and the perceptual speech quality against prior arts.

title: On End to end Multi channel Time Domain Speech Separation in Reverberant Environments
abstract: This paper introduces a new method for multi channel time domain speech separation in reverberant environments. A fully convolutional neural network structure has been used to directly separate speech from multiple microphone recordings, with no need of conventional spatial feature extraction. To reduce the influence of reverberation on spatial feature extraction, a dereverberation pre processing method has been applied to further improve the separation performance. A spatialized version of wsj0 2mix dataset has been simulated to evaluate the proposed system. Both source separation and speech recognition performance of the separated signals have been evaluated objectively. Experiments show that the proposed fully convolutional network improves the source separation metric and the word error rate (WER) by more than 13% and 50% relative, respectively, over a reference system with conventional features. Applying dereverberation as pre processing to the proposed system can further reduce the WER by 29% relative using an acoustic model trained on clean and reverberated data.

title: Deep Audio Visual Speech Separation with Attention Mechanism
abstract: Previous work shows that audio visual fusion is a practical approach to deal with the speech separation task in the cocktail party problem. In this paper, we explore a better strategy to utilize visual representations with the attention mechanism. Compared to the previous baseline only using one visual stream of the target speaker, both speaker dependent visual streams in the mixed audio are fed into the model, and it also predicts two separated speech streams simultaneously. To further enhance the performance, the attention mechanism is designed on the audio visual speech separation architecture. The results show that the proposed approach works well in audio visual speech separation. Our best model achieves an obvious and consistent improvement in speech separation when compared to the traditional method only using the target speaker visual stream.

title: Deep Casa for Talker independent Monaural Speech Separation
abstract: Monaural speech separation is the task of separating target speech from interference in single channel recordings. Although substantial progress has been made recently in deep learning based speech separation, previous studies usually focus on a single type of interference, either background noise or competing speakers. In this study, we address both speech and nonspeech interference, i.e. monaural speaker separation in noise, in a talker independent fashion. We extend a recently proposed deep CASA system to deal with noisy speaker mixtures. To facilitate speech enhancement, a denoising module is added to deep CASA as a front end processor. The proposed systems achieve state of the art results on a benchmark noisy two speaker separation dataset. The denoising module leads to substantial performance gain across various noise types, and even better generalization in noise free conditions.

title: Neural Speech Separation Using Spatially Distributed Microphones
abstract: This paper proposes a neural network based speech separation method using spatially distributed microphones. Unlike with traditional microphone array settings, neither the number of microphones nor their spatial arrangement is known in advance, which hinders the use of conventional multi channel speech separation neural networks based on fixed size input. To overcome this, a novel network architecture is proposed that interleaves inter channel processing layers and temporal processing layers. The inter channel processing layers apply a self attention mechanism along the channel dimension to exploit the information obtained with a varying number of microphones. The temporal processing layers are based on a bidirectional long short term memory (BLSTM) model and applied to each channel independently. The proposed network leverages information across time and space by stacking these two kinds of layers alternately. Our network estimates time frequency (TF) masks for each speaker, which are then used to generate enhanced speech signals either with TF masking or beamforming. Speech recognition experimental results show that the proposed method significantly outperforms baseline multi channel speech separation systems.