GenAI4ELab/papercli-papers huggingface dataset:
PhonMatchNet: Phoneme-Guided Zero-Shot Keyword Spotting for User-Defined Keywords
Authors: Yong-Hyeok Lee; Namhyun Cho
Venue: Interspeech 2023 (main)

Abstract: This study presents a novel zero-shot user-defined keyword spotting model that utilizes the audio-phoneme relationship of the keyword to improve performance. Unlike the previous approach that estimates at utterance level, we use both utterance and phoneme level information. Our proposed method comprises a two-stream speech encoder architecture, self-attention-based pattern extractor, and phoneme-level detection loss for high performance in various pronunciation environments. Based on experimental results, our proposed model outperforms the baseline model and achieves competitive performance compared with full-shot keyword spotting models. Our proposed model significantly improves the EER and AUC across all datasets, including familiar words, proper nouns, and indistinguishable pronunciations, with an average relative improvement of 67% and 80%, respectively. The implementation code of our proposed model is available at https://github.com/ncsoft/PhonMatchNet.
Mix before Align: Towards Zero-shot Cross-lingual Sentiment Analysis via Soft-Mix and Multi-View Learning
Authors: Zhihong Zhu; Xuxin Cheng; Dongsheng Chen; Zhiqi Huang; Hongxiang Li; Yuexian Zou
Venue: Interspeech 2023 (main)

Abstract: Due to the insufficient sentiment corpus in many languages, recent studies have proposed cross-lingual sentiment analysis to transfer sentiment analysis models from rich-resource languages to low-resource ones. However, existing models heavily rely on code-switched sentences to reduce the alignment discrepancy of cross-lingual embeddings, which could be limited by their inherent constraints. In this paper, we propose a novel method dubbed SOUL (short for Softmix and Multiview learning) to enhance zero-shot cross-lingual sentiment analysis. Instead of using the embeddings of code-switched sentences directly, SOUL first mixes them softly with the embeddings of original sentences. Furthermore, SOUL utilizes multi-view learning to encourage contextualized embeddings to align into a refined language-invariant space. Experimental results on four cross-lingual benchmarks across five languages clearly verify the effectiveness of our proposed SOUL.
AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translation
Authors: Sara Papi; Marco Turchi; Matteo Negri
Venue: Interspeech 2023 (main)

Abstract: Attention is the core mechanism of today's most used architectures for natural language processing and has been analyzed from many perspectives, including its effectiveness for machine translation-related tasks. Among these studies, attention resulted to be a useful source of information to get insights about word alignment also when the input text is substituted with audio segments, as in the case of the speech translation (ST) task. In this paper, we propose AlignAtt, a novel policy for simultaneous ST (SimulST) that exploits the attention information to generate source-target alignments that guide the model during inference. Through experiments on the 8 language pairs of MuST-C v1.0, we show that AlignAtt outperforms previous state-of-the-art SimulST policies applied to offline-trained models with gains in terms of BLEU of 2 points and latency reductions ranging from 0.5s to 0.8s across the 8 languages.
Incremental Blockwise Beam Search for Simultaneous Speech Translation with Controllable Quality-Latency Tradeoff
Authors: Peter PolÃ¡k; Brian Yan; Shinji Watanabe; Alex Waibel; OndÅej Bojar
Venue: Interspeech 2023 (main)

Abstract: Blockwise self-attentional encoder models have recently emerged as one promising end-to-end approach to simultaneous speech translation. These models employ a blockwise beam search with hypothesis reliability scoring to determine when to wait for more input speech before translating further. However, this method maintains multiple hypotheses until the entire speech input is consumed - this scheme cannot directly show a single incremental translation to users. Further, this method lacks mechanisms for controlling the quality vs. latency tradeoff. We propose a modified incremental blockwise beam search incorporating local agreement or hold-n policies for quality-latency control. We apply our framework to models trained for online or offline translation and demonstrate that both types can be effectively used in online mode. Experimental results on MuST-C show 0.6-3.6 BLEU improvement without changing latency or 0.8-1.4 s latency improvement without changing quality.
Zambezi Voice: A Multilingual Speech Corpus for Zambian Languages
Authors: Claytone Sikasote; Kalinda Siaminwe; Stanly Mwape; Bangiwe Zulu; Mofya Phiri; Martin Phiri; David Zulu; Mayumbo Nyirenda; Antonios Anastasopoulos
Venue: Interspeech 2023 (main)

Abstract: This work introduces Zambezi Voice, an open-source multilingual speech resource for Zambian languages. It contains two collections of datasets: unlabelled audio recordings of radio news and talk shows programs (160 hours) and labelled data (over 80 hours) consisting of read speech recorded from text sourced from publicly available literature books. The dataset is created for speech recognition but can be extended to multilingual speech processing research for both supervised and unsupervised learning approaches. To our knowledge, this is the first multilingual speech dataset created for Zambian languages. We exploit pretraining and cross-lingual transfer learning by finetuning the Wav2Vec2.0 large-scale multilingual pretrained model to build end-to-end (E2E) speech recognition models for our baseline models. The dataset is released publicly under a Creative Commons BY-NC-ND 4.0 license and can be accessed through the project repository.
Towards Single Integrated Spoofing-aware Speaker Verification Embeddings
Authors: Sung Hwan Mun; Hye-jin Shim; Hemlata Tak; Xin Wang; Xuechen Liu; Md Sahidullah; Myeonghun Jeong; Min Hyun Han; Massimiliano Todisco; Kong Aik Lee; Junichi Yamagishi; Nicholas Evans; Tomi Kinnunen; Nam Soo Kim; Jee-weon Jung
Venue: Interspeech 2023 (main)

Abstract: This study aims to develop a single integrated spoofing-aware speaker verification (SASV) embeddings that satisfy two aspects. First, rejecting non-target speakers' input as well as target speakers' spoofed inputs should be addressed. Second, competitive performance should be demonstrated compared to the fusion of automatic speaker verification (ASV) and countermeasure (CM) embeddings, which outperformed single embedding solutions by a large margin in the SASV2022 challenge. We analyze that the inferior performance of single SASV embeddings comes from insufficient amount of training data and distinct nature of ASV and CM tasks. To this end, we propose a novel framework that includes multi-stage training and a combination of loss functions. Copy synthesis, combined with several vocoders, is also exploited to address the lack of spoofed data. Experimental results show dramatic improvements, achieving an SASV-EER of 1.06% on the evaluation protocol of the SASV2022 challenge.
Pseudo-Siamese Network based Timbre-reserved Black-box Adversarial Attack in Speaker Identification
Authors: Qing Wang; Jixun Yao; Ziqian Wang; Pengcheng Guo; Lei Xie
Venue: Interspeech 2023 (main)

Abstract: In this study, we propose a timbre-reserved adversarial attack approach for speaker identification (SID) to not only exploit the weakness of the SID model but also preserve the timbre of the target speaker in a black-box attack setting. Particularly, we generate timbre-reserved fake audio by adding an adversarial constraint during the training of the voice conversion model. Then, we leverage a pseudo-Siamese network architecture to learn from the black-box SID model constraining both intrinsic similarity and structural similarity simultaneously. The intrinsic similarity loss is to learn an intrinsic invariance, while the structural similarity loss is to ensure that the substitute SID model shares a similar decision boundary to the fixed black-box SID model. The substitute model can be used as a proxy to generate timbre-reserved fake audio for attacking. Experimental results on the Audio Deepfake Detection (ADD) challenge dataset indicate that the attack success rate of our proposed approach yields up to 60.58% and 55.38% in the white-box and black-box scenarios, respectively, and can deceive both human beings and machines.
Betray Oneself: A Novel Audio DeepFake Detection Model via Mono-to-Stereo Conversion
Authors: Rui Liu; Jinhua Zhang; Guanglai Gao; Haizhou Li
Venue: Interspeech 2023 (main)

Abstract: Audio Deepfake Detection (ADD) aims to detect the fake audio generated by text-to-speech (TTS), voice conversion (VC) and replay, etc., which is an emerging topic. Traditionally we take the mono signal as input and focus on robust feature extraction and effective classifier design. However, the dual-channel stereo information in the audio signal also includes important cues for deepfake, which has not been studied in the prior work. In this paper, we propose a novel ADD model, termed as M2S-ADD, that attempts to discover audio authenticity cues during the mono-to-stereo conversion process. We first projects the mono to a stereo signal using a pretrained stereo synthesizer, then employs a dual-branch neural architecture to process the left and right channel signals, respectively. In this way, we effectively reveal the artifacts in the fake audio, thus improve the ADD performance. The experiments on the ASVspoof2019 database show that M2S-ADD outperforms all baselines that input mono. We release the source code at https://github.com/AI-S2-Lab/M2S-ADD.
Robust Audio Anti-spoofing Countermeasure with Joint Training of Front-end and Back-end Models
Authors: Xingming Wang; Bang Zeng; Suo Hongbin; Yulong Wan; Ming Li
Venue: Interspeech 2023 (main)

Abstract: The accuracy and reliability of many speech processing systems may deteriorate under noisy conditions. This paper discusses robust audio anti-spoofing countermeasure for audio in noisy environments. Firstly, we attempt to use a pre-trained speech enhancement model as the front-end module and build a cascaded system. However, the independent denoising process of enhancement models may distort the synthesis artifacts or anti-spoofing related information included in utterances, leading to performance degradation. Therefore, we proposes a new framework for robust audio anti-spoofing by joint training the integrated speech enhancement front-end and anti-spoofing back-end. The final results demonstrate that the joint training framework is more effective than the cascaded framework. Additionally, we propose a cross-joint training scheme, which allows the single-model performance to exceed the result of score level fusion, making the joint framework more effective and efficient.
Improved DeepFake Detection Using Whisper Features
Authors: Piotr Kawa; Marcin Plata; MichaÅ Czuba; Piotr SzymaÅski; Piotr Syga
Venue: Interspeech 2023 (main)

Abstract: With a recent influx of voice generation methods, the threat introduced by audio DeepFake (DF) is ever-increasing. Several different detection methods have been presented as a countermeasure. Many methods are based on so-called front-ends, which, by transforming the raw audio, emphasize features crucial for assessing the genuineness of the audio sample. Our contribution contains investigating the influence of the state-of-the-art Whisper automatic speech recognition model as a DF detection front-end. We compare various combinations of Whisper and well-established front-ends by training 3 detection models (LCNN, SpecRNet, and MesoNet) on a widely used ASVspoof 2021 DF dataset and later evaluating them on the DF In-The-Wild dataset. We show that using Whisper-based features improves the detection for each model and outperforms recent results on the In-The-Wild dataset by reducing Equal Error Rate by 21%.
DoubleDeceiver: Deceiving the Speaker Verification System Protected by Spoofing Countermeasures
Authors: Mengao Zhang; Ke Xu; Hao Li; Lei Wang; Chengfang Fang; Jie Shi
Venue: Interspeech 2023 (main)

Abstract: Automatic Speaker Verification (ASV) systems are vulnerable to various attacks, especially spoofing attacks, and therefore are typically protected by spoofing countermeasures. However, both spoofing countermeasures and ASV models are vulnerable to adversarial attacks. We propose DoubleDeceiver - a novel black-box attack method that incorporates text-to-speech synthesis and adversarial attack to deceive ASV systems even with the protection of spoofing countermeasures. Although the surrogate models and victim models differ in architectures, DoubleDeceiver achieved a successful attack rate (SAR) as high as 98.3%. DoubleDeceiver identified the vulnerabilities of ASV systems and issued a warning that solely relying on the spoofing countermeasures is not reliable to protect ASV systems' security. This work encourages the development of more secure anti-spoofing and ASV systems by highlighting the need to consider composite attacks in future designs.
On Training a Neural Residual Acoustic Echo Suppressor for Improved ASR
Authors: Sankaran Panchapagesan; Turaj Zakizadeh Shabestary; Arun Narayanan
Venue: Interspeech 2023 (main)

Abstract: Acoustic Echo Cancellation (AEC) is critical for accurate recognition of speech directed at a smart device playing audio. Previous work has showed that neural AEC models can significantly improve Automatic Speech Recognition (ASR) accuracy. In this paper, we train a conformer-based waveform-domain neural model to perform residual acoustic echo suppression (RAES) on the output of a linear AEC. We focus specifically on improving ASR accuracy in realistic mismatched test conditions, when training on large-scale simulated training data, as needed for production voice-interaction systems. Our key finding is that instead of naively using the best evaluation-time linear AEC configuration during neural RAES model training, using a weaker linear AEC generalizes significantly better, with 17-30% lower word error rate (WER) on a realistic re-recorded test set. Overall, the neural RAES model yields 38-53% WER reduction over the linear AEC alone.
Extending DNN-based Multiplicative Masking to Deep Subband Filtering for Improved Dereverberation
Authors: Jean-Marie Lemercier; Julian Tobergte; Timo Gerkmann
Venue: Interspeech 2023 (main)

Abstract: In this paper, we present a scheme for extending deep neural network-based multiplicative maskers to deep subband filters for speech restoration in the time-frequency domain. The resulting method can be generically applied to any deep neural network providing masks in the time-frequency domain, while requiring only few more trainable parameters and a computational overhead that is negligible for state-of-the-art neural networks. We demonstrate that the resulting deep subband filtering scheme outperforms multiplicative masking for dereverberation, while leaving the denoising performance virtually the same. We argue that this is because deep subband filtering in the time-frequency domain fits the subband approximation often assumed in the dereverberation literature, whereas multiplicative masking corresponds to the narrowband approximation generally employed for denoising.
UnSE: Unsupervised Speech Enhancement Using Optimal Transport
Authors: Wenbin Jiang; Fei Wen; Yifan Zhang; Kai Yu
Venue: Interspeech 2023 (main)

Abstract: Most deep learning-based speech enhancement methods usually use supervised learning, which requires massive noisy-to-clean training pairs. However, the synthesized training data can only partially cover some realistic environments, and it is generally difficult or almost impossible to collect pairs of noisy and ground-truth clean speech in some scenarios. To address this problem, we propose an unsupervised speech enhancement method that does not require any paired noisy-to-clean training data. Specifically, based on the optimal transport criterion, the speech enhancement model is trained in an unsupervised manner only using a noisy speech based fidelity loss and a distribution divergence loss, by which the divergence between the output and (unpaired) clean speech is minimized. Experimental results show that the proposed unsupervised method can achieve competitive performance with supervised methods on the VCTK + DEMAND benchmark and better performance on the CHiME4 benchmark.
MC-SpEx: Towards Effective Speaker Extraction with Multi-Scale Interfusion and Conditional Speaker Modulation
Authors: Jun Chen; Wei Rao; Zilin Wang; Jiuxin Lin; Yukai Ju; Shulin He; Yannan Wang; Zhiyong Wu
Venue: Interspeech 2023 (main)

Abstract: The previous SpEx+ has yielded outstanding performance in speaker extraction and attracted much attention. However, it still encounters inadequate utilization of multi-scale information and speaker embedding. To this end, this paper proposes a new effective speaker extraction system with multi-scale interfusion and conditional speaker modulation (ConSM), which is called MC-SpEx. First of all, we design the weight-share multi-scale fusers (ScaleFusers) for efficiently leveraging multi-scale information as well as ensuring consistency of the model's feature space. Then, to consider different scale information while generating masks, the multi-scale interactive mask generator (ScaleInterMG) is presented. Moreover, we introduce ConSM module to fully exploit speaker embedding in the speech extractor. Experimental results on the Libri2Mix dataset demonstrate the effectiveness of our improvements and the state-of-the-art performance of our proposed MC-SpEx.
Causal Signal-Based DCCRN with Overlapped-Frame Prediction for Online Speech Enhancement
Authors: Julitta Bartolewska; StanisÅaw Kacprzak; Konrad Kowalczyk
Venue: Interspeech 2023 (main)

Abstract: The aim of speech enhancement is to improve speech signal quality and intelligibility from a noisy microphone signal. In many applications, it is crucial to enable processing with small computational complexity and minimal requirements regarding access to future signal samples (look-ahead). This paper presents signal-based causal DCCRN that improves online single-channel speech enhancement by reducing the required look-ahead and the number of network parameters. The proposed modifications include complex filtering of the signal, application of overlapped-frame prediction, causal convolutions and deconvolutions, and modification of the loss function. Results of performed experiments indicate that the proposed model with overlapped signal prediction and additional adjustments, achieves similar or better performance than the original DCCRN in terms of various speech enhancement metrics, while it reduces the latency and network parameter number by around 30%.
Gesper: A Restoration-Enhancement Framework for General Speech Reconstruction
Authors: Wenzhe Liu; Yupeng Shi; Jun Chen; Wei Rao; Shulin He; Andong Li; Yannan Wang; Zhiyong Wu
Venue: Interspeech 2023 (main)

Abstract: This paper describes a real-time General Speech Reconstruction (Gesper) system submitted to the ICASSP 2023 Speech Signal Improvement (SSI) Challenge. This novel proposed system is a two-stage architecture, in which the speech restoration is performed, and then cascaded by speech enhancement. We propose a complex spectral mapping-based generative adversarial network (CSM-GAN) as the speech restoration module for the first time. For noise suppression and dereverberation, the enhancement module is performed with fullband-wideband parallel processing. On the blind test set of ICASSP 2023 SSI Challenge, the proposed Gesper system, which satisfies the real-time condition, achieves 3.27 P.804 overall mean opinion score (MOS) and 3.35 P.835 overall MOS, ranked 1st in both track 1 and track 2.
Multimodal Personality Traits Assessment (MuPTA) Corpus: The Impact of Spontaneous and Read Speech
Authors: Elena Ryumina; Dmitry Ryumin; Maxim Markitantov; Heysem Kaya; Alexey Karpov
Venue: Interspeech 2023 (main)

Abstract: Automatic personality traits assessment (PTA) provides high-level, intelligible predictive inputs for subsequent critical downstream tasks, such as job interview recommendations and mental healthcare monitoring. In this work, we introduce a novel Multimodal Personality Traits Assessment (MuPTA) corpus. Our MuPTA corpus is unique in that it contains both spontaneous and read speech collected in the midly-resourced Russian language. We present a novel audio-visual approach for PTA that is used in order to set up baseline results on this corpus. We further analyze the impact of both spontaneous and read speech types on the PTA predictive performance. We find that for the audio modality, the PTA predictive performances on short signals are almost equal regardless of the speech type, while PTA using video modality is more accurate with spontaneous speech compared to read one regardless of the signal length.
MOCKS 1.0: Multilingual Open Custom Keyword Spotting Testset
Authors: MikoÅaj Pudo; Mateusz Wosik; Adam CieÅlak; Justyna Krzywdziak; Bozena Lukasiak; Artur Janicki
Venue: Interspeech 2023 (main)

Abstract: The main purpose of this work is to create a comprehensive audio testset that can be used to evaluate custom keyword spotting (KWS) models and to benchmark different KWS solutions. We also propose a set of requirements that should be followed while creating testsets to evaluate custom KWS models. We consider multiple versions of the problem: text and audio-based keyword spotting, as well as offline and online (streaming) modes. Our testset named MOCKS is based on LibriSpeech and Mozilla Common Voice datasets. We used automatically generated alignments to extract parts of the recordings, which were split into keywords and test samples. The resulting testset contains almost 50,000 keywords. It contains audio data in English, French, German, Italian, and Spanish, but can be easily extended to other languages. MOCKS has been made publicly available to the research community. Initial KWS experiments run on MOCKS suggest that it can serve as a challenging testset for future research.
MD3: The Multi-Dialect Dataset of Dialogues
Authors: Jacob Eisenstein; Vinodkumar Prabhakaran; Clara Rivera; Dorottya Demszky; Devyani Sharma
Venue: Interspeech 2023 (main)

Abstract: We introduce a new dataset of conversational speech representing English from India, Nigeria, and the United States. The Multi-Dialect Dataset of Dialogues (MD3) strikes a new balance between open-ended conversational speech and task-oriented dialogue by prompting participants to perform a series of short information-sharing tasks. This facilitates quantitative cross-dialectal comparison, while avoiding the imposition of a restrictive task structure that might inhibit the expression of dialect features. Preliminary analysis of the dataset reveals significant differences in syntax and in the use of discourse markers. The dataset, which will be made publicly available with the publication of this paper, includes more than 20 hours of audio and more than 200,000 orthographically-transcribed tokens.
MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation
Authors: Mohamed Anwar; Bowen Shi; Vedanuj Goswami; Wei-Ning Hsu; Juan Pino; Changhan Wang
Venue: Interspeech 2023 (main)

Abstract: We introduce MuAViC, a multilingual audio-visual corpus for robust speech recognition and robust speech-to-text translation providing 1200 hours of audio-visual speech in 9 languages. It is fully transcribed and covers 6 English-to-X translation as well as 6 X-to-English translation directions. To the best of our knowledge, this is the first open benchmark for audio-visual speech-to-text translation and the largest open benchmark for multilingual audio-visual speech recognition. Our baseline results show that MuAViC is effective for building noise-robust speech recognition and translation models. We make the corpus available at https://github.com/facebookresearch/muavic.