GenAI4ELab/papercli-papers huggingface dataset:
Speak & Improve: L2 English Speaking Practice Tool
Authors: Diane Nicholls; Kate M. Knill; Mark J. F. Gales; Anton Ragni; Paul Ricketts
Venue: Interspeech 2023 (main)

Abstract: A problem for building and studying approaches to automatically assess and give feedback to L2 English learners on their speaking ability is a lack of suitable data sets and platforms to run experiments on. This paper describes the Speak & Improve (S&I) speaking practice tool, which has been designed to meet the needs of researchers while also offering learners the opportunity to practise their English speaking and improve their confidence. S&I has tasks that allow the learner to demonstrate and improve their proficiency across the English speaking construct. Most of the tasks are free speaking, that is, the learner is not constrained in what they have to say. Over 400,000 learners worldwide have tried the alpha version of S&I. This paper presents the next version which provides more choice and flexibility to the users and more feedback on performance.
Measuring prosody in child speech using SoapBox Fluency API
Authors: Mauro Nicolao; Brenda McGuirk; Declan Moore; Niall Mullally; Lora Lynn OâMahony; Emma OâNeill; Amelia C. Kelly
Venue: Interspeech 2023 (main)

Abstract: SoapBox Fluency API uses automatic speech recognition (ASR) to return data points that enable educators to assess a child's oral reading fluency (ORF). Prosody is a key component of ORF assessment together with accuracy and speed, but manual assessment of prosody is largely subjective. A quantitative prosody score would benefit fair assessment. In this show and tell paper and accompanying video, we describe the new prosody feature of the API and demonstrate how the outputs can be used to assess expressiveness in ORF.
Teaching Non-native Sound Contrasts using Visual Biofeedback
Authors: Shawn Nissen
Venue: Interspeech 2023 (main)

Abstract: This demonstration will explain how non-native sound contrasts can be taught to second language (L2) learners using real-time visual biofeedback through an electropalatographic (EPG) sensor. How the EPG sensor is created from a dental mold and is subsequently produced will be shown during the presentation. An explanation of how a student can visualize the contact patterns of their speech and how a tutor might use the various features of the associated software to customize the instruction to an individual speaker will be illustrated during the presentation. Possible limitations, costs, and drawbacks to using the technology will also be discussed.
Large-Scale Automatic Audiobook Creation
Authors: Brendan Walsh; Mark Hamilton; Greg Newby; Xi Wang; Serena Ruan; Sheng Zhao; Lei He; Shaofei Zhang; Eric Dettinger; William T. Freeman; Markus Weimer
Venue: Interspeech 2023 (main)

Abstract: An audiobook can dramatically improve a work of literature's accessibility and improve reader engagement. However, audiobooks can take hundreds of hours of human effort to create, edit, and publish. In this work, we present a system that can automatically generate high-quality audiobooks from online e-books. In particular, we leverage recent advances in neural text-to-speech to create and release thousands of human-quality, open-license audiobooks from the Project Gutenberg e-book collection. Our method can identify the proper subset of e-book content to read for a wide collection of diversely structured books and can operate on hundreds of books in parallel. Our system allows users to customize an audiobook's speaking speed and style, emotional intonation, and can even match a desired voice using a small amount of sample audio. This work contributed over five thousand open-license audiobooks and an interactive demo that allows users to quickly create their own customized audiobooks. To listen to the audiobook collection visit https://aka.ms/audiobook.
QVoice: Arabic Speech Pronunciation Learning Application
Authors: Yassine El Kheir; Fouad Khnaisser; Shammur Absar Chowdhury; Hamdy Mubarak; Shazia Afzal; Ahmed M. Ali
Venue: Interspeech 2023 (main)

Abstract: This paper introduces a novel Arabic pronunciation learning application QVoice, powered with end-to-end mispronunciation detection and feedback generator module. The application is designed to support non-native Arabic speakers in enhancing their pronunciation skills, while also helping native speakers mitigate any potential influence from regional dialects on their Modern Standard Arabic (MSA) pronunciation. QVoice employs various learning cues to aid learners in comprehending meaning, drawing connections with their existing knowledge of English language, and offers detailed feedback for pronunciation correction, along with contextual examples showcasing word usage. The learning cues featured in QVoice encompass a wide range of meaningful information, such as visualizations of phrases/words and their translations, as well as phonetic transcriptions and transliterations. QVoice provides pronunciation feedback at the character level and assesses performance at the word level.
Asking Questions: an Innovative Way to Interact with Oral History Archives
Authors: Jan Å vec; Martin BulÃ­n; Adam FrÃ©mund; Filip PolÃ¡k
Venue: Interspeech 2023 (main)

Abstract: The paper describes our initial effort to use Transformer-based neural networks for understanding and presenting oral history archives. Such archives of interviews often contain large passages of the interviewee's speech. Our approach automatically generates relevant questions, which enrich such monotonous parts and allows the listener to better orient in the interview. The generated questions also allow for finding interesting parts of the interview without changing the original meaning of the testimony. We present our working pipeline consisting of a Wav2Vec speech recognizer, BERT-based punctuation detection, T5 asking questions model and BERT-based semantic continuity model.
DisfluencyFixer: A tool to enhance Language Learning through Speech To Speech Disfluency Correction
Authors: Vineet Bhat; Preethi Jyothi; Pushpak Bhattacharyya
Venue: Interspeech 2023 (main)

Abstract: Conversational speech often consists of deviations from the speech plan, producing disfluent utterances that affect downstream NLP tasks. Removing these disfluencies is necessary to create fluent and coherent speech. This paper presents DisfluencyFixer, a tool that performs speech-to-speech disfluency correction in English and Hindi using a pipeline of Automatic Speech Recognition (ASR), Disfluency Correction (DC) and Text-To-Speech (TTS) models. Our proposed system removes disfluencies from input speech and returns fluent speech as output along with its transcript, disfluency type and total disfluency count in source utterance, providing a one-stop destination for language learners to improve the fluency of their speech. We evaluate the performance of our tool subjectively and receive scores of 4.26, 4.29 and 4.42 out of 5 in ASR performance, DC performance and ease-of-use of the system. Our tool can be accessed openly at the following link.
Technology Pipeline for Large Scale Cross-Lingual Dubbing of Lecture Videos into Multiple Indian Languages
Authors: Anusha Prakash; Arun Kumar; Ashish Seth; Bhagyashree Mukherjee; Ishika Gupta; Jom Kuriakose; Jordan F; K V Vikram; Mano R Kumar M; Metilda Sagaya Mary; Mohammad Wajahat; Mohana N; Mudit Batra; Navina K; Nihal John George; Nithya Ravi; Pruthwik Mishra; Sudhanshu Srivastava; Vasista Sai Lodagala; Vandan Mujadia; Kada Sai Venkata Vineeth; Vrunda N. Sukhadia; Dipti Sharma; Hema Murthy; Pushpak Bhattacharyya; S Umesh; Rajeev Sangal
Venue: Interspeech 2023 (main)

Abstract: Cross-lingual dubbing of lecture videos requires the transcription of the original audio, correction and removal of disfluencies, domain term discovery, text-to-text translation into the target language, text-to-speech synthesis followed by isochronous lipsyncing to the original video. This task becomes challenging when the source and target languages belong to different language families, resulting in differences in generated audio duration. This is further compounded by the original speaker's rhythm, especially for extempore speech. This paper describes the challenges in regenerating English lecture videos in Indian languages semi-automatically. A prototype is developed for dubbing lectures into Indian languages. A demo for dubbing with supervision is available online.
MyVoice: Arabic Speech Resource Collaboration Platform
Authors: Yousseif Elshahawy; Yassine El Kheir; Shammur Absar Chowdhury; Ahmed M. Ali
Venue: Interspeech 2023 (main)

Abstract: We introduce MyVoice, a crowdsourcing platform designed to collect Arabic speech to enhance dialectal speech technologies. This platform offers an opportunity to design large dialectal speech datasets; and makes them publicly available. MyVoice allows contributors to select city/country-level fine-grained dialect and record the displayed utterances. Users can switch roles between contributors and annotators. The platform incorporates a quality assurance system that filters out low-quality and spurious recordings before sending them for validation. During the validation phase, contributors can assess the quality of recordings, annotate them, and provide feedback which is then reviewed by administrators. Furthermore, the platform offers flexibility to admin roles to add new data or tasks beyond dialectal speech and word collection, which are displayed to contributors. Thus, enabling collaborative efforts in gathering diverse and large Arabic speech data.
Personal Primer Prototype 1: Invitation to Make Your Own Embooked Speech-Based Educational Artifact
Authors: Daniel D. Hromada; Hyungjoong Kim
Venue: Interspeech 2023 (main)

Abstract: In our show & tell contribution, we present more closely seven properties of a digital educational artifact known as Personal Primer. These are: speech-based, embooked, voluminous, modular, circadian, SDG-compliant, edge-computing. Furthermore, we provide link to code repository as well as enumerate off-the-shelf components which can be combined together in order to yield personal Primer prototypes of the first generation. Thus, this show& tell contribution can be understood as an invitation addressing any motivated teacher, student, parent or engineer willing to make an own copy of a Primer for oneself or others.
Time-frequency Domain Filter-and-sum Network for Multi-channel Speech Separation
Authors: Zhewen Deng; Yi Zhou; Hongqing Liu
Venue: Interspeech 2023 (main)

Abstract: Learning-based methods have made impressive strides in speech separation, and the implicit filter-and-sum network (iFaSNet) stands out as a reliable multi-channel solution. Meanwhile, the TF-GridNet has achieved state-of-the-art performance on the WSJ0-2mix dataset, indicating the underlying capability of time-frequency (T-F) domain speech separation methods. This paper investigates the possibility of constructing a T-F domain filter-and-sum network that improves upon the iFaSNet. In addition to optimizing the separation module, we develop a narrow-band spatial feature as a cross-channel feature and a convolution module for context decoding. With these enhancements, we redesign each module under the iFaSNet architecture, which entirely operates in the T-F domain. Thus, the proposed method is referred to as the TF-FaSNet. Experimental results on fixed microphone array geometries show that the TF-FaSNet outperforms the standard iFaSNet under all conditions with similar model complexity.
Audio-Visual Fusion using Multiscale Temporal Convolutional Attention for Time-Domain Speech Separation
Authors: Debang Liu; Tianqi Zhang; Mads GrÃ¦sbÃ¸ll Christensen; Ying Wei; Zeliang An
Venue: Interspeech 2023 (main)

Abstract: Audio-only speech separation methods cannot fully exploit audio-visual correlation information of speaker, which limits separation performance. Additionally, audio-visual separation methods usually adopt traditional idea of feature splicing and linear mapping to fuse audio-visual features, this approach requires us to think more about fusion process. Therefore, in this paper, combining with the changes of speaker mouth landmarks, we propose a time-domain audio-visual temporal convolution attention speech separation method (AVTA). In AVTA, we design a multiscale temporal convolutional attention (MTCA) to better focus on contextual dependencies of time sequences. We then use sequence learning and fusion network composed of MTCA to build a separation model for speech separation task. On different datasets, AVTA achieves competitive performance, and compared to baseline methods, AVTA is better balanced in training cost, computational complexity and separation performance.
An Efficient Speech Separation Network Based on Recurrent Fusion Dilated Convolution and Channel Attention
Authors: Junyu Wang
Venue: Interspeech 2023 (main)

Abstract: We present an efficient speech separation neural network, ARFDCN, which combines dilated convolutions, multi-scale fusion (MSF), and channel attention to overcome the limited receptive field of convolution-based networks and the high computational cost of transformer-based networks. The suggested network architecture is encoder-decoder based. By using dilated convolutions with gradually increasing dilation value to learn local and global features and fusing them at adjacent stages, the model can learn rich feature content. Meanwhile, by adding channel attention modules to the network, the model can extract channel weights, learn more important features, and thus improve its expressive power and robustness. Experimental results indicate that the model achieves a decent balance between performance and computational efficiency, making it a promising alternative to current mainstream models for practical applications.
Binaural Sound Localization in Noisy Environments Using Frequency-Based Audio Vision Transformer (FAViT)
Authors: Waradon Phokhinanan; Nicolas Obin; Sylvain Argentieri
Venue: Interspeech 2023 (main)

Abstract: Binaural sound source localization (BSSL) aims to locate sound as the way human does, but it falls short due to acoustic interferences. While Convolutional Neural Networks (CNNs) have shown promise in localizing sounds corrupted by noise, their large parameter and training data requirements make them unsuitable for real-time processing on devices like hearing aids and robots. In this paper, we propose an adapted Vision Transformer (ViT) model for BSSL in noisy environments. Inspired by the Duplex Theory, our model uses selective attention mechanisms to the frequency range of binaural features to aid in sound localization. Our model outperformed recent CNNs and standard audio ViT models in localizing speech in unseen noises and speakers, even in challenging conditions with low training data and parameters. The attention heatmap results suggest differences in how humans and machines process binaural cues, opening up for further investigation.
Contrastive Learning based Deep Latent Masking for Music Source Separation
Authors: Jihyun Kim; Hong-Goo Kang
Venue: Interspeech 2023 (main)

Abstract: Recent studies on music source separation have extended their applicability to generic audio signals. Real-time applications for music source separation are necessary to provide services such as custom equalizers or to improve the sound of live streaming with diverse effects. However, most prior methods are unsuitable for real-time applications due to their high computational complexity, large memory usage, or long latency. To overcome these problems, we propose a Wave-U-Net type of music source separation network that utilizes high-dimensional masking for the deep latent domain features. We also introduce a contrastive learning technique to estimate the salient latent space embedding of each target source using a masking-based approach. The performance of our proposed model is evaluated on the MUSDB18HQ dataset in comparison with several baselines. The experiments confirm that our proposed model is capable of real-time processing and outperforms existing models.
Speaker Extraction with Detection of Presence and Absence of Target Speakers
Authors: Ke Zhang; Marvin Borsdorf; Zexu Pan; Haizhou Li; Yangjie Wei; Yi Wang
Venue: Interspeech 2023 (main)

Abstract: Target speaker extraction extracts a target voice from a given cocktail party mixture signal. Most studies are restricted to conditions in which the target speaker is present in the mixture (PT), which often fail when the target speaker is absent (AT). Training on both PT and AT situations helps, but degrades the PT performance as the model intrinsically tries to detect the target presence. We propose a new model, called TSEJoint, that jointly performs target speaker detection and extraction. Both tasks share the low-level modules, allowing the detection branch to use a pre-separated signal and keeping the overall processing pipeline length similar, while at the high-level they have different branches to ensure the performance of each task. We evaluate our proposed methods under PT and AT conditions comprising one and two talkers. The TSEJoint model shows better extraction performance under the PT condition and better detection performance on all conditions compared with the baseline.
PIAVE: A Pose-Invariant Audio-Visual Speaker Extraction Network
Authors: Qinghua Liu; Meng Ge; Zhizheng Wu; Haizhou Li
Venue: Interspeech 2023 (main)

Abstract: It is common in everyday spoken communication that we look at the turning head of a talker to listen to his/her voice. Humans see the talker to listen better, so do machines. However, previous studies on audio-visual speaker extraction have not effectively handled the varying talking face. This paper studies how to take full advantage of the varying talking face. We propose a Pose-Invariant Audio-Visual Speaker Extraction Network (PIAVE) that incorporates an additional pose-invariant view to improve audio-visual speaker extraction. Specifically, we generate the pose-invariant view from each original pose orientation, which enables the model to receive a consistent frontal view of the talker regardless of his/her head pose, therefore, forming a multi-view visual input for the speaker. Experiments on the multi-view MEAD and in-the-wild LRS3 dataset demonstrate that PIAVE outperforms the state-of-the-art and is more robust to pose variations.
Spatial LibriSpeech: An Augmented Dataset for Spatial Audio Learning
Authors: Miguel Sarabia; Elena Menyaylenko; Alessandro Toso; Skyler Seto; Zakaria Aldeneh; Shadi Pirhosseinloo; Luca Zappella; Barry-John Theobald; Nicholas Apostoloff; Jonathan Sheaffer
Venue: Interspeech 2023 (main)

Abstract: We present Spatial LibriSpeech, a spatial audio dataset with over 650 hours of 19-channel audio, first-order ambisonics, and optional distractor noise. Spatial LibriSpeech is designed for machine learning model training, and it includes labels for source position, speaking direction, room acoustics and geometry. Spatial LibriSpeech is generated by augmenting LibriSpeech samples with 200k+ simulated acoustic conditions across 8k+ synthetic rooms. To demonstrate the utility of our dataset, we train models on four spatial audio tasks, resulting in a median absolute error of 6.60Â° on 3D source localization, 0.43m on distance, 90.66ms on T30, and 2.74dB on direct-to-reverberant ratio estimation. We show that the same models generalize well to widely-used evaluation datasets, e.g., obtaining a median absolute error of 12.43Â° on 3D source localization on TUT Sound Events 2018, and 157.32ms on T30 estimation on ACE Challenge.
Image-driven Audio-visual Universal Source Separation
Authors: Chenxing Li; Ye Bai; Yang Wang; Feng Deng; Yuanyuan Zhao; Zhuo Zhang; Xiaorui Wang
Venue: Interspeech 2023 (main)

Abstract: This paper introduces an image-driven audio-visual universal source separation (ID-USS) and proposes ID-USS-Conformer. ID-USS aims to separate a target source from the mixture based on the input image that is consistent with the target. Importantly, ID-USS only focuses on the sound made by the target in this image, not on the description of the target or the semantic information of the picture. In detail, ID-USS-Conformer mainly consists of an Efficient-b3-based visual branch and a Conformer-based audio branch. The visual branch extracts the visual clue of the target from the input image. After the audio branch fuses the visual features, ID-USS-Conformer separates the target source from the mixture. We launch an ID-USS dataset and verify the effectiveness of ID-USS-Conformer on it. The ID-USS-Conformer has achieved a 10.139 dB signal-to-distortion ratio improvement in the test set and outperformed the compared methods.
Joint Blind Source Separation and Dereverberation for Automatic Speech Recognition using Delayed-Subsource MNMF with Localization Prior
Authors: Mieszko FraÅ; Marcin Witkowski; Konrad Kowalczyk
Venue: Interspeech 2023 (main)

Abstract: Overlapping speech and high room reverberation deteriorate the accuracy of automatic speech recognition (ASR). This paper proposes a method for jointly optimum source separation and dereverberation using delayed subsource multichannel nonnegative matrix factorization (MNMF). We formulate a subsource-based signal model that accounts for late room reverberation using time-delayed microphone signals from several past time frames. We then propose a maximum a posteriori (MaP) estimator based on MNMF with localization prior on the mixing matrix suitable for direct-path and reverberant signal components estimation. Finally, two algorithms are derived, namely the original and simplified delayed subsource MNMF, which are shown to outperform many state-of-the-art approaches. The results of experimental evaluations, performed using real and simulated data, indicate superior performance of the proposed processing in terms of the word error rate (WER) as well as signal-to-distortion ratio (SDR).
SDNet: Stream-attention and Dual-feature Learning Network for Ad-hoc Array Speech Separation
Authors: Honglong Wang; Chengyun Deng; Yanjie Fu; Meng Ge; Longbiao Wang; Gaoyan Zhang; Jianwu Dang; Fei Wang
Venue: Interspeech 2023 (main)

Abstract: Considerable progress has been made in multi-channel speech separation for fixed arrays. In this paper, we aim to develop a robust system for ad-hoc arrays to deal with uncertainties of microphone locations and numbers. Previous works commonly used the averaging method for ad-hoc arrays, overlooking the diversity of microphones in various positions. Some studies suggest that microphones with high signal-to-noise ratio(SNR) are more helpful in improving speech quality. Motivated by this, we propose stream-attention and dual-feature learning network called SDNet. The key points are as follows: 1) We propose a dual-feature learning block with fewer parameters to learn the long-term dependency better. 2) Based on this high-quality speech representation, we further propose stream attention that effectively handles microphone variability and allocates more attention to microphones with higher SNR. Experiments show that our proposed model outperforms other advanced baselines.