GenAI4ELab/papercli-papers huggingface dataset:
Array Geometry-Robust Attention-Based Neural Beamformer for Moving Speakers
Authors: Marvin Tammen; Tsubasa Ochiai; Marc Delcroix; Tomohiro Nakatani; Shoko Araki; Simon Doclo
Venue: Interspeech 2024 (main)

Abstract: Although mask-based beamforming is a powerful speech enhancement approach, it often requires manual parameter tuning to handle moving speakers. Recently, this approach was augmented with an attention-based spatial covariance matrix aggregator (ASA) module, enabling accurate tracking of moving speakers without manual tuning. However, the deep neural network model used in this module is limited to specific microphone arrays, necessitating a different model for varying channel permutations, numbers, or geometries. To improve the robustness of the ASA module against such variations, in this paper we investigate three approaches: training with random channel configurations, employing the transform-average-concatenate method to process multi-channel input features, and utilizing robust input features. Our experiments on the CHiME-3 and DEMAND datasets show that these approaches enable the ASA-augmented beamformer to track moving speakers across different microphone arrays unseen in training.
FoVNet: Configurable Field-of-View Speech Enhancement with Low Computation and Distortion for Smart Glasses
Authors: Zhongweiyang Xu; Ali Aroudi; Ke Tan; Ashutosh Pandey; Jung-Suk Lee; Buye Xu; Francesco Nesta
Venue: Interspeech 2024 (main)

Abstract: This paper presents a novel multi-channel speech enhancement approach, FoVNet, that enables highly efficient speech enhancement within a configurable field of view (FoV) of a smart-glasses user without needing specific target-talker(s) directions. It advances over prior works by enhancing all speakers within any given FoV, with a hybrid signal processing and deep learning approach designed with high computational efficiency. The neural network component is designed with ultra-low computation (about 50 MMACS). A multi-channel Wiener filter and a post-processing module are further used to improve perceptual quality. We evaluate our algorithm with a microphone array on smart glasses, providing a configurable, efficient solution for augmented hearing on energy-constrained devices. FoVNet excels in both computational efficiency and speech quality across multiple scenarios, making it a promising solution for smart glasses applications.
Audio Enhancement from Multiple Crowdsourced Recordings: A Simple and Effective Baseline
Authors: Shiran Aziz; Yossi Adi; Shmuel Peleg
Venue: Interspeech 2024 (main)

Abstract: With the popularity of cellular phones, events are often recorded by multiple devices from different locations and shared on social media. Several different recordings could be found for many events. Such recordings are usually noisy, where noise for each device is local and unrelated to others. This case of multiple microphones at unknown locations, capturing local, uncorrelated noise, was rarely treated in the literature. In this work we propose a simple and effective crowdsourced audio enhancement method to remove local noises at each input audio signal. Then, averaging all cleaned source signals gives an improved audio of the event. We demonstrate the effectiveness of our method using synthetic audio signals, together with real-world recordings. This simple approach can set a new baseline for crowdsourced audio enhancement for more sophisticated methods which we hope will be developed by the research community. Code, dataset, and models are available.
DeFTAN-AA: Array Geometry Agnostic Multichannel Speech Enhancement
Authors: Dongheon Lee; Jung-Woo Choi
Venue: Interspeech 2024 (main)

Abstract: We propose an array geometry agnostic multichannel speech enhancement model, which is trained on a single microphone array but can enhance speech in various arrays with different shapes and numbers of microphones. To enable array-agnostic processing, the model employs a gated split dense block (GSDB) that separates foreground speech and background noise regardless of array geometry. Furthermore, to design an array-agnostic encoder compatible with different numbers of microphones, we introduce the spatial transformer (ST) that aggregates spatial information by channel-wise self-attention. The proposed space-object cross-attention (SOCA) block alleviates overfitting to a specific array configuration through cross-attention between spatial features and object features. Experimental results demonstrate the efficacy of the proposed model across various array geometries in both simulated and real-world datasets.
PLDNet: PLD-Guided Lightweight Deep Network Boosted by Efï¬cient Attention for Handheld Dual-Microphone Speech Enhancement
Authors: Nan Zhou; Youhai Jiang; Jialin Tan; Chongmin Qi
Venue: Interspeech 2024 (main)

Abstract: Low-complexity speech enhancement on mobile phones is crucial in the era of 5G. Thus, focusing on handheld mobile phone communication scenario, based on power level difference (PLD) algorithm and lightweight U-Net, we propose PLD-guided lightweight deep network (PLDNet), an extremely lightweight dual-microphone speech enhancement method that integrates the guidance of signal processing algorithm and lightweight attention-augmented U-Net. For the guidance information, we employ PLD algorithm to pre-process dual-microphone spectrum, and feed the output into subsequent deep neural network, which utilizes a lightweight U-Net with our proposed gated convolution augmented frequency attention (GCAFA) module to extract desired clean speech. Experimental results demonstrate that our proposed method achieves competitive performance with recent top-performing models while reducing computational cost by over 90%, highlighting the potential for low-complexity speech enhancement on mobile phones.
Highly Intelligible Speaker-Independent Articulatory Synthesis
Authors: Charles McGhee; Kate Knill; Mark Gales
Venue: Interspeech 2024 (main)

Abstract: An articulatory synthesiser which could accurately map vocal tract features to speech would enable novel evaluation of acoustic-to-articulatory inversion models beyond the small, typically monolingual, articulatory datasets available. However, current deep articulatory synthesisers and physical simulation-based synthesisers struggle to produce consistently intelligible speech, with Word Error Rates (WER) of around 20% for real or hand-crafted articulatory input. Additionally, deep learning methods have often only achieved this level of intelligibility when training and evaluating on the same speaker (speaker-dependent training). In this paper, we create a highly intelligible (WER 7% for real data and 10% for synthetic), speaker-independent articulatory synthesiser by training a deep synthesiser on a combination of high-quality real data and synthetic data generated by inversion. We then perform a multilingual evaluation of the joint inversion-synthesis system.
An Attribute Interpolation Method in Speech Synthesis by Model Merging
Authors: Masato Murata; Koichi Miyazaki; Tomoki Koriyama
Venue: Interspeech 2024 (main)

Abstract: With the development of speech synthesis, recent research has focused on challenging tasks, such as speaker generation and emotion intensity control. Attribute interpolation is a common approach to these tasks. However, most previous methods for attribute interpolation require specific modules or training methods. We propose an attribute interpolation method in speech synthesis by model merging. Model merging is a method that creates new parameters by only averaging the parameters of base models. The merged model can generate an output with an intermediate feature of the base models. This method is easily applicable without specific modules or training methods, as it uses only existing trained base models. We merged two text-to-speech models to achieve attribute interpolation and evaluated its performance on speaker generation and emotion intensity control tasks. As a result, our proposed method achieved smooth attribute interpolation while keeping the linguistic content in both tasks.
Low-dimensional Style Token Control for Hyperarticulated Speech Synthesis
Authors: Miku Nishihara; Dan Wells; Korin Richmond; Aidan Pine
Venue: Interspeech 2024 (main)

Abstract: Global style tokens (GSTs) allow for rich modelling of the variation in a speech corpus and subsequent control of text-tospeech synthesis (TTS). However, certain styles of speech may be marked by variation along multiple dimensions, complicating the interpretation and control of learned style tokens. One example is hyperarticulated or âclearâ speech, for example as directed toward listeners with hearing impairments or language learners in the classroom, which in English is characterised by reduced speaking rate, increased F0, more careful articulation of vowels and plosive consonants, and other factors. We present a method for simplifying control of style tokens by applying principal components analysis (PCA) to GST weights from a TTS system trained on both plain and clear speech. We identify the axes of variation in PCA space with the acoustic correlates of clear speech in English and show that we can synthesise either style by moving along a single dimension in that space.
Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generation
Authors: Hanzhao Li; Liumeng Xue; Haohan Guo; Xinfa Zhu; Yuanjun Lv; Lei Xie; Yunlin Chen; Hao Yin; Zhifei Li
Venue: Interspeech 2024 (main)

Abstract: The multi-codebook speech codec enables the application of large language models (LLM) in TTS but bottlenecks efficiency and robustness due to multi-sequence prediction. To avoid this obstacle, we propose Single-Codec, a singlecodebook single-sequence codec, which employs a disentangled VQ-VAE to decouple speech into a time-invariant embedding and a phonetically-rich discrete sequence. Furthermore, the encoder is enhanced with 1) contextual modeling with a BLSTM module to exploit the temporal information, 2) a hybrid sampling module to alleviate distortion from upsampling and downsampling, and 3) a resampling module to encourage discrete units to carry more phonetic information. Compared with multi-codebook codecs, e.g., EnCodec and TiCodec, Single- Codec demonstrates higher reconstruction quality with a lower bandwidth of only 304bps. The effectiveness of Single-Code is further validated by LLM-TTS experiments, showing improved naturalness and intelligibility.
LiveSpeech: Low-Latency Zero-shot Text-to-Speech via Autoregressive Modeling of Audio Discrete Codes
Authors: Trung Dang; David Aponte; Dung Tran; Kazuhito Koishida
Venue: Interspeech 2024 (main)

Abstract: Prior works have demonstrated zero-shot text-to-speech by using a generative language model on audio tokens obtained via a neural audio codec. It is still challenging, however, to adapt them to low-latency scenarios. In this paper, we present LiveSpeech - a fully autoregressive language model-based approach for zero-shot text-to-speech, enabling low-latency streaming of the output audio. To allow multiple token prediction within a single decoding step, we propose (1) using adaptive codebook loss weights that consider codebook contribution in each frame and focus on hard instances, and (2) grouping codebooks and processing groups in parallel. Experiments show our proposed models achieve competitive results to state-of-the-art baselines in terms of content accuracy, speaker similarity, audio quality, and inference speed while being suitable for low-latency streaming applications.
ClariTTS: Feature-ratio Normalization and Duration Stabilization for Code-mixed Multi-speaker Speech Synthesis
Authors: Changhwan Kim
Venue: Interspeech 2024 (main)

Abstract: Recent text-to-speech (TTS) models have synthesized remarkably natural speech for code-mixed TTS as well as cross-lingual TTS. However, code-mixed texts are synthesized with unnatural accents for each word because speaker-related features can include linguistic features from the speaker's source language. To solve the problems, we propose ClariTTS, which synthesizes speech with appropriate accents for the language of each word in code-mixed texts. Specifically, we propose feature-ratio normalized affine coupling layer in the flow-based TTS model, which disentangles speaker and linguistic features to prevent the accent of the target speaker's source language from being included in the target language. Furthermore, we introduce a duration stabilization training objectives to ensure stable duration prediction in code-mixed TTS. From the experimental results, we demonstrate that ClariTTS reliably generates code-mixed speech with clear pronunciation while preserving speaker identity.
Multi-modal Adversarial Training for Zero-Shot Voice Cloning
Authors: John Janiczek; Dading Chong; Dongyang Dai; Arlo Faria; Chao Wang; Tao Wang; Yuzong Liu
Venue: Interspeech 2024 (main)

Abstract: A text-to-speech (TTS) model trained to reconstruct speech given text tends towards predictions that are close to the average characteristics of a dataset, failing to model the variations that make human speech sound natural. This problem is magnified for zero-shot voice cloning, a task that requires training data with high variance in speaking styles. We build off of recent works which have used Generative Advsarial Networks (GAN) by proposing a Transformer encoder-decoder architecture to conditionally discriminates between real and generated speech features. The discriminator is used in a training pipeline that improves both the acoustic and prosodic features of a TTS model. We introduce our novel adversarial training technique by applying it to a FastSpeech2 acoustic model and training on Libriheavy, a large multi-speaker dataset, for the task of zero-shot voice cloning. Our model achieves improvements over the baseline in terms of speech quality and speaker similarity.
Learning Fine-Grained Controllability on Speech Generation via Efficient Fine-Tuning
Authors: Chung-Ming Chien; Andros Tjandra; Apoorv Vyas; Matt Le; Bowen Shi; Wei-Ning Hsu
Venue: Interspeech 2024 (main)

Abstract: As the scale of generative models continues to grow, efficient reuse and adaptation of pre-trained models have become crucial considerations. In this work, we propose Voicebox Adapter, a novel approach that integrates fine-grained conditions into a pre-trained Voicebox speech generation model using a crossattention module. To ensure a smooth integration of newly added modules with pre-trained ones, we explore various efficient fine-tuning approaches. Our experiment shows that the LoRA with bias-tuning configuration yields the best performance, enhancing controllability without compromising speech quality. Across three fine-grained conditional generation tasks, we demonstrate the effectiveness and resource efficiency of Voicebox Adapter. Follow-up experiments further highlight the robustness of Voicebox Adapter across diverse data setups.
Modeling Vocal Tract Like Acoustic Tubes Using the Immersed Boundary Method
Authors: Rongshuai Wu; Debasish Ray Mohapatra; Sidney Fels
Venue: Interspeech 2024 (main)

Abstract: The standard finite-difference time-domain (FDTD) wave solver employs a regular grid that approximates vocal tract boundaries in a stair-stepped manner, making it less effective for modeling complex and dynamic geometry. We present a novel 2D wave solver that integrates a unique immersed boundary method with FDTD, enabling precise approximation of wave propagation in acoustic tubes. The solver uses Lagrangian points to define vocal tract boundaries on a regular grid, eliminating their stair-stepped discretization. The boundary and the flow equations interact via additional forcing terms characterized by boundary immittances. The results show that the formant frequencies of a single-segment tube closely align with its analytical solutions, exhibiting a percentage deviation of less than 3%. Similarly, the frequency responses of a two-segment tube and actual vocal tract geometries show good agreement with existing state-of-the-art 2D and 3D wave solvers, except for a few discrepancies.
Small-E: Small Language Model with Linear Attention for Efficient Speech Synthesis
Authors: ThÃ©odor Lemerle; Nicolas Obin; Axel Roebel
Venue: Interspeech 2024 (main)

Abstract: Recent advancements in text-to-speech (TTS) powered by language models have showcased remarkable capabilities in achieving naturalness and zero-shot voice cloning. Notably, the decoder-only transformer is the prominent architecture in this domain. However, transformers face challenges stemming from their quadratic complexity in sequence length, impeding training on lengthy sequences and resource-constrained hardware. Moreover they lack specific inductive bias with regards to the monotonic nature of TTS alignments. In response, we propose to replace transformers with emerging recurrent architectures and introduce specialized cross-attention mechanisms for reducing repeating and skipping issues. Consequently our architecture can be efficiently trained on long samples and achieve state-of-the-art zero-shot voice cloning against baselines of comparable size. Our implementation and demos are available at https://github.com/theodorblackbird/lina-speech.
Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignment
Authors: Paarth Neekhara; Shehzeen Hussain; Subhankar Ghosh; Jason Li; Boris Ginsburg
Venue: Interspeech 2024 (main)

Abstract: Large Language Model (LLM) based text-to-speech (TTS) systems have demonstrated remarkable capabilities in handling large speech datasets and generating natural speech for new speakers. However, LLM-based TTS models are not robust as the generated output can contain repeating words, missing words and mis-aligned speech (referred to as hallucinations or attention errors), especially when the text contains multiple occurrences of the same token. We examine these challenges in an encoder-decoder transformer model and find that certain cross-attention heads in such models implicitly learn the text and speech alignment when trained for predicting speech tokens for a given text. To make the alignment more robust, we propose techniques utilizing CTC loss and attention priors that encourage monotonic cross-attention over the text tokens. Our guided attention training technique does not introduce any new learnable parameters and significantly improves robustness of LLM-based TTS models.
Synthesizing Long-Form Speech merely from Sentence-Level Corpus with Content Extrapolation and LLM Contextual Enrichment
Authors: Shijie Lai; Minglu He; Zijing Zhao; Kai Wang; Hao Huang; Jichen Yang
Venue: Interspeech 2024 (main)

Abstract: Current text-to-speech (TTS) models can produce natural speech but often fail to synthesize long-form speech properly when only sentence-level corpus is available. The failure is mainly due to 1) poor length generalization of the acoustic model, 2) lack of appropriate pause marks in the inference text, and 3) absence of contextual information during training. We propose Content Extrapolation, which includes introducing Moving Average Equipped Gated Attention (MEGA) to improve the model's generalization for addressing 1) and presenting the Global-information-enhanced Classification Pause Insertion model (GCPI) to address 2). For 3), we propose LLM-based Contextual Enrichment (LLM-CE) to generate multiple sets of different contexts. Experiments show that the proposed methods solve the above issues and successfully generate long-form speech with clear pronunciation and natural prosody using only sentence-level corpus, reducing training costs.
FluentEditor: Text-based Speech Editing by Considering Acoustic and Prosody Consistency
Authors: Rui Liu; Jiatian Xi; Ziyue Jiang; Haizhou Li
Venue: Interspeech 2024 (main)

Abstract: The current Text-based Speech Editing (TSE) techniques have focused on reducing the difference between the generated speech segment and the reference target in the editing region, ignoring its local and global fluency in the context and original utterance. To maintain the speech fluency, we propose a fluency speech editing model, termed FluentEditor, by considering fluency-aware training criterion in the TSE training. Specifically, the acoustic consistency constraint aims to smooth the transition between the edited region and its neighboring acoustic segments consistent with the ground truth, while the prosody consistency constraint seeks to ensure that the prosody attributes within the edited regions remain consistent with the overall style of the original utterance. The subjective and objective experimental results on VCTK demonstrate that our FluentEditor outperforms all advanced baselines in terms of naturalness and fluency. The audio samples and code are available at https://github.com/AI-S2-Lab/FluentEditor.
Phonetic Enhanced Language Modeling for Text-to-Speech Synthesis
Authors: Kun Zhou; Shengkui Zhao; Yukun Ma; Chong Zhang; Hao Wang; Dianwen Ng; Chongjia Ni; Trung Hieu Nguyen; Jia Qi Yip; Bin Ma
Venue: Interspeech 2024 (main)

Abstract: Recent language model-based text-to-speech (TTS) frameworks demonstrate scalability and in-context learning capabilities. However, they suffer from robustness issues due to the accumulation of errors in speech unit predictions during autoregressive language modeling. In this paper, we propose a phonetic enhanced language modeling method to improve the performance of TTS models. We leverage self-supervised representations that are phonetically rich as the training target for the autoregressive language model. Subsequently, a non-autoregressive model is employed to predict discrete acoustic codecs that contain fine-grained acoustic details. The TTS model focuses solely on linguistic modeling during autoregressive training, thereby reducing the error propagation that occurs in non-autoregressive training. Both objective and subjective evaluations validate the effectiveness of our proposed method.
High Fidelity Text-to-Speech Via Discrete Tokens Using Token Transducer and Group Masked Language Model
Authors: Joun Yeop Lee; Myeonghun Jeong; Minchan Kim; Ji-Hyun Lee; Hoon-Young Cho; Nam Soo Kim
Venue: Interspeech 2024 (main)

Abstract: We propose a novel two-stage text-to-speech (TTS) framework with two types of discrete tokens, i.e., semantic and acoustic tokens, for high-fidelity speech synthesis. It features two core components: the Interpreting module, which processes text and a speech prompt into semantic tokens focusing on linguistic contents and alignment, and the Speaking module, which captures the timbre of the target voice to generate acoustic tokens from semantic tokens, enriching speech reconstruction. The Interpreting stage employs a transducer for its robustness in aligning text to speech. In contrast, the Speaking stage utilizes a Conformer-based architecture integrated with a Grouped Masked Language Model (G-MLM) to boost computational efficiency. Our experiments verify that this innovative structure surpasses the conventional models in the zero-shot scenario in terms of speech quality and speaker similarity.
FastLips: an End-to-End Audiovisual Text-to-Speech System with Lip Features Prediction for Virtual Avatars
Authors: Martin Lenglet; Olivier Perrotin; Gerard Bailly
Venue: Interspeech 2024 (main)

Abstract: In this paper, we introduce FastLips, an end-to-end neural model designed to generate speech and co-verbal facial movements from text, animating a virtual avatar. Based on the FastSpeech2 Text-to-Speech model, FastLips integrates an audiovisual Transformer-based encoder with distinct audio and visual neural decoders. This model combines audiovisual representations computed by the shared encoder with asynchronous generation of audio and visual features. Furthermore, we enhance the model with explicit predictors of lip aperture and spreading, adapted from prosodic FastSpeech2's variance adaptor. The proposed model generates mel-spectrograms and facial features (head, eyes, jaw and lip movements) to drive the virtual avatar's action units. In our evaluation, we compare FastLips with a baseline audiovisual-Tacotron2, demonstrating the advantages of the FastSpeech2 architecture for lip generation. This benefit becomes particularly prominent when implementing explicit lip prediction.