GenAI4ELab/papercli-papers huggingface dataset:
ERes2NetV2: Boosting Short-Duration Speaker Verification Performance with Computational Efficiency
Authors: Yafeng Chen; Siqi Zheng; Hui Wang; Luyao Cheng; Qian Chen; Shiliang Zhang; Junjie Li
Venue: Interspeech 2024 (main)

Abstract: Speaker verification systems experience significant performance degradation when tasked with short-duration trial recordings. To address this challenge, a multi-scale feature fusion approach has been proposed to effectively capture speaker characteristics from short utterances. Constrained by the modelâs size, a robust backbone Enhanced Res2Net (ERes2Net) combining global and local feature fusion demonstrates sub-optimal performance in short-duration speaker verification. To further improve the short-duration feature extraction capability of ERes2Net, we expand the channel dimension within each stage. However, this modification also increases the number of model parameters and computational complexity. To alleviate this problem, we propose an improved ERes2NetV21 by pruning redundant structures, ultimately reducing both the model parameters and its computational cost. A range of experiments conducted on the VoxCeleb datasets exhibits the superiority of ERes2NetV2, which achieves EER of 0.61% for the full duration trial, 0.98% for the 3s-duration trial, and 1.48% for the 2s-duration trial on VoxCeleb1-O, respectively.
Classification of Room Impulse Responses and its application for channel verification and diarization
Authors: Yuri Khokhlov; Tatiana Prisyach; Anton Mitrofanov; Dmitry Dutov; Igor Agafonov; Tatiana Timofeeva; Aleksei Romanenko; Maxim Korenevsky
Venue: Interspeech 2024 (main)

Abstract: This paper describes experiments on a classification of Room Impulse Responses (RIRs) from recordings where acoustic channel is described by corresponding RIRs. The classifiers are trained on large sets of synthetic RIRs and then used as extractors of RIR embeddings bearing information about acoustic characteristic of a room as well as the locations of sound source and microphone. Experiments on different datasets demonstrate that RIR embeddings can be used for verification of acoustic channel and for the diarization in meeting-like scenarios where speakers' positions are fixed. The verification experiments on VoxCeleb1 and BUT Speech@FIT Reverb show reasonable performance of RIR embeddings. The diarization results with RIR embeddings on LibriCSS dataset are better than those with state-of-the-art speaker embeddings that shows the potential of the proposed approach.
RIR-in-a-Box: Estimating Room Acoustics from 3D Mesh Data through Shoebox Approximation
Authors: Liam Kelley; Diego Di Carlo; Aditya Arie Nugraha; Mathieu Fontaine; Yoshiaki Bando; Kazuyoshi Yoshii
Venue: Interspeech 2024 (main)

Abstract: This paper describes a method for estimating the room impulse response (RIR) for a microphone and a sound source located at arbitrary positions from the 3D mesh data of the room. Simulating realistic RIRs with pure physics-driven methods often fails the balance between physical consistency and computational efficiency, hindering application to real time speech processing. Alternatively, one can use MESH2IR, a fast black-box estimator that consists of an encoder extracting latent code from mesh data with a graph convolutional network (GCN) and a decoder generating the RIR from the latent code. Combining these two approaches, we propose a fast yet physically coherent estimator with interpretable latent code based on differentiable digital signal processing (DDSP). Specifically, the encoder estimates a virtual shoebox room scene that acoustically approximates the real scene, accelerating physical simulation with the differentiable image-source model in the decoder. Our experiments showed that our method outperformed MESH2IR for real mesh data obtained with the depth scanner of Microsoft HoloLens 2, and can provide correct spatial consistency for binaural RIRs.
Novel-view Acoustic Synthesis From 3D Reconstructed Rooms
Authors: Byeongjoo Ahn; Karren Yang; Brian Hamilton; Jonathan Sheaffer; Anurag Ranjan; Miguel Sarabia; Oncel Tuzel; Jen-Hao Rick Chang
Venue: Interspeech 2024 (main)

Abstract: We investigate the benefit of combining blind audio recordings with 3D scene information for novel-view acoustic synthesis. Given audio recordings from 2â4 microphones and the 3D geometry and material of a scene containing multiple unknown sound sources, we estimate the sound anywhere in the scene. We identify the main challenges of novel-view acoustic synthesis as sound source localization, separation, and dereverberation. While naively training an end-to-end network fails to produce high-quality results, we show that incorporating room impulse responses (RIRs) derived from 3D reconstructed rooms enables the same network to jointly tackle these tasks. Our method outperforms existing methods designed for the individual tasks, demonstrating its effectiveness at utilizing 3D visual information. In a simulated study on the Matterport3D-NVAS dataset, our model achieves near-perfect accuracy on source localization, a PSNR of 26.44 dB and a SDR of 14.23 dB for source separation and dereverberation, resulting in a PSNR of 25.55 dB and a SDR of 14.20 dB on novel-view acoustic synthesis. We release our code and model on our project website at https://github.com/apple/ml-nvas3d. Please wear headphones when listening to the results.
Spatial Acoustic Enhancement Using Unbiased Relative Harmonic Coefficients
Authors: Liang Tao; Maoshen Jia; Yonggang Hu; Changchun Bao
Venue: Interspeech 2024 (main)

Abstract: This paper targets at enhancing the noisy soundfield over the entire recording area and all the individual channels, while preserving the spatial clues of the original soundfield. For the goal, we utilize a recently proposed spherical harmonics (SH) domain feature denoted relative harmonic coefficients (RHC) as it compactly contains the sourceâs spatial information. Specifically, we (i) propose an unbiased estimator of RHC in noisy environments; (ii) estimate the source signal in noisy environments using a SH domain beamformer; (iii) enhance the SH coefficients by multiplying the estimated RHC and source signal; and (iv) reconstruct the entire soundfield based on the enhanced SH coefficients. Finally, we evaluate and validate the performance of the enhancement algorithm using extensive simulations.
Design of Feedback Active Noise Cancellation Filter Using Nested Recurrent Neural Networks
Authors: Alireza Bayestehtashk; Amit Kumar; Mike Wurtz
Venue: Interspeech 2024 (main)

Abstract: We examine the problem of Feedback (FB) Active Noise Cancellation (ANC) and propose a novel DSP-inspired neural network for designing FB ANC filters. It reinterprets the ANC problem as a neural network optimization issue, thus allowing the use of modern machine learning tools. At the same time it also benefits from classical DSP techniques as it retains and extends the state of the art for efficient generation of anti-noise. The proposed approach leverages machine learning algorithms to learn stochastically optimal FB ANC filter coefficients under a variety of conditions and constraints that make the problem intractable for classical methods. One challenge with FB ANC filter design is to guarantee the stability of the IIR filter structure as well as the feedback loop around it. The proposed method meets this stability requirement while providing new avenues for improved system efficiency and adaptability. We demonstrate its effectiveness through simulations using real-world models.
Neuromorphic Keyword Spotting with Pulse Density Modulation MEMS Microphones
Authors: Sidi Yaya Arnaud Yarga; Sean U N Wood
Venue: Interspeech 2024 (main)

Abstract: The Keyword Spotting (KWS) task involves continuous audio stream monitoring to detect predefined words, requiring low energy devices for continuous processing. Neuromorphic devices effectively address this energy challenge. However, the general neuromorphic KWS pipeline, from microphone to Spiking Neural Network (SNN), entails multiple processing stages. Leveraging the popularity of Pulse Density Modulation (PDM) microphones in modern devices and their similarity to spiking neurons, we propose a direct microphone-to-SNN connection. This approach eliminates intermediate stages, notably reducing computational costs. The system achieved an accuracy of 91.54\% on the Google Speech Command (GSC) dataset, surpassing the state-of-the-art for the Spiking Speech Command (SSC) dataset which is a bio-inspired encoded GSC. Furthermore, the observed sparsity in network activity and connectivity indicates potential for remarkably low energy consumption in a neuromorphic device implementation.
RevRIR: Joint Reverberant Speech and Room Impulse Response Embedding using Contrastive Learning with Application to Room Shape Classification
Authors: Jacob Bitterman; Daniel Levi; Hilel Hagai Diamandi; Sharon Gannot; Tal Rosenwein
Venue: Interspeech 2024 (main)

Abstract: This paper focuses on room fingerprinting, a task involving the analysis of an audio recording to determine the specific volume and shape of the room in which it was captured. While it is relatively straightforward to determine the basic room parameters from the Room Impulse Responses (RIR), doing so from a speech signal is a cumbersome task. To address this challenge, we introduce a dual-encoder architecture that facilitates the estimation of room parameters directly from speech utterances. During pre-training, one encoder receives the RIR while the other processes the reverberant speech signal. A contrastive loss function is employed to embed the speech and the acoustic response jointly. In the fine-tuning stage, the specific classification task is trained. In the test phase, only the reverberant utterance is available, and its embedding is used for the task of room shape classification. The proposed scheme is extensively evaluated using simulated acoustic environments.
ConsistencyTTA: Accelerating Diffusion-Based Text-to-Audio Generation with Consistency Distillation
Authors: Yatong Bai; Trung Dang; Dung Tran; Kazuhito Koishida; Somayeh Sojoudi
Venue: Interspeech 2024 (main)

Abstract: Diffusion models are instrumental in text-to-audio (TTA) generation. Unfortunately, they suffer from slow inference due to an excessive number of queries to the underlying denoising network per generation. To address this bottleneck, we introduce ConsistencyTTA, a framework requiring only a single non-autoregressive network query, thereby accelerating TTA by hundreds of times. We achieve so by proposing âCFG-aware latent consistency model,â which adapts consistency generation into a latent space and incorporates classifier-free guidance (CFG) into model training. Moreover, unlike diffusion models, ConsistencyTTA can be finetuned closed-loop with audio-space text-aware metrics, such as CLAP score, to further enhance the generations. Our objective and subjective evaluation on the AudioCaps dataset shows that compared to diffusion-based counterparts, ConsistencyTTA reduces inference computation by 400x while retaining generation quality and diversity.
Audio Editing with Non-Rigid Text Prompts
Authors: Francesco Paissan; Luca Della Libera; Zhepei Wang; Paris Smaragdis; Mirco Ravanelli; Cem Subakan
Venue: Interspeech 2024 (main)

Abstract: In this paper, we explore audio editing with non-rigid text prompts via Latent Diffusion Models. Our methodology is based on carrying out a fine-tuning step on the latent diffusion model, which increases the overall faithfulness of the generated edits to the input audio. We quantitatively and qualitatively show that our pipeline obtains results which outperform current state-of-the-art neural audio editing pipelines for addition, style transfer, and inpainting. Through a user study, we show that our method results in higher user preference compared to several baselines. We also show that the produced edits obtain better trade-offs in terms of fidelity to the text prompt and to the input audio compared to the baselines. Finally, we benchmark the impact of LoRA to improve editing speed while maintaining edits quality.
Phoneme Discretized Saliency Maps for Explainable Detection of AI-Generated Voice
Authors: Shubham Gupta; Mirco Ravanelli; Pascal Germain; Cem Subakan
Venue: Interspeech 2024 (main)

Abstract: In this paper, we propose Phoneme Discretized Saliency Maps (PDSM), a discretization algorithm for saliency maps that takes advantage of phoneme boundaries for explainable detection of AI-generated voice. We experimentally show with two different Text-to-Speech systems (i.e., Tacotron2 and Fastspeech2) that the proposed algorithm produces saliency maps that result in more faithful explanations compared to standard posthoc explanation methods. Moreover, by associating the saliency maps to the phoneme representations, this methodology generates explanations that tend to be more understandable than standard saliency maps on magnitude spectrograms.
Exploring compressibility of transformer based text-to-music (TTM) models
Authors: Vasileios Moschopoulos; Thanasis Kotsiopoulos; Pablo Peso Parada; Konstantinos Nikiforidis; Alexandros Stergiadis; Gerasimos Papakostas; Md Asif Jalal; Jisi Zhang; Anastasios Drosou; Karthikeyan Saravanan
Venue: Interspeech 2024 (main)

Abstract: State-of-the art Text-To-Music (TTM) generative AI models are large and require desktop or server class compute, making them infeasible for deployment on mobile phones. This paper presents an analysis of trade-offs between model compression and generation performance of TTM models. We study compression through knowledge distillation and specific modifications that enable applicability over the various components of the TTM model (encoder, generative model and the decoder). Leveraging these methods we create TinyTTM (89.2M params) that achieves a FAD of 3.66 and KL of 1.32 on MusicBench dataset, better than MusicGen-Small (557.6M params) but not lower than MusicGen-small fine-tuned on MusicBench.
Sound of Vision: Audio Generation from Visual Text Embedding through Training Domain Discriminator
Authors: Jaewon Kim; Won-Gook Choi; Seyun Ahn; Joon-Hyuk Chang
Venue: Interspeech 2024 (main)

Abstract: Recent advancements in text-to-audio (TTA) models have demonstrated their ability to generate sound that aligns with user intentions. Despite this advancement, a notable limitation arises from the models' inability to effectively synthesize audio from visual-domain texts. In this study, we address this challenge by utilizing a novel dataset that pairs visual and acoustic-domain texts, derived using ChatGPT-3.5, and encoding switch through a domain discriminator. This approach ensures not only computational efficiency but also enhances the model's generalization, adaptability, and flexibility. It addresses concerns that training exclusively with visual texts might compromise audio generation quality from audio texts. This study presents a novel methodology for enhancing text-to-audio synthesis, demonstrating significant improvements in audio output fidelity from visual-text inputs.
Retrieval-Augmented Classifier Guidance for Audio Generation
Authors: Ho-Young Choi; Won-Gook Choi; Joon-Hyuk Chang
Venue: Interspeech 2024 (main)

Abstract: Most audio datasets utilized for training in the audio generation fields are low-quality, leading to difficulties in the generation of high-quality, single-event audio. However, to acquire single-event audio with noise-free, high costs are incurred. In this paper, we propose a simple retrieval-augmented classifier-guided sampling strategy for foley sound synthesis. Specifically, to guide the diffusion model during sampling with classifier guidance, given an input class, we first retrieve relevant audio features by utilizing a Contrastive Language-Audio Pretraining model. The gradients from a classifier for the retrieved audio features are then calculated to serve as additional guidance. Our evaluation, conducted on the DCASE 2023 challenge task 7 dataset, demonstrates that our proposed method overall improves a Frechet audio distance score.
Efficient Fine-tuning of Audio Spectrogram Transformers via Soft Mixture of Adapters
Authors: Umberto Cappellazzo; Daniele Falavigna; Alessio Brutti
Venue: Interspeech 2024 (main)

Abstract: Mixture of Experts (MoE) architectures have recently started burgeoning due to their ability to scale modelâs capacity while maintaining the computational cost affordable, leading to state-of-the-art results in numerous fields. While MoE has been mostly investigated for the pre-training stage, its use in parameter-efficient transfer learning (PETL) settings is underexplored. To narrow this gap, this paper attempts to demystify the use of MoE for PETL of Audio Spectrogram Transformers to audio and speech downstream tasks. Specifically, we propose Soft Mixture of Adapters (Soft-MoA). It exploits adapters as the experts and, leveraging the recent Soft MoE method, it relies on a soft assignment between the input tokens and experts to keep the computational time limited. Extensive experiments across 4 benchmarks demonstrate that Soft-MoA outperforms the single adapter method and performs on par with the dense MoA counterpart. We finally present ablation studies on key elements of Soft-MoA. Our code is available at https://github.com/umbertocappellazzo/PETL_AST.
PAM: Prompting Audio-Language Models for Audio Quality Assessment
Authors: Soham Deshmukh; Dareen Alharthi; Benjamin Elizalde; Hannes Gamper; Mahmoud Al Ismail; Rita Singh; Bhiksha Raj; Huaming Wang
Venue: Interspeech 2024 (main)

Abstract: Audio quality is a key performance metric for various audio processing tasks, including generative modeling, however its objective measurement remains a challenge. Audio-Language Models (ALM) are pre-trained on millions of audio-text pairs that may contain information about audio quality, the presence of artifacts or noise. Given an audio input and a text prompt about quality, an ALM can calculate a similarity score between the two. We exploit this capability and introduce PAM, a truly reference-free metric for assessing audio quality for different audio processing tasks. Contrary to other âreference-freeâ metrics, PAM does not require computing embeddings on a reference dataset nor training a task-specific model on a costly set of human listening scores. We extensively evaluate PAM against established metrics and newly collected human listening scores on four tasks: text- to-audio (TTA), text-to-music generation (TTM), text-to-speech (TTS), and deep noise suppression (DNS). We perform multiple ablation studies with controlled audio distortions, in-the-wild setups, and prompt choices. Our evaluation shows that overall, PAM correlates strongly with human listening scores and performs better than existing metrics. These results demonstrate the potential of ALM for computing a general-purpose audio quality metric. Code and human listening scores will be released.
GenDistiller: Distilling Pre-trained Language Models based on an Autoregressive Generative Model
Authors: Yingying Gao; Shilei Zhang; Chao Deng; Junlan Feng
Venue: Interspeech 2024 (main)

Abstract: Pre-trained speech language models such as HuBERT and WavLM leverage unlabeled speech data for self-supervised learning and offer powerful representations for numerous downstream tasks. Despite the success of these models, their high requirements for memory and computing resource hinder their application on resource restricted devices. Therefore, this paper introduces GenDistiller, a novel knowledge distillation framework which generates the hidden representations of the pretrained teacher model directly by a much smaller student network. The proposed method takes the previous hidden layer as history and implements a layer-by-layer prediction of the teacher model autoregressively. Experiments on SUPERB reveal the advantage of GenDistiller over the baseline distilling method without an autoregressive framework, with 33% fewer parameters, similar time consumption and better performance on most of the SUPERB tasks. Ultimately, the proposed GenDistiller reduces the size of WavLM by 82%.
Gender and Language Identification in Multilingual Models of Speech: Exploring the Genericity and Robustness of Speech Representations
Authors: SÃ©verine Guillaume; Maxime Fily; Alexis Michaud; Guillaume Wisniewski
Venue: Interspeech 2024 (main)

Abstract: Models such as XLS-R and UniSpeech have proven effective in speech processing across diverse languages, even with limited annotated data, enabling, for instance, the development of transcription systems for some under-documented languages. This work aims to test the hypothesis that these models can build âgenericâ representations of an audio snippet that do not depend on characteristics that are irrelevant to understanding the message conveyed. Through two sets of experiments, we assess their ability to abstract away from speaker-specific details and distill core informational contents â in an informational-communicational sense to be refined further: all the information contained in the audio signal that contributes evidence on the speaker's communicative intent. The results of our experiments show that pre-trained  models of speech such as XLS-R do not necessarily encode information in the same way, depending on the speaker's gender.
Neural Compression Augmentation for Contrastive Audio Representation Learning
Authors: Zhaoyu Wang; Haohe Liu; Harry Coppock; BjÃ¶rn Schuller; Mark D. Plumbley
Venue: Interspeech 2024 (main)

Abstract: The choice of data augmentation is pivotal in contrastive self-supervised learning. Current augmentation techniques for audio data, such as the widely used Random Resize Crop (RRC), underperform in pitch-sensitive music tasks and lack generalisation across various types of audio. This study aims to address these limitations by introducing Neural Compression Augmentation (NCA), an approach based on lossy neural compression. We use the Audio Barlow Twins (ABT), a contrastive self-supervised framework for audio, as our backbone. We experiment with both NCA and several baseline augmentation methods in the augmentation block of ABT and train the models on AudioSet. Experimental results show that models integrated with NCA considerably surpass the original performance of ABT, especially in the music tasks of the HEAR benchmark, demonstrating the effectiveness of compression-based augmentation for audio contrastive self-supervised learning.
Post-Net: A linguistically inspired sequence-dependent transformed neural architecture for automatic syllable stress detection
Authors: Sai Harshitha Aluru; Jhansi Mallela; Chiranjeevi Yarra
Venue: Interspeech 2024 (main)

Abstract: Automatic syllable stress detection methods typically consider syllable-level features as independent. However, as per linguistic studies, there is a dependency among the syllables within a word. In this work, we address this issue by proposing a Post-Net approach using Time-Delay Neural Networks to exploit the syllable dependency in a word for stress detection task. For this, we propose a loss function to incorporate the dependency by ensuring only one stressed syllable in a word. The proposed Post-Net leverages the existing SOTA sequence-independent stress detection models and learns in both supervised and unsupervised settings. We compare the Post-Net with three existing SOTA sequence-independent models and also with sequential model (LSTMs).  Experiments conducted on ISLE corpus show the highest relative accuracy improvement of  2.1% and 20.28%  with the proposed Post-Net compared to the best sequence-independent SOTA model in supervised and unsupervised manners, respectively.