GenAI4ELab/papercli-papers huggingface dataset:
Wav2ToBI: a new approach to automatic ToBI transcription
Authors: Wanyue Zhai; Mark Hasegawa-Johnson
Venue: Interspeech 2023 (main)

Abstract: ToBI is a prosody labeling system that transcribes American English prosody in terms of phonological tones and break indices. Previous works on automatic ToBI transcription require additional information such as word boundaries and use modular feature extraction with separately optimized feature detectors and classifiers. We are interested in investigating if a neural network-based approach would also result in high performance on automatic ToBI transcription without additional information. In this paper, we investigate the problem of pitch accent detection and prosody boundary detection using the Wav2vec 2.0 model with only acoustic information. Our model is trained on the Boston University Radio News Corpus and evaluated on both the Boston University Radio News Corpus and the Boston Directions Corpus. We show that it achieves an F1 score of 0.82 on pitch accent detection and 0.86 on phrase boundary detection. Code and model weights are available.
Joint-Former: Jointly Regularized and Locally Down-sampled Conformer for Semi-supervised Sound Event Detection
Authors: Lijian Gao; Qirong Mao; Ming Dong
Venue: Interspeech 2023 (main)

Abstract: Semi-supervised Sound Event Detection (SSED) is to recognize the categories of events and mark their onset and offset times using a small amount of weakly-labeled and a large-scale of unlabeled data. To exploit unlabeled data effectively and reduce over-fitting, regularization techniques play a critical role in SSED. In this paper, we proposed a novel jointly regularized and locally down-sampled Conformer (Joint-Former) model for SSED. Joint-Former first locally down-samples the spectrogram and learns the token representations with high temporal resolution and low computational cost. Then, Joint-Former effectively exploits unlabelled data in SSED by integrating Mean-Teacher and Masked Spectrogram Modeling using joint regularization through a multitask learning framework. Extensive experiments on DCASE 2019, DCASE 2020, and DCASE 2021 task4 SSED datasets show that Joint-Former greatly outperformed existing methods.
Towards Attention-based Contrastive Learning for Audio Spoof Detection
Authors: Chirag Goel; Surya Koppisetti; Ben Colman; Ali Shahriyari; Gaurav Bharaj
Venue: Interspeech 2023 (main)

Abstract: Vision transformers (ViT) have made substantial progress for classification tasks in computer vision. Recently, Gong et. al. '21, introduced attention-based modeling for several audio tasks. However, relatively unexplored is the use of a ViT for audio spoof detection task. We bridge this gap and introduce ViTs for this task. A vanilla baseline built on fine-tuning the SSAST (Gong et. al. '22) audio ViT model achieves sub-optimal equal error rates (EERs). To improve performance, we propose a novel attention-based contrastive learning framework (SSAST-CL) that uses cross-attention to aid the representation learning. Experiments show that our framework successfully disentangles the bonafide and spoof classes and helps learn better classifiers for the task. With appropriate data augmentations policy, a model trained on our framework achieves competitive performance on the ASVSpoof 2021 challenge. We provide comparisons and ablation studies to justify our claim.
Masked Audio Modeling with CLAP and Multi-Objective Learning
Authors: Yifei Xin; Xiulian Peng; Yan Lu
Venue: Interspeech 2023 (main)

Abstract: Most existing masked audio modeling (MAM) methods learn audio representations by masking and reconstructing local spectrogram patches. However, the reconstruction loss mainly accounts for the signal-level quality of the reconstructed spectrogram and is still limited in extracting high-level audio semantics. In this paper, we propose to enhance the semantic modeling of MAM by distilling cross-modality knowledge from contrastive language-audio pretraining (CLAP) representations for both masked and unmasked regions (MAM-CLAP) and leveraging a multi-objective learning strategy with a supervised classification branch (SupMAM), thereby providing more semantic knowledge for MAM and enabling it to effectively learn global features from labels. Experiments show that our methods significantly improve the performance on multiple downstream tasks. Furthermore, by combining our MAM-CLAP with SupMAM, we can achieve new state-of-the-art on various audio and speech classification tasks, exceeding previous self-supervised and supervised pretraining methods.
Few-Shot Open-Set Learning for On-Device Customization of KeyWord Spotting Systems
Authors: Manuele Rusci; Tinne Tuytelaars
Venue: Interspeech 2023 (main)

Abstract: A personalized KeyWord Spotting (KWS) pipeline typically requires the training of a Deep Learning model on a large set of user-defined speech utterances, preventing fast customization directly applied on-device. To fill this gap, this paper investigates few-shot learning methods for open-set KWS classification by combining a deep feature encoder with a prototype-based classifier. With user-defined keywords from 10 classes of the Google Speech Command dataset, our study reports an accuracy of up to 76% in a 10-shot scenario while the false acceptance rate of unknown data is kept to 5%. In the analyzed settings, the usage of the triplet loss to train an encoder with normalized output features performs better than the prototypical networks jointly trained with a generator of dummy unknown-class prototypes. This design is also more effective than encoders trained on a classification problem and features fewer parameters than other iso-accuracy approaches.
Self-Supervised Dataset Pruning for Efficient Training in Audio Anti-spoofing
Authors: Abdul Hameed Azeemi; Ihsan Ayyub Qazi; Agha Ali Raza
Venue: Interspeech 2023 (main)

Abstract: The computational cost for training neural anti-spoofing models has rapidly increased due to larger network architectures. Several dataset-pruning metrics have been proposed to increase the training efficiency of these models. However, these metrics require example labels and an initial training step to compute example scores which is computationally intensive. We propose a novel self-supervised pruning metric for efficient dataset pruning in neural anti-spoofing models. Our method identifies important examples and prunes the dataset in an efficient, self-supervised manner using the clustered embedding representation of audios. We demonstrate that our method exceeds the performance of four other pruning metrics on the ASVSpoof 2019 dataset across two anti-spoofing models while being 91% computationally more efficient. We also find differences in the distribution of certain attacks, which helps explain the better performance of self-supervised pruning over other metrics.
Semantic Segmentation with Bidirectional Language Models Improves Long-form ASR
Authors: W. Ronny Huang; Hao Zhang; Shankar Kumar; Shuo-Yiin Chang; Tara Sainath
Venue: Interspeech 2023 (main)

Abstract: We propose a method of segmenting long-form speech by separating semantically complete sentences within the utterance. This prevents the ASR decoder from needlessly processing faraway context while also preventing it from missing relevant context within the current sentence. Semantically complete sentence boundaries are typically demarcated by punctuation in written text; but unfortunately, spoken real-world utterances rarely contain punctuation. We address this limitation by distilling punctuation knowledge from a bidirectional teacher language model (LM) trained on written, punctuated text. We compare our segmenter, which is distilled from the LM teacher, against a segmenter distilled from a acoustic-pause-based teacher used in other works, on a streaming ASR pipeline. The pipeline with our segmenter achieves a 3.2% relative WER gain along with a 60 ms median end-of-segment latency reduction on a YouTube captioning task.
Multi-microphone Automatic Speech Segmentation in Meetings Based on Circular Harmonics Features
Authors: ThÃ©o Mariotte; Anthony Larcher; Silvio MontrÃ©sor; Jean-Hugh Thomas
Venue: Interspeech 2023 (main)

Abstract: Speaker diarization is the task of answering Who spoke and when? in an audio stream. Pipeline systems rely on speech segmentation to extract speakers' segments and achieve robust speaker diarization. This paper proposes a common framework to solve three segmentation tasks in the distant speech scenario: Voice Activity Detection (VAD), Overlapped Speech Detection (OSD), and Speaker Change Detection (SCD). In the literature, a few studies investigate the multi-microphone distant speech scenario. In this work, we propose a new set of spatial features based on direction-of-arrival estimations in the circular harmonic domain (CH-DOA). These spatial features are extracted from multi-microphone audio data and combined with standard acoustic features. Experiments on the AMI meeting corpus show that CH-DOA can improve the segmentation while being robust in case of deactivated microphones.
Advanced RawNet2 with Attention-based Channel Masking for Synthetic Speech Detection
Authors: Jing Li; Yanhua Long; Yijie Li; Dongxing Xu
Venue: Interspeech 2023 (main)

Abstract: Automatic speaker verification (ASV) systems are often vulnerable to spoofing attacks, particularly unseen attacks. Due to the diversity of text-to-speech and voice conversion algorithms, how to improve the generalization ability of synthetic speech detection systems is a challenging issue. To address this issue, we propose an advanced RawNet2 (ARawNet2) by introducing an attention-based channel masking (ACM) block to improve the RawNet2, with three main components: the squeeze-and-excitation, the channel masking, and a global-local feature aggregation. The effectiveness of the proposed system is evaluated on both the ASVspoof 2019 and ASVspoof 2021 datasets. Specifically, the ARawNet2 achieves an EER of 4.61% on the ASVspoof 2019 logical access (LA) task, and on the ASVspoof 2021 LA and speech deepfake (DF) tasks, it achieves EER of 8.36% and 19.03%, which obtains relative 12.00% and 14.97% EER reductions over the RawNet2 baseline, respectively.
Insights into end-to-end audio-to-score transcription with real recordings: A case study with saxophone works
Authors: Juan Carlos MartÃ­nez-Sevilla; MarÃ­a Alfaro-Contreras; Jose J. Valero-Mas; Jorge Calvo-Zaragoza
Venue: Interspeech 2023 (main)

Abstract: Neural end-to-end Audio-to-Score (A2S) transcription aims to retrieve a score that encodes the music content of an audio recording in a single step. Due to the recentness of this formulation, the existing works have exclusively addressed controlled scenarios with synthetic data that fail to provide conclusions applicable to real-world cases. In response to this gap in the literature, this work introduces a novel assortment of real saxophone recordings---together with their digital scores---and poses several experimental scenarios involving real and synthetic data. The obtained results confirm the adequacy of this A2S framework to deal with real data as well as proving the relevance of leveraging synthetic interpretations to improve the recognition rate in scenarios with real-data scarcity.
Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers
Authors: Yuan Gong; Sameer Khurana; Leonid Karlinsky; James Glass
Venue: Interspeech 2023 (main)

Abstract: In this paper, we focus on Whisper, a recent automatic speech recognition model trained with a massive 680k hour labeled speech corpus recorded in diverse conditions. We first show an interesting finding that while Whisper is very robust against real-world background sounds (e.g., music), its audio representation is actually not noise-invariant, but is instead highly correlated to non-speech sounds, indicating that Whisper recognizes speech conditioned on the noise type. With this finding, we build a unified audio tagging and speech recognition model Whisper-AT by freezing the backbone of Whisper, and training a lightweight audio tagging model on top of it. With <1% extra computational cost, Whisper-AT can recognize audio events, in addition to spoken text, in a single forward pass.
Synthetic Voice Spoofing Detection based on Feature Pyramid Conformer
Authors: Jingran Gong; Ning Chen
Venue: Interspeech 2023 (main)

Abstract: In speech anti-spoofing, artefacts used to detect spoofed speech are often located in specific sub-bands. Previous works often use Convolution Neural Networks (CNNs) as backbone which are good at capturing local features. However, if artefacts simultaneously exist in different sub-bands, CNNs cannot model this kind of information. Thus, we propose to use Feature Pyramid Conformer to solve this issue. Conformer can capture both local and global features. We aggregate the outputs of each Conformer block with Feature Pyramid Module. Through addition and lateral connection, the aggregation can be better integrated. Besides, to improve generalization of detecting unknown attacks, we propose to adopt Elastic penalty Margin Softmax. It can enhance intra-class compactness and inter-class discrepancy flexibly. Without data augmentaion, our system achieve an Equal Error Rate (EER) of 1.65% on the evaluation set of ASVspooof 2019 logical access, outperforming most existing systems.
Learning A Self-Supervised Domain-Invariant Feature Representation for Generalized Audio Deepfake Detection
Authors: Yuankun Xie; Haonan Cheng; Yutian Wang; Long Ye
Venue: Interspeech 2023 (main)

Abstract: In recent years, Audio Deepfake Detection (ADD) models have shown promising results in intra-domain. However, they do not perform well in cross-domain scenarios. This is mainly due to the limited variety of domain types and attack methods in training data, as well as insufficient research on hidden feature representation. To address these issues, we present W2VASDG, a generalized ADD system including a self-supervised representation front-end and a domain generalization backbone. Furthermore, we try to learn an ideal feature space which aggregates real speech and separates fake speech. Fake speech varies significantly by different forgery methods, while real speech varies less. In light of this, we further propose the aggregation and separation domain generalization (ASDG) method as the back-end to learn a domain invariant feature representation. Experiments show that our W2V-ASDG outperforms baseline models in cross-domains and gets the lowest average equal error rates (EER) of 4.60%.
Application of Knowledge Distillation to Multi-Task Speech Representation Learning
Authors: Mine Kerpicci; Van Nguyen; Shuhua Zhang; Erik Visser
Venue: Interspeech 2023 (main)

Abstract: Model architectures such as wav2vec 2.0 and HuBERT have been proposed to learn speech representations from audio waveforms in a self-supervised manner. When they are combined with downstream tasks such as keyword spotting and speaker verification, they provide state-of-the-art performance. However, these models use a large number of parameters, the smallest version of which has 95 million parameters. This constitutes a challenge for edge AI device deployments. In this paper, we investigate the application of knowledge distillation to speech representation learning (SRL) models followed by joint fine-tuning with multiple downstream voice-activated tasks. In our experiments on two such tasks, our approach results in nearly 75% reduction in model size while suffering only 0.1% accuracy and 0.9% equal error rate degradation compared to the full-size model. In addition, we show that fine-tuning the SRL models results in a significant performance boost compared to using frozen SRL models.
DeCoR: Defy Knowledge Forgetting by Predicting Earlier Audio Codes
Authors: Xilin Jiang; Yinghao Aaron Li; Nima Mesgarani
Venue: Interspeech 2023 (main)

Abstract: Lifelong audio feature extraction involves learning new sound classes incrementally, which is essential for adapting to new data distributions over time. However, optimizing the model only on new data can lead to catastrophic forgetting of previously learned tasks, which undermines the model's ability to perform well over the long term. This paper introduces a new approach to continual audio representation learning called DeCoR. Unlike other methods that store previous data, features, or models, DeCoR indirectly distills knowledge from an earlier model to the latest by predicting quantization indices from a delayed codebook. We demonstrate that DeCoR improves acoustic scene classification accuracy and integrates well with continual self-supervised representation learning. Our approach introduces minimal storage and computation overhead, making it a lightweight and efficient solution for continual learning.
Variational Classifier for Unsupervised Anomalous Sound Detection under Domain Generalization
Authors: Antonio AlmudÃ©var; Alfonso Ortega; Luis Vicente; Antonio Miguel; Eduardo Lleida
Venue: Interspeech 2023 (main)

Abstract: Unsupervised anomalous sound detection typically involves using a classifier with the last layer removed to extract embeddings. After that the cosine distance between train and test embeddings as anomaly score is used. In this paper, we propose a new idea which we call variational classifier that force the embeddings to follow a distribution imposed by design that can depend on the class of the input among other factors. To achieve this goal, in addition to the cross-entropy, we add to the loss function the KL divergence between these distributions and the one followed by the training embeddings. This enhances the ability of the system to differentiate between classes and it allows us to use sampling methods and to calculate the log-likelihood of a test embedding in the train embeddings distributions. We tested this proposal on the DCASE 2022 Task 2 dataset and observed improvements in both classification and unsupervised anomaly detection, which is the primary task.
FlexiAST: Flexibility is What AST Needs
Authors: Jiu Feng; Mehmet Hamza Erol; Joon Son Chung; Arda Senocak
Venue: Interspeech 2023 (main)

Abstract: The objective of this work is to give patch-size flexibility to Audio Spectrogram Transformers (AST). Recent advancements in ASTs have shown superior performance in various audio-based tasks. However, the performance of standard ASTs drastically drops when evaluated at different patch sizes than they were trained at. As a result, AST models are typically retrained to accommodate changes in patch sizes. To overcome this limitation, this paper demonstrates a training procedure to provide flexibility to standard AST models without architectural changes, allowing them to work with various patch sizes at the inference stage - FlexiAST. This proposed training approach simply utilizes random patch size selection and resizing of patch and positional embedding weights. Our experiments show that FlexiAST gives similar performance to standard AST models while maintaining its evaluation ability at various patch sizes on different datasets for audio classification tasks.
MCR-Data2vec 2.0: Improving Self-supervised Speech Pre-training via Model-level Consistency Regularization
Authors: Ji Won Yoon; Seok Min Kim; Nam Soo Kim
Venue: Interspeech 2023 (main)

Abstract: Self-supervised learning (SSL) has shown significant progress in speech processing tasks. However, despite the intrinsic randomness in the Transformer structure, such as dropout variants and layer-drop, improving the model-level consistency remains under-explored in the speech SSL literature. To address this, we propose a new pre-training method that uses consistency regularization to improve Data2vec 2.0, the recent state-of-the-art (SOTA) SSL model. Specifically, the proposed method involves sampling two different student sub-models within the Data2vec 2.0 framework, enabling two output variants derived from a single input without additional parameters. Subsequently, we regularize the outputs from the student sub-models to be consistent and require them to predict the representation of the teacher model. Our experimental results demonstrate that the proposed approach improves the SSL model's robustness and generalization ability, resulting in SOTA results on the SUPERB benchmark.
Visually-Aware Audio Captioning With Adaptive Audio-Visual Attention
Authors: Xubo Liu; Qiushi Huang; Xinhao Mei; Haohe Liu; Qiuqiang Kong; Jianyuan Sun; Shengchen Li; Tom Ko; Yu Zhang; Lilian H. Tang; Mark D. Plumbley; Volkan KÄ±lÄ±Ã§; Wenwu Wang
Venue: Interspeech 2023 (main)

Abstract: Audio captioning aims to generate text descriptions of audio clips. In the real world, many objects produce similar sounds. How to accurately recognize ambiguous sounds is a major challenge for audio captioning. In this work, inspired by inherent human multimodal perception, we propose visually-aware audio captioning, which makes use of visual information to help the description of ambiguous sounding objects. Specifically, we introduce an off-the-shelf visual encoder to extract video features and incorporate the visual features into an audio captioning system. Furthermore, to better exploit complementary audio-visual contexts, we propose an audio-visual attention mechanism that adaptively integrates audio and visual context and removes the redundant information in the latent space. Experimental results on AudioCaps, the largest audio captioning dataset, show that our proposed method achieves state-of-the-art results on machine translation metrics.
Masking Kernel for Learning Energy-Efficient Representations for Speaker Recognition and Mobile Health
Authors: Apiwat Ditthapron; Emmanuel O. Agu; Adam C. Lammert
Venue: Interspeech 2023 (main)

Abstract: Modern smartphones possess hardware for audio acquisition and to perform speech processing tasks such as speaker recognition and health assessment. However, energy consumption remains a concern, especially for resource-intensive DNNs. Prior work has improved the DNN energy efficiency by utilizing a compact model or reducing the dimensions of speech features. Both approaches reduced energy consumption during DNN inference but not during speech acquisition. This paper proposes using a masking kernel integrated into gradient descent during DNN training to learn the most energy-efficient speech length and sampling rate for windowing, a common step for sample construction. To determine the most energy-optimal parameters, a masking function with non-zero derivatives was combined with a low-pass filter. The proposed approach minimizes the energy consumption of both data collection and inference by 57%, and is competitive with speaker recognition and traumatic brain injury detection baselines.
eSTImate: A Real-time Speech Transmission Index Estimator With Speech Enhancement Auxiliary Task Using Self-Attention Feature Pyramid Network
Authors: Bajian Xiang; Hongkun Liu; Zedong Wu; Su Shen; Xiangdong Zhang
Venue: Interspeech 2023 (main)

Abstract: The Speech Transmission Index (STI) is a crucial metric for evaluating speech intelligibility, but its standard measurement method is too complicated for real-time applications. Though recently proposed deep learning based STI estimation schemes can effectively address the problem, existing methods still fall short of covering all possible STI scenarios. This paper presents eSTImate: an end-to-end deep learning system for real-time STI blind estimation that integrates the tasks of STI estimation and speech enhancement through a feature pyramid auxiliary learning architecture and incorporates multi-head attention mechanisms. The proposed model demonstrates the performance of state-of-the-art, achieving a low mean absolute error of 0.016 and root mean square error of 0.021 on the constructed dataset that covers the whole range of STI, highlighting its potential to provide accurate and consistent real-time STI estimation across diverse real-world scenarios.