GenAI4ELab/papercli-papers huggingface dataset:
Controlling Multi-Class Human Vocalization Generation via a Simple Segment-based Labeling Scheme
Authors: Hieu-Thi Luong; Junichi Yamagishi
Venue: Interspeech 2023 (main)

Abstract: As prompt-based generative models have received much attention, many studies have proposed a similar model for sound generation. While prompt-based generative models have an intuitive interface for non-professional users to experiment with, they lack the ability to control the generated sounds via a more direct means. In this work, we investigated the use of a simple segment-based labeling scheme for human vocalization generation, which is a specific subset of sound generation. By conditioning the generative models on the label sequence which marks the vocalization class of the segment, the generated sound can be controlled in a more detailed manner while maintaining a simple and intuitive input interface. Our experiments showed that simply switching the label scheme from global to segment-based does not degrade the quality of the generated samples in any way and provides a new method of controlling the generation process.
Vistaar: Diverse Benchmarks and Training Sets for Indian Language ASR
Authors: Kaushal Bhogale; Sai Sundaresan; Abhigyan Raman; Tahir Javed; Mitesh M. Khapra; Pratyush Kumar
Venue: Interspeech 2023 (main)

Abstract: Improving ASR systems is necessary to make new LLM-based use-cases accessible to people across the globe. In this paper, we focus on Indian languages, and make the case that diverse benchmarks are required to evaluate and improve ASR systems for Indian languages. To address this, we collate Vistaar as a set of 59 benchmarks across various language and domain combinations, on which we evaluate 3 publicly available ASR systems and 2 commercial systems. We also train IndicWhisper models by fine-tuning the Whisper models on publicly available training datasets across 12 Indian languages totalling to 10.7K hours. We show that IndicWhisper significantly improves on considered ASR systems on the Vistaar benchmark. Indeed, IndicWhisper has the lowest WER in 39 out of the 59 benchmarks, with an average reduction of 4.1 WER. We open-source all datasets, code and models - https://github.com/AI4Bharat/vistaar
Domain Adaptive Self-supervised Training of Automatic Speech Recognition
Authors: Cong-Thanh Do; Rama Doddipatla; Mohan Li; Thomas Hain
Venue: Interspeech 2023 (main)

Abstract: This paper explores domain adaptive self-supervised training of automatic speech recognition (ASR). Unlabeled data from the target domain can either be used in training the self-supervised pre-trained model or in the fine-tuning stage using semi-supervised approaches for the ASR task or both. Here we specifically focus on how semi-supervised approaches can enhance domain adaptation of pre-trained models built using self-supervised learning (SSL). For the purpose of this study, we use variants of English accents as the data from different domains. ASR experiments targeting single domain achieve relative word error rate (WER) reduction in the range 2.7-41.8% based on the extent of domain mismatch, while in the multiple-domain setting we achieve a relative WER reduction of 8% on average using semi-supervised fine-tuning on top of the model pre-trained with target domain using SSL.
There is more than one kind of robustness: Fooling Whisper with adversarial examples
Authors: Raphael Olivier; Bhiksha Raj
Venue: Interspeech 2023 (main)

Abstract: Whisper is a recent Automatic Speech Recognition (ASR) model displaying impressive robustness to both out-of-distribution inputs and random noise. In this work, we show that this robustness does not carry over to adversarial noise. We generate very small input perturbations with Signal Noise Ratio of up to 45dB, with which we can degrade Whisper performance dramatically, or even transcribe a target sentence of our choice. We also show that by fooling the Whisper language detector we can very easily degrade the performance of multilingual models. These vulnerabilities of a widely popular open-source model have practical security implications, and emphasize the need for adversarially robust ASR.
MT-SLVR: Multi-Task Self-Supervised Learning for Transformation In(Variant) Representations
Authors: Calum Heggan; Tim Hospedales; Sam Budgett; Mehrdad Yaghoobi
Venue: Interspeech 2023 (main)

Abstract: Contrastive self-supervised learning has gained attention for its ability to create high-quality representations from large unlabelled data sets. A key reason that these powerful features enable data-efficient learning of downstream tasks is that they provide augmentation invariance, which is often a useful inductive bias. However, the amount and type of invariances preferred is not known apriori, and varies across different downstream tasks. We therefore propose a multi-task self-supervised framework (MT-SLVR) that learns both variant and invariant features in a parameter-efficient manner. Our multi-task representation provides a strong and flexible feature that benefits diverse downstream tasks. We evaluate our approach on few-shot classification tasks drawn from a variety of audio domains and demonstrate improved classification performance on all of them.
Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Compute
Authors: William Chen; Xuankai Chang; Yifan Peng; Zhaoheng Ni; Soumi Maiti; Shinji Watanabe
Venue: Interspeech 2023 (main)

Abstract: Self-supervised learning (SSL) has led to great strides in speech processing. However, the resources needed to train these models has become prohibitively large as they continue to scale. Currently, only a few groups with substantial resources are capable of creating SSL models, which harms reproducibility. In this work, we optimize HuBERT SSL to fit in academic constraints. We reproduce HuBERT independently from the original implementation, with no performance loss. Our code and training optimizations make SSL feasible with only 8 GPUs, instead of the 32 used in the original work. We also explore a semi-supervised route, using an ASR model to skip the first pre-training iteration. Within one iteration of pre-training, our models improve over HuBERT on several tasks. Furthermore, our HuBERT Large variant requires only 8 GPUs, achieving similar performance to the original trained on 128. As our contribution to the community, all models, configurations, and code are made opensource in ESPnet.
Blank-regularized CTC for Frame Skipping in Neural Transducer
Authors: Yifan Yang; Xiaoyu Yang; Liyong Guo; Zengwei Yao; Wei Kang; Fangjun Kuang; Long Lin; Xie Chen; Daniel Povey
Venue: Interspeech 2023 (main)

Abstract: Neural Transducer and connectionist temporal classification (CTC) are popular end-to-end automatic speech recognition systems. Due to their frame-synchronous design, blank symbols are introduced to address the length mismatch between acoustic frames and output tokens, which might bring redundant computation. Previous studies managed to accelerate the training and inference of neural Transducers by discarding frames based on the blank symbols predicted by a co-trained CTC. However, there is no guarantee that the co-trained CTC can maximize the ratio of blank symbols. This paper proposes two novel regularization methods to explicitly encourage more blanks by constraining the self-loop of non-blank symbols in the CTC. It is interesting to find that the frame reduction ratio of the neural Transducer can approach the theoretical boundary. Experiments on LibriSpeech corpus show that our proposed method accelerates the inference of neural Transducer by 4 times without sacrificing performance.
The Tag-Team Approach: Leveraging CLS and Language Tagging for Enhancing Multilingual ASR
Authors: Kaousheik Jayakumar; Vrunda N. Sukhadia; A Arunkumar; S Umesh
Venue: Interspeech 2023 (main)

Abstract: Building a multilingual Automated Speech Recognition (ASR) system in a linguistically diverse country like India can be a challenging task due to the differences in scripts and the limited availability of speech data. This problem can be solved by exploiting the fact that many of these languages are phonetically similar. These languages can be converted into a Common Label Set (CLS) by mapping similar sounds to common labels. In this paper, new approaches are explored and compared to improve the performance of CLS based multilingual ASR model. Specific language information is infused in the ASR model by giving Language ID or using CLS to Native script converter on top of the CLS Multilingual model. These methods give a significant improvement in Word Error Rate (WER) compared to the CLS baseline. These methods are further tried on out-of-distribution data to check their robustness.
Improving RNN-Transducers with Acoustic LookAhead
Authors: Vinit S. Unni; Ashish Mittal; Preethi Jyothi; Sunita Sarawagi
Venue: Interspeech 2023 (main)

Abstract: RNN-Transducers (RNN-Ts) have gained widespread acceptance as an end-to-end model for speech to text conversion because of their high accuracy and streaming capabilities. A typical RNN-T independently encodes the input audio and the text context, and combines the two encodings by a thin joint network. While this architecture provides SOTA streaming accuracy, it also makes the model vulnerable to strong LM biasing which manifests as multi-step hallucination of text without acoustic evidence. In this paper we propose LOOKAHEAD that makes text representations more acoustically grounded by looking ahead into the future within the audio input. This technique yields a significant 5% - 20% relative reduction in word error rate on both in-domain and out-of-domain evaluation sets.
Everyone has an accent
Authors: Nina Markl; Catherine Lai
Venue: Interspeech 2023 (main)

Abstract: In this paper, we consider how the notion "accent" in particular in the context of "accented speech" has been discussed in Interspeech publications between 2004 and 2022. We contrast the way speech technology research published in the conference has conceptualised these terms with their usage in linguistics. The point of this comparison is to: highlight significant inter-disciplinary differences in the way apparently core terms are used, discuss disadvantages of using inexact language in research, and encourage researchers to be more mindful about the use of particular short-hands.
Some Voices are Too Common: Building Fair Speech Recognition Systems Using the CommonVoice Dataset
Authors: Lucas Maison; Yannick EstÃ¨ve
Venue: Interspeech 2023 (main)

Abstract: Automatic speech recognition (ASR) systems become increasingly efficient thanks to new advances in neural network training like self-supervised learning. However, they are known to be unfair toward certain groups, for instance, people speaking with an accent. In this work, we use the French Common Voice dataset to quantify the biases of a pre-trained wav2vec 2.0 model toward several demographic groups. By fine-tuning the pre-trained model on a variety of fixed-size, carefully crafted training sets, we demonstrate the importance of speaker diversity. We also run an in-depth analysis of the Common Voice corpus and identify important shortcomings that should be taken into account by users of this dataset.
Information Magnitude Based Dynamic Sub-sampling for Speech-to-text
Authors: Yuhao Zhang; Chenghao Gao; Kaiqi Kou; Chen Xu; Tong Xiao; Jingbo Zhu
Venue: Interspeech 2023 (main)

Abstract: Attention-based models have achieved new state-of-the-art in many tasks while the computational cost of these models increases drastically compared with previous methods. For most acoustic tasks, excessively long speech sequences exacerbate this problem and do not benefit a lot from the attention mechanism. We propose the information magnitude (IM) based dynamic stride convolution (IM-DSC) method. This method first calculates the IM according to the importance of each frame, then dynamically squeezes the redundant frames. We carry on experiments on speech translation and automatic speech recognition tasks. Our results show that we achieve 0.5 BLEU and 0.4 BLEU improvements on the MuST-C En-De and En-Fr data with a 22% compression ratio. For the ASR task, we gain a 0.2 WER reduction with a 21% compression ratio on the Librispeech data.
Whatâs in a Rise? The Relevance of Intonation for Attention Orienting
Authors: Martine Grice
Venue: Interspeech 2023 (main)

Abstract: In this talk I will explore why and how intonational rises are used to orient attention towards the words and phrases bearing them. The attention orienting function of rising pitch is known outside the linguistic domain, with evidence from auditory looming, a phenomenon whereby a signal that increases in loudness or pitch appears to be approaching the listener and is perceived as an immediate threat.This attention orienting function extends to speech communication, where rises in pitch are crucial for directing listenersâ attention to the most important parts of the linguistic message. I will provide evidence from event related brain potentials that such rises affect both preattentive and conscious attention. Moreover, the lack of a rise can, in some situations, direct attention away from parts of the message, leading to information being missed. I will also discuss the influence of intonational rises on short-term memory, showing that rises can boost recall of items in a list. This effect can be local to a particular item if the rise is accentual, or more global if the rise is at the edge of a domain. However, despite the cross-linguistic effect of rises on attention, their influence can be impacted by language specific prosodic structure and linguistic expectations.
HierVST: Hierarchical Adaptive Zero-shot Voice Style Transfer
Authors: Sang-Hoon Lee; Ha-Yeong Choi; Hyung-Seok Oh; Seong-Whan Lee
Venue: Interspeech 2023 (main)

Abstract: Despite rapid progress in the voice style transfer (VST) field, recent zero-shot VST systems still lack the ability to transfer the voice style of a novel speaker. In this paper, we present HierVST, a hierarchical adaptive end-to-end zero-shot VST model. Without any text transcripts, we only use the speech dataset to train the model by utilizing hierarchical variational inference and self-supervised representation. In addition, we adopt a hierarchical adaptive generator that generates the pitch representation and waveform audio sequentially. Moreover, we utilize unconditional generation to improve the speaker-relative acoustic capacity in the acoustic representation. With a hierarchical adaptive structure, the model can adapt to a novel voice style and convert speech progressively. The experimental results demonstrate that our method outperforms other VST models in zero-shot VST scenarios. Audio samples are available at https://hiervst.github.io/.
VISinger2: High-Fidelity End-to-End Singing Voice Synthesis Enhanced by Digital Signal Processing Synthesizer
Authors: Yongmao Zhang; Heyang Xue; Hanzhao Li; Lei Xie; Tingwei Guo; Ruixiong Zhang; Caixia Gong
Venue: Interspeech 2023 (main)

Abstract: End-to-end singing voice synthesis (SVS) model VISinger can achieve better performance than the typical two-stage model with fewer parameters. However, VISinger has several problems: text-to-phase problem, the end-to-end model learns the meaningless mapping of text-to-phase; glitches problem, the harmonic components corresponding to the periodic signal of the voiced segment occurs a sudden change with audible artefacts; low sampling rate, the sampling rate of 24KHz does not meet the application needs of high-fidelity generation with the full-band rate (44.1KHz or higher). In this paper, we propose VISinger 2 to address these issues by integrating the digital signal processing (DSP) methods with VISinger. Specifically, inspired by recent advances in differentiable digital signal processing (DDSP), we incorporate a DSP synthesizer into the decoder to solve the above issues. The DSP synthesizer consists of a harmonic synthesizer and a noise synthesizer to generate periodic and aperiodic signals, respectively, from the latent representation z in VISinger. It supervises the posterior encoder to extract the latent representation without phase information and avoid the prior encoder modelling text-to-phase mapping. To avoid glitch artefacts, the HiFiGAN is modified to accept the waveforms generated by the DSP synthesizer as a condition to produce the singing voice. Moreover, with the improved waveform decoder, VISinger 2 manages to generate 44.1kHz singing audio with richer expression and better quality. Experiments on OpenCpop corpus show that VISinger 2 outperforms VISinger, CpopSing and RefineSinger in both subjective and objective metrics. Our audio samples and source code are available.
EdenTTS: A Simple and Efficient Parallel Text-to-speech Architecture with Collaborative Duration-alignment Learning
Authors: Youneng Ma; Junyi He; Meimei Wu; Guangyue Hu; Haojun Fei
Venue: Interspeech 2023 (main)

Abstract: In pursuit of high inference speed, many non-autoregressive neural text-to-speech (TTS) models have been proposed for parallel speech synthesis recently. A critical challenge of parallel speech generation lies in the learning of text-speech alignment. Existing methods usually require an external aligner for guidance or involve complex training process. In this work, we propose Eden-TTS, a simple and efficient parallel TTS architecture which jointly learns duration prediction, text-speech alignment and speech generation in a single fully-differentiable model. The alignment is learned implicitly in our architecture. A novel energy-modulated attention mechanism is proposed for alignment guidance which leads to fast and stable convergence of our model. Our model can be easily implemented and trained. Experiments demonstrate that our method can generate speech of high quality with high training efficiency.
Generalizable Zero-Shot Speaker Adaptive Speech Synthesis with Disentangled Representations
Authors: Wenbin Wang; Yang Song; Sanjay Jha
Venue: Interspeech 2023 (main)

Abstract: While most research into speech synthesis has focused on synthesizing high-quality speech for in-dataset speakers, an equally essential yet unsolved problem is synthesizing speech for unseen speakers who are out-of-dataset with limited reference data, i.e., speaker adaptive speech synthesis. Many studies have proposed zero-shot speaker adaptive text-to-speech and voice conversion approaches aimed at this task. However, most current approaches suffer from the degradation of naturalness and speaker similarity when synthesizing speech for unseen speakers (i.e., speakers not in the training dataset) due to the poor generalizability of the model in out-of-distribution data. To address this problem, we propose GZS-TV, a generalizable zero-shot speaker adaptive text-to-speech and voice conversion model. GZS-TV introduces disentangled representation learning for both speaker embedding extraction and timbre transformation to improve model generalization and leverages the representation learning capability of the variational autoencoder to enhance the speaker encoder. Our experiments demonstrate that GZS-TV reduces performance degradation on unseen speakers and outperforms all baseline models in multiple datasets. The audio samples, detailed network architecture and pre-trained model are available at https://gzs-tv.github.io/.
Speech inpainting: Context-based speech synthesis guided by video
Authors: Juan Felipe Montesinos; Daniel Michelsanti; Gloria Haro; Zheng-Hua Tan; Jesper Jensen
Venue: Interspeech 2023 (main)

Abstract: Audio and visual modalities are inherently connected in speech signals: lip movements and facial expressions are correlated with speech sounds. This motivates studies that incorporate the visual modality to enhance an acoustic speech signal or even restore missing audio information. Specifically, this paper focuses on the problem of audio-visual speech inpainting, which is the task of synthesizing the speech in a corrupted audio segment in a way that it is consistent with the corresponding visual content and the uncorrupted audio context. We present an audio-visual transformer-based deep learning model that leverages visual cues that provide information about the content of the corrupted audio. It outperforms the previous state-of-the-art audio-visual model and audio-only baselines. We also show how visual features extracted with AV-HuBERT, a large audio-visual transformer for speech recognition, are suitable for synthesizing speech.
STEN-TTS: Improving Zero-shot Cross-Lingual Transfer for Multi-Lingual TTS with Style-Enhanced Normalization Diffusion Framework
Authors: Chung Tran; Chi Mai Luong; Sakriani Sakti
Venue: Interspeech 2023 (main)

Abstract: The prevalence of personalized multilingual tools plays an important role in learning aids and virtual assistants. The existing works on multilingual adaptive text-to-speech (TTS) mainly focus on fine-tuning models or extracting personal styles, such as prosody, emotion, and identity, with the aim of adapting to new speakers. This paper introduces the Style-Enhanced Normalization TTS (STEN-TTS) approach to synthesizing multilingual voice and maintaining personal styles with only 3 seconds of input reference. By presenting an integrated module (STEN) into the diffusion model, the proposed method can simulate the speaker's style and eliminate white noise in the synthesized speech. The experimental results show that our model achieves good performance, at above 3.5 on SMOS for cross-lingual switching. Furthermore, when using speaker verification to assess the similarity between the ground truth and synthesized voices, the accuracy reaches 82.4% with 3 seconds of audio reference.
Average Token Delay: A Latency Metric for Simultaneous Translation
Authors: Yasumasa Kano; Katsuhito Sudoh; Satoshi Nakamura
Venue: Interspeech 2023 (main)

Abstract: In simultaneous translation, translation begins before the speaker has finished speaking. In its evaluation, we have to consider the latency of the translation in addition to the quality, with latency preferably as small as possible for users to comprehend what the speaker says with a small delay. Existing latency metrics focus on when the translation starts but do not consider adequately when the translation ends. This means such metrics do not penalize the latency caused by a long translation output, which delays user comprehension. In this work, we propose a novel latency evaluation metric called Average Token Delay (ATD) that focuses on the end timings of partial translations in simultaneous translation. We discuss the advantage of ATD using simulated examples and investigate the differences between ATD and Average Lagging with simultaneous translation experiments.
Automatic Speech Recognition Transformer with Global Contextual Information Decoder
Authors: Yukun Qian; Xuyi Zhuang; Mingjiang Wang
Venue: Interspeech 2023 (main)

Abstract: Most current automatic speech recognition (ASR) models use decoders that do not have access to global contextual information at the token level. Therefore, we propose a decoder structure with text-level global contextual information. We construct the global information encoder based on non-autoregressive recognition. To eliminate the non-autoregressive independence assumption, we add a self-attention layer with rotary position encoding. The obtained text-level global contextual information and the decoder are fused as cross-attention to construct a decoder with contextual information. Our model can achieve a character error rate of 3.92% on the AISHELL-1 validation set and 4.35% on the test set, reducing the error rate by 1.72%(dev)/2.13%(test) compared to the baseline model, achieving SOTA performance. Finally, we also use visualization techniques to explain the role of global information in the decoder.