GenAI4ELab/papercli-papers huggingface dataset:
J-ToneNet: A Transformer-based Encoding Network for Improving Tone Classification in Continuous Speech via F0 Sequences
Authors: Yi-Fen Liu; Xiang-Li Lu
Venue: Interspeech 2023 (main)

Abstract: Currently, tone classification studies mainly focus on training classifiers by using intrinsic features of isolated segments, i.e. often the syllables. Mostly, the works are not merely in use of fundamental frequency (f0) but utilizing more information on the spectrograms, MFCCs, or energy to improve model accuracy. However, as we know, more challenges on tone classification lie on modeling the complex f0 variations from the tonal coarticulations and the interactive effects among tonality in continuous speech. To tackle down this issue, we first aim at in using the sequence of f0 samples in speech utterance only. In addition, we propose a transformer based network with an extendable BERT input architecture and a joint learning technique to consolidate the contour representations of consecutive tones. Leveraging or fusing more information affected from speech rhythm in utterance, the experiments show that the proposed J-ToneNet is very robust for read speech.
Towards Cross-Language Prosody Transfer for Dialog
Authors: Jonathan E. Avila; Nigel G. Ward
Venue: Interspeech 2023 (main)

Abstract: Speech-to-speech translation systems today do not adequately support use for dialog purposes. In particular, nuances of speaker intent and stance can be lost due to improper prosody transfer. We present an exploration of what needs to be done to overcome this. First, we developed a data collection protocol in which bilingual speakers re-enact utterances from an earlier conversation in their other language, and used this to collect an English-Spanish corpus, so far comprising 1871 matched utterance pairs. Second, we developed a simple prosodic dissimilarity metric based on Euclidean distance over a broad set of prosodic features. We then used these to investigate cross-language prosodic differences, measure the likely utility of three simple baseline models, and identify phenomena which will require more powerful modeling. Our findings should inform future research on cross-language prosody and the design of speech-to-speech translation systems capable of effective prosody transfer.
Strategies for Improving Low Resource Speech to Text Translation Relying on Pre-trained ASR Models
Authors: Santosh Kesiraju; Marek SarvaÅ¡; TomÃ¡Å¡ PavlÃ­Äek; CÃ©cile Macaire; Alejandro Ciuba
Venue: Interspeech 2023 (main)

Abstract: This paper presents techniques and findings for improving the performance of low-resource speech to text translation (ST). We conducted experiments on both simulated and real-low resource setups, on language pairs English - Portuguese, and Tamasheq - French respectively. Using the encoder-decoder framework for ST, our results show that a multilingual automatic speech recognition system acts as a good initialization under low-resource scenarios. Furthermore, using the CTC as an additional objective for translation during training and decoding helps to reorder the internal representations and improves the final translation. Through our experiments, we try to identify various factors (initializations, objectives, and hyper-parameters) that contribute the most for improvements in low-resource setups. With only 300 hours of pre-training data, our model achieved 7.3 BLEU score on Tamasheq - French data, outperforming prior published works from IWSLT 2022 by 1.6 points.
ITALIC: An Italian Intent Classification Dataset
Authors: Alkis Koudounas; Moreno La Quatra; Lorenzo Vaiani; Luca Colomba; Giuseppe Attanasio; Eliana Pastor; Luca Cagliero; Elena Baralis
Venue: Interspeech 2023 (main)

Abstract: Recent large-scale Spoken Language Understanding datasets focus predominantly on English and do not account for language-specific phenomena such as particular phonemes or words in different lects. We introduce ITALIC, the first large-scale speech dataset designed for intent classification in Italian. The dataset comprises 16,521 crowdsourced audio samples recorded by 70 speakers from various Italian regions and annotated with intent labels and additional metadata. We explore the versatility of ITALIC by evaluating current state-of-the-art speech and text models. Results on intent classification suggest that increasing scale and running language adaptation yield better speech models, monolingual text models outscore multilingual ones, and that speech recognition on ITALIC is more challenging than on existing Italian benchmarks. We release both the dataset and the annotation scheme to streamline the development of new Italian SLU models and language-specific datasets.
Perceptual and Task-Oriented Assessment of a Semantic Metric for ASR Evaluation
Authors: Janine Rugayan; Giampiero Salvi; TorbjÃ¸rn Svendsen
Venue: Interspeech 2023 (main)

Abstract: Automatic speech recognition (ASR) systems have become a vital part of our everyday lives through their many applications. However, as much as we have developed in this regard, our most common evaluation method for ASR systems still remains to be word error rate (WER). WER does not give information on the severity of errors, which strongly impacts practical performance. As such, we examine a semantic-based metric called Aligned Semantic Distance (ASD) against WER and demonstrate its advantage over WER in two facets. First, we conduct a survey asking participants to score reference text and ASR transcription pairs. We perform a correlation analysis and show that ASD is more correlated to the human evaluation scores compared to WER. We also explore the feasibility of predicting human perception using ASD. Second, we demonstrate that ASD is more effective than WER as an indicator of performance on downstream NLP tasks such as named entity recognition and sentiment classification.
How ChatGPT is Robust for Spoken Language Understanding?
Authors: Guangpeng Li; Lu Chen; Kai Yu
Venue: Interspeech 2023 (main)

Abstract: Large language models (LLMs), e.g. ChatGPT, have shown super performance on various NLP tasks.  There is a doubt whether these LLMs, which are trained on a large corpus of written text, can show the robustness of understanding the spoken text.  Therefore in this paper, we give a detailed investigation of robust spoken language understanding (SLU) with ChatGPT. In our experiments, we evaluate ChatGPT on two sets of public datasets, Spoken SQuAD and ASR-GLUE, in which there are ASR errors in the text.  Quantitative and qualitative analyses on the experimental results are conducted to show that ChatGPT not only performs very well for SLU tasks but also can recover some ASR errors with its super reason ability.
GigaST: A 10,000-hour Pseudo Speech Translation Corpus
Authors: Rong Ye; Chengqi Zhao; Tom Ko; Chutong Meng; Tao Wang; Mingxuan Wang; Jun Cao
Venue: Interspeech 2023 (main)

Abstract: This paper introduces GigaST, a large-scale pseudo speech-to-text translation (ST) corpus. We create the corpus by translating the transcript in GigaSpeech, an English ASR corpus, into German and Chinese. The training set is translated by a strong machine translation system and the test set is translated by human. ST models trained with an addition of our corpus obtain new state-of-the-art results on the MuST-C English-German benchmark test set. We provide a detailed description of the translation process and verify its quality. We make the translated text data public and hope to facilitate research in speech translation. Additionally, we also release the training scripts on NeurST1 to make it easy to replicate our systems. GigaST dataset is available at https://st-benchmark.github.io/resources/GigaST.
Boosting Chinese ASR Error Correction with Dynamic Error Scaling Mechanism
Authors: Jiaxin Fan; Yong Zhang; Hanzhang Li; Jianzong Wang; Zhitao Li; Sheng Ouyang; Ning Cheng; Jing Xiao
Venue: Interspeech 2023 (main)

Abstract: Chinese Automatic Speech Recognition (ASR) error correction presents significant challenges due to the Chinese language's unique features, including a large character set and borderless, morpheme-based structure. Current mainstream models often struggle with effectively utilizing word-level features and phonetic information. This paper introduces a novel approach that incorporates a dynamic error scaling mechanism to detect and correct phonetically erroneous text generated by ASR output. This mechanism operates by dynamically fusing word-level features and phonetic information, thereby enriching the model with additional semantic data. Furthermore, our method implements unique error reduction and amplification strategies to address the issues of matching wrong words caused by incorrect characters. Experimental results indicate substantial improvements in ASR error correction, demonstrating the effectiveness of our proposed method and yielding promising results on established datasets.
Crowdsource-based Validation of the Audio Cocktail as a Sound Browsing Tool
Authors: Per Fallgren; Jens Edlund
Venue: Interspeech 2023 (main)

Abstract: We conduct two crowdsourcing experiments designed to examine the usefulness of audio cocktails to quickly find out information on the contents of large audio data. Several thousand crowd workers were engaged to listen to audio cocktails with systematically varied composition. They were then asked to state either which sound out of four categories (Children, Women, Men, Orchestra) they heard the most of, or if they heard anything of a specific category at all. The results show that their responses have high reliability and provide information as to whether a specific task can be performed using audio cocktails. We also propose that the combination of crowd workers and audio cocktails can be used directly as a tool to investigate the contents of large audio data.
PunCantonese: A Benchmark Corpus for Low-Resource Cantonese Punctuation Restoration from Speech Transcripts
Authors: Yunxiang Li; Pengfei Liu; Xixin Wu; Helen Meng
Venue: Interspeech 2023 (main)

Abstract: Punctuation restoration from unsegmented speech transcripts is an essential task to improve the readability of transcripts and can facilitate various downstream NLP tasks. However, there is still lack of systematic studies on punctuation restoration for Cantonese as a low-resource language. This paper introduces a new Cantonese punctuation corpus named PunCantonese, which consists of annotated spoken transcripts and written-style Wikipedia sentences, covering the major punctuations such as â,.?!â and code-switched sentences in Cantonese and English. We also propose a Transformer-based punctuation model which exploits pre-trained multilingual language models, adopts multitask learning for style and punctuation prediction, and introduces a novel Jyutping embedding layer to inject the
phonetic features not explicitly available in Cantonese characters. Experimental results show that these methods are effective in improving punctuation restoration, and the Jyutping embedding layer brings an absolute F1 increase by more than 2%.
Speech-to-Face Conversion Using Denoising Diffusion Probabilistic Models
Authors: Shuhei Kato; Taiichi Hashimoto
Venue: Interspeech 2023 (main)

Abstract: Speech-to-face conversion is the task of generating face images from speech signals. Many studies have been conducted to address this task, and achieved good performances. In this paper, we introduce denoising diffusion probabilistic models (DDPMs) to generate face images instead of generative adversarial networks (GANs) or autoencoders, which are used in most of the prior studies. Moreover, unlike prior studies, several components of our system are designed to use high-resolution face image datasets instead of audio-visual paired data. As a result, our system can generate high-resolution face images from speech signals with an architecture that is simpler and more flexible than the ones used in prior studies. In addition, introducing DDPMs enables us to utilize techniques that control out- puts of DDPMs or improve performance of them in succeeding studies.
Inter-connection: Effective Connection between Pre-trained Encoder and Decoder for Speech Translation
Authors: Yuta Nishikawa; Satoshi Nakamura
Venue: Interspeech 2023 (main)

Abstract: In end-to-end speech translation, speech and text pre-trained models improve translation quality. Recently proposed models simply connect the pre-trained models of speech and text as encoder and decoder. Therefore, only the information from the final layer of encoders is input to the decoder. Since it is clear that the speech pre-trained model outputs different information from each layer, the simple connection method cannot fully utilize the information that the speech pre-trained model has. In this study, we propose an inter-connection mechanism that aggregates the information from each layer of the speech pre-trained model by weighted sums and inputs into the decoder. This mechanism increased BLEU by approximately 2 points in en-de, en-ja, and en-zh by increasing parameters by 2K when the speech pre-trained model was frozen. Furthermore, we investigated the contribution of each layer for each language by visualizing layer weights and found that the contributions were different.
Conmer: Streaming Conformer Without Self-attention for Interactive Voice Assistants
Authors: Martin Radfar; Paulina Lyskawa; Brandon Trujillo; Yi Xie; Kai Zhen; Jahn Heymann; Denis Filimonov; Grant P. Strimel; Nathan Susanj; Athanasios Mouchtaris
Venue: Interspeech 2023 (main)

Abstract: Conformer is an extension of transformer-based neural ASR models whose fundamental component is the self-attention module. In this paper, we show that we can remove the self-attention module from Conformer and achieve the same or even better recognition performance  for utterances whose length is up to around 10 seconds. This is particularly important for streaming interactive voice assistants as input is often very short and a fast response is expected. Since the computational complexity of self-attention is quadratic, this modification allows for faster, smaller sized models, two requirements for on-device applications. Using this finding, we propose Conmer, a neural architecture based on Conformer but without self-attention for streaming interactive voice assistants. We conduct experiments on public and real-world data and show the streaming Conmer  reduces the WER and computational complexity relatively by 4.03% and 10%, respectively.
Intra-ensemble: A New Method for Combining Intermediate Outputs in Transformer-based Automatic Speech Recognition
Authors: Dohee Kim; Jieun Choi; Joon-Hyuk Chang
Venue: Interspeech 2023 (main)

Abstract: Deep learning models employ various regularization techniques to prevent overfitting and enhance generalization. In particular, an auxiliary loss, as proposed for connectionist temporal classification (CTC) models, demonstrated the potential for intermediate prediction to be useful by enabling sub-models to recognize speech accurately. We propose a new method called Intra-ensemble, which combines these accurate intermediate outputs into a single output for both training and inference, considering the importance of the intermediate layer using learnable parameters. Our approach is applicable to CTC models, attention-based encoder-decoder models, and transducer structures and demonstrated performance improvements of 13.5%, 3.0%, and 4.1% respectively, in the LibriSpeech evaluation. Furthermore, through various analytical experiments, we found that the sub-models contributed significantly to performance improvement.
A Comparative Study on E-Branchformer vs Conformer in Speech Recognition, Translation, and Understanding Tasks
Authors: Yifan Peng; Kwangyoun Kim; Felix Wu; Brian Yan; Siddhant Arora; William Chen; Jiyang Tang; Suwon Shon; Prashant Sridhar; Shinji Watanabe
Venue: Interspeech 2023 (main)

Abstract: Conformer, a convolution-augmented Transformer variant, has become the de facto encoder architecture for speech processing due to its superior performance in various tasks, including automatic speech recognition (ASR), speech translation (ST) and spoken language understanding (SLU). Recently, a new encoder called E-Branchformer has outperformed Conformer in the LibriSpeech ASR benchmark, making it promising for more general speech applications. This work compares E-Branchformer and Conformer through extensive experiments using different types of end-to-end sequence-to-sequence models. Results demonstrate that E-Branchformer achieves comparable or better performance than Conformer in almost all evaluation sets across 15 ASR, 2 ST, and 3 SLU benchmarks, while being more stable during training. We will release our training configurations and pre-trained models for reproducibility, which can benefit the speech community.
HyperConformer: Multi-head HyperMixer for Efficient Speech Recognition
Authors: Florian Mai; Juan Zuluaga-Gomez; Titouan Parcollet; Petr Motlicek
Venue: Interspeech 2023 (main)

Abstract: State-of-the-art ASR systems have achieved promising results by modeling local and global interactions separately. While the former can be computed efficiently, global interactions are usually modeled via attention mechanisms, which are expensive for long input sequences. Here, we address this by extending HyperMixer, an efficient alternative to attention exhibiting linear complexity, to the Conformer architecture for speech recognition, leading to HyperConformer. In particular, multi-head HyperConformer achieves comparable or higher recognition performance while being more efficient than Conformer in terms of inference speed, memory, parameter count, and available training data. HyperConformer achieves a word error rate of 2.9% on LibriSpeech test-clean with less than 8M neural parameters and a peak memory during training of 5.7GB, hence trainable with accessible hardware. Inference speed is between 38% on mid-length speech and 56% on long speech faster than an equivalent Conformer.
Memory-augmented conformer for improved end-to-end long-form ASR
Authors: Carlos Carvalho; Alberto Abad
Venue: Interspeech 2023 (main)

Abstract: Conformers have recently been proposed as a promising modelling approach for automatic speech recognition (ASR), outperforming recurrent neural network-based approaches and transformers. Nevertheless, in general, the performance of these end-to-end models, especially attention-based models, is particularly degraded in the case of long utterances. To address this limitation, we propose adding a fully-differentiable memory-augmented neural network between the encoder and decoder of a conformer. This external memory can enrich the generalization for longer utterances since it allows the system to store and retrieve more information recurrently. Notably, we explore the neural Turing machine (NTM) that results in our proposed Conformer-NTM model architecture for ASR. Experimental results using Librispeech train-clean-100 and train-960 sets show that the proposed system outperforms the baseline conformer without memory for long utterances.
Towards Effective and Compact Contextual Representation for Conformer Transducer Speech Recognition Systems
Authors: Mingyu Cui; Jiawen Kang; Jiajun Deng; Xi Yin; Yutao Xie; Xie Chen; Xunying Liu
Venue: Interspeech 2023 (main)

Abstract: Current ASR systems are mainly trained and evaluated at the utterance level. Long range cross utterance context can be incorporated. A key task is to derive a suitable compact representation of the most relevant history contexts. In contrast to previous researches based on either LSTM-RNN encoded histories that attenuate the information from longer range contexts, or frame level concatenation of transformer context embeddings, in this paper compact low-dimensional cross utterance contextual features are learned in the Conformer-Transducer Encoder using specially designed attention pooling layers that are applied over efficiently cached preceding utterances' history vectors. Experiments on the 1000-hr Gigaspeech corpus demonstrate that the proposed contextualized streaming Conformer-Transducers outperform the baseline using utterance internal context only with statistically significant WER reductions of 0.7% to 0.5% absolute (4.3% to 3.1% relative) on the dev and test data.
An Enhanced Res2Net with Local and Global Feature Fusion for Speaker Verification
Authors: Yafeng Chen; Siqi Zheng; Hui Wang; Luyao Cheng; Qian Chen; Jiajun Qi
Venue: Interspeech 2023 (main)

Abstract: Effective fusion of multi-scale features is crucial for improving speaker verification performance. While most existing methods aggregate multi-scale features in a layer-wise manner via simple operations, such as summation or concatenation. This paper proposes a novel architecture called Enhanced Res2Net (ERes2Net), which incorporates both local and global feature fusion techniques to improve the performance. The local feature fusion (LFF) fuses the features within one single residual block to extract the local signal. The global feature fusion (GFF) takes acoustic features of different scales as input to aggregate global signal. To facilitate effective feature fusion in both LFF and GFF, an attentional feature fusion module is employed in the ERes2Net architecture, replacing summation or concatenation operations. A range of experiments conducted on the VoxCeleb datasets demonstrate the superiority of the ERes2Net in speaker verification.
A Study on Visualization of Voiceprint Feature
Authors: Jian Zhang; Liang He; Xiaochen Guo; Jing Ma
Venue: Interspeech 2023 (main)

Abstract: Despite the remarkable success of convolutional neural networks (CNNs) in voiceprint recognition, we still lack a comprehensive understanding of the specific features extracted by these models. To address this issue, we adopt an attribution approach in this paper to explain the voiceprint identification model and visualize the relevant features. Using five attribution methods, we successfully identify the features extracted by the ECAPA-TDNN model and confirm the reliability of our attribution techniques.We also explore two distinct methods for visualizing voiceprint features, with one approach aimed at interpreting features in unknown speech and the other focused on known speech. Through the attribution method, we are able to more precisely capture voiceprint features within speech data without significantly impacting the performance of the voiceprint recognition model. It would help us to do a more detailed study of the voiceprint features in the future.
VoxTube: a multilingual speaker recognition dataset
Authors: Ivan Yakovlev; Anton Okhotnikov; Nikita Torgashov; Rostislav Makarov; Yuri Voevodin; Konstantin Simonchik
Venue: Interspeech 2023 (main)

Abstract: The objective of this paper is to advance the development of technologies in the fields of speaker recognition and speaker identification by introducing a large labeled audio database VoxTube collected from the open-source media. We propose a fully automated unsupervised approach for audio labeling that requires any pre-trained speaker recognition model. Collected with this approach from videos with CC BY license the VoxTube dataset contains more than 5.000 speakers with more than 4 million utterances pronounced in more than 10 languages. In our paper we show the VoxTube's high generalization ability across multiple domains by evaluating the accuracy metrics on various speaker recognition benchmarks. We also show how well this dataset complements an already existing VoxCeleb2 dataset.