GenAI4ELab/papercli-papers huggingface dataset:
InterFormer: Interactive Local and Global Features Fusion for Automatic Speech Recognition
Authors: Zhi-Hao Lai; Tian-Hao Zhang; Qi Liu; Xinyuan Qian; Li-Fang Wei; Feng Chen; Song-Lu Chen; Xu-Cheng Yin
Venue: Interspeech 2023 (main)

Abstract: The local and global features are both essential for automatic speech recognition (ASR). Many recent methods have verified that simply combining local and global features can further promote ASR performance. However, these methods pay less attention to the interaction of local and global features, and their series architectures are rigid to reflect local and global relationships. To address these issues, this paper proposes InterFormer for interactive local and global features fusion to learn a better representation for ASR. Specifically, we combine the convolution block with the transformer block in a parallel design. Besides, we propose a bidirectional feature interaction module (BFIM) and a selective fusion module (SFM) to implement the interaction and fusion of local and global features, respectively. Extensive experiments on public ASR datasets demonstrate the effectiveness of our proposed InterFormer and its superior performance over the baseline models
Transductive Feature Space Regularization for Few-shot Bioacoustic Event Detection
Authors: Yizhou Tan; Haojun Ai; Shengchen Li; Feng Zhang
Venue: Interspeech 2023 (main)

Abstract: In few-shot bioacoustic event detection, besides interested target events, background noises and various uninterested sound events lead to complex decision boundaries, which require regularized feature distributions in feature space. Due to the low label availability of uncertain noise events, existing few-shot learning methods with entropy-based regularizers suffer from overfitting during optimization. In this paper, we propose a transductive inference model with a prior knowledge based regularizer (PKR) to overcome the overfitting problem. We use a task-adaptive feature extractor to reconstruct a regularized feature space. A PKR is proposed to minimize the divergence between the original and reconstructed feature space. The development set of DCASE 2022 Task 5 is adopted as the experimental dataset. With the increasing iterations, the proposed model performs with long-lasting results around 55.43 F-measure, and well solves the overfitting problem in transductive inference.
Incorporating L2 Phonemes Using Articulatory Features for Robust Speech Recognition
Authors: Jisung Wang; Haram Lee; Myungwoo Oh
Venue: Interspeech 2023 (main)

Abstract: The limited availability of non-native speech datasets presents a major challenge in automatic speech recognition (ASR) to narrow the performance gap between native and non-native speakers. To address this, the focus of this study is on the efficient incorporation of the L2 phonemes, which in this work refer to Korean phonemes, through articulatory feature analysis. This not only enables accurate modeling of pronunciation variants but also allows for the utilization of both native Korean and English speech datasets. We employ the lattice-free maximum mutual information (LF-MMI) objective in an end-to-end manner, to train the acoustic model to align and predict one of multiple pronunciation candidates. Experimental results show that the proposed method improves ASR accuracy for Korean L2 speech by training solely on L1 speech data. Furthermore, fine-tuning on L2 speech improves recognition accuracy for both L1 and L2 speech without performance trade-offs.
On the (In)Efficiency of Acoustic Feature Extractors for Self-Supervised Speech Representation Learning
Authors: Titouan Parcollet; Shucong Zhang; Rogier van Dalen; Alberto Gil C. P. Ramos; Sourav Bhattacharya
Venue: Interspeech 2023 (main)

Abstract: Speech representations learned with self-supervised learning (SSL) have the potential to significantly improve the performance of a number of audio applications, especially when availability of labeled data from the deployment domain is limited. Despite their successes, SSL training methods are compute- and memory-heavy, and require large investments in computing infrastructure, thus putting it out of the reach of most institutions. Therefore, building efficient model architectures is essential for the wide-scale adoption of SSL in speech technologies. CNN-based Acoustic Feature Extractors (AFE), which are widely used as encoders of acoustic waveforms, remain one of the main efficiency bottlenecks. This work proposes replacing CNN-based AFEs with more efficient ones and demonstrates that SSL pre-training time and memory consumption can be reduced by a factor of two to three over existing methods while preserving performances in speech-, command-, and speaker-recognition tasks.
Phonemic competition in end-to-end ASR models
Authors: Louis ten Bosch; Martijn Bentum; Lou Boves
Venue: Interspeech 2023 (main)

Abstract: Advanced end-to-end ASR systems encode speech signals by means of a multi-layer network architecture. In Wav2vec2.0, for example, a CNN is used as feature encoder on top of which transformer layers are used to map the high-dimensional CNN representations to the elements of some lexicon. Compared to the previous generation of 'modular' ASR systems it is much more difficult to interpret the processing and representations in an end-to-end system from a phonetic point of view. We built a Wav2vec2.0-based end-to-end system for producing broad phonetic transcriptions of Dutch. In this paper we investigate to what extent the CNN features and the representations on several transformer layers of a pre-trained and fine-tuned model reflect widely-shared phonetic knowledge. For that purpose we analyze distances between phones and the phonetic features of the most-activated phones in the output of an MLP classifier operating on the representations in several layers.
Automatic speaker recognition with variation across vocal conditions: a controlled experiment with implications for forensics
Authors: Vincent Hughes; Jessica Wormald; Paul Foulkes; Philip Harrison; Finnian Kelly; David van der Vloed; Poppy Welch; Chenzi Xu
Venue: Interspeech 2023 (main)

Abstract: Automatic Speaker Recognition (ASR) involves a complex range of processes to extract, model, and compare speaker-specific information from a pair of voice samples. Using heavily controlled recordings, this paper explores the impact of specific vocal conditions (i.e. vocal setting, disguise, accent guises) on ASR performance. When vocal conditions are matched, ASR performance is generally excellent (whisper is an exception). When conditions are mismatched, as in most forensic cases, we see an increase in discrimination and calibration error in some cases. The most problematic mismatches are those involving whisper and supralaryngeal vocal settings; these produce the greatest phonetic changes to speech. Mismatches involving high pitch also produce poor performance, although this appears to be driven by speaker-specific differences in articulatory implementation. We discuss the implications of the findings for the use of ASR in forensic casework and the interpretability of system output.
Exploring Graph Theory Methods For the Analysis of Pronunciation Variation in Spontaneous Speech
Authors: Bernhard C. Geiger; Barbara Schuppler
Venue: Interspeech 2023 (main)

Abstract: Given the development of automatic speech recognition based techniques for creating phonetic annotations of large speech corpora, there has been a growing interest in investigating the frequencies of occurrence of phonological and reduction processes. Given that most studies have analyzed these processes separately, they did not provide insights about their co-occurrences. This paper contributes with introducing graph theory methods for the analysis of pronunciation variation in a large corpus of Austrian German conversational speech. More specifically, we investigate how reduction processes that are typical for spontaneous German in general co-occur with phonological processes typical for the Austrian German variety. Whereas our concrete findings are of special interest to scientists investigating variation in German, the approach presented opens new possibilities to analyze pronunciation variation in large corpora of different speaking styles in any language.
Automatic Speaker Recognition performance with matched and mismatched female bilingual speech data
Authors: Bryony Nuttall; Philip Harrison; Vincent Hughes
Venue: Interspeech 2023 (main)

Abstract: Validation of forensic voice comparison methods requires testing using speech samples that are representative of forensic casework conditions. Increasingly, around the world, forensic voice comparison casework is being undertaken using automatic speaker recognition (ASR) systems. However, multilingualism remains a key issue in applying automatic systems to forensic casework. This research aims to consider the effect of language on ASR performance, testing developers' claims of 'language independency'. Specifically, we examine the extent to which language mismatch either between the known and questioned samples, or between the evidential samples and the calibration data, affects overall system performance and the resulting strength of evidence (i.e., likelihood ratios for individual comparisons). Results indicate that mixed language trials produce more errors than single language trials which makes drawing evidential conclusions based on bilingual data challenging.
FACTSpeech: Speaking a Foreign Language Pronunciation Using Only Your Native Characters
Authors: Hong-Sun Yang; Ji-Hoon Kim; Yoon-Cheol Ju; Il-Hwan Kim; Byeong-Yeol Kim; Shuk-Jae Choi; Hyung-Yong Kim
Venue: Interspeech 2023 (main)

Abstract: Recent text-to-speech models have been requested to synthesize natural speech from language-mixed sentences because they are commonly used in real-world applications. However, most models do not consider transliterated words as input. When generating speech from transliterated text, it is not always natural to pronounce transliterated words as they are written, such as in the case of song titles. To address this issue, we introduce FACTSpeech, a system that can synthesize natural speech from transliterated text while allowing users to control the pronunciation between native and literal languages. Specifically, we propose a new language shift embedding to control the pronunciation of input text between native or literal pronunciation. Moreover, we leverage conditional instance normalization to improve pronunciation while preserving the speaker identity. The experimental results show that FACTSpeech generates native speech even from the sentences of transliterated form.
Cross-Lingual Transfer Learning for Phrase Break Prediction with Multilingual Language Model
Authors: Hoyeon Lee; Hyun-Wook Yoon; Jong-Hwan Kim; Jae-Min Kim
Venue: Interspeech 2023 (main)

Abstract: Phrase break prediction is a crucial task for improving the prosody naturalness of a text-to-speech (TTS) system. However, most proposed phrase break prediction models are monolingual, trained exclusively on a large amount of labeled data. In this paper, we address this issue for low-resource languages with limited labeled data using cross-lingual transfer. We investigate the effectiveness of zero-shot and few-shot cross-lingual transfer for phrase break prediction using a pre-trained multilingual language model. We use manually collected datasets in four Indo-European languages: one high-resource language and three with limited resources. Our findings demonstrate that cross-lingual transfer learning can be a particularly effective approach, especially in the few-shot setting, for improving performance in low-resource languages. This suggests that cross-lingual transfer can be inexpensive and effective for developing TTS front-end in resource-poor languages.
DSE-TTS: Dual Speaker Embedding for Cross-Lingual Text-to-Speech
Authors: Sen Liu; Yiwei Guo; Chenpeng Du; Xie Chen; Kai Yu
Venue: Interspeech 2023 (main)

Abstract: Although high-fidelity speech can be obtained for intralingual speech synthesis, cross-lingual text-to-speech (CTTS) is still far from satisfactory as it is difficult to accurately retain the speaker timbres (i.e. speaker similarity) and eliminate the accents from their first language (i.e. nativeness). In this paper, we demonstrated that vector-quantized (VQ) acoustic feature contains less speaker information than mel-spectrogram. Based on this finding, we propose a novel dual speaker embedding TTS (DSE-TTS) framework for CTTS with authentic speaking style. Here, one embedding is fed to the acoustic model to learn the linguistic speaking style, while the other one is integrated into the vocoder to mimic the target speaker's timbre. Experiments show that by combining both embeddings, DSE-TTS significantly outperforms the state-of-the-art SANE-TTS in cross-lingual synthesis, especially in terms of nativeness.
Generating Multilingual Gender-Ambiguous Text-to-Speech Voices
Authors: Konstantinos Markopoulos; Georgia Maniati; Georgios Vamvoukakis; Nikolaos Ellinas; Georgios Vardaxoglou; Panos Kakoulidis; Junkwang Oh; Gunu Jho; Inchul Hwang; Aimilios Chalamandaris; Pirros Tsiakoulis; Spyros Raptis
Venue: Interspeech 2023 (main)

Abstract: The gender of any voice user interface is a key element of its perceived identity. Recently there has been increasing interest in interfaces where the gender is ambiguous rather than clearly identifying as female or male. This work addresses the task of generating novel gender-ambiguous TTS voices in a multi-speaker, multilingual setting. This is accomplished by efficiently sampling from a latent speaker embedding space using a proposed gender-aware method. Extensive objective and subjective evaluations clearly indicate that this method is able to efficiently generate a range of novel, diverse voices that are consistent and perceived as more gender-ambiguous than a baseline voice across all the languages examined. Interestingly, the gender perception is found to be robust across two demographic factors of the listeners: native language and gender. To our knowledge, this is the first systematic and validated approach that can reliably generate a variety of gender-ambiguous voices.
RAD-MMM: Multilingual Multiaccented Multispeaker Text To Speech
Authors: Rohan Badlani; Rafael Valle; Kevin J. Shih; JoÃ£o Felipe Santos; Siddharth Gururani; Bryan Catanzaro
Venue: Interspeech 2023 (main)

Abstract: We create a multilingual speech synthesis system that can generate speech with a native accent in any seen language while retaining the characteristics of an individual's voice. It is expensive to obtain bilingual training data for a speaker and the lack of such data results in strong correlations that entangle speaker, language, and accent, resulting in poor transfer capabilities. To overcome this, we present RADMMM, a speech synthesis model based on RADTTS with explicit control over accent, language, speaker, and fine-grained F0 and energy features. Our proposed model does not rely on bilingual training data. We demonstrate an ability to control synthesized accent for any speaker in an open-source dataset comprising of 7 languages, with one native speaker per language. Human subjective evaluation demonstrates that, when compared to controlled baselines, our model better retains a speaker's voice and target accent, while synthesizing fluent speech in all target languages and accents in our dataset.
Multilingual context-based pronunciation learning for Text-to-Speech
Authors: Giulia Comini; Sam Ribeiro; Fan Yang; Heereen Shim; Jaime Lorenzo-Trueba
Venue: Interspeech 2023 (main)

Abstract: Phonetic information and linguistic knowledge are an essential component of a Text-to-speech (TTS) front-end. Given a language, a lexicon can be collected offline and Grapheme-to-Phoneme (G2P) relationships are usually modeled in order to predict the pronunciation for out-of-vocabulary (OOV) words. Additionally, post-lexical phonology, often defined in the form of rule-based systems, is used to correct pronunciation within or between words. In this work we showcase a multilingual unified front-end system that addresses any pronunciation related task, typically handled by separate modules. We evaluate the proposed model on G2P conversion and other language-specific challenges, such as homograph and polyphones disambiguation, post-lexical rules and implicit diacritization. We find that the multilingual model is competitive across languages and tasks, however, some trade-offs exists when compared to equivalent monolingual solutions.
Personalized Adaptation with Pre-trained Speech Encoders for Continuous Emotion Recognition
Authors: Minh Tran; Yufeng Yin; Mohammad Soleymani
Venue: Interspeech 2023 (main)

Abstract: There are individual differences in expressive behaviors driven by cultural norms and personality. This between-person variation can result in reduced emotion recognition performance. Therefore, personalization is an important step in improving the generalization and robustness of speech emotion recognition. In this paper, to achieve unsupervised personalized emotion recognition, we first pre-train an encoder with learnable speaker embeddings in a self-supervised manner to learn robust speech representations conditioned on speakers. Second, we propose an unsupervised method to compensate for the label distribution shifts by finding similar speakers and leveraging their label distributions from the training set. Extensive experimental results on the MSP-Podcast corpus indicate that our method consistently outperforms strong personalization baselines and achieves state-of-the-art performance for valence estimation.
The Importance of Calibration: Rethinking Confidence and Performance of Speech Multi-label Emotion Classifiers
Authors: Huang-Cheng Chou; Lucas Goncalves; Seong-Gyun Leem; Chi-Chun Lee; Carlos Busso
Venue: Interspeech 2023 (main)

Abstract: The uncertainty in modeling emotions makes speech emotion recognition (SER) systems less reliable. An intuitive way to increase trust in SER is to reject predictions with low confidence. This approach assumes that an SER system is well calibrated, where highly confident predictions are often right and low confident predictions are often wrong. Hence, it is desirable to calibrate the confidence of SER classifiers. We evaluate the reliability of SER systems by exploring the relationship between confidence and accuracy, using the expected calibration error (ECE) metric. We develop a multi-label variant of the post-hoc temperature scaling (TS) method to calibrate SER systems, while preserving their accuracy. The best method combines an emotion co-occurrence weight penalty function, a class-balanced objective function, and the proposed multi-label TS calibration method. The experiments show the effectiveness of our developed multi-label calibration method in terms of accuracy and ECE.
A Preliminary Study on Augmenting Speech Emotion Recognition using a Diffusion Model
Authors: Mohammad Ibrahim Malik; Siddique Latif; Raja Jurdak; BjÃ¶rn W. Schuller
Venue: Interspeech 2023 (main)

Abstract: In this paper, we propose to utilise diffusion models for data augmentation in speech emotion recognition (SER). In particular, we present an effective approach to utilise improved denoising diffusion probabilistic models (IDDPM) to generate synthetic emotional data. We condition the IDDPM with the textual embedding from bidirectional encoder representations from transformers (BERT) to generate high-quality synthetic emotional samples in different speakers' voiceswe uploaded the synthetic samples for reviewers to listen.. We implement a series of experiments and show that better quality synthetic data helps improve SER performance. We compare results with generative adversarial networks (GANs) and show that the proposed model generates better-quality synthetic samples that can considerably improve the performance of SER when augmented with synthetic data.
Privacy Risks in Speech Emotion Recognition: A Systematic Study on Gender Inference Attack
Authors: Basmah Alsenani; Tanaya Guha; Alessandro Vinciarelli
Venue: Interspeech 2023 (main)

Abstract: Increasingly more applications now use deep networks to analyse speaker's affective states. An undesirable side effect is that models trained to perform one task (e.g, emotion from speech) can be attacked to infer other, possibly privacy-sensitive attributes (e.g., gender) of the speaker. The amount of information an attacker can infer through such attacks is called leakage, and this article presents the first systematic study of the interplay between gender leakage and the main characteristics of the attacker model (family, architecture and training condition). To this end, we define various attack scenarios, and perform extensive experiments to analyse privacy risks in Speech Emotion Recognition (SER). Results show that SER models can leak a speaker's gender with an accuracy of 51% to 95% (upper bound) depending on the attack condition. Furthermore, our results provide fresh insights on how to limit the effectiveness of possible attacks and, thereby, to ensure privacy preservation.
Episodic Memory For Domain-Adaptable, Robust Speech Emotion Recognition
Authors: James Tavernor; Matthew Perez; Emily Mower Provost
Venue: Interspeech 2023 (main)

Abstract: Emotion conveys abundant information that can improve the user experience of various automated systems, in addition to communicating information important for managing well-being. Human speech conveys emotion, but speech emotion recognition models do not perform well in unseen environments. This limits the ubiquitous use of speech emotion recognition models. In this paper, we investigate how a model can be adapted to unseen environments without forgetting previously learned environments. We show that memory-based methods maintain performance on previously seen environments while still being able to adapt to new environments. These methods enable continual training of speech emotion recognition models following deployment while retaining previous knowledge, working towards a more general, adaptable, acoustic model.
Stable Speech Emotion Recognition with Head-k-Pooling Loss
Authors: Chaoyue Ding; Jiakui Li; Daoming Zong; Baoxiang Li; Tian-Hao Zhang; Qunyan Zhou
Venue: Interspeech 2023 (main)

Abstract: Speech emotion recognition (SER) aims to detect the emotion of the speaker involved in a given utterance. Most existing SER methods focus on local speech features by stacking convolutions and training all segments of an utterance with an utterance-level label. Two deficiencies exist in these methods: i) learning only local speech features may be insufficient for SER due to the ambiguity of emotions; ii) consistent supervision of each segment may lead to label error propagation, as the true emotions of some segments may not match the utterance label. To solve the two issues, we first devise a global-local fusion network to model both long- and short-range relations in speech. Second, we tailor a novel head-k-pooling loss for SER tasks, which dynamically assigns labels for each segment and selectively performs loss calculation across segments. We test our method on the IEMOCAP and the newly collected ST-EMO dataset, and the results show its superiority and stability.