GenAI4ELab/papercli-papers huggingface dataset:
Self-FiLM: Conditioning GANs with self-supervised representations for bandwidth extension based speaker recognition
Authors: Saurabh Kataria; JesÃºs Villalba; Laureano Moro-Velazquez; Thomas Thebaud; Najim Dehak
Venue: Interspeech 2023 (main)

Abstract: Speech super-resolution/Bandwidth Extension (BWE) can improve downstream tasks like Automatic Speaker Verification (ASV). We introduce a simple novel technique called Self-FiLM to inject self-supervision information in existing BWE models via Feature-wise Linear Modulation. We hypothesize that such information contains domain/environment information, which can help BWE deliver zero-shot generalization. Self-FiLM improves conditional GAN-based BWE by 18% (relative) in Equal Error Rate and 8.5% in minimum Decision Cost Function on the x-vector & Probabilistic Linear Discriminant Analysis based state-of-the-art ASV system on SRE21 test. We further improve it by using deep feature losses from time-domain models and re-training data2vec 2.0 models on naturalistic wideband (VoxCeleb) and telephone data (SRE Superset etc.). Lastly, we integrate Self-FiLM in CycleGAN to obtain a completely unsupervised solution that matches the CGAN-based semi-supervised performance.
Curriculum Learning for Self-supervised Speaker Verification
Authors: Hee-Soo Heo; Jee-weon Jung; Jingu Kang; Young-ki Kwon; Bong-Jin Lee; You Jin Kim; Joon Son Chung
Venue: Interspeech 2023 (main)

Abstract: The goal of this paper is to train effective self-supervised speaker representations without identity labels. We propose two curriculum learning strategies within a self-supervised learning framework. The first strategy aims to gradually increase the number of speakers in the training phase by enlarging the used portion of the train dataset. The second strategy applies various data augmentations to more utterances within a mini-batch as the training proceeds. A range of experiments conducted using the DINO self-supervised framework on the VoxCeleb1 evaluation protocol demonstrates the effectiveness of our proposed curriculum learning strategies. We report a competitive equal error rate of 4.47% with a single-phase training, and we also demonstrate that the performance further improves to 1.84% by fine-tuning on a small labelled dataset.
Introducing Self-Supervised Phonetic Information for Text-Independent Speaker Verification
Authors: Ziyang Zhang; Wu Guo; Bin Gu
Venue: Interspeech 2023 (main)

Abstract: This paper presents a novel multi-task learning framework by introducing self-supervised phonetic information for deep speaker embedding extraction. The primary task is still to classify speakers, but we consider an auxiliary task to identify phoneme boundaries in speech signals following the Noise Contrastive Estimation principle. To further utilize self-supervised information to assist speaker feature learning, the features of intermediate layers in the main task are refined by the features of corresponding layers in the auxiliary task through masking and biasing operations. We use the VoxCeleb1 and CN-Celeb datasets for performance evaluation, which consistently verifies the efficacy of the proposed method.
A Teacher-Student Approach for Extracting Informative Speaker Embeddings From Speech Mixtures
Authors: Tobias Cord-Landwehr; Christoph Boeddeker; CÄtÄlin ZorilÄ; Rama Doddipatla; Reinhold Haeb-Umbach
Venue: Interspeech 2023 (main)

Abstract: We introduce a monaural neural speaker embeddings extractor that computes an embedding for each speaker present in a speech mixture. To allow for supervised training, a teacher-student approach is employed: the teacher computes the target embeddings from each speaker's utterance before the utterances are added to form the mixture, and the student embedding extractor is then tasked to reproduce those embeddings from the speech mixture at its input. The system much more reliably verifies the presence or absence of a given speaker in a mixture than a conventional speaker embedding extractor, and even exhibits comparable performance to a multi-channel approach that exploits spatial information for embedding extraction. Further, it is shown that a speaker embedding computed from a mixture can be used to check for the presence of that speaker in another mixture.
Experimenting with Additive Margins for Contrastive Self-Supervised Speaker Verification
Authors: Theo Lepage; Reda Dehak
Venue: Interspeech 2023 (main)

Abstract: Most state-of-the-art self-supervised speaker verification systems rely on a contrastive-based objective function to learn speaker representations from unlabeled speech data. We explore different ways to improve the performance of these methods by: (1) revisiting how positive and negative pairs are sampled through a âsymmetricâ formulation of the contrastive loss; (2) introducing margins similar to AM-Softmax and AAM-Softmax that have been widely adopted in the supervised setting. We demonstrate the effectiveness of the symmetric contrastive loss which provides more supervision for the self-supervised task. Moreover, we show that Additive Margin and Additive Angular Margin allow reducing the overall number of false negatives and false positives by improving speaker separability. Finally, by combining both techniques and training a larger model we achieve 7.50% EER and 0.5804 minDCF on the VoxCeleb1 test set, which outperforms other contrastive self supervised methods on speaker verification.
Nonbinary American English speakers encode gender in vowel acoustics
Authors: Maxwell Hope; Charlotte Ward; Jason Lilley
Venue: Interspeech 2023 (main)

Abstract: Encoding of gender in speech is a well-researched phenomenon, especially as it concerns men and women. Men tend to produce certain acoustic characteristics in certain ways (low fundamental frequency (F0), lower formant frequencies, lower center of gravity for [s] in English) compared to women, though these characteristics also differ based on other group memberships (e.g. race, sexuality, etc). Those who are more feminine, regardless of categorical gender, have been found to produce an increase in F0 and a larger vowel space. However, these previous studies used largely cisgender women and men or only examined encoding of binary gender in speech and did not consider encoding of "other" or nonbinary gender in speech. This study recruited American English nonbinary speakers to record 400 utterances and correlated acoustic characteristics with multidimensional gender. Masculine, feminine, and "other" gender are significantly correlated with vowel acoustics.
Coarticulation of Sibe Vowels and Dorsal Fricatives in Spontaneous Speech: An Acoustic Study
Authors: Jared Sharp; Matthew Faytak; Hasutai Fei Xiong Liu
Venue: Interspeech 2023 (main)

Abstract: Previous phonological analyses of dorsal segments in Sibe (Tungusic; Xinjiang, China) have treated the velar and uvular series as allophonic, with velar segments adjacent to /i É u/ and uvular segments adjacent to /a o/. In this paper, we use a spontaneous speech corpus to examine the acoustic correlates of coarticulation of dorsal fricatives and vowels in Sibe. The first two spectral moments and mid-frequency spectral peak of dorsal fricatives were measured over the fricative onset in VC sequences and offset in CV sequences. Differences in spectral measures suggest that dorsal fricatives coarticulate with both preceding and following vowels primarily in terms of tongue dorsum backness, but with some role of height possible. These findings reflect a more complex relationship between Sibe vowels and dorsals than previously described; the distribution of labels used for the dorsal fricatives also suggests a gradient assimilation process.
Using speech synthesis to explain automatic speaker recognition: a new application of synthetic speech
Authors: Georgina Brown; Christin KirchhÃ¼bel; Ramiz Cuthbert
Venue: Interspeech 2023 (main)

Abstract: Some speech synthesis systems make use of zero-shot adaptation to generate speech based on a target speaker. These systems produce speaker embeddings in the same way that speaker embeddings (often called 'x-vectors') are produced in automatic speaker recognition systems. This commonality between the two technologies could lower barriers that constrain the use of automatic speaker recognition systems in forensic speech analysis casework. A key barrier to the use of automatic speaker recognition in the forensic context is the issue of explainability, including what information about the voice a system uses in order to arrive at conclusions. This paper sets out a new approach that could be used to effectively communicate this type of information to audiences in the legal setting. Specifically, it is proposed that exposing listeners to synthetic speech produced by a zero-shot adaptation system could illustrate what aspects of the voice an automatic speaker recognition system captures.
Same F0, Different Tones: A Multidimensional Investigation of Zhangzhou Tones
Authors: Yishan Huang
Venue: Interspeech 2023 (main)

Abstract: This paper explores tonal encoding in Zhangzhou Southern Min, a Sinitic dialect spoken in South Fujian province of China. Results show that tones sharing an identical F0 contour can differ considerably in duration, vowel quality, voice quality, syllable coda type, and sandhi behavior. It is proper to consider tones are multidimensionally distinguished and treat each tone as a complex of phonetic features. The discussion stretches and deepens our understanding of the phonetic nature of tone, while questions the conventional definition of tone as a contrastive use of pitch. It contributes vital empirical data to the typology of tone as an important phenomenon in world's languages, and sheds important light on how tones should be defined.
Discovering Phonetic Feature Event Patterns in Transformer Embeddings
Authors: Patrick Cormac English; John D. Kelleher; Julie Carson-Berndsen
Venue: Interspeech 2023 (main)

Abstract: Domain-informed probing of large speech recognition transformer-based models offers an opportunity to investigate how phonetic information is captured and transformed in the information-rich embeddings that emerge as part of the recognition process. Previous work in this area has established the efficacy of probing these embeddings with simple multi-layer perceptron models to identify the information patterns encoded at each layer of the transformer. This paper explores phonetic feature event patterns which evolve at each layer of a transformer model. Probing models are trained with phonetic embeddings, which are averaged and labelled at the phone level using the TIMIT dataset, to detect the presence of certain phonetic features in time-steps of a speech signal. This paper demonstrates how the detection of phonetic features within the embeddings of transformer models, such as voicing, frication and nasal, provides insights in relation to the encoding of speech patterns in these models.
A System for Generating Voice Source Signals that Implements the Transformed LF-model Parameter Control
Authors: Zihan Wang; Christer Gobl
Venue: Interspeech 2023 (main)

Abstract: This paper describes a system which fully implements the transformed LF glottal flow model, incorporating also the often-overlooked k-factors. A problem with the original proposal is that the global waveshape parameter Rd, central to the transformed model and used to predict some of the other parameters, cannot be directly controlled. Instead, a stylisation of the model was used to indirectly control Rd. However, this approach can yield substantial errors in the Rd value of the pulse, thus undermining the usefulness of the model. To overcome this problem, an iterative algorithm is presented, which ensures that for a given Rd input value, a pulse with that Rd value will be produced. Using this new Rd control, it transpired that some of the original parameter predictions give rise to combinations of values that are incompatible with the LF model. Modifications to the original predictions of the Rk parameter were incorporated to ensure model conformity.
Speaker-independent Speech Inversion for Estimation of Nasalance
Authors: Yashish M Siriwardena; Carol Espy-Wilson; Suzanne Boyce; Mark Tiede; Liran Oren
Venue: Interspeech 2023 (main)

Abstract: The velopharyngeal (VP) valve regulates the opening between the nasal and oral cavities. This valve opens and closes through a coordinated motion of the velum and pharyngeal walls. Nasalance is an objective measure derived from the oral and nasal acoustic signals that correlate with nasality. In this work, we evaluate the degree to which the nasalance measure reflects fine-grained patterns of VP movement by comparison with simultaneously collected direct measures of VP opening using high-speed nasopharyngoscopy (HSN). We show that nasalance is significantly correlated with the HSN signal, and that both match expected patterns of nasality. We then train a temporal convolution-based speech inversion system in a speaker-independent fashion to estimate VP movement for nasality, using nasalance as the ground truth. In further experiments, we also show the importance of incorporating source features (from glottal activity) to improve nasality prediction.
Effects of Tonal Coarticulation and Prosodic Positions on Tonal Contours of Low Rising Tones: In the Case of Xiamen Dialect
Authors: Yiying Hu; Hui Feng; Qinghua Zhao; Aijun Li
Venue: Interspeech 2023 (main)

Abstract: Few studies have worked on the effects of tonal coarticulation and prosodic positions on the low rising tone in Xiamen Dialect. This study addressed such an issue. To do so, a new method, the Tonal Contour Analysis in Tonal Triangle, was proposed to measure the subtle curvature of the tonal contour. Findings are as follows: (1) The low rising tone in Xiamen Dialect has a tendency towards the falling-rising tone, which is significantly affected by the tonal coarticulation and prosodic positions. (2) The low rising tone presents as a falling-rising tone when preceded by a tone with a high offset, and as a low rising tone when preceded by a tone that ends up low. (3) The curvature of the low rising tone is greatest in the sentence-initial position, and is positively correlated to its own duration.
Durational and Non-durational Correlates of Lexical and Derived Geminates in Arabic
Authors: Amel Issa
Venue: Interspeech 2023 (main)

Abstract: This paper reports on the phonetic and phonological patterns of gemination in Tripolitanian Libyan Arabic (TLA). While previous studies on Arabic gemination have either focused on lexical geminates or reported results on data that contains both lexical and derived geminates, without investigating its effect on the phonetic output, the present study investigates the effect of the phonological status of a geminate on the phonetic realization. Several measurements were obtained including target segments duration, RMS amplitude and F1, F2 and F3 for the target consonants. Preliminary results suggest that the acoustic distinction between singleton and geminate consonants in TLA is dependent mainly on durational correlates. There was no evidence of differences in RMS amplitude between singleton and geminate consonants of any type. F1, F2 and F3 frequencies are found to show similar patterns for singleton and geminate types for all sounds, suggesting no gestural effects of gemination in TLA.
Mapping Phonemes to Acoustic Symbols and Codes Using Synchrony in Speech Modulation Vectors Estimated by the Travellingwave Filter Bank
Authors: Ashwin Rao
Venue: Interspeech 2023 (main)

Abstract: A hybrid vector representation for speech resonances is defined using the modulation model and the sum of sinusoids model. An adaptive filter bank, whose channels utilize resonance localized modulation tracking, to robustly estimate temporal variations in these vectors, is then presented. The synchrony in modulations, within and across resonance channels, is subsequently used to derive acoustic symbols and codes that map fundamental units of languages, phonemes. Such an acoustic-phonetic mapping has never been demonstrated before. It has potential applications in speech recognition and voice analytics.
Rhythmic Characteristics of L2 German Speech by Advanced Chinese Learners
Authors: Lindun Ge; Min Xu; Hongwei Ding
Venue: Interspeech 2023 (main)

Abstract: The rhythm of speech produced by advanced Chinese learners can still be problematic. In order to improve their pronunciation, it is necessary to investigate L2 rhythmic deviance. In this study, read-aloud materials of 10 German L1 speakers selected from MULTEXT and parallel recordings of 14 Chinese L2 learners were analyzed by comparing six widely used rhythmic metrics, speech rate, and vowel deletion ratio. A correlation analysis was also conducted between these rhythmic features and the degree of perceived foreign accent. The results indicated that speech rate, varcoC, and vowel deletion ratio were significantly different between L1 and L2 speakers, and that speech rate and vowel deletion ratio had a strong correlation with perceived foreign accent within L2 speech. These findings suggest that complex consonantal intervals and vowel deletion are still challenging for advanced learners and should be given more attention in pronunciation training.
(Dis)agreement and Preference Structure are Reflected in Matching Along Distinct Acoustic-prosodic Features
Authors: Anneliese Kelterer; Margaret Zellers; Barbara Schuppler
Venue: Interspeech 2023 (main)

Abstract: This paper presents an investigation of acoustic-prosodic alignment in conversational speech and its relationship to functional inter-speaker alignment. While most previous research studied global alignment over whole conversations between strangers, the focus of this paper is on alignment between friends, partners and colleagues as a more local phenomenon related to affiliation and preference structure. Based on 359 turn-pairs from assessment sequences, we analyzed three prosodic matching features between adjacent turns in logistic and linear regression models. We found that disagreements tend to be produced with less F0 span matching than agreements and with less F0 median matching in some parts of the conversation. Preferred responses were more likely to be marked by higher F0 median matching than dispreferred responses. These results indicate that different aspects of functional inter-speaker alignment are reflected in matching along distinct acoustic-prosodic features.
Vowel reduction by Greek-speaking children: The effect of stress and word length
Authors: Polychronia Christodoulidou; Katerina Nicolaidis; Dimitrios Stamovlasis
Venue: Interspeech 2023 (main)

Abstract: This study examines vowel reduction as a function of stress and word length in 3 male and 3 female 3-, 5- and 7-year-olds, and adult controls. Participants produced triplets of two- and three-syllable words, where each Greek vowel /i, Îµ, É, Î¿, u/ was studied in the first syllable in both stressed and unstressed conditions. Measurements were made for absolute and relative duration and (non-)normalized vowel space areas. The results showed that children had longer absolute and relative vowel duration and larger non-normalized vowel space areas than adults. Vowel space areas were lower and to the left in the acoustic vowel space in relation to adults. Longer duration and larger vowel space areas were found in stressed position across all ages with an adult-like spectral reduction already present at 3 years of age but an adult-like temporal reduction from 5 years of age and onwards for unstressed vowels in both two- and three-syllable words.
Pitch distributions in a very large corpus of spontaneous Finnish speech
Authors: Mietta Lennes; Minnaleena Toivola
Venue: Interspeech 2023 (main)

Abstract: Speakers differ in the pitch range they use in their speech. In order to analyze the functional aspects of pitch, the typical pitch range of each individual is needed as reference. However, systematically collected pitch data from a sufficiently large corpus have not been previously available. We analyze the pitch distributions of individual speakers in a subset of the Donate Speech Corpus, collected from speakers of Finnish in 2020-2021. We report pitch analysis results based on samples from 8197 speakers and 1475 hours of speech. We compare the results obtained from male and female speakers in different age groups.
Speech Enhancement Patterns in Human-Robot Interaction: A Cross-Linguistic Perspective
Authors: Jacek Kudera; Katharina Zahner-Ritter; Jakob Engel; Nathalie ElsÃ¤sser; Philipp Hutmacher; Carolin Worstbrock
Venue: Interspeech 2023 (main)

Abstract: This paper presents the results of the human-robot interaction (HRI) study with German native speakers addressing the robot in their L1 and in L2 English. The aim of the experiment is to test the strategies of providing clarifications when talking to the voice assistant in a task involving teaching complex vocabulary. The analyses is based on spectral (F1, F2, and mean F0) and temporal (vowel length) features excerpted from the target words. With reference to a theoretical framework of hyperarticulation and hypoarticulation, these acoustic measures were compared across the iterations of the target words (first vs. second iteration). Results showed that participants, when asked for clarification by an inanimate interlocutor, do not hyperarticulate, but try to preserve the surface representation of target words across the iterations. These findings suggest that acoustic characteristics of clarifications directed to voice assistants differ from the ones directed to human interlocutors.
Controllable Generation of Artificial Speaker Embeddings through Discovery of Principal Directions
Authors: Florian Lux; Pascal Tilli; Sarina Meyer; Ngoc Thang Vu
Venue: Interspeech 2023 (main)

Abstract: Customizing voice and speaking style in a speech synthesis system with intuitive and fine-grained controls is challenging, given that little data with appropriate labels is available. Furthermore, editing an existing human's voice also comes with ethical concerns. In this paper, we propose a method to generate artificial speaker embeddings that cannot be linked to a real human while offering intuitive and fine-grained control over the voice and speaking style of the embeddings, without requiring any labels for speaker or style. The artificial and controllable embeddings can be fed to a speech synthesis system, conditioned on embeddings of real humans during training, without sacrificing privacy during inference.
Dual Audio Encoders Based Mandarin Prosodic Boundary Prediction by Using Multi-Granularity Prosodic Representations
Authors: Ruishan Li; Yingming Gao; Yanlu Xie; Dengfeng Ke; Jinsong Zhang
Venue: Interspeech 2023 (main)

Abstract: Prosodic boundary prediction plays an important role in speech synthesis, phonetic understanding, etc. In previous studies, supra-segmental features such as pitch, energy, and duration have been widely used to explicitly model Mandarin prosodic boundaries. In this paper, we propose to refine implicit prosodic representations with fine-grained information from complex acoustic features including mel-spectrogram and context vectors obtained from a pre-trained model. Pitch and energy are encoded as explicit prosodic representations. These two representations extracted by dual audio encoders are fused by the decoder mainly composed of cross-attention layers. Then the fused representations are used to predict Mandarin prosodic boundaries. The results indicate that our proposed method outperforms the baselines in the Mandarin prosodic boundary prediction task, particularly for the minor prosodic phrases (#2).