GenAI4ELab/papercli-papers huggingface dataset:
NAST: Noise Aware Speech Tokenization for Speech Language Models
Authors: Shoval Messica; Yossi Adi
Venue: Interspeech 2024 (main)

Abstract: Speech tokenization is the task of representing speech signals as a sequence of discrete units. Such representations can be later used for various downstream tasks including automatic speech recognition, text-to-speech, etc. More relevant to this study, such representation serves as the basis of Speech Language Models. In this work, we tackle the task of speech tokenization under the noisy setup and present NAST: Noise Aware Speech Tokenization for Speech Language Models. NAST is composed of three main components: (i) a predictor; (ii) a residual encoder; and (iii) a decoder. We evaluate the efficiency of NAST considering several spoken language modeling tasks and show that NAST is superior to the evaluated baselines across all setups. Lastly, we analyze NAST and show its disentanglement properties and robustness to signal variations in the form of noise, reverberation, pitch-shift, and time-stretch. Code and pre-trained models are available at https://github.com/ShovalMessica/NAST.
Low Bitrate High-Quality RVQGAN-based Discrete Speech Tokenizer
Authors: Slava Shechtman; Avihu Dekel
Venue: Interspeech 2024 (main)

Abstract: Discrete Audio codecs (or audio tokenizers) have recently regained interest due to the ability of Large Language Models (LLMs) to learn their compressed acoustic representations. Various publicly available trainable discrete tokenizers recently demonstrated impressive results for audio tokenization, yet they mostly require high token rates to gain high-quality reconstruction. In this study, we fine-tuned an open-source general audio RVQGAN model using diverse open-source speech data, considering various recording conditions and quality levels. The resulting wideband (24kHz) speech-only model achieves speech reconstruction, which is nearly indistinguishable from PCM (pulse-code modulation) with a rate of 150-300 tokens per second (1500-3000 bps). The evaluation used comprehensive English speech data encompassing different recording conditions, including studio settings. Speech samples and details on the reproducible trained models are made publicly available.
Perception of music and speech: Focus on rhythm processing
Authors: Barbara Tillmann
Venue: Interspeech 2024 (main)

Abstract: Research in cognitive neuroscience has revealed similarities in neural and cognitive correlates of music and language processing. Investigations focusing on temporal processing, in particular, rhythmic and metrical processing, have revealed interesting connections between music and speech. These observations have led to several theoretical frameworks and hypotheses about underlying mechanisms and neural functioning, and has motivated applications to clinical research. I will present research that has demonstrated beneficial effects of rhythmic stimulation or training to improve language processing in populations of adults and children with typical development and with developmental language disorder or dyslexia. A recent hypothesis highlights the potential value of early detection of atypical rhythmic processing as indicative of increased risk for language disorders. This research domain provides perspectives for creating rhythm-based training programs for rehabilitation and also for early intervention, aiming to decrease language deficits during development.
Acquisition of high vowel devoicing in Japanese: A production experiment with three and four year olds
Authors: Hyun Kyung Hwang; Manami Hirayama
Venue: Interspeech 2024 (main)

Abstract: The purpose of this study is to investigate the developmental path of high vowel devoicing (HVD) in Japanese. A picture-naming task was conducted with Japanese-learning preschoolers of three and four years old. The empirical data presented in this study allow us not only to make comparisons with the data from 4 year-olds in a previous study, but also to address the devoicing patterns for the 3-year old children, which have received little attention in the developmental literature on HVD. The results of twenty children reveal distinct patterns depending on position; Word-medially, the overall occurrence of HVD increases if we compare the average HVD rates between the ages of 3 and 4, but their rate is not yet reached at the adult-like level at the age of four. Word-finally, on the other hand, the rates are overall lower than the word-medial devoicing. Further, unlike the incremental pattern observed in the rates for the word-medial devoicing, no clear developmental advancement is found in this position. The presence or absence of the developmental advancement appears to support the qualitative differences between two types of HVDs in distinct positions.
The Production of Contrastive Focus by 7 to 13-year-olds Learning Mandarin Chinese
Authors: Zimeng Li; Zhongxuan Mao; Shengting Shen; Ivan Yuen; Ping Tang
Venue: Interspeech 2024 (main)

Abstract: Contrastive focus highlights the important information in discourse, acoustically manifested in fundamental frequency (F0) and duration. In tonal languages, F0 conveys both lexical (lexical tones) and discourse meanings (contrastive focus). Given the importance of tones in word recognition, children acquire tonal productions early before age 3, but it was unclear when and how children acquire contrastive focus. This study tested 190 7-13-year-olds and 20 adult controls. Their productions of noun phrases with and without contrastive focus were elicited and perceptually evaluated for the presence/absence of focus. The perceptual judgement revealed that the perceived accuracy improved from 76% at age 7 to 93% at age 13. The acoustic analysis on the âperceived focusâ productions revealed that 7-year-olds were able to use both F0 and durational cues, although fine-tuning continues until 13 years. These findings revealed that focus acquisition in Mandarin might be a late and gradual process.
Cross-Linguistic Intelligibility of Non-Compositional Expressions in Spoken Context
Authors: Iuliia Zaitova; Irina Stenger; Wei Xue; Tania Avgustinova; Bernd MÃ¶bius; Dietrich Klakow
Venue: Interspeech 2024 (main)

Abstract: This study investigates intelligibility of non-compositional expressions in spoken context for five closely related Slavic languages (Belarusian, Bulgarian, Czech, Polish, and Ukrainian) by native Russian speakers. Our investigation employs a web-based experiment involving free-response and multiple-choice translation tasks. Drawing on prior research, two factors were examined: (1) linguistic similarities (orthographic and phonological distances), and (2) surprisal scores obtained from two multilingual speech representation (SR) models fine-tuned for Russian (Wav2Vec2-Large-Ru-Golos-With-LM and Whisper Medium Russian).  According to the results of Pearson correlation and regression analyses, phonological distance appears to be a better predictor of intelligibility scores than SR surprisal.
On the relationship between speech production and vocabulary size in 3-5 year olds
Authors: Alexis DeMaere; Nicole van Rootselaar; Fangfang Li; Robbin Gibb; Claudia L. R. Gonzalez
Venue: Interspeech 2024 (main)

Abstract: Language acquisition is a multifaceted process, including distinct yet complimentary elements such as word comprehension and speech production . There is evidence that these skills develop together. We studied a group of preschool children to measure the strength of the relationship between receptive vocabulary and speech production at the start and the end of a six-month period, as well as to document any improvement in these domains. A linear regression revealed that vocabulary size at the initial testing time predicted speech production accuracy at the final session, and vice versa. We discuss the null result of no sex differences for either speech or vocabulary. Our results highlight the inter-connectedness of speech production and vocabulary.
Towards Classifying Mother Tongue from Infant Cries - Findings Substantiating Prenatal Learning Theory
Authors: Tim Polzehl; Tim Herzig; Friedrich Wicke; Kathleen Wermke; Razieh Khamsehashari; Michiko Dahlem; Sebastian MÃ¶ller
Venue: Interspeech 2024 (main)

Abstract: In this work we introduce automatic mother tongue classification based on infant cries.  We use data of 63 German and Japanese healthy, term-born neonates, and model their cries with the help of data augmentation and Pre-trained Audio Neural Networks (PANNs), leveraging transfer learning methods suited to the very limited data at hand.  Applying small models on top of PANNs we obtain F1 scores of 85% and above on a held-out test set. We conduct several experiments analyzing model reliability, all of which indicate the network focuses on the actual infant cries rather than on confounding factors. We visualize the network focus to adhere to pitch contour and harmonics thereof, rendering prosody central for our model prediction.  Eventually, our models add a novel objectively obtained perspective to neonate crying analysis, while our results substantiate an extremely early vocal learning indication.
Effect of Complex Boundary Tones on Tone Identification: An Experimental Study with Mandarin-speaking Preschool Children
Authors: Aijun Li; Jun Gao; Zhiwei Wang
Venue: Interspeech 2024 (main)

Abstract: In Chinese, boundary tones can be categorized into simultaneous addition boundary tone (SiABT) and successive addition boundary tone (SuABT). This study focuses on the use of falling SuABT in child-directed speech for preschoolers' word learning scenarios and examines whether SuABT affects children's decoding of lexical tone. Forty-eight Mandarin-speaking preschoolers participated in an identification experiment. A total of 90 familiar monosyllabic words were designed, comprising two types of word pairs: with or without tonal contrast. The target words were embedded in the two types of boundary tones. The results indicate that complex SuABT does not impede the decoding of tonal categories but rather assists preschoolers in acquiring lexical information. Age demonstrates a correlation with the ability to decode tones under both boundary tones, with a more pronounced correlation observed under the SuABT condition. As age increases, this ability gradually stabilizes after 32 months.
Ethnolinguistic Identification of Vietnamese-German Heritage Speech
Authors: Thanh Lan Truong; Andrea Weber
Venue: Interspeech 2024 (main)

Abstract: We examined whether listeners can identify the heritage of speakers based on their voices. In Experiment 1, native German listeners with and without an Asian heritage listened to sentences that had either been produced by native German speakers with a Vietnamese heritage or by Germans without any Asian heritage. Asian heritage listeners performed more accurately when speakers matched their heritage background, but they could identify German speakers with no Asian heritage with above chance accuracy. Listeners with no Asian heritage performed more accurately when the speakers had no Vietnamese background than when they did, but they could not identify speakers with a Vietnamese heritage with above chance accuracy. In Experiment 2, the pattern for German listeners with no Asian heritage was replicated with monotonized stimuli. Thus, matching heritage background facilitates ethnolinguistic identification for Vietnamese-German heritage speech, which persists even when pitch is absent.
Joint prediction of subjective listening effort and speech intelligibility based on end-to-end learning
Authors: Dirk Eike Hoffner; Jana RoÃbach; Bernd T. Meyer
Venue: Interspeech 2024 (main)

Abstract: Subjective listening effort and speech intelligibility are crucial aspects in human communication. Models that can predict these metrics are important tools to develop speech enhancement or compression algorithms. To make predictions in real-life situations, non-intrusive models are required which do not use a clean reference signal as additional input. This paper explores a non-intrusive model for joint prediction of listening effort and speech intelligibility, which is based on character probabilities obtained from an end-to-end automatic speech recognition system. The uncertainty of the character classification is quantified using an entropy-based measure and compared to subjective data from normal-hearing and hearing-impaired listeners. The proposed model achieves correlation values of at least 0.9 and a root-mean-square error at or below 5 percentage points for speech intelligibility, and outperforms an intrusive baseline in four out of six conditions.
Depression Enhances Internal Inconsistency between Spoken and Semantic Emotion: Evidence from the Analysis of Emotion Expression in Conversation
Authors: Xinyi Wu; Changqing Xu; Nan Li; Rongfeng Su; Lan Wang; Nan Yan
Venue: Interspeech 2024 (main)

Abstract: Spoken emotion and semantic emotion are two components of emotion expression. In human conversation, emotions expressed by these two modalities are similar in healthy individuals. However, rich evidence documents that depression might affect emotional expression. Nevertheless, the consistency between spoken and semantic emotion in depressed patients has rarely been studied previously. In the present study, we investigated the consistency between emotions expressed by acoustical features and text content in depressed and healthy individuals during natural conversations. It was found that depressed patients tended to talk about negative topics in a neutral emotional tone and talk about neutral or positive topics in a depressed tone. These findings suggest that depression not only affects the emotion expression of a single modality but also results in an inconsistency between emotions expressed by these two modalities.
Listeners' F0 preferences in quiet and stationary noise
Authors: Olympia Simantiraki; Martin Cooke
Venue: Interspeech 2024 (main)

Abstract: Talkers - and increasingly speech output technology - typically alter speech characteristics when faced with challenging communicative conditions, but the impact of these changes on interlocutors is not fully understood. One such characteristic is fundamental frequency (F0), whose mean and range tend to increase when talking in noise or when communicating with inexperienced listeners. However, speech perception experiments have yet to demonstrate any intelligibility advantage for F0 modifications. The current study asked listeners to alter mean F0 or F0 variation with real-time feedback, in order to maximise comprehensibility in quiet and noise. Listeners chose a lower mean F0 than that of the original, and F0 variation similar to the original. Masker level had no effect on preference, suggesting that while listeners express clear choices, adjustment of F0 has no impact on intelligibility, and may instead reflect considerations such as naturalness or listening effort.
Effects of talker and playback rate of reverberation-induced speech on speech intelligibility of older adults
Authors: Nao Hodoshima
Venue: Interspeech 2024 (main)

Abstract: Speech intelligibility in noise and reverberation is generally lower for older adults (OAs) than for young adults (YAs). One solution might be the Lombard effect or reverberation-induced speech. This study investigates whether the intelligibility of reverberation-induced speech depends on talkers and playback rates for OAs. Four YAs recorded sentences under quiet (Q) and reverberation (R). In R, the reverberant speech was fed back to the talkers via headphones. The playback rate was 80% (slow), 100% (original), and 120% (fast). Twenty-four OAs carried out word identification tests in reverberation. The results showed that R was significantly more intelligible than Q for two of the four talkers. Slow speech was significantly more intelligible than original and fast speech, and original speech was significantly more intelligible than fast speech. The results suggest that reverberation-induced speech might increase the intelligibility of public-address announcements for OAs.
Age-related Differences in Acoustic Cues for the Perception of Checked Syllables in Shengzhou Wu
Authors: Bingliang Zhao; Jiangping Kong; Xiyu Wu
Venue: Interspeech 2024 (main)

Abstract: The transition from checked to unchecked syllables is a crucial phenomenon of sound change in Wu Chinese. This study aims to discover the underlying mechanisms of this evolution, specifically by investigating perceptual age-related differences in Shengzhou Wu. After conducting acoustic and articulatory analyses of checked syllables, an identification task was performed to assess the perceptual importance of F0, vowel duration, and the glottal stop coda in two generations. The results show that older individuals tend to rely more on the glottal stop coda when identifying checked syllables, whereas youngsters place more perceptual weight on vowel duration. Additionally, younger people have a reduced ability to recognize checked syllables, potentially due to their exposure to standard Mandarin. These findings suggest that the glottal stop coda may weaken and disappear before the lengthening of vowel duration in the evolution of checked syllables in Wu Chinese.
Quantity-sensitivity affects recall performance of word stress
Authors: Constantijn Kaland; Maria Lialiou
Venue: Interspeech 2024 (main)

Abstract: Previous studies showed that word stress patterns need to be lexically stored when they are highly variable. Hence, listeners are better at memorizing stress patterns in recall tasks when their native language has variable stress patterns than when their native language has fixed stress patterns. The current study hypothesizes that in quantity-sensitive stress languages, segmental information facilitates stress perception and therefore makes smaller demands on lexical storage than in languages with quantity-insensitive stress patterns. This prediction is tested in two stress recall tasks with German (var. stress, q-sensitive) and Greek (var. stress, q-insensitive) listeners. Results show that German listeners indeed perform worse than Greek listeners. The outcomes provide an important novel perspective on the interaction between segmental and suprasegmental information in speech perception and nuance lexical statistics accounts of word stress.
Phonological Symmetry Does Not Predict Generalization of Perceptual Adaptation to Vowels
Authors: Zuheyra Tokac; Jennifer Cole
Venue: Interspeech 2024 (main)

Abstract: Speech perception is inherently adaptive, with context-dependent rather than fixed perceptual phoneme boundaries. Exposure to novel vowel variants in lexically biasing contexts induces lasting boundary shifts toward the novel variant. Studies have found mixed results as to whether perceptual adaptation generalizes to phonologically related vowels and suggest that phonological symmetry might predict generalization. We test this hypothesis in Turkish, which has a fully symmetrical 8-vowel inventory. Listeners were exposed to words with either lowered /i/s or raised /É/s and identified vowels on /i-e/ and /u-o/ continua to assess perceptual adaptation and generalization. We found perceptual adaptation to only lowered /i/s in /i-e/ identification and no generalization of perceptual adaptation in /u-o/ identification. We argue that phonological symmetry might not be sufficient for generalization, and that vowel inventory size and organization might also play a role.
Perceptual Learning in Lexical Tone: Phonetic Similarity vs. Phonological Categories
Authors: AriÃ«lle Reitsema; Chenxin Li; Leanne van Lambalgen; Laura Preining; Saskia Galindo Jong; Qing Yang; Xinyi Wen; Yiya Chen
Venue: Interspeech 2024 (main)

Abstract: In speech comprehension, listeners recalibrate their interpretation of variable speech signals through exposure and disambiguating information. Recalibration is attested both segmentally and suprasegmentally, but little is known about what constrains it in lexical tone. This project investigated the effects of phonological categories and phonetic similarity on perceptual learning. We exposed Chinese listeners to pitch contours ambiguous between two tone categories (realised with a level or rising pitch contour) and lexically biased their perception to one interpretation. Crucially, the rising pitch contour could be from two different phonological tone categories. Perceptual learning was observed not only in the rising contours used for exposure but also across phonetically similar but phonologically different rising contours, suggesting that perceptual learning in tone is not constrained by phonological tone categories but is facilitated by the phonetic similarity of pitch contours.
Modeling probabilistic reduction across domains with Naive Discriminative Learning
Authors: Anna Stein; Kevin Tang
Venue: Interspeech 2024 (main)

Abstract: The predictability of a word modulates its acoustic duration. Such probabilistic effects can compete across linguistic domains (segments, syllables and adjacent-word contexts e.g., frequent words with infrequent syllables) and across local and aggregate contexts (e.g., a generally unpredictable word in a predictable context). This study aims to tease apart competing effects using Naive Discriminative Learning, which incorporates cue competition. The model was trained on English conversational speech from the Buckeye Corpus, using words as outcomes and segments, syllables, and adjacent words as cues. The connections between cues and outcomes were used to predict acoustic word duration. Results show that a word's duration is more strongly predicted by its syllables than its segments, and  a word's predictability aggregated over all contexts is a stronger predictor than its specific local contexts. Our study presents a unified approach to modeling competition in probabilistic reduction.
Do we EXPECT TO find phonetic traces for syntactic traces?
Authors: Jonathan Him Nok Lee; Mark Liberman; Martin Salzmann
Venue: Interspeech 2024 (main)

Abstract: In syntactic theories, the existence of silent/inaudible syntactic elements, such as movement traces, has been hypothesized to impact phonetic outcomes. One classic example involves the contraction of âwant toâ into âwanna,â where intervening syntactic traces are thought to prevent such contraction. This study extends this inquiry to âexpect to,â a similar construction but without established morpho-phonological contractions, and investigates whether there are any phonetic effects of the presence of traces. Drawing on findings from wanna- contraction, syntactic theories predict that intervening traces will hinder phonetic reduction in âexpect to.â To investigate, we randomly sampled 300 utterances containing âexpect toâ from a corpus of NPR podcasts. Multivariate linear regression shows no relationship between the presence of intervening syntactic traces and the duration of any phones in âexpect to.â However, multinomial logistic regression showed that the lenition of âexpect toâ (âtoâ /tu:/ â [tÊ] or [tÉª]) was significantly more likely to occur across intervening syntactic traces. Our findings contradict the predictions posited by syntactic theories, suggesting a more intricate interplay between syntax and phonetics.