GenAI4ELab/papercli-papers huggingface dataset:
Adapting a ConvNeXt Model to Audio Classification on AudioSet
Authors: Thomas Pellegrini; Ismail Khalfaoui-Hassani; Etienne LabbÃ©; TimothÃ©e Masquelier
Venue: Interspeech 2023 (main)

Abstract: In computer vision, convolutional neural networks (CNN) such as ConvNeXt, have been able to surpass state-of-the-art transformers, partly thanks to depthwise separable convolutions (DSC). DSC, as an approximation of the regular convolution, has made CNNs more efficient in time and memory complexity without deteriorating their accuracy, and sometimes even improving it. In this paper, we first implement DSC into the Pretrained Audio Neural Networks (PANN) family for audio classification on AudioSet, to show its benefits in terms of accuracy/model size trade-off. Second, we adapt the now famous ConvNeXt model to the same task. It rapidly overfits, so we report on techniques that improve the learning process. Our best ConvNeXt model reached 0.471 mean-average precision on AudioSet, which is better than or equivalent to recent large audio transformers, while using three times less parameters. We also achieved positive results in audio captioning and audio retrieval with this model.
Few-shot Class-incremental Audio Classification Using Stochastic Classifier
Authors: Yanxiong Li; Wenchang Cao; Jialong Li; Wei Xie; Qianhua He
Venue: Interspeech 2023 (main)

Abstract: It is generally assumed that number of classes is fixed in current audio classification methods, and the model can recognize pre-given classes only. When new classes emerge, the model needs to be retrained with adequate samples of all classes. If new classes continually emerge, these methods will not work well and even infeasible. In this study, we propose a method for few-shot class-incremental audio classification, which continually recognizes new classes and remember old ones. The proposed model consists of an embedding extractor and a stochastic classifier. The former is trained in base session and frozen in incremental sessions, while the latter is incrementally expanded in all sessions. Two datasets (NS-100 and LS-100) are built by choosing samples from audio corpora of NSynth and LibriSpeech, respectively. Results show that our method exceeds four baseline ones in average accuracy and performance dropping rate. Code is at https://github.com/vinceasvp/meta-sc.
Enhance Temporal Relations in Audio Captioning with Sound Event Detection
Authors: Zeyu Xie; Xuenan Xu; Mengyue Wu; Kai Yu
Venue: Interspeech 2023 (main)

Abstract: Automated audio captioning aims at generating natural language descriptions for given audio clips, not only detecting and classifying sounds, but also summarizing the relationships between audio events. Recent research advances in audio captioning have introduced additional guidance to improve the accuracy of audio events in generated sentences. However, temporal relations between audio events have received little attention while revealing complex relations is a key component in summarizing audio content. Therefore, this paper aims to better capture temporal relationships in caption generation with sound event detection (SED), a task that locates events' timestamps. We investigate the best approach to integrate temporal information in a captioning model and propose a temporal tag system to transform the timestamps into comprehensible relations. Results evaluated by the proposed temporal metrics suggest that great improvement is achieved in terms of temporal relation generation.
First Language Effects on Second Language Perception: Evidence from English Low-vowel Nasal Sequences Perceived by L1 Mandarin Chinese Listeners
Authors: Sijia Zhang
Venue: Interspeech 2023 (main)

Abstract: First language (L1) sound systems can shape second language (L2) perception of non-native phonological contrasts. This study examines how L1 Mandarin listeners perceive English low vowel + nasal (VN) sequences that are not contrastive in Mandarin. A speeded AX discrimination task tested listeners' low-level processing of English VN sequences, and a perceptual similarity rating task assessed listeners' higher-level phonological knowledge. Despite high discrimination accuracy, reaction times showed that L1 Mandarin listeners had a harder time processing nasal-different sequences compared to L1 English listeners. Moreover, L1 Mandarin listeners perceived vowel-different sequences as more distinct than English listeners. Overall, listeners' L1 phonological system influences L2 perception, suggested by group-level differences in both perceived phonological distinctiveness and phonetic discrimination.
Motor Control Similarity Between Speakers Saying âA Soukâ Using Inverse Atlas Tongue Modeling
Authors: Ursa Maity; Fangxu Xing; Jerry Prince; Maureen Stone; El Fakhri Georges; Jonghye Woo; Sidney Fels
Venue: Interspeech 2023 (main)

Abstract: Finite element models (FEM) of the tongue have facilitated speech studies through analysis of internal muscle forces indirectly derived from imaging data. In this work, we build a uniform hexahedral FEM of a tongue atlas constructed from magnetic resonance imaging data of a healthy population. The FEM is driven by inverse internal tongue tissue kinematics of speakers temporally aligned and deformed into the same atlas space, while performing the speech task âa soukâ allowing muscle activation predictions. This work aims to investigate the commonalities in tongue motor strategies in the articulation of âa soukâ predicted by the inverse tongue atlas model. Our findings report variability among five speakers for estimated muscle activations with a similarity index using a dynamic time warp function. Two speakers show similarity index > 0.9 and two others < 0.7 with respect to a reference speaker for most tongue muscles. The relative motion tracking error of the model is less than 2% which is promising for speech study applications.
Assessing Phrase Break of ESL Speech with Pre-trained Language Models and Large Language Models
Authors: Zhiyi Wang; Shaoguang Mao; Wenshan Wu; Yan Xia; Yan Deng; Jonathan Tien
Venue: Interspeech 2023 (main)

Abstract: This work introduces approaches to assessing phrase breaks in ESL learners' speech using pre-trained language models (PLMs) and large language models (LLMs). There are two tasks: overall assessment of phrase break for a speech clip and fine-grained assessment of every possible phrase break position. To leverage NLP models, speech input is first force-aligned with texts, and then pre-processed into a token sequence, including words and phrase break information. To utilize PLMs, we propose a pre-training and fine-tuning pipeline with the processed tokens. This process includes pre-training with a replaced break token detection module and fine-tuning with text classification and sequence labeling. To employ LLMs, we design prompts for ChatGPT. The experiments show that with the PLMs, the dependence on labeled training data has been greatly reduced, and the performance has improved. Meanwhile, we verify that ChatGPT, a renowned LLM, has potential for further advancement in this area.
A Relationship Between Vocal Fold Vibration and Droplet Production
Authors: Tsukasa Yoshinaga; Takayuki Arai; Akiyoshi Iida
Venue: Interspeech 2023 (main)

Abstract: While some aerosol droplets causing airborne transmissions are argued to be produced by vocal fold vibrations, the detailed production mechanisms were unclear due to the difficulty of direct observation. In this study, by using a transparent acrylic vocal fold model and high-speed imaging, we observed vocal fold vibrations that produce droplets of artificial mucus to clarify the relationship between the vocal fold vibration and the droplet production. The vocal fold model was set on a lung model which has a manual diaphragm. The artificial mucus in between the vocal folds was lighted up by a laser sheet. The results showed that droplets were produced when the sound amplitudes were decreased or the fundamental frequency became unstable. We observed that mucus drops attached to the middle of the vocal fold wall splashed and formed a droplet that flew from the vocal fold wall. This suggests that the shear forces of turbulent airflow passing on the mucus mainly produced the droplets.
Audio, Visual and Audiovisual intelligibility of vowels produced in noise
Authors: Maeva Garnier
Venue: Interspeech 2023 (main)

Abstract: Why do speakers amplify articulatory movements when communicating in noisy environments? This study examines the hypothesis that hyper-articulation contributes to improved vowel intelligibility in audio, visual and audiovisual domains. A perceptual test was conducted with Audio-Only (AO), Visual-Only (VO) and Audiovisual (AV) stimuli of vowels produced in conversational and Lombard speech. The average score of vowel recognition was significantly increased in Lombard speech, compared to normal speech, for all perceptual modalities (AO, VO and AV). Specifically, the distinctive features of vowel height and backness were better perceived in Lombard speech in both the audio and visual domains. Changes in speech articulation in noise did not affect the perception of the rounding feature in the visual domain, but degraded it in the audio domain. On the contrary, the perception of the spreading feature was decreased in Lombard speech in the visual domain, but improved in the audio domain.
Optimal control of speech with context-dependent articulatory targets
Authors: Benjamin Elie; Juraj Å imko; Alice Turk
Venue: Interspeech 2023 (main)

Abstract: This paper presents a computational implementation of phonetic planning which consists of choosing the position of articulatory targets which satisfy conflicting linguistic and extra-linguistic requirements. We present a minimal model that considers intelligibility and least effort as task requirements. To achieve the context-dependent variability of targets, our model approximates intelligibility as a function of target phoneme recognition probability given a vector of articulatory parameters. Preliminary experiments show that our minimal computational model of phonetic planning is able to predict two types of hypoarticulation by adjusting the weight assigned to effort: vowel centralization and stop consonant lenition.
Computational modeling of auditory brainstem responses derived from modified speech
Authors: Tzu-Han Zoe Cheng; Paul Calamia
Venue: Interspeech 2023 (main)

Abstract: The auditory brainstem response (ABR) is a powerful neurophysiological measure to diagnose hearing deficits along the auditory pathway. Wave I of the ABRs is particularly critical for assessing early hearing loss, though hard to observe in humans. The major downside of ABR is that most protocols are very boring since they use thousands of clicks to elicit ABRs. Here, we derived modeled ABRs with continuous speech from an audiobook. Unlike other studies involving computationally intensive modification that made their speech stimuli unnatural-sounding and unlikely to be used in real-life applications, we applied a fast and efficient algorithm that enhances speech transients to better elicit ABRs. Using the auditory periphery model that simulates human brains, we derived ABRs from our transient speech and showed a significantly larger Wave I-V ratio compared to other stimuli. These results demonstrated a potential of assessing hearing conditions in a more objective and naturalistic way.
Leveraging Label Information for Multimodal Emotion Recognition
Authors: Peiying Wang; Sunlu Zeng; Junqing Chen; Lu Fan; Meng Chen; Youzheng Wu; Xiaodong He
Venue: Interspeech 2023 (main)

Abstract: Multimodal emotion recognition (MER) aims to detect the emotional status of a given expression by combining the speech and text information. Intuitively, label information should be capable of helping the model locate the salient tokens/frames relevant to the specific emotion, which finally facilitates the MER task. Inspired by this, we propose a novel approach for MER by leveraging label information. Specifically, we first obtain the representative label embeddings for both text and speech modalities, then learn the label-enhanced text/speech representations for each utterance via label-token and label-frame interactions. Finally, we devise a novel label-guided attentive fusion module to fuse the label-aware text and speech representations for emotion classification. Extensive experiments were conducted on the public IEMOCAP dataset, and experimental results demonstrate that our proposed approach outperforms existing baselines and achieves new state-of-the-art performance.
Improving End-to-End Modeling For Mandarin-English Code-Switching Using Lightweight Switch-Routing Mixture-of-Experts
Authors: Fengyun Tan; Chaofeng Feng; Tao Wei; Shuai Gong; Jinqiang Leng; Wei Chu; Jun Ma; Shaojun Wang; Jing Xiao
Venue: Interspeech 2023 (main)

Abstract: Code-switching is a common phenomenon in multilingual communities. In this paper, we study end-to-end model for Mandarin-English intra-sentential code-switching speech recognition. A lightweight Switch-Routing network is proposed, which includes two experts and a switch router. Two experts, representing Mandarin and English learners, implicitly provide language identification information and skillfully use monolingual data to assist code-switching task training, which solves the problem of data sparsity. In addition, our network is a lightweight structure, which makes use of the advantages of Switch Transformer and discards its weakness of increasing model capacity. Finally, we study the effect of using lightweight Switch Routing in different blocks of encoder and decoder. Compared with Bi-Encoder, proposed model has a better performance on the ASRU code-switching test set, and the most important thing is that it requires much less inference time with RTF decreasing by 31.39%.
Frequency Patterns of Individual Speaker Characteristics at Higher and Lower Spectral Ranges
Authors: Zhao Zhang; Ju Zhang; Ziyu Zhu; Yujie Chi; Kiyoshi Honda; Jianguo Wei
Venue: Interspeech 2023 (main)

Abstract: Acoustic characteristics of speech exhibit variability across individuals, while preserving shared phonetic information to listeners. In this paper, the general time-frequency pattern of individual speaker characteristics is discussed based on our previous research. The main target here is set at speaker-specific acoustic effects of the vocal tract in both higher and lower frequency ranges. To address the under-explored phenomena, two experiments were conducted. Firstly, simulations based on the transmission line model are used to explore how resonances in higher frequencies vary with different hypopharyngeal-cavity shapes. Secondly, speech signals emitted from the mouth and nostrils are recorded separately to observe potential factors for spectral irregularity in lower frequencies. From our findings, a time-frequency model of individual speaker characteristics is proposed that provides insights into how individuality is manifested in speech spectral patterns.
Adaptation to predictive prosodic cues in non-native standard dialect
Authors: Sabine Gosselke Berthelsen
Venue: Interspeech 2023 (main)

Abstract: Predicting word or sentence structure from prosodic cues (e.g., pitch, pauses, or duration) is an important mechanism in speech processing. The mechanism is readily employed by native listeners but considerably less so by second language learners except with extensive training. This prompts the question of how flexible and adaptive the predictive system is. Can listeners adjust to accommodate, for example, diverging predictive cues in a different dialect? The present paper tests this for prosody-based word structure prediction in mainland Scandinavian. Neurophysiological and behavioural results suggest that listeners from non-standard dialect areas can in fact predict word forms in the standard variety even when the predictive cues and their validity differ considerably. This suggests a certain degree of adaptability in the predictive system, potentially depending on factors like familiarity, prestige, and exposure onset.
Head movements in two- and four-person interactive conversational tasks in noisy and moderately reverberant conditions
Authors: Alan Archer-Boyd; Rainer Martin
Venue: Interspeech 2023 (main)

Abstract: Multi-modal processing schemes are of increasing importance for adaptive hearing devices. However, more data is required to understand interactions in complex application scenarios. In this study, the speech and head movements of eight normal-hearing participants were recorded in two- and four-person interactive conversational tasks, with and without 4-talker babble noise at 75 dB(A) and reverberation times of 0.25 and 0.6 s. Two-person conversations showed a head movement (yaw) interquartile range of 11.6Â° while four-person conversations showed a statistically significantly different interquartile range of 21.9Â°. No effect of acoustic condition was observed. The recorded data were also successfully used to test a previously published hearing-device direction of arrival estimation algorithm that utilized head movement information and correlation lag between acoustic signals from the left and right ear.
Second language identification of Vietnamese tones by native Mandarin learners
Authors: Juqiang Chen; Ailing Qin; Hui Chang; Hua Chen
Venue: Interspeech 2023 (main)

Abstract: In this study, we examined native language phonological and phonetic factors in identification of second language (L2) tones by learners with different learning experience. Results show that when L2 tones were Categorised with high percent choice and goodness ratings into native categories, it can be accurately identified. Uncategorised L2 tones tend to be mis-identified as other L2 tones that were assimilated into overlapping native response categories. L2 tones with unique phonetic characteristics can be easily identified even though they were Uncategorised. Thus, our research has theoretical implications for L2 speech learning models and ramifications for L2 lexical tone teaching and learning.
Nasal vowel production and grammatical processing in French-speaking children with cochlear implants and normal-hearing peers.
Authors: Sophie Fagniart; VÃ©ronique Delvaux; Brigitte Charlier; Bernard Harmegnies; Anne Huberlant; Myriam Piccaluga; Kathy Huet
Venue: Interspeech 2023 (main)

Abstract: Our study investigates morphemic receptive skills, quality of phonological representations, and production abilities of the acoustic nasal feature in French, a language with nasal vocalic phonemes. We examine these skills in two groups: children with cochlear implants (CI group) and their normal hearing peers (NH group). The results reveal weaker receptive skills and more phonological errors in production among the CI group. Additionally, the CI group shows a specific use of different acoustic cues associated with vowel nasality, suggesting a perceptual-productive profile that focuses more on perceptually salient cues. This profile may be related to language skills, as the use of subtle acoustic cues is associated with better morphemic processing skills.
Emotion Classification with EEG Responses Evoked by Emotional Prosody of Speech
Authors: Zechen Zhang; Xihong Wu; Jing Chen
Venue: Interspeech 2023 (main)

Abstract: Emotion classification with EEG responses can be used in human-computer interaction, security, medical treatment, etc. Neural responses recorded via EEG can reflect more direct and objective emotional information than other behavioral signals (i.e., facial expression...). In most previous studies, only features of EEG were used as input for machine learning models. In this work, we assumed that the emotional features included in speech stimuli could assist in emotion recognition with EEG when the emotion is evoked by the emotional prosody of speech. An EEG data corpus was collected with specific speech stimuli, in which emotion was represented with only speech prosody and without semantic context. A novel EEG-Prosody CRNN model was proposed to classify four types of typical emotions. The classification accuracy can achieve at 82.85% when the prosody features of speech were integrated as input, which outperformed most audio-evoked EEG-based emotion classification methods.
L2-Mandarin regional accent variability during Mandarin tone-word training facilitates English listenersâ subsequent tone categorizations
Authors: Yanping Li; Michael D. Tyler; Denis Burnham; Catherine T. Best
Venue: Interspeech 2023 (main)

Abstract: We examined how accent variability during training on minimal-tone-contrast Mandarin words affects English listeners' subsequent generalization of tone categorization and discrimination to new talkers and accents. English listeners underwent 6 days of training on 16 pseudowords (4 tones 4 sets) produced by 12 talkers of either Beijing (n = 24) or a mix of Beijing, Yantai, and Guangzhou (n = 24) accents. In a post-training test, they used tone contour icons to categorize the tones produced by new talkers with a familiar and an unfamiliar accent, and to discriminate all possible tone contrasts for the new talkers. While both training groups discriminated all six tone contrasts for both new talkers, only the multiple accent group reliably categorized all four tones. The single accent group failed to correctly categorize the falling tone in both generalization tests. These results suggest that accent variability during tone-word training can facilitate subsequent tone categorization.
HumanDiffusion: diffusion model using perceptual gradients
Authors: Yota Ueda; Shinnosuke Takamichi; Yuki Saito; Norihiro Takamune; Hiroshi Saruwatari
Venue: Interspeech 2023 (main)

Abstract: We propose HumanDiffusion, a diffusion model trained from humans' perceptual gradients to learn an acceptable range of data for humans (i.e., human-acceptable distribution). Conventional HumanGAN aims to model the human-acceptable distribution wider than the real-data distribution by training a neural network-based generator with human-based discriminators. However, HumanGAN training tends to converge in a meaningless distribution due to the gradient vanishing or mode collapse and requires careful heuristics. In contrast, our HumanDiffusion learns the human-acceptable distribution through Langevin dynamics based on gradients of human perceptual evaluations. Our training iterates a process to diffuse real data to cover a wider human-acceptable distribution and can avoid the issues in the HumanGAN training. The evaluation results demonstrate that our HumanDiffusion can successfully represent the human-acceptable distribution without any heuristics for the training.
Queer Events, Relationships, and Sports: Does Topic Influence Speakersâ Acoustic Expression of Sexual Orientation?
Authors: Sven Kachel; Manuel PÃ¶hlmann; Christine Nussbaum
Venue: Interspeech 2023 (main)

Abstract: Studies on the acoustic parameterization of actual and perceived sexual orientation yielded inconclusive findings. One reason for this could be different linguistic and situational factors underlying these studies. In the present research, we aim to illuminate inconsistent findings by systematically varying the way sexual orientation was made salient in interview topics: Lesbian/gay and straight women and men (n = 72) were asked to answer questions referring to lesbian/gay issues, to their own sexual orientation, and to a non-sexual orientation topic. Applying a person perceptions approach that provides a holistic measure for phonetic variation across topics, raters (n = 35) were asked to judge speakers' sexual orientations. Overall, straight speakers were rated as straighter than lesbian/gay speakers. Contrary to expectations, this difference was largest in the control condition. Results are discussed in terms of the same topic having differential effects on different speaker groups.