GenAI4ELab/papercli-papers huggingface dataset:
The effect of masking noise on listenersâ spectral tilt preferences
Authors: Olympia Simantiraki; Yannis Pantazis; Martin Cooke
Venue: Interspeech 2023 (main)

Abstract: Speech enhancement algorithms often focus on optimising intelligibility while neglecting other aspects of speech such as naturalness, quality and listening effort which may affect a listener's experience. This paper investigates the impact of spectral tilt on listeners' preferences, using a new corpus of Greek utterances. Participants adjusted spectral tilt with real-time feedback to select their preferred tilt in quiet and in the presence of speech-shaped noise at eight signal-to-noise ratios. Listeners displayed distinct preferences, with a tendency to select flatter tilts with increasing noise. Preferences were not random even for constant intelligibility, indicating that their adjustments were influenced by factors beyond the need to maintain comprehensibility. These findings have the potential to inform the design of speech enhancement algorithms that jointly optimise intelligibility and a listener's overall experience.
The Effect of Whistled Vowels on Whistled Word Categorization for Naive Listeners
Authors: Anais Tran Ngoc; Fanny Meunier; Julien Meyer
Venue: Interspeech 2023 (main)

Abstract: In this paper, we explore whistled word perception by naive French speakers. In whistled words of non-tonal languages, vowels are transposed to relatively stable pitches, which contrast with consonant movements or interruptions. Previous studies on whistled speech with naive listeners have tested vowels and consonants separately. Other studies on spoken word recognition have found that vowels and consonants contribute differently to intelligibility, where the role of vowels was highly mediated by the context. Here, naive participants recognize disyllabic whistled words above chance, and vowels are shown to contribute differently than consonants. When focusing on the role of vowels, we found different scales of performance between the vowels tested, mediated by their position in the word. We also highlighted the importance of the vowels' relative frequency difference (called 'interval') in the word.
Automatic Deep Neural Network-Based Segmental Pronunciation Error Detection of L2 English Speech (L1 Bengali)
Authors: Puja Bharati; Sabyasachi Chandra; Shayamal Kumar Das Mandal
Venue: Interspeech 2023 (main)

Abstract: In the last few decades, English has become a popular language as it helps us to communicate with the global world. A large population of English learners find it challenging to achieve an 'acceptable' and 'intelligible' pronunciation. To overcome these issues, various computer-assisted pronunciation training tools are designed where automatic pronunciation error detection (APED) is a core component of the system. Most of the works of APED are based on European English speech, but there is no such work reported for Bengali English speech. This paper proposes a system for pronunciation error detection of L2 English speech (L1 Bengali) at phoneme/segmental level using a hybrid convolutional neural network and long short-term memory modules with CTC loss. Experiments are done based on newly created L2 English speaker (L1 Bengali) speech data. The results demonstrate that the proposed system outperforms the goodness of pronunciation-based methods by 15% in terms of F1 score using fbank.
The effect of stress on Mandarin tonal perception in continuous speech for Spanish-speaking learners
Authors: Lixia Hao; Qi Gong; Jinsong Zhang
Venue: Interspeech 2023 (main)

Abstract: The perception of lexical tones is a well-known challenge for L2 learners especially in continuous speech where the tonal variations are complicated. To investigate whether the stress, by affecting the acoustic manifestations of tones, has an effect on Mandarin tonal perception for L2 learners, we carried out a perceptual experiment based on the Annotated Speech Corpus of Chinese Discourse with 25 native Spanish-speaking participants. The results indicate that: the perceptual accuracy of stressed tones is significantly higher than that of the unstressed ones in general; T3 is the most difficult one to be perceived among the four Mandarin tones (T1-T4) in both stressed and unstressed syllables, and presumably the Spanish-speaking learners' perceptual order of Mandarin tones is T4-T1-T2-T3; the significant interactive effect found between tone and tonal context in continuous speech may lead to great confusion of tonal perception, especially when T2 and T3 are adjacent with one another.
Combining acoustic and aerodynamic data collection: A perceptual evaluation of acoustic distortions
Authors: AmÃ©lie Elmerich; Jiayin Gao; Angelique Amelot; Lise Crevier-Buchman; Shinji Maeda
Venue: Interspeech 2023 (main)

Abstract: Combining acoustic and aerodynamic data acquisitions is challenging. Devices for aerodynamic measurements often create severe acoustic distortions, which make it impossible to analyse the simultaneously recorded acoustic data. An improved technique, a pneumotachograph mask made of synthetic fibers, is acoustically transparent while ensuring a high-quality aerodynamic data acquisition. A previous acoustic study confirms the minimal acoustic distortions caused by this technique. The present study evaluates the impact of different aerodynamic devices on the human perception of vowels. Results show that vowels recorded with a fiber mask are almost as accurately categorised as acoustic-only recordings, compared with rigid masks that result in perceptual confusions. Listeners are also less likely to perceive the presence of a mask. Overall, our study provides the perceptual validation of the fiber mask technique, which will be of a great value in the field of speech sciences.
Estimating virtual targets for lingual stop consonants using general Tau theory
Authors: Benjamin Elie; Alice Turk
Venue: Interspeech 2023 (main)

Abstract: This paper investigates the existence and position of virtual targets during the production of stop consonants. Using the equations from general Tau theory to model the time-course of tongue constriction formation movements, targets were estimated by fitting these equations on observed tongue constriction variables extracted from real EMA data from 2 native speakers of English. Results suggest that targets are virtual for 50 to 60% of movements. For these movements, virtual targets of the tongue tip constriction are predicted to occur around 0.1 cm beyond the palate, and virtual targets for the tongue dorsum constriction are predicted to occur between 0.05 and 0.2 cm beyond the palate. Our results suggest that the time-course of movement is planned so that the onset of closure occurs with relatively high velocity: closure onset is generally located very close in time to the time of peak velocity.
Using Random Forests to classify language as a function of syllable timing in two groups: children with cochlear implants and with normal hearing
Authors: Mark Gibson
Venue: Interspeech 2023 (main)

Abstract: We trained a series of Random Forest models in a supervised learning environment on different temporal parameters related to syllable structure: voice onset time (VOT), vowel duration following simplex and complex onsets, and lateral duration in word initial (/lV) position and as the second consonant in a C1C2 cluster (where C means consonant). Capitalizing on previous work we trained the models on data from monolingual Spanish- and English-speaking adults. We asked whether the timing productions used by bilingual children with normal hearing (NH) and children with cochlear implants (CI) can be classified as pertaining to the same timing system (i.e. language), or whether the children are applying the same basic timing plan to two different languages. We also asked whether there were differences between the CI and NH groups. Our results indicate that the children from both groups produce qualitatively distinct timing plans for each language with no interference from the other language.
An Improved End-to-End Audio-Visual Speech Recognition Model
Authors: Sheng Yang; Zheng Gong; Jia Kang
Venue: Interspeech 2023 (main)

Abstract: By incorporating lip language, audio-visual speech recognition can effectively improve the recognition effect in noisy environments, and will slightly improve the recognition effect in quiet environments. we use a frequency domain attention based residual network (Fca-Net) as the model of the vision front-end module, which extracts more features that are helpful to the AVSR and VSR system at a small cost. And use the powerful speech pre-training model Hu-BERT as the recognition front-end model of ASR. We compare the impact of different model as visual back-end modules and fusion modules on the AVSR system. Our experiments show that the model selection of the fusion module is critical to the performance of the AVSR system. Ultimately, our proposed model achieves state-of-the-art results on audio-visual speech recognition tasks using the LRS2 dataset.
What influences the foreign accent strength? Phonological and grammatical errors in the perception of accentedness
Authors: Sarah WesoÅek; Piotr Gulgowski; Joanna BÅaszczak; Marzena Å»ygis
Venue: Interspeech 2023 (main)

Abstract: The present study investigates the influence of grammatical and phonological errors on the perceived degree of foreign accent strength. German and Polish participants listened to speech in their native language produced with foreign and native accent. They rated the accent strength of each sentence on a 7-point scale. Grammatical errors consisted of gender agreement violations and phonological errors consisted of controlled vowel substitutions. Both error types significantly affected the perception of accent strength in the foreign and native-accented condition. In Polish, phonological anomalies had significantly more impact than grammatical violations in native-accented sentences. In German, there was no significant difference between phonological and grammatical violations. The study provides evidence that the presence of phonological and grammatical errors increases the perceived accentedness of speech. The weighting of both errors for accent perception can vary between languages.
Investigating the Perception Production Link through Perceptual Adaptation and Phonetic Convergence
Authors: Lena-Marie Huttner; NoÃ«l Nguyen; Martin J. Pickering
Venue: Interspeech 2023 (main)

Abstract: Speech perception and production may be linked. In this pilot study, we aim to investigate the nature of this link by examining how perception and production of VOT adjust in social interaction. In an online experiment with 2x2 conditions participants were asked to categorize nine stimuli on a VOT continuum between /tin/ and /din/ and produce the words din and tin before and after playing a game with a bot. During the game participants were trained on a sound to word correspondence with a bias either towards /din/ or /tin/. In two conditions participants alternated categorizing and producing the stimuli, while in the other two participants only categorized the stimuli. Significant differences in categorization between conditions occurred only during the game. Whereas significant changes in VOT can be observed between pre and post-test for three conditions. This could mean that perception and production do not adjust symmetrically over the course of an interaction.
Emotion Prompting for Speech Emotion Recognition
Authors: Xingfa Zhou; Min Li; Lan Yang; Rui Sun; Xin Wang; Huayi Zhan
Venue: Interspeech 2023 (main)

Abstract: Speech Emotion Recognition (SER) classifies speech into emotion categories such as: Happy, Angry. Most prior works for SER focused on how to mine compelling features to improve performance. However, these methods ignore the influence of emotional label information on SER. Recent studies have attempted to prompt pre-trained language models and yield good performance for NLP tasks. Nevertheless, few works have attempted to prompt pre-trained speech models (PSM) on speech tasks. In light of these, we propose a simple but effective prompt-based method that prompts PSM for SER. Firstly, we reframe SER as an entailment task. Next, we generate speech prompts and combine them with the raw audio to form the input for PSM. Finally, we build a multi-task learning framework to extract more compelling features by simultaneously performing automatic speech recognition (ASR) and SER. Experiments on the IEMOCAP benchmark show that our method outperforms state-of-the-art baselines on the SER task.
Speech-in-Speech Recognition is Modulated by Familiarity to Dialect
Authors: Jessica L. L. Chin; Elena Talevska; Mark Antoniou
Venue: Interspeech 2023 (main)

Abstract: Listening to speech in competing background speech can be difficult due to elements such as the linguistic content of the signal. Linguistic release from masking occurs when altering the masker language results in less interference in speech recognition. The greater the linguistic differences between the target and masker, the higher the speech recognition accuracy. However, for dialectal variations of the same language, these patterns are less consistent. This study examined speech-in-speech recognition in Australian English monolinguals when the target speech was in either Australian (AU) or American English, and when the masker speech was in either of the dialects or a foreign language (Swedish). Speech recognition performance was greatest when AU was the target and poorest when AU was the masker. There were fewer differences in performance between the Swedish and dialect maskers. Results indicate that speech recognition is modulated by a listener's familiarity to a dialect.
BASEN: Time-Domain Brain-Assisted Speech Enhancement Network with Convolutional Cross Attention in Multi-talker Conditions
Authors: Jie Zhang; QingTian Xu; Qiu-Shi Zhu; Zhen-Hua Ling
Venue: Interspeech 2023 (main)

Abstract: Time-domain single-channel speech enhancement (SE) still remains challenging to extract the target speaker without any prior information on multi-talker conditions. It has been shown via auditory attention decoding that the brain activity of the listener contains the auditory information of the attended speaker. In this paper, we thus propose a novel time-domain brain-assisted SE network (BASEN) incorporating electroencephalography (EEG) signals recorded from the listener for extracting the target speaker from monaural speech mixtures. The proposed BASEN is based on the fully-convolutional time-domain audio separation network. In order to fully leverage the complementary information contained in the EEG signals, we further propose a convolutional multi-layer cross attention module to fuse the dual-branch features. Experimental results on a public dataset show that the proposed model outperforms the state-of-the-art method in several evaluation metrics. The reproducible code is available at https://github.com/jzhangU/Basen.
Are retroflex-to-dental sibilant substitutions in Polish children's speech an example of a covert contrast? A preliminary acoustic study
Authors: Zuzanna Miodonska; Claartje Levelt; Natalia Mocko; MichaÅ KrÄcichwost; Agata Sage; Pawel Badura
Venue: Interspeech 2023 (main)

Abstract: The study aimed to investigate whether the dental substitutions of retroflex voiceless fricatives (/Ê/ to [s]) in Polish children's speech are an example of a covert contrast. We analyzed speech samples collected through a picture naming test from 11 children showing this retroflex-to-dental production pattern. The language material included words with /Ê/ and /s/ in diversified word positions. We extracted a set of spectrum-based acoustic features from the recorded sibilants and conducted the analysis using linear mixed-effect models. The models showed that significant acoustic differences (p < 0.05) can be found between realizations of /s/ and /Ê/ substituted by [s]. The main differences were detected in the amplitudes of fricative formants and the energy levels in specific subbands of the frication noise. The study provides preliminary evidence of the existence of covert contrasts in the analyzed substitutions.
Reversible Neural Networks for Memory-Efficient Speaker Verification
Authors: Bei Liu; Yanmin Qian
Venue: Interspeech 2023 (main)

Abstract: Training large-scale speaker verification systems on consumer GPUs is difficult due to the memory consumption of the existing networks being proportional to the number of layers. In this paper, a novel family of Reversible Neural Networks (RevNets) is proposed for memory-efficient speaker verification. Specifically, we introduce two types of RevNets, namely partially and fully reversible networks, which alleviate the need to store activations in memory during back-propagation. Consequently, RevNets require nearly constant memory costs as the network depth increases. Experiments on Voxceleb show that RevNets achieve up to 15.7x memory savings, while maintaining nearly identical parameters and performance when compared to the vanilla ResNets. To our knowledge, this is the first work to investigate memory-efficient training for speaker verification. Our results indicate the potential of reversible networks as a more efficient backbone for resource-limited training scenarios.
ECAPA++: Fine-grained Deep Embedding Learning for TDNN Based Speaker Verification
Authors: Bei Liu; Yanmin Qian
Venue: Interspeech 2023 (main)

Abstract: In this paper, we aim to bridge the performance gap between TDNN and 2D CNN based speaker verification systems. Specifically, three types of architectural enhancements to ECAPA-TDNN are proposed: 1) follow depth-first design to significantly increase network depth while maintaining its complexity. 2) introduce recursive convolution to better capture fine-grained speaker information. 3) propose pyramid-based multi-path feature enhancement module to yield more discriminative speaker representation. Experiments on Voxceleb show that our final model, named ECAPA++, achieves 25%, 23% and 24% relative improvements on Vox1-O, E and H respectively, while with 2.4x fewer parameters and 2.3x fewer FLOPs over the previous best TDNN-based system. Meanwhile, it is comparable to the state-of-the-art ResNet-based systems with higher computational efficiency. In addition, further performance gains can be achieved by fusing ECAPA++ and ResNet-based systems.
TO-Rawnet: Improving RawNet with TCN and Orthogonal Regularization for Fake Audio Detection
Authors: Chenglong Wang; Jiangyan Yi; Jianhua Tao; Chu Yuan Zhang; Shuai Zhang; Ruibo Fu; Xun Chen
Venue: Interspeech 2023 (main)

Abstract: Current fake audio detection relies on hand-crafted features, which lose information during extraction. To overcome this, recent studies use direct feature extraction from raw audio signals. For example, RawNet is one of the representative works in end-to-end fake audio detection. However, existing work on RawNet does not optimize the parameters of the Sinc-conv during training, which limited its performance. In this paper, we propose to incorporate orthogonal convolution into RawNet, which reduces the correlation between filters when optimizing the parameters of Sinc-conv, thus improving discriminability. Additionally, we introduce temporal convolutional networks (TCN) to capture long-term dependencies in speech signals. Experiments on the ASVspoof 2019 show that the Our TO-RawNet system can relatively reduce EER by 66.09% on logical access scenario compared with the RawNet, demonstrating its effectiveness in detecting fake audio attacks.
Fooling Speaker Identification Systems with Adversarial Background Music
Authors: Chu-Xiao Zuo; Jia-Yi Leng; Wu-Jun Li
Venue: Interspeech 2023 (main)

Abstract: Speaker identification (SI) systems are widely used in real-world scenarios but are vulnerable to attacks from malicious users. Although existing attacks mainly focus on speech-shaped inputs, SI models can also be broken by speech-unrelated background music (BGM) in practical use. In this paper, we propose a new attack, called BGM Attack (BGMA), that generates auditorily natural music to deceive SI models. BGMA integrates a music generation model and a SI model to modify the music-level semantic features. We propose a linear transform called differentiable spectrogram reconstruction (DSR) that acts as a bridge for conveying gradient information between the two models in BGMA. Our experiments show that BGMA can effectively break state-of-the-art SI models with generated auditorily natural music. The result of this paper highlights the need for SI models to be robust against attacks from non-speech inputs and provides a novel attack method for testing the security of SI systems.
Mutual Information-based Embedding Decoupling for Generalizable Speaker Verification
Authors: Jianchen Li; Jiqing Han; Shiwen Deng; Tieran Zheng; Yongjun He; Guibin Zheng
Venue: Interspeech 2023 (main)

Abstract: Domain shift is a challenging problem in speaker verification, especially when dealing with unseen target domains. Recently, embedding decoupling-based methods have shown their effectiveness. Typically, domain information is extracted by a domain classification loss and then decoupled from speaker embeddings. However, the domain classification loss fails to ensure that only domain information is encoded in domain embeddings. This paper proposes a novel mutual information-based embedding decoupling framework, in which the domain information is extracted by maximizing the mutual information between different speaker sample pairs in the same domain. Then the domain information is removed from speaker embeddings by minimizing mutual information between speaker and domain embeddings. Experiments indicate that our method can improve the generalization and outperform domain classification-based decoupling methods.
Target Active Speaker Detection with Audio-visual Cues
Authors: Yidi Jiang; Ruijie Tao; Zexu Pan; Haizhou Li
Venue: Interspeech 2023 (main)

Abstract: In active speaker detection (ASD), we would like to detect whether an on-screen person is speaking based on audio-visual cues. Previous studies have primarily focused on modeling audio-visual synchronization cue, which depends on the video quality of the lip region of a speaker. In real-world applications, it is possible that we can also have the reference speech of the on-screen speaker. To benefit from both facial cue and reference speech, we propose the Target Speaker TalkNet (TS-TalkNet), which leverages a pre-enrolled speaker embedding to complement the audio-visual synchronization cue in detecting whether the target speaker is speaking. Our framework outperforms the popular model, TalkNet on two datasets, achieving absolute improvements of 1.6% in mAP on the AVA-ActiveSpeaker validation set, and 0.8%, 0.4%, and 0.8% in terms of AP, AUC and EER on the ASW test set, respectively.Code is available at https://github.com/Jiang-Yidi/TS-TalkNet/.
Improving End-to-End Neural Diarization Using Conversational Summary Representations
Authors: Samuel J. Broughton; Lahiru Samarakoon
Venue: Interspeech 2023 (main)

Abstract: Speaker diarization is a task concerned with partitioning an audio recording by speaker identity. End-to-end neural diarization with encoder-decoder based attractor calculation (EEND-EDA) aims to solve this problem by directly outputting diarization results for a flexible number of speakers. Currently, the EDA module responsible for generating speaker-wise attractors is conditioned on zero vectors providing no relevant information to the network. In this work, we extend EEND-EDA by replacing the input zero vectors to the decoder with learned conversational summary representations. The updated EDA module sequentially generates speaker-wise attractors based on utterance-level information. We propose three methods to initialize the summary vector and conduct an investigation into varying input recording lengths. On a range of publicly available test sets, our model achieves an absolute DER performance improvement of 1.90 % when compared to the baseline.