GenAI4ELab/papercli-papers huggingface dataset:
MOSLight: A Lightweight Data-Efficient System for Non-Intrusive Speech Quality Assessment
Authors: Zitong Li; Wei Li
Venue: Interspeech 2023 (main)

Abstract: Automatically predicting the mean opinion score (MOS) of a synthesized speech without the reference signal with deep learning systems has been studied extensively recently and shown great results. However, previous best systems are mostly based on self-supervised learned (SSL) models consisting of up to hundreds of millions of parameters making them unsuitable for mobile or embedded applications. In this paper, we propose MOSLight, a non-SSL-based lightweight yet powerful system for MOS prediction. We argue that 2D convolutions are inefficient for audio feature processing and not ideal for tasks where training data are scarce. To build MOSLight, we utilized depthwise separable dilated 1D convolutions and incorporated multi-task learning and non-strict frame-level score clipping. We conducted experiments on the Voice Conversion Challenge 2018 (VCC2018) and BVCC. Results show MOSLight achieves great effectiveness despite being a lightweight model trained with limited training data.
A Multi-Scale Attentive Transformer for Multi-Instrument Symbolic Music Generation
Authors: Xipin Wei; Junhui Chen; Zirui Zheng; Li Guo; Lantian Li; Dong Wang
Venue: Interspeech 2023 (main)

Abstract: Recently, multi-instrument music generation has become a hot topic. Different from single-instrument generation, multi-instrument generation needs to consider inter-track harmony besides intra-track coherence. This is usually achieved by composing note segments from different instruments into a signal sequence. This composition could be on different scales, such as note, bar, or track. Most existing work focuses on a particular scale, leading to shortage in modeling music with diverse temporal and track dependencies.  This paper proposes a multi-scale attentive Transformer model to improve the quality of multi-instrument generation. We first employ multiple Transformer encoders to learn multi-instrument representations of different scales and then design an attentive mechanism to fuse the multi-scale information. Experiments conducted on SOD and LMD datasets show that our model improves both quantitative and qualitative performance compared to models based on single-scale information. The source code and some generated samples can be found at https://github.com/HaRryqaq/MSAT.
MTANet: Multi-band Time-frequency Attention Network for Singing Melody Extraction from Polyphonic Music
Authors: Yuan Gao; Ying Hu; Liusong Wang; Hao Huang; Liang He
Venue: Interspeech 2023 (main)

Abstract: Singing melody extraction is an important task in music information retrieval. In this paper, we propose a multi-band time-frequency attention network (MTANet) for singing melody extraction from polyphonic music, which can generate the feature representation to characterize the fundamental frequency (F0) component. Moreover, a band partition scheme is proposed to fit the position distribution of the F0 component. Further, three hourglass sub-networks are used to capture various multi-band features. Then, a feature fusion module (FFM) is employed to fuse the multi-band features. Visualization analysis shows that the multi-band feature extraction branch can generate the feature representation for characterizing the F0 component effectively. Experimental results show that the MTANet outperforms the existing state-of-the-art methods, while keeping with fewer network parameters. Visualized results intuitively show that the MTANet can reduce the octave and melody detection errors.
Xiaoicesing 2: A High-Fidelity Singing Voice Synthesizer Based on Generative Adversarial Network
Authors: Wang Chunhui; Chang Zeng; Xing He
Venue: Interspeech 2023 (main)

Abstract: The over-smoothing problem in the middle- and high-frequency areas prevents the acoustic model from generating high-quality singing voices. In this paper, we propose XiaoiceSing2, which is a generative adversarial network consisting of a FastSpeech-based generator and a multi-band discriminator, to generate the full-band mel-spectrogram. Specifically, we improve the feed-forward Transformer (FFT) block by adding multiple residual convolutional blocks in parallel with the self-attention block to balance the local and global features. The multi-band discriminator contains three sub-discriminators responsible for low-, middle-, and high-frequency parts of the mel-spectrogram, respectively. Each sub-discriminator is composed of several segment discriminators (SD) and detail discriminators (DD) to distinguish the audio from different aspects. The experiment on our internal 48kHz singing voice dataset shows XiaoiceSing2 significantly improves the quality of the singing voice over XiaoiceSing
Do Vocal Breath Sounds Encode Gender Cues for Automatic Gender Classification?
Authors: Mohammad Shaique Solanki; Ashutosh Bharadwaj; Jeevan Kylash; Prasanta Kumar Ghosh
Venue: Interspeech 2023 (main)

Abstract: The acoustic features of continuous speech, such as pitch (F0) and formant frequencies (F1, F2) have been utilized for gender classification. However, non-speech signals including vocal breath sounds have not been explored due to the absence of gender-specific acoustic features. This study investigates if vocal breath sounds carry gender information and if they can be used for automatic gender classification. The study examines the use of data-driven and knowledge-based features from breath sounds, classifier complexity, and the importance of breath signal segment location and duration. Results from experiments on 54 minutes of male and 52 minutes of female breath sounds demonstrate that classifiers with low-complexity and knowledge-based features (MFCC statistics) perform similarly to high-complexity classifiers with data-driven features. Breath segments of around 3 seconds are found to be the most suitable choice regardless of location, eliminating the need for breath cycle boundary marking.
Automatic Exploration of Optimal Data Processing Operations for Sound Data Augmentation Using Improved Differentiable Automatic Data Augmentation
Authors: Toki Sugiura; Hiromitsu Nishizaki
Venue: Interspeech 2023 (main)

Abstract: Data augmentation is one of the methods used to robustly train machine learning models with a small dataset. This method randomly applies pre-defined data processing operations to input data, regardless of the characteristics of the input data. However, some data processing operations may be inappropriate for certain data. In this study, we propose a new method to automatically search for the best data processing operations for each sound file to be input into a sound classification neural network. The proposed method is an improvement on the previously proposed differentiable automatic data augmentation (DADA), which uses a differentiable neural network to select the optimal data processing operations. We evaluated our proposed method on an acoustic scene classification task on the ESC-50 dataset and demonstrated that the proposed method can train a more robust model compared to the original DADA-based data augmentation.
A Snoring Sound Dataset for Body Position Recognition: Collection, Annotation, and Analysis
Authors: Li Xiao; Xiuping Yang; Xinhong Li; Weiping Tu; Xiong Chen; Weiyan Yi; Jie Lin; Yuhong Yang; Yanzhen Ren
Venue: Interspeech 2023 (main)

Abstract: Obstructive Sleep Apnea-Hypopnea Syndrome (OSAHS) is a chronic breathing disorder caused by a blockage in the upper airways. Snoring is a prominent symptom of OSAHS, and previous studies have attempted to identify the obstruction site of the upper airways by snoring sounds. Despite some progress, the classification of the obstruction site remains challenging in real-world clinical settings due to the influence of sleep body position on upper airways. To address this challenge, this paper proposes a snore-based sleep body position recognition dataset (SSBPR) consisting of 7570 snoring recordings, which comprises six distinct labels for sleep body position: supine, supine but left lateral head, supine but right lateral head, left-side lying, right-side lying and prone. Experimental results show that snoring sounds exhibit certain acoustic features that enable their effective utilization for identifying body posture during sleep in real-world scenarios.
RMVPE: A Robust Model for Vocal Pitch Estimation in Polyphonic Music
Authors: Haojie Wei; Xueke Cao; Tangpeng Dan; Yueguo Chen
Venue: Interspeech 2023 (main)

Abstract: Vocal pitch is an important high-level feature in music audio processing. However, extracting vocal pitch in polyphonic music is more challenging due to the presence of accompaniment. To eliminate the influence of the accompaniment, most previous methods adopt music source separation models to obtain clean vocals from polyphonic music before predicting vocal pitches. As a result, the performance of vocal pitch estimation is affected by the music source separation models. To address this issue and directly extract vocal pitches from polyphonic music, we propose a robust model named RMVPE. This model can extract effective hidden features and accurately predict vocal pitches from polyphonic music. The experimental results demonstrate the superiority of RMVPE in terms of raw pitch accuracy (RPA) and raw chroma accuracy (RCA). Additionally, experiments conducted with different types of noise show that RMVPE is robust across all signal-to-noise ratio (SNR) levels.
Spatialization Quality Metric for Binaural Speech
Authors: Pranay Manocha; Israel Dejene Gebru; Anurag Kumar; Dejan Markovic; Alexander Richard
Venue: Interspeech 2023 (main)

Abstract: In spatial-audio enabled systems, evaluating the quality of spatialization is an essential process. This paper proposes a new objective metric to measure the spatialization quality (SQ) between any pair of binaural signals while being agnostic to speech content and signal duration. We formulate SQ as a metric learning problem and compute deep-feature distance on embeddings learned using triplet loss and multi-task learning with direction-of-arrival and binaural speech synthesis as auxiliary tasks. We show the robustness of our model on localization in (un)seen contexts, monotonicity with increasing angular distance, content in-variance and retrieval performance. Experiments show that our metric correlates well with publicly available subjective ratings, and it yields improvements when used as a differentiable loss in a binaural speech enhancement system.
AsthmaSCELNet: A Lightweight Supervised Contrastive Embedding Learning Framework for Asthma Classification Using Lung Sounds
Authors: Arka Roy; Udit Satija
Venue: Interspeech 2023 (main)

Abstract: Asthma is one of the most prevalent respiratory disorders, which can be identified by different modalities such as speech, wheezing of lung sounds (LSs), spirometric measures, etc. In this paper, we propose AsthmaSCELNet, a lightweight supervised contrastive embedding learning framework, to classify asthmatic LSs by providing adequate classification margin across the embeddings of healthy and asthma LS, in contrast to vanilla supervised learning. Our proposed framework consists of three steps: pre-processing, melspectrogram extraction, and classification. The AsthmaSCELNet consists of two stages: embedding learning using a lightweight embedding extraction backbone module that extracts compact embedding from the melspectrogram, and classification by the learnt embeddings using multi-layer perceptrons. The proposed framework achieves an accuracy, sensitivity, and specificity of 98.54%, 98.27%, and 98.73% respectively, that outperforms existing methods based on LSs and other modalities.
Patch-Mix Contrastive Learning with Audio Spectrogram Transformer on Respiratory Sound Classification
Authors: Sangmin Bae; June-Woo Kim; Won-Yang Cho; Hyerim Baek; Soyoun Son; Byungjo Lee; Changwan Ha; Kyongpil Tae; Sungnyun Kim; Se-Young Yun
Venue: Interspeech 2023 (main)

Abstract: Respiratory sound contains crucial information for the early diagnosis of fatal lung diseases. Since the COVID-19 pandemic, there has been a growing interest in contact-free medical care based on electronic stethoscopes. To this end, cutting-edge deep learning models have been developed to diagnose lung diseases; however, it is still challenging due to the scarcity of medical data. In this study, we demonstrate that the pretrained model on large-scale visual and audio datasets can be generalized to the respiratory sound classification task. In addition, we introduce a straightforward Patch-Mix augmentation, which randomly mixes patches between different samples, with Audio Spectrogram Transformer (AST). We further propose a novel and effective Patch-Mix Contrastive Learning to distinguish the mixed representations in the latent space. Our method achieves state-of-the-art performance on the ICBHI dataset, outperforming the prior leading score by an improvement of 4.08%.
Remote Assessment for ALS using Multimodal Dialog Agents: Data Quality, Feasibility and Task Compliance
Authors: Vanessa Richter; Michael Neumann; Jordan Green; Brian Richburg; Oliver Roesler; Hardik Kothare; Vikram Ramanarayanan
Venue: Interspeech 2023 (main)

Abstract: We investigate the feasibility, task compliance and audiovisual data quality of a multimodal dialog-based solution for remote assessment of Amyotrophic Lateral Sclerosis (ALS). 53 people with ALS and 52 healthy controls interacted with Tina, a cloud-based conversational agent, in performing speech tasks designed to probe various aspects of motor speech function while their audio and video was recorded. We rated a total of 250 recordings for audio/video quality and participant task compliance, along with the relative frequency of different issues observed. We observed excellent compliance (98%) and audio (95.2%) and visual quality rates (84.8%), resulting in an overall yield of 80.8% recordings that were both compliant and of high quality. Furthermore, recording quality and compliance were not affected by level of speech severity and did not differ significantly across end devices. These findings support the utility of dialog systems for remote monitoring of speech in ALS.
Adaptation of Text-Conditioned Diffusion Models for Audio-to-Image Generation
Authors: Guy Yariv; Itai Gat; Lior Wolf; Yossi Adi; Idan Schwartz
Venue: Interspeech 2023 (main)

Abstract: In recent years, image generation has shown a great leap in performance, where diffusion models play a central role. Although generating high-quality images, such models are mainly conditioned on textual descriptions. This begs the question: how can we adopt such models to be conditioned on other modalities? In this paper, we propose a novel method utilizing latent diffusion models, trained for text-to-image-generation, to generate images, conditioned on audio recordings. Using a pre-trained audio encoding model, the proposed method encodes audio into a new token, which can be considered as an adaptation layer between the audio and text representations. Such a modeling paradigm requires a small number of trainable parameters, making the proposed approach appealing for lightweight optimization. Results suggest the proposed method is superior to the evaluated baseline methods, considering both objective and subjective metrics. Code and samples will be publicly available.
Obstructive sleep apnea screening with breathing sounds and respiratory effort: a multimodal deep learning approach
Authors: Hector E. Romero; Ning Ma; Guy J. Brown; Sam Johnson
Venue: Interspeech 2023 (main)

Abstract: Obstructive sleep apnea (OSA) is a chronic and prevalent condition with well-established comorbidities. Due to limited diagnostic resources and high cost, a significant OSA population lives undiagnosed, and accurate and low-cost methods to screen for OSA are needed. We propose a novel screening method based on breathing sounds recorded with a smartphone and respiratory effort. Whole night recordings are divided into 30-s segments, each of which is classified for the presence or absence of OSA events by a multimodal deep neural network. Data fusion techniques were investigated and evaluated based on the apnea-hypopnea index estimated from whole night recordings. Real-world recordings made during home sleep apnea testing from 103 participants were used to develop and evaluate the proposed system. The late fusion system achieved the best sensitivity and specificity when screening for severe OSA, at 0.93 and 0.92, respectively. This offers the prospect of inexpensive OSA screening at home.
Investigation of Music Emotion Recognition Based on Segmented Semi-Supervised Learning
Authors: Yifu Sun; Xulong Zhang; Jianzong Wang; Ning Cheng; Kaiyu Hu; Jing Xiao
Venue: Interspeech 2023 (main)

Abstract: The production and annotation of music datasets requires very specialized background knowledge, which is difficult for most people to complete. Therefore, the number of annotated music samples is at a premium for Music Information Retrieval (MIR) tasks. Recently, segment-based methods for emotion-related tasks have been proposed, which train backbone networks on shorter segments instead of entire audio clips, thereby naturally augmenting training samples without requiring additional resources. However, when training at the segment level, segment labels are the major problem. The most commonly used method is that segment inherits the label of the clip containing it, but as we all know, music emotion is not constant during the whole clip. Doing so will introduce label noise and make the training overfit easily. To handle the noisy label issue, we propose a semi-supervised self-learning method and achieve better results than previous methods.
The Effects of Input Type and Pronunciation Dictionary Usage in Transfer Learning for Low-Resource Text-to-Speech
Authors: Phat Do; Matt Coler; Jelske DÄ³kstra; Esther Klabbers
Venue: Interspeech 2023 (main)

Abstract: We compare phone labels and articulatory features as input for cross-lingual transfer learning in text-to-speech (TTS) for low-resource languages (LRLs). Experiments with FastSpeech 2 and the LRL West Frisian show that using articulatory features outperformed using phone labels in both intelligibility and naturalness. For LRLs without pronunciation dictionaries, we propose two novel approaches: a) using a massively multilingual model to convert grapheme-to-phone (G2P) in both training and synthesizing, and b) using a universal phone recognizer to create a makeshift dictionary. Results show that the G2P approach performs largely on par with using a ground-truth dictionary and the phone recognition approach, while performing generally worse, remains a viable option for LRLs less suitable for the G2P approach. Within each approach, using articulatory features as input outperforms using phone labels.
Resource-Efficient Fine-Tuning Strategies for Automatic MOS Prediction in Text-to-Speech for Low-Resource Languages
Authors: Phat Do; Matt Coler; Jelske DÄ³kstra; Esther Klabbers
Venue: Interspeech 2023 (main)

Abstract: We train a MOS prediction model based on wav2vec 2.0 using the open-access data sets BVCC and SOMOS. Our test with neural TTS data in the low-resource language (LRL) West Frisian shows that pre-training on BVCC before fine-tuning on SOMOS leads to the best accuracy for both fine-tuned and zero-shot prediction. Further fine-tuning experiments show that using more than 30 percent of the total data does not lead to significant improvements. In addition, fine-tuning with data from a single listener shows promising system-level accuracy, supporting the viability of one-participant pilot tests. These findings can all assist the resource-conscious development of TTS for LRLs by progressing towards better zero-shot MOS prediction and informing the design of listening tests, especially in early-stage evaluation.
Robust Feature Decoupling in Voice Conversion by Using Locality-Based Instance Normalization
Authors: Yewei Gu; Xianfeng Zhao; Xiaowei Yi
Venue: Interspeech 2023 (main)

Abstract: Extensive style transfer methods have shown that instance normalization (IN) is a simple yet effective way to remove style information. However, few studies have focused on whether these channel-wise feature statistics, such as mean and standard deviation (std) are consistent locally and globally, which ultimately leads to insufficient feature decoupling. In this paper, we first propose locality-based instance normalization (LoIN) to impose statistical feature consistency constraints on latent feature maps. LoIN performs normalization using local feature statistics which are calculated on randomly selected frames rather than the entire
set of frames used in the training phase. In particular, LoIN is lightweight, less computationally intensive, and transferable to any IN-driven VC method. Experimental results show the superiority of LoIN in disentanglement and transfer performance and show improvement in both speaker similarity and content consistency.
Zero-Shot Accent Conversion using Pseudo Siamese Disentanglement Network
Authors: Dongya Jia; Qiao Tian; Kainan Peng; Jiaxin Li; Yuanzhe Chen; Mingbo Ma; Yuping Wang; Yuxuan Wang
Venue: Interspeech 2023 (main)

Abstract: The goal of accent conversion (AC) is to convert the accent of speech into the target accent while preserving the content and speaker identity. AC enables a variety of applications, such as language learning, speech content creation, and data augmentation. Previous methods rely on reference utterances in the inference phase or are unable to preserve speaker identity. To address these issues, we propose a zero-shot reference-free accent conversion method, which is able to convert unseen speakers' utterances into a target accent. Pseudo Siamese Disentanglement Network (PSDN) is proposed to disentangle the accent from the content representation. Experimental results show that our model generates speech samples with much higher accentedness than the input and comparable naturalness, on two-way conversion including foreign-to-native and native-to-foreign.
Automatic Evaluation of Turn-taking Cues in Conversational Speech Synthesis
Authors: Erik Ekstedt; Siyang Wang; Ãva SzÃ©kely; Joakim Gustafson; Gabriel Skantze
Venue: Interspeech 2023 (main)

Abstract: Turn-taking is a fundamental aspect of human communication where speakers convey their intention to either hold, or yield, their turn through prosodic cues. Using the recently proposed Voice Activity Projection model, we propose an automatic evaluation approach to measure these aspects for conversational speech synthesis. We investigate the ability of three commercial, and two open-source, Text-To-Speech (TTS) systems ability to generate turn-taking cues over simulated turns. By varying the stimuli, or controlling the prosody, we analyze the models performances. We show that while commercial TTS largely provide appropriate cues, they often produce ambiguous signals, and that further improvements are possible. TTS, trained on read or spontaneous speech, produce strong turn-hold but weak turn-yield cues. We argue that this approach, that focus on functional aspects of interaction, provides a useful addition to other important speech metrics, such as intelligibility and naturalness.