A Multimodal Dataset for Vocal Pedagogy
Creators: name: Häärä, Suvi
Publisher: name: Zenodo
Year: 2024
Subjects: Vocal Pedagogy
Machine Learning
Multimodal Data
Crossmodal Analysis
Music Technology
Sound and Music Computing

description: The voice is a complex instrument, requiring singers to understand the intricate relationships between physical actions, sensory feedback, and sound production. This complexity presents a unique opportunity to explore how technology can enhance musical learning. Beginners face numerous challenges in singing, including breath control, vocal fold closure, laryngeal positioning, articulation, resonance and posture.Current singing voice datasets are often audio-centric and focus on tasks such as source separation or singer identification, neglecting the physical aspects and missing appropriate labels crucial for developing technologies for vocal pedagogy.

This thesis presents a multimodal dataset designed to address this gap by capturingaudio, video, and biosignal data. The dataset was created by recording audio from three sources (a 2017 MacBook Air, an iPhone 14 Pro, and a Behringer C-3 icrophone), video from two sources (a 2017 MacBook Air and an iPhone 14 Pro), an electromyography sensor and a piezoelectric respiration sensor. We collected data from 14 participants, including professional, intermediate, and inexperienced singers, who sang various scales to ensure variability in vowel sounds, note lengths, and pitches.

We developed Python scripts and Jupyter Notebooks, available on GitHub, for data alignment and cleaning (1), feature extraction (2), and the development of various machine learning models (3). We compared K-Nearest Neighbour (KNN), Multilayer Perceptrón (MLP), and Random Forest (RF) classifier architectures to determine the most effective models, modalities and features for classifying phonation and vocal technique errors (VTEs).Individual models for two participants, a combined model, and their results were compared to identify the most relevant features for vocal pedagogical applications.

Our results demonstrate the dataset’s effectiveness in phonation and VTE classification.A 2-way phonation KNN classifier achieved approximately 75% accuracy with both individual and combined data models using 5 features. The optimal approach for crossmodal VTE classification was a set of six 2-way RF models using Recursive Feature Elimination (RFE), which achieved average accuracies of around 71% with only five features.

This research advances vocal pedagogical technologies by providing a multimodal dataset that includes audio, video, and biosignal data, capturing both expressive resources and vocal technique errors. Future work should focus on developing a generalised model that personalises feedback as it learns from individual users’ voices. Such a system could start with a general model and progressively refine its feedback with user-specific data.

The code and resources developed for this study are available on GitHub, facilitating further research and reproducibility.

 _______________________________

(1) https://github.com/suvimh/vocal_data_cleaning(2) https://github.com/suvimh/vocal_data_feature_extraction(3) https://github.com/suvimh/vocal_data_feature_based_classification_for_colab