###### Abstract  
The subjectivity of listeners' emotional responses to music is at the crux of optimizing emotion-aware music recommendation. To address this challenge, we constructed a new multimodal dataset (\"HKU956\") with aligned peripheral physiological signals (i.e., heart rate, skin conductance, blood volume pulse, skin temperature) and self-reported emotion collected from 30 participants, as well as original audio of 956 music pieces listened to by the participants. A comprehensive set of features was extracted from physiological signals using methods in physiological computing. This study then compared performances of three feature sets (i.e., acoustic, physiological, and combined) on the task of classifying music-induced emotion. Moreover, the classifiers were also trained on subgroups of users with different Big-Five personality traits for further customized modeling. The results reveal that (1) physiological features contribute to improving performance on valence classification with statistical significance; (2) classification models built for users in different personality groups could sometimes further improve arousal prediction; and (3) the multimodal classifier outperformed single-modality ones on valence classification for most user groups. This study contributes to designing music retrieval systems which incorporate user physiological data and model listeners' emotional responses to music in a customized manner. multimodal recognition; music retrieval and generation; physiological measures; sound and music computing; customization +
Footnote †: journal: Medical Imaging and Biomedicine and Biomedicine and Biomedicine  
such as physiological signals. With physiological signals collected via wireless and low-profile devices (e.g., wristband) during music listening, it would be possible to monitor and measure listeners' emotional responses to music in an objective and unobtrusive manner based on which music recommendation could be more customized and context-aware. To this end, this study aimed to explore the potentials for incorporating physiological sensing for detecting music-induced emotion. Particularly, we focused on exploiting peripheral physiological signals such as heart rate, blood volume pulse, skin conductance, and skin temperature as these signals were reported to be associated with people's emotional status [6] and can be collected by portable wearable and unobtrusive devices. The research questions of this study could be summarized as:  
**RQ1:** Do features extracted from peripheral physiological signals from listeners improve the performance of music-induced emotion classification compared to acoustic features extracted from music content? Furthermore, motivated by studies on the role of personality traits in affective perception of music in the psychology literature [7], this study further investigated:  
**RQ2:** Does building multimodal models on subgroups of users with similar personality traits improve classification performance of music-induced emotion? To answer these research questions, a user experiment was conducted to record users' interactions with a novel MIR system, in which participants' self-reported emotional status and a series of peripheral physiological signals (e.g., heart rate, skin conductance, skin temperature) was collected by a research-grade wristband. These data were then aligned with 956 music pieces traced from participants' music listening history. In response to RQ1, we trained classification models on three distinct feature sets, including acoustic features extracted from music content, physiological features extracted from listeners' physiological signals, and the combination of the two, and compared the performances across these sources of different modalities. Moreover, classifiers were also trained on subgroups of users with similar Big-Five personality traits for further customized modeling. Contributions of this study are four-fold.