DOI: 10.5281/zenodo.17623118
title: EMOSIC: A Multimodal Musical Performance Dataset for Exploring the Emotional Synergy between Music and Motion

description: EMOSIC: Emotional Motion and Music Dataset


Overview

The **EMOSIC Dataset** features over **7.5 hours** of high-precision, synchronized audio and upper-body motion capture data. It captures the performances of **10 professional musicians** from a top-tier conservatory playing **traditional Chinese plucked instruments** (Zhong Ruan, Liuqin, Soprano Ruan, and Bass Ruan).


This dataset provides a valuable **non-Eurocentric resource** to the affective computing and human-robot interaction communities. It is specifically designed to facilitate research into:

- The **"Dominant-Amplifier"** mechanism in multimodal emotion perception.

- Cross-cultural generalizability of emotion recognition models.

- Generative motion synthesis for naturalistic robotic behaviors.


Key Features

- 90 Hz optical motion capture (15-marker upper-body skeletal model).

- 16-bit, 16.0 kHz synchronized recordings.

- Focuses on non-Western musical instrumentation and expressive gestures.


---


Dataset Organization

The dataset is organized into 3 main directories:


### 1. Data

The complete dataset containing all **3,388 unique 8-second clips**.

*   **bvh/**: Processed 3D motion capture data in Biovision Hierarchy (`.bvh`) format.

*   **wav/**: Corresponding audio recordings in Waveform Audio File (`.wav`) format (16-bit, 16.0 kHz, mono).


### 2. Stimuli

The subset of **60 clips** used as stimuli in the perceptual validation experiment described in the associated paper. These are partitioned into three modality conditions and three counterbalanced sets.

*   **A/**: Audio-Only clips (`.wav`).

*   **V/**: Visual-Only clips, rendered as silent skeletal animation videos (`.mp4`).

*   **AV/**: Audio-Visual clips, with synchronized audio and skeletal animations (`.mp4`).


### 3. Validation

Contains raw data and metadata from the perceptual user study ($N=30$), provided in tabular (`.csv`) format.

*   **annotations_A.csv**: Raw Valence-Arousal ratings for Audio-Only condition.

*   **annotations_V.csv**: Raw Valence-Arousal ratings for Visual-Only condition.

*   **annotations_AV.csv**: Raw Valence-Arousal ratings for Audio-Visual condition.

*   **demographics.csv**: Participant demographic info (age, gender, musical background), linked by unique ID.

*   **participants.csv**: Counterbalanced group assignment for each participant.

*   **post_survey.csv**: Post-experiment qualitative feedback.


---


Dataset Structure

Once unzipped, the directory structure is as follows:


```text

EmosicDataset_v1.0/

├── README.md

├── Data/

│   ├── bvh/           # 3,388 .bvh files

│   │   ├── 0000.bvh

│   │   └── ...

│   └── wav/           # 3,388 .wav files

│       ├── 0000.wav

│       └── ...

├── Stimuli/           # 60 clips used in user study

│   ├── A/

│   │   ├── set1/

│   │   ├── set2/

│   │   └── set3/

│   ├── AV/ ...

│   └── V/ ...

└── Validation/        # User study results

    ├── annotations_A.csv

    ├── ...

    └── post_survey.csv

```


File Naming Convention

The core data files in `Data/bvh` and `Data/wav` use a consistent 4-digit numerical identifier (from `0000` to `3387`).

*   **Example:** `0000.bvh` is the motion data corresponding strictly to `0000.wav`.


Files within the `Stimuli` folders utilize the same identifiers, allowing easy mapping back to the full dataset.


Citation

If you use this dataset in your research, please cite the following paper:


> **Asymmetrical Emotional Synergy between Music and Body Motion: Insights from the EMOSIC Musical Performance Dataset**  

> *Authors: Kaiyuan Ma, Biyun Zhang, Jun He*  

> *Scientific Reports, Under review*  


Contact

For questions regarding the dataset, please contact:

**Jun He** (jhe@sjtu.edu.cn)

School of Mechanical Engineering, Shanghai Jiao Tong University