DOI: 10.5281/zenodo.17990636
title: Multimodal Dataset for Evaluating Driving Distractors

description: Driver safety has never been as important as it is now, due to the changes our society is facing. New technologies such as AI have been successfully employed in various technological fields. It is natural to want to replicate that success in the field of driving.

The dataset is multimodal, consisting of synchronized audio, video, and annotation streams. Audio was recorded using four microphones, while video data was obtained from multiple cameras, including a dashboard camera and eye trackers. Two distinct scenarios are available, with durations of 41:12 (S1) and 39:55 (S2), respectively.

This dataset was developed to quantify the influence of auditory distractions on driving performance. For this reason, we focused on auditory elements — both on the different classes identified as distractions and on the quantity of audio events present in the dataset.  

Objectives of the Dataset

This dataset provides samples that allow users to:



Train models based on noises and sound events.

Train models to detect distracting elements inside the vehicle.

Train models to identify distracting elements outside the vehicle.


Data Collection Methodology

The dataset was formed from videos and audio recordings obtained in two scenarios. In these, participants are observed during normal driving, which is interrupted by different sound effects.

Equipment Used



Neon - Pupil Labs.  

ZOOM h6 Recorder. 

Behringer ECM8000.

Samsung Galaxy Watch 5.


Data Format and Structure

The dataset is organized into two distinct scenarios, identified using the abbreviations S1 and S2, respectively.

The information is divided into raw data and labels.

Audio_Elements Folder

Divided into data and labels.

Data

Contains 8 audio elements in .wav format:



3 from Dash cameras.(Sx_audio_D_25fps,Sx_audio_F_25fps,Sx_audio_M_25fps)

1 recorded by Pupil Labs.(SX_audio_Eyetracker)

4 from the Zoom recorder microphones.(Sx_audio_Zoom_Left,Sx_audio_Zoom_Right,Sx_audio_Zoom_Tr1,Sx_audio_Zoom_Tr3)


Labels

A .txt file containing:



Start and end offset of each sound event.

Class name.


This table presents information on the number of occurrences of each class across different scenarios (Instances_Sx), as well as the total accumulated duration of those instances(Duration_Sx).

  




ID
Name
Instances_S1
Instances_S2
Duration_S1
Duration_S2


0
Manipulating Objects
5
1
3.2
5.2


1
Speech
268
155
1514.71
1830.18


2
Laughter
39
43
51.27
82.39


3
Music 
8
12
80.11
212.8


4
Warning Beeps  
5
4
25.72
18.7


5
Ring Tone
12
4
69.39
5.37


6
Notifications 
14
15
14.6
12.41


7


Cough 

3
3
2.1
3.09


8
Cry 
3
2
63.24
20.64


9
Hit 
1
2
0.24
1.1


10
Yell
1
0
1.5 
0


11
Dog 
6
2
2.99
2.3


12
Car Horn 
4
2
4.5
2.14


13
Scream  
0
1
0
1.39


14
Microphone Noise 
0
1
0
1.9


15
Stone 
0
10
0
9.34


16
Engine 
0
2
0
19.22




  

Additional metadata for the audio labels is also provided.

Videos_Raw Folder

Videos are separated based on their file size and stored in .mp4 format.

There are 4 video sources.



3 from the Dash camera.(Sx_Video_D_25fps,Sx_Video_D_25fps,Sx_Video_D_25fps)

1 from the Pupil Labs.(Sx_Video_Eyetracker_25fps)


Additional information about each video is included here.

  

Video_Labels Folder

Includes the following subfolders:

Coordinates_Eye_Tracker

Contains gaze data recorded with Pupil Eye Tracker for each scenario, stored in .csv files.

  

First CSV includes:



Unix timestamp.

Video time in seconds.

Video frame number

X and Y gaze coordinates.

Fixation ID (if any).


Labeled coordinates CSV includes additionally:



Object being viewed (refers to interior_classes; left empty if none)

Sound event.

Fixation indicator.

Fixation ID.

 Fixation coordinates (X, Y).

Pixel variation between the current gaze and the fixation point.


Object_Segmentation_labels

Video frames were extracted using FFmpeg with the following command:

  



ffmpeg -i video_name.mp4 -vf fps=1 -start_number 1 frames/test_%06d.png


  

One frame per second was extracted (instead of 25 fps), which is reflected in the frame naming.

  

The .txt file contains:



Class ID.

Object position coordinates within the frame. 


Additional information about the segmentation labels is also included.

  

The first table refers to exterior objects and distractors, while the second one corresponds to interior objects and distractions. These tables summarize the instances in which each class appears.

    

Interior_Classes

 




ID
Name
Instances_S1
Instances_S2 


0
Car
9719
8426 


1
Traffic light
1148
1547


2
Traffic signal
4297
3619


3
Person
678
573


4
Road
4991
3605


5
Pedestrian crossing
1205
958




 

  

Exterior_Classes

  




ID
Name
Instances_S1
Instances_S2 


0
Interior rear-view mirror
2127
2145  


1
Outside left
2388
1946  


2
Outside Central
2479
2400 


3
Screen
2212
2209 


4
Speedometer
2298
2237 


5
Left rear-view visor
2035
1536 


6
Right rear-view mirror
682
713 


7
Outside Right
447
630 


8
Co-driver
2189
1901 


9
Radio
2157
1528 


10
Steering wheel
2378
2016 




 

  

Extra Folder

Contains supplementary information not directly related to the audio or video elements.

It includes JSON files for both scenarios and heart rate data.

Each file stores:



Average, maximum, and minimum heart rate.

Time period in Unix timestamp format


Warning

The heart rate data may be inaccurate or unreliable. We recommend exercising caution when using or interpreting this information.

Authors

C. Castorena, A. Roche, J. Lopez-Ballester, J. A. De Rus, J. J. Perez-Solano, S. Roger, C. Botella-Mascarell, J. J. Lopez, J. M. Mossi, F. J. Ferri and M. Cobos

 Acknolegments 

Dataset collected within the AEOLIAN-CONNECT (TED2021-131003B-C21) project, funded by MICIU/AEI /10.13039/501100011033 and by European Union NextGenerationEU/ PRTR.