DOI: 10.5281/zenodo.14267884
title: HRI-SENSE: A Multimodal Dataset on Social and Emotional Responses to Robot Behaviour

description: This is the dataset for the paper "HRI-SENSE: A Multimodal Dataset on Social and Emotional Responses to Robot Behaviour" – available at the Proceedings of the 2025 ACM/IEEE International Conference on Human-Robot Interaction.

 

The dataset captures various modalities of user behaviour (facial landmarks, facial action units, head pose, body pose landmarks, depth footage) exhibited by participants interacting with a TIAGo Steel robot following different behaviour models on a collaborative manipulation "Burger Assembly" task. Details of the task can be found in the paper. The non-verbal modalities are complemented by explicit feedback signals (verbal dialogue transcripts), robot joint movements data, interaction event labels and self-assessed questionnaires (pre-study and post-interaction questionnaires) on sociodemographics and perceived user impressions (e.g. frustration, satisfaction).

 

The dataset's contents have been collected from over 6 hours of verbal and physical human-robot interactions in over 146 sessions with 18 participants.

Data Modalities

User facial expression data (facial landmarks, facial action units and head pose) have been calculated by OpenFace:



head location: pose_Tx, pose_Ty, pose_Tz

head rotation: pose_Rx, pose_Ry, pose_Rz

facial landmark 3d locations: X_0, ... X_67, Y_0,...Y_67, Z_0,...Z_67

facial action units intensity (r) and presence (c): AU01_r, AU02_r,...AU25_r, AU26_r, AU45_r, AU01_c, AU02_c,...AU28_c, AU45_c


User pose landmarks have been calculated by MediaPipe Pose Landmarker:





3d coordinates of 33 pose landmarks



Verbal dialogue transcripts have been produced by OpenAI's Whisper model.

 

Depth information and robot joint data has been recorded by the TIAGo robot's sensors.

 

Self-assessed questionnaire data has been collected using a 5-point Likert scale, following question items and practices established in previous Human-Robot Interactions and Psychology works. Details of this can be found in the paper.

 

For more details on the data collection and processing pipeline, please see the paper.

Contents

The dataset is organised as:

 





README.md

questionnaire-data/



pre-study-questionnaire/



pre-study-questionnaire.pdf

pre-study-questionnaire-responses.csv: participant-id,A1-age,A2-occupation,B1,B2,B3,B4,B5,B6,C1,C2,C3,C4,C5,C6,C7,C8,C9,C10,C11,C12













post-interaction-questionnaire/



post-interaction-questionnaire.pdf

post-interaction-questionnaire-responses.csv: participant-id,model-id,Q1,Q2,Q3,Q4,Q5,Q6,Q7,Q8,Q9,Q10,Q11,Q12,Q13,Q14,Q15,Q16,Q17





sensory-data/



depth-data/



P[participant_id]-M[model_id]-[trial_number].mp4



user-face-data/



P[participant_id]-M[model_id]-[trial_number].csv: timestamp,confidence,success,pose_Tx,pose_Ty,pose_Tz,pose_Rx,pose_Ry,pose_Rz,X_0,X_1,X_2,X_3,X_4,X_5,X_6,X_7,X_8,X_9,X_10,X_11,X_12,X_13,X_14,X_15,X_16,X_17,X_18,X_19,X_20,X_21,X_22,X_23,X_24,X_25,X_26,X_27,X_28,X_29,X_30,X_31,X_32,X_33,X_34,X_35,X_36,X_37,X_38,X_39,X_40,X_41,X_42,X_43,X_44,X_45,X_46,X_47,X_48,X_49,X_50,X_51,X_52,X_53,X_54,X_55,X_56,X_57,X_58,X_59,X_60,X_61,X_62,X_63,X_64,X_65,X_66,X_67,Y_0,Y_1,Y_2,Y_3,Y_4,Y_5,Y_6,Y_7,Y_8,Y_9,Y_10,Y_11,Y_12,Y_13,Y_14,Y_15,Y_16,Y_17,Y_18,Y_19,Y_20,Y_21,Y_22,Y_23,Y_24,Y_25,Y_26,Y_27,Y_28,Y_29,Y_30,Y_31,Y_32,Y_33,Y_34,Y_35,Y_36,Y_37,Y_38,Y_39,Y_40,Y_41,Y_42,Y_43,Y_44,Y_45,Y_46,Y_47,Y_48,Y_49,Y_50,Y_51,Y_52,Y_53,Y_54,Y_55,Y_56,Y_57,Y_58,Y_59,Y_60,Y_61,Y_62,Y_63,Y_64,Y_65,Y_66,Y_67,Z_0,Z_1,Z_2,Z_3,Z_4,Z_5,Z_6,Z_7,Z_8,Z_9,Z_10,Z_11,Z_12,Z_13,Z_14,Z_15,Z_16,Z_17,Z_18,Z_19,Z_20,Z_21,Z_22,Z_23,Z_24,Z_25,Z_26,Z_27,Z_28,Z_29,Z_30,Z_31,Z_32,Z_33,Z_34,Z_35,Z_36,Z_37,Z_38,Z_39,Z_40,Z_41,Z_42,Z_43,Z_44,Z_45,Z_46,Z_47,Z_48,Z_49,Z_50,Z_51,Z_52,Z_53,Z_54,Z_55,Z_56,Z_57,Z_58,Z_59,Z_60,Z_61,Z_62,Z_63,Z_64,Z_65,Z_66,Z_67,AU01_r,AU02_r,AU04_r,AU05_r,AU06_r,AU07_r,AU09_r,AU10_r,AU12_r,AU14_r,AU15_r,AU17_r,AU20_r,AU23_r,AU25_r,AU26_r,AU45_r,AU01_c,AU02_c,AU04_c,AU05_c,AU06_c,AU07_c,AU09_c,AU10_c,AU12_c,AU14_c,AU15_c,AU17_c,AU20_c,AU23_c,AU25_c,AU26_c,AU28_c,AU45_c



user-pose-data/



P[participant_id]-M[model_id]-[trial_number]-[camera_id].csv: time,L0-x,L0-y,L0-z,L1-x,L1-y,L1-z,L2-x,L2-y,L2-z,L3-x,L3-y,L3-z,L4-x,L4-y,L4-z,L5-x,L5-y,L5-z,L6-x,L6-y,L6-z,L7-x,L7-y,L7-z,L8-x,L8-y,L8-z,L9-x,L9-y,L9-z,L10-x,L10-y,L10-z,L11-x,L11-y,L11-z,L12-x,L12-y,L12-z,L13-x,L13-y,L13-z,L14-x,L14-y,L14-z,L15-x,L15-y,L15-z,L16-x,L16-y,L16-z,L17-x,L17-y,L17-z,L18-x,L18-y,L18-z,L19-x,L19-y,L19-z,L20-x,L20-y,L20-z,L21-x,L21-y,L21-z,L22-x,L22-y,L22-z,L23-x,L23-y,L23-z,L24-x,L24-y,L24-z,L25-x,L25-y,L25-z,L26-x,L26-y,L26-z,L27-x,L27-y,L27-z,L28-x,L28-y,L28-z,L29-x,L29-y,L29-z,L30-x,L30-y,L30-z,L31-x,L31-y,L31-z,L32-x,L32-y,L32-z



robot-joint-data/



P[participant_id]-M[model_id]-[trial_number].csv: time, arm_1_joint, arm_2_joint, arm_3_joint, arm_4_joint, arm_5_joint, arm_6_joint, arm_7_joint 



dialogue-transcript/



P[participant_id]-M[model_id]-[trial_number].csv: start,end,text





interaction-event-labels/



P[participant_id]-M[model_id]-[trial_number].csv: event,start,end




Limitations

Due to recording sensor malfunctions and processing library limitations, not all interaction scenario data contains all modalities. Verbal dialogue transcripts or user pose data may be incomplete or missing for a small number of interactions or recording angles.

 

Acknowledgements

This work was supported by UK Research and Innovation [EP/S024298/1].