DOI: 10.5281/zenodo.17121106
title: Investigating the Role of Implicit Signals in Adaptive User-Aware Human-Robot Interactions: A Multimodal Dataset

description: The paper "Investigating the Role of Implicit Signals in Adaptive User-Aware Human-Robot Interactions" associated with this dataset is currently under double-blind review for the 2026 IEEE International Conference on Robotics & Automation (ICRA 2026). This entry will be updated with proper affiliations following the review process.

The dataset captures various modalities of user behaviour (facial landmarks, facial action units, head pose and body pose landmarks) exhibited by participants interacting with a Pepper robot following different behaviour models on a social human-robot interaction task taking place in a cafeteria-style scenario. Details of the scenario can be found in the paper. The non-verbal modalities are complemented by explicit feedback signals captured in the form of verbal dialogue transcripts.

The dataset's contents have been collected from over 5 hours of verbal and physical human-robot interactions with 20 participants.

Data Modalities

User facial expression data (facial landmarks, facial action units and head pose) have been calculated by OpenFace:



head location: pose_Tx, pose_Ty, pose_Tz

head rotation: pose_Rx, pose_Ry, pose_Rz

facial landmark 3d locations: X_0, ... X_67, Y_0,...Y_67, Z_0,...Z_67

facial action units intensity (r) and presence (c): AU01_r, AU02_r,...AU25_r, AU26_r, AU45_r, AU01_c, AU02_c,...AU28_c, AU45_c


User pose landmarks have been calculated by MediaPipe Pose Landmarker:





3d coordinates of 33 pose landmarks



Verbal dialogue transcripts have been produced by OpenAI's Whisper model.

 

For more details on the data collection and processing pipeline, please see the paper.

Contents

The dataset is organised as:





README.md





sensory-data/



user-face-data/



P[participant_id]-C[camera_id]-[trial_number].csv: timestamp,confidence,success,pose_Tx,pose_Ty,pose_Tz,pose_Rx,pose_Ry,pose_Rz,X_0,X_1,X_2,X_3,X_4,X_5,X_6,X_7,X_8,X_9,X_10,X_11,X_12,X_13,X_14,X_15,X_16,X_17,X_18,X_19,X_20,X_21,X_22,X_23,X_24,X_25,X_26,X_27,X_28,X_29,X_30,X_31,X_32,X_33,X_34,X_35,X_36,X_37,X_38,X_39,X_40,X_41,X_42,X_43,X_44,X_45,X_46,X_47,X_48,X_49,X_50,X_51,X_52,X_53,X_54,X_55,X_56,X_57,X_58,X_59,X_60,X_61,X_62,X_63,X_64,X_65,X_66,X_67,Y_0,Y_1,Y_2,Y_3,Y_4,Y_5,Y_6,Y_7,Y_8,Y_9,Y_10,Y_11,Y_12,Y_13,Y_14,Y_15,Y_16,Y_17,Y_18,Y_19,Y_20,Y_21,Y_22,Y_23,Y_24,Y_25,Y_26,Y_27,Y_28,Y_29,Y_30,Y_31,Y_32,Y_33,Y_34,Y_35,Y_36,Y_37,Y_38,Y_39,Y_40,Y_41,Y_42,Y_43,Y_44,Y_45,Y_46,Y_47,Y_48,Y_49,Y_50,Y_51,Y_52,Y_53,Y_54,Y_55,Y_56,Y_57,Y_58,Y_59,Y_60,Y_61,Y_62,Y_63,Y_64,Y_65,Y_66,Y_67,Z_0,Z_1,Z_2,Z_3,Z_4,Z_5,Z_6,Z_7,Z_8,Z_9,Z_10,Z_11,Z_12,Z_13,Z_14,Z_15,Z_16,Z_17,Z_18,Z_19,Z_20,Z_21,Z_22,Z_23,Z_24,Z_25,Z_26,Z_27,Z_28,Z_29,Z_30,Z_31,Z_32,Z_33,Z_34,Z_35,Z_36,Z_37,Z_38,Z_39,Z_40,Z_41,Z_42,Z_43,Z_44,Z_45,Z_46,Z_47,Z_48,Z_49,Z_50,Z_51,Z_52,Z_53,Z_54,Z_55,Z_56,Z_57,Z_58,Z_59,Z_60,Z_61,Z_62,Z_63,Z_64,Z_65,Z_66,Z_67,AU01_r,AU02_r,AU04_r,AU05_r,AU06_r,AU07_r,AU09_r,AU10_r,AU12_r,AU14_r,AU15_r,AU17_r,AU20_r,AU23_r,AU25_r,AU26_r,AU45_r,AU01_c,AU02_c,AU04_c,AU05_c,AU06_c,AU07_c,AU09_c,AU10_c,AU12_c,AU14_c,AU15_c,AU17_c,AU20_c,AU23_c,AU25_c,AU26_c,AU28_c,AU45_c



user-pose-data/



P[participant_id]-C[camera_id]-[trial_number]-pose.csv: time,L0-x,L0-y,L0-z,L1-x,L1-y,L1-z,L2-x,L2-y,L2-z,L3-x,L3-y,L3-z,L4-x,L4-y,L4-z,L5-x,L5-y,L5-z,L6-x,L6-y,L6-z,L7-x,L7-y,L7-z,L8-x,L8-y,L8-z,L9-x,L9-y,L9-z,L10-x,L10-y,L10-z,L11-x,L11-y,L11-z,L12-x,L12-y,L12-z,L13-x,L13-y,L13-z,L14-x,L14-y,L14-z,L15-x,L15-y,L15-z,L16-x,L16-y,L16-z,L17-x,L17-y,L17-z,L18-x,L18-y,L18-z,L19-x,L19-y,L19-z,L20-x,L20-y,L20-z,L21-x,L21-y,L21-z,L22-x,L22-y,L22-z,L23-x,L23-y,L23-z,L24-x,L24-y,L24-z,L25-x,L25-y,L25-z,L26-x,L26-y,L26-z,L27-x,L27-y,L27-z,L28-x,L28-y,L28-z,L29-x,L29-y,L29-z,L30-x,L30-y,L30-z,L31-x,L31-y,L31-z,L32-x,L32-y,L32-z



dialogue-transcript/



P[participant_id]-[trial_number].csv: start,end,text






Limitations

Due to recording sensor malfunctions and processing library limitations, not all interaction scenario data contains all modalities. Verbal dialogue transcripts or user pose data may be incomplete or missing for a small number of interactions or recording angles.