DOI: 10.82270/robotrio
title: RoboTrio2
title: RoboTrio2

description: Le corpus a été enregistré en 2018 (stage de M1 de Nathan Loudjani). Il est constitué de 23 enregistrements, où un robot téléopéré par la même personne (pilote) interagit à chaque fois avec 2 joueurs humains différents.

Ont été enregistrés : 3 canaux audio, 2 canaux vidéo + la vision stéréo subjective du robot 
(yeux et têtes mobiles), ainsi que tous les paramètres de mouvement (3 
degrés de liberté pour le cou) et d'articulation (6 degrés de liberté 
pour la mâchoire et les lèvres) du robot. Cela représente plus de 8h d'enregistrement multimodal. 

Les échanges verbaux de chaque paire de sujets ont été retranscrits en 2019 au LPLLa version 2023 du corpus (RoboTrio2) inclut des annotations supplémentaires, avec notamment les transcriptions hiérarchiques pour le pilote, avec les actes de langages parmi 23 catégories, et des annotations supplémentaires pour les deux joueurs humains (entre autres, différentes catégories de feedback).La documentation en anglais détaille tous les fichiers inclus, et notamment les ajouts de la V2.
description: 
The RoboTrio corpus:

These resources -- the RoboTrio corpus -- were recorded at the GIPSA laboratory during the 2018 Summer, in French.
They belong to the RoboTrio project (funded by a CNRS PEPS). The project
 involves GIPSA-Lab (Grenoble), LPL (Aix-en-Provence) and INT 
(Marseille). 

More details can be found here: http://www.gipsa-lab.grenoble-inp.fr/projet/RoboTrio/home.html

The corpus involves a collaborative game, played simultaneously by two humans. They sit in front of a social robot that plays as a game animator and referee. This robot is teleoperated by a human pilot:
 Gaze for the robot, eye vergence, head orientation, lip and jaw 
articulations, speech are captured on the human pilot in real-time and 
drive the robot. In this immersive teleoperation setup, the pilot
 sees through the robot cameras (stereo) and perceives audio through the
 ears of the robot, leading to a high level of embodiment. What is 
demonstrated by the pilot is a viable solution for the robot 
sensors/actuators to conduct a natural interaction with humans and 
successfully perform the intended task (social interaction with gaze and speech turnovers
 in a gaming scenario). Data streams and events that link to the 
perception as well as to the action are logged. These were primarily 
intended to build  autonomous behaviour models for a social robot (Nina robot, a modified iCub).

The recording conditions and the live corpus can be judged in photo or in video from these two files:
Example_Interaction-Photo.jpg
Example_interaction-From_expe_13.mp4

23 experiments were recorded (around 20 minutes each). The
 pilot is always the same, the human players are always different. The 
two players in an experiment are either both male or both female.

Content of the directories:


Example_Interaction-Photo.jpg:shows the setup, with the 
two humans facing the robot. One can also see the two cameras in the 
mobile eyes of the robot, and it's human-like ears (rich HRTF to support
 audio localization). In front of the robot, on the table, the support 
(Bamp;W square marker) allows for an augmented reality display for the 
pilot to conduct the game, announce the themes, and validate the player 
answers. One can also see 2 fixed cameras, directed toward one human 
player each. Not used by the robot, they are used afterward to better 
analyze the corpus (head tracking, human gaze, prephonatory gestures, 
face analysis).Example_Interaction-From_expe_13.mp4:this video excerpt 
illustrates the nature of the in-game interaction. The montage depicts 
what was recorded by the two fixed cameras (top line), what the pilot 
sees when he directs his head/gaze (bottom line : stereo view, virtual 
tablet) and what humans see (center: robot face with gaze and blinks, 
neck movements, and speech articulation).Audio_Humans:23 wav files, with stereo recording of the human players. 1 file per experimentTranscriptions_Humans:46 Praat files, one for every human player. 2 files per experimentAudio_Pilot:23 wav mono files. Recording used an ambient 
microphone near the robot, capturing mainly the pilot but also the robot
 motors, the robot power supplies, and the human players.Transcriptions_Pilot:11 ELAN files and video files (mkv 
format), for the selected experiments.
These are the full transcription of the corpus, including the pilot 
speech, his intents and his actions/keywords/references to users, 
direction, theme of the game, answers of the players.
Please contact us if you need more of these data.OpenFace_trackings:
11 trackings for the corresponding sequences, from three view points: 
for the left player (*_droite.csv files) and for the right player 
(*_gauche.csv files), using the HD fixed cameras. And using the mobile 
stereo pair that is embeded in the robot eyes (*_icub.csv files).
OpenFace documentation details how they are computed. These csv files 
have been gzip compressed. First CSV line is a header with the fields 
name.
OpenFace_videos:
This directory contains some of the HD videos (1280x768 @ 60Hz) with the points from the OpenFace tracking superimposed.
CSV_combined_annotations:
These gzip compressed files are time-resampled informations about all 
the streams and annotated states for the pilot and the two players. 
Easier to use for machine learning for example than to exploit the ELAN 
file along with the openFace trackings.
The first line of each csv file is a header that explicits the fields:
Time,Tilt_eye,Version_eye,Vergence_eye,X_eye,Y_eye,Robot_Gaze,Fixation_Duration,Pitch_head,Roll_head,Yaw_head,X_head,Y_head,Robot_Head,Robot_Speech,Robot_Intent,Robot_Addressee,Last_User,UserL_Gaze,UserL_Speech,UserL_Intent,UserR_Gaze,UserR_Speech,UserR_Intent,Seq_num



Contact address:

mailto:/frederic.elisei@gipsa-lab.grenoble-inp.fr