DOI: 10.25349/d9pp5j
title: Robot Control Gestures (RoCoG)

description: Building successful collaboration between humans and robots requires
 efficient, effective, and natural communication. This dataset
 supports the study of RGB-based deep learning models for
 controlling robots through gestures (e.g., “follow me”). To
 address the challenge of collecting high-quality annotated data from human
 subjects, synthetic data was considered for this domain. This
 dataset of gestures includes real videos with human subjects and synthetic
 videos from our custom simulator. This dataset can be used
 as a benchmark for studying how ML models for activity
 perception can be improved with synthetic data. Reference: de Melo C,
 Rothrock B, Gurram P, Ulutan O, Manjunath BS (2020) Vision-based gesture
 recognition in human-robot teams using synthetic data. In Proc. IROS 2020.
description: For effective human-robot interaction, the gestures need to have
 clear meaning, be easy to interpret, and have intuitive shape and motion
 profiles. To accomplish this, we selected standard gestures from the US
 Army Field Manual, which describes efficient, effective, and
 tried-and-tested gestures that are appropriate for various types of
 operating environments. Specifically, we consider seven gestures:
 <em>Move in reverse</em>, instructs the robot to move back in
 the opposite direction; <em>Halt</em>, stops the robot;
 <em>Attention</em>, instructs the robot to halt its current
 operation and pay attention to the human; <em>Advance</em>,
 instructs the robot to move towards its target position in the context of
 the ongoing mission; <em>Follow me</em>, instructs the robot
 to follow the human; and, <em>Move forward</em>, instructs the
 robot to move forward. The human dataset consists of
 recordings for 14 subjects (4 females, 10 males). Subjects performed each
 gesture twice, once for each of eight camera orientations (0º, 45º, ...,
 315º). Some gestures can only be performed with one repetition (halt,
 advance), whereas others can have multiple repetitions (e.g., move in
 reverse); in the latter case, we instructed subjects to perform the
 gestures with as many repetitions as it felt natural to them. The videos
 were recorded in open environments over four different sessions. The
 procedure for the data collection was approved by the US Army Research
 Laboratory IRB, and the subjects gave informed consent to share the data.
 The average length of each gesture performance varied from 2 to 5 seconds
 and 1,574 video segments of gestures were collected. The video frames were
 manually annotated using custom tools we developed. The frames before and
 after the gesture performance were labelled 'Idle'. Notice that
 since the duration of the actual gesture - i.e., non-idle motion - varied
 per subject and gesture type, the dataset includes comparable, but not
 equal, number of frames for each gesture. To synthesize
 the gestures, we built a virtual human simulator using a commercial game
 engine, namely Unity. The 3D models for the character bodies were
 retrieved from Mixamo, the 3D models for the face were generated on
 FaceGen, and the characters were assembled using 3ds Max. The character
 bodies were already rigged and ready for animation. We created four
 characters representative of the domains we were interested in: male in
 civilian and camouflage uniforms, and female in civilian and camouflage
 uniforms. Each character can be changed to reflect a Caucasian,
 African-American, and East Indian skin color. The simulator also supports
 two different body shapes: thin and thick. The seven gestures were
 animated using standard skeleton-animation techniques. Three animations,
 using the human data as reference, were created for each gesture. The
 simulator supports performance of the gestures with an arbitrary number of
 repetitions and at arbitrary speeds. The characters were also endowed with
 subtle random motion for the body. The background environments were
 retrieved from the Ultimate PBR Terrain Collection available at the Unity
 Asset Store. Finally, the simulator supports arbitrary camera orientations
 and lighting conditions. The synthetic dataset was
 generated by systematically varying the aforementioned parameters. In
 total, 117,504 videos were synthesized. The average video duration was
 between 3 to 5 seconds. To generate the dataset, we ran several instances
 of Unity, across multiple machines, over the course of two days. The
 labels for these videos were automatically generated, without any need for
 manual annotation.
description: The following files are available with the dataset:
 <strong>rocog_s00.zip, ..., rocog_s12.zip
 </strong>(26.2 GB): Raw videos for the human subjects performing the
 gestures and annotations 
 <strong>rocog_human_frames.zip, ...,
 rocog_human_frames.z02</strong> (18.7 GB): Frames for human data
 used for training and testing. Each folder also has annotations for
 gesture (label.bin), orientation (orientation.bin), and the number of
 times the gesture is repeated (repetitions.bin) 
 <strong>rocog_synth_frames.zip, ...,
 rocog_synth_frames.z09</strong> (~85.0 GB): Frames for synthetic
 data used for training and testing. Each folder also has annotations for
 gesture (label.bin), orientation (orientation.bin), and the number of
 times the gesture is repeated (repetitions.bin) 
 The labels are saved into Python binary
 struct arrays. Each file contains one entry per frame in the
 corresponding directory. Here's Python sample code to open these
 files: <pre style="margin-left:40px;"> import
 glob import os import struct frames_dir =
 'FemaleCivilian\\10_Advance_11_1_2019_19_6_29' frame_count =
 len(glob.glob(os.path.join(frames_dir, '*[0-9].png'))) fmt_str =
 '&lt;{}i'.format(frame_count) labels =
 list(struct.unpack(fmt_str, open(os.path.join(frames_dir,
 'label.bin'), 'rb').read()))</pre> The
 gesture labels are as follows: 0, advance; 1, attention; 2, rally; 3, move
 forward; 4, halt; 5, follow me; 6, move in reverse.