#  2\. Introduction

The DE-ENIGMA DB will be the very first of its kind to be released for
research of behaviours shown by children with autism spectrum conditions
(ASC). It will contain (manually / semi-automatically) annotated audio-visual
recordings with respect to facial points, facial gestures, body postures and
gestures, various vocalisations, verbal cues, continuously valued target
affective states, valence, arousal, interest, stress, and prototypic examples
(templates) of rapport behaviour.

The Data Management Plan has been developed and agreed upon by the consortium
members, based on the DE-ENIGMA commitment to open access and to advancement
of the state of the art in the field by means of release of as much data and
as many software tools as possible.

#  3\. Objectives

The Multi-Modal Human-Robot Interaction for Teaching and Expanding Social
Imagination in Autistic Children (DE-ENIGMA) project aims to build robotic
technologies that can robustly and accurately track and recognise children’s
facial, bodily, and vocal behaviours and naturalistic interactions “in the
wild”, and react appropriately based on the observed child’s behaviour, with
the ultimate goal of helping autistic children and young people to enhance
their social communication skills in structured teaching with a therapist and
in everyday interactions.

Specifically, the project seeks to develop multimodal human-robot interaction
(HRI) methods that learn from interactions to:

  1. model the child’s behaviour, 

  

  2. map multimodal input to estimate the child’s affect, interest, physical response and rapport, and 

  

  3. adapt the interaction to the current context (the child’s cultural background, the task, and his / her level of interest and stress) in order to maximise the effectiveness of teaching socio-emotional skills and social imagination to autistic children. 

  

  

The robot will learn to understand the child’s vocalisation, their choice of
words, facial gestures, head and body gestures and postures and how these
modalities are combined to convey meaning. It will also examine the best ways
of changing the robot’s interactive behaviours in cases where there is lack of
engagement, lack of rapport, and increased behavioural responses by the child.

Learning models of the aforementioned behaviour suitable for machine analysis
depends on having suitable data recordings to learn from. Hence, an important
aspect of the DE-ENIGMA project lies in collecting suitable datasets of enough
labelled examples for building robust tools.

#  4\. Dataset

4.1. Data set reference and name

The dataset collected during the project will be denoted as the “DE-ENIGMA
DB”.

##  4.2. Data set description

A database of annotated audio, 2D and 3D recordings of interactions between
autistic children and (a) the robot, (b) the researcher and (c) their parents
made in structured teaching settings will be collected in the DE-ENIGMA
project.

In this project, we aim to recruit a total of 128 children on the autism
spectrum, half (n = 64) from London and South East of UK, and the other half
from Serbia. In each culture group, 32 children will be aged between 5 and 8
years and the other 32 between 9 and 12 years. The children from the two
cultures will take part in identical experiment settings. Namely, for each
culture, half of the children will be involved in robot-led teaching and the
other half will be involved in researcher / clinician-led teaching.

During the experiment, children within each age group will be randomly
assigned to either robot-led or researcher / clinician-led teaching
intervention, which will be implemented across multiple short sessions (10-15
minutes long) every 1-2 days for a maximum period of 3 weeks. We will follow
Howlin et al.’s (1998) approach to teaching perception, expression,
understanding, and social imagination related to four affective states:
surprise, happiness, anger and sadness. Specifically, the children will work
through the following Howlin et al.’s 6 Phases of teaching at their own pace,
with feedback given either by the robot or the researcher / clinician:

  1. Matching across same static emotional images. 

  

  2. Matching across different static emotional images. 

  

  3. Matching from dynamic “real” emotional displays to static images. 

  

  4. Identifying dynamic “real” emotional displays and expression that emotion. 

  

  5. Identifying dynamic “real” emotional displays and expressing that emotion in the same way. 6) Understanding own / others’ emotional states. 

  

  

After the intervention, the children will also take part in two teaching
sessions with their parents in the format as the earlier robot-led or
researcher / clinician-led sessions. These additional sessions will allow us
to examine whether the child has retained the skills learned during the
intervention and generalised such skill across instructional partners (in this
case, parents).

In the data collection experiment, all robot-led sessions will be facilitated
through a “Wizzard of Oz” (WoZ) setup. Namely, the robot will be controlled
directly by the researcher / clinician using a small keypad hidden from the
child’s view. Nevertheless, the robot will also perform a set of idle
animations autonomously, such as eye-blinks, head-turns, and minor hand
movements, to achieve a more “life-like” appearance.

All teaching sessions will be recorded in 3 modalities: audio, 2D video, and
3D video. For each modality, the following devices will be used:

  1. Audio: 4 professional omnidirectional microphones will be used as the main data sources. Among these microphones, two will be mounted close to the child and the researcher / clinician respectively. 

  

  

Another one will be mounted on the ceiling of the room directly above the
experiment setup. And the last one will be a wireless microphone carried
either by the child or the researcher / clinician (in case if the child is
unwilling or unsuitable to carry the microphone). In addition to these
professional microphones, we will also use the 2D and 3D video cameras’ built-
in consumer-grade microphones to make extra audio recordings. Each 2D camera
has 2 built-in microphones (except the one mounted on the robot’s chest that
only has 1), and the 3D camera (Microsoft Kinect) has 4 built-in microphones.
Therefore, for each session, a total of 18 (in researcher / clinician-led
sessions) or 19 (in robot-led sessions) distinct audio recordings will be
made.

  2. 2D Video: 5 (in researcher / clinician-led sessions) or 6 (in robot-led sessions) 720p HD webcams will be used to make video recordings at approximately 30 frames per second. The placement of these cameras is as follows: 2 cameras will be mounted at the opposite corners of the room to record from overview perspectives; 3 cameras will be placed close to the researcher / clinician and the child to capture their facial expressions (1 facing the researcher / clinician, 1 facing the child, and 1 facing both); and, in robot-led sessions, 1 camera will be mounted on the robot to capture the scene from the robot’s perspective. In addition, the 2D video captured by the 3D camera will also be recorded. These add up to 6 (in researcher / clinician-led sessions) or 7 (in robot-led sessions) 2D video recordings per session. 

  

  3. 3D Video: We will use one Microsoft Kinect to record 2D and 3D video and sound data. We will record the monocular image, the registered depth field of the scene, and the 4-channel sound, at a sample rate of approximately 30 Hz. 

  

  

The sensor placement is further illustrated in Figure 1. Note that the two
overview webcams at the corners and the microphone on the ceiling are not
visible in this picture.

Figure 1. Sensor placement in the experiment setup for robot-led sessions.

All recorded data streams will be time-stamped and synchronised. Specifically,
the internal clock of all data capturing machines will be synchronised to
universal time coordinated (UTC) using network time protocol (NTP). These
clocks will then serve as the reference clock to time-stamp all recorded data
on either perframe basis (for 2D and 3D video data) or per-buffer basis (for
audio data).

The DE-ENIGMA database will also include annotations of the recordings in
terms of facial landmarks and gestures, body postures and gestures, vocal and
verbal cues, continuously valued emotion dimensions, and rapport behaviours.
The data will be annotated in an iterative fashion, starting with a sufficient
number of examples to be annotated in a semi-automated manner and to be used
to train the algorithms in WP2WP4, and ending with a large database of
annotated facial and bodily behaviour recorded in the wild.

##  4.3. Ethical issues

The DE-ENIGMA project has obtained full ethical approval from UCL IOE Research
Ethics Committee and the Ethics Committee of Serbian Institute of Mental
Health. More details about the ethics approval can be found in DE-ENIGMA
Deliverable 1.1.

##  4.4. DB design and metadata

The DE-ENIGMA database will be organised into a flat list of folders, each
storing the data recorded during a single teaching session. The folders will
be named sequentially, reflecting the order of the sessions being conducted.
The layout of the folder’s content will be as follows.

  1. An index file detailing the meta-data of all files saved in the folder. This file will be parsed by the database web portal to generate the overall database catalogue. 

  

  2. The participant’s demographic information and their parents’ answers to various pre-intervention questionnaires. All information will be saved in Java-script object notation (JSON) files with a strictly defined semantics to support automatic search and filtering. The participants’ information will be anonymised by replacing their name with a randomly generated unique identifier (ID). 

  

  3. A set of 6 or 7 AVI files storing the 2D videos recorded during the session. All video data will be recorded at a frame rate of approximately 30 frames per second and will have a resolution of at least 1280 x 720 pixels. Each AVI file will be accompanied by a text file containing all frames’ time-stamp. 

  

  4. A set of 18 or 19 WAV files storing the audio data recorded during the session, all sampled at 44.1 kHz, except the data recorded from the Microsoft Kinect which is sampled at 16 kHz. Similar to the video recording, each WAV file will also be accompanied by a text file containing time-stamp information. 

  

  5. Time-stamped images together with depth field and 4-channel sound captured using the Kinect device stored as multiple files, including raw data for each modality, and RGB to depth mapping information. 

  

  6. A folder containing all available annotations as described in the previous section. Each type of labels will be saved in its own subfolder, of which the exact folder structure and / or file format may vary. "ReadMe" files will be included in the subfolders to explain the specific data organisation method. 

  

  

Along with the data, a comprehensive help document will be provided to give
detailed explanation on the format and semantics of all files included in the
database.

##  4.5. Data sharing

A web-portal will be developed for the DE-ENIGMA database, allowing easy
access and search of the available recordings according to various evidences
(i.e. annotations of key cues like facial actions, expressions, rapport) and
according to various metadata (gender, age, cultural background, occlusions,
etc.). This will facilitate investigations during and beyond the project in
the field of machine analysis of autistic children’s behaviours as well as in
other research fields.

The DE-ENIGMA database will be made available to researchers for academic-use
only. To comply with clauses stated in the Informed Consent signed by the
recorded participants, all non-academic / commercial use of the data is
prohibited. To enforce this retraction, an end-user license agreement (EULA)
has been prepared (see Appendix). Only researchers who have signed the EULA
will be granted access to the database. In order to ensure secure transfer of
data from the database to an authorised user’s PC, data will be protected by
SSL (Secure Sockets Layer) with an encryption key. If at any point, the
administrators of the DE-ENIGMA database and / or DE-ENIGMA researchers have a
reasonable doubt that an authorised user does not act in accordance to the
signed EULA, he/she will be declined the access to the database.

To increase the impact of the DE-ENIGMA project, we plan to organise data-
based research competitions. Partners of the project have done so previously
at the INTERSPEECH major speech conference (the INTERSPEECH ComParE 2009-2016
annual competitions) and premier ACM Multimedia venue (the AVEC series on
Audio / Visual Emotion Challenge has been organised six times up to now by
members of the consortium) and the premier IEEE Int’l Conf. Computer Vision
(satellite events on facial landmark localisation in static images and in
videos, in ICCV 2013 and ICCV 2015 respectively). These events have reached up
to 65 registered teams per event by now, thus generating significant impact in
the field. The number of downloads of data made available for these
competitions exceeds 2500 per data set. To be able to organise such events,
part of the data and labels need to be hidden temporarily from the outer
community. We plan to use parts of the DE-ENIGMA database to organise such
data-based research competitions.

##  4.6. Archiving and preservation (including storage and backup)

The DE-ENIGMA database will be stored on a data server hosted by the
Department of Computing, Imperial College London. The web-portal of the
database will be attached to the DE-ENIGMA project website. Both services will
continue to function indefinitely after the end of project without additional
cost. As a fail-safe measure, an additional backup copy of the DE-ENIGMA
database will be created and saved in external hard-drives.

##  4.7. Data destruction policy

The central repository of DE-ENIGMA database will be maintained indefinitely.
However, it is inevitable that parts of the DE-ENIGMA data may be stored
temporarily at other locations. For instance, during the data collection
experiment, pieces of raw data may reside on the data capturing machines’
local disk before they can be transferred to the central repository. To
prevent unauthorized access of the DEENIGMA data, all local copies of the data
will be permanently deleted once they are no longer in use. In addition, all
disks used to store these temporary copies will be labelled. At the end of the
project, these disks will be formatted and filled with random data repeatedly
(~10 times) to render their previous data content unrecoverable.

#  5\. Conclusion

The goal is for the DE-ENIGMA DB to become a publicly available benchmark
multilingual dataset of annotated atypical facial, bodily, vocal and verbal
interactive behaviour recordings made in naturalistic settings representing a
benchmark for efforts in automatic analysis of audio-visual behaviour in the
wild.

#  6\. EULA

**End User License Agreement**

**DE-ENIGMA Database** (www.de-enigma.eu)

By signing this document the user, he or she who will make use of the database
or the database interface, agrees to the following terms.

With database, we denote both the actual data as well as the interface to the
database.

##  1\. Commercial use

The user may not use the database for any non-academic purpose. Non-academic
purposes include, but are not limited to:

  * proving the efficiency of commercial systems 

  

  * training or testing of commercial systems 

  

  * using screenshots of subjects from the dataset in advertisements 

  

  * selling data from the dataset 

  

  * creating military applications 

  

  * developing governmental systems used in public spaces 

  

##  2\. Responsibility

This document must be signed by a person with a permanent position at an
academic institution (the signee). Up to five other researchers affiliated
with the same institution for whom the signee is responsible may be named at
the end of this document which will allow them to work with this dataset.

##  3\. Distribution

The user may not distribute the database or portions thereof in any way, with
the exception of using small portions of data for the exclusive purpose of
clarifying academic publications or presentations. **Only data from
participants who gave consent to have their data used in publications and
presentations may be used for this purpose.** Note that publications will have
to comply with the terms stated in article 5.

##  4\. Access

The user may only use the database after this End User License Agreement
(EULA) has been signed and returned to the Centre for Research in Autism and
Education at UCL Institute of Education, University College London. The user
may return the signed EULA by traditional mail or by email in portable
document format (pdf).

The signed EULA can be send to any of the following addresses:

Traditional mail:

Prof. Liz Pellicano

Centre for Research in Autism and Education (CRAE)

UCL Institute of Education

55-59 Gordon Square

London WC1H 0NU

United Kingdom

E-mail (pdf of EULA, after signing):

eula@de-enigma-db.eu

The user may not grant anyone access to the database by giving out their user
name and password.

##  5\. Publications

Publications include not only papers, but also presentations for conferences
or educational purposes.

**The user may only use data of subjects in publications if that particular
subject has explicitly granted permission for this. This is specified with
every database element.**

All documents and papers that report on research that use any of the DE-ENIGMA
Database will acknowledge this as follows:

“(Portions of) the research in this paper uses the DE-ENIGMA database
collected jointly by a European academic consortium consisting of Prof. Liz
Pellicano and her team at University College London, Prof. Evers and her team
of University of Twente, Prof. Maja Pantic and her team at Imperial College
London, Suncica Petrovic and her team at the Serbian Society for Autism, Prof.
Schuller and his team at the University of Passau, Prof. Sminchisescu and his
team at the Institute of Mathematics of the Romanian Academy, within the scope
of the ‘DE-ENIGMA: Multi-Modal Human-Robot Interaction for Teaching and
Expanding Social Imagination in Autistic Children’ project, financially
supported by the European Council under the European Council’s Horizon 2020
Work Programme (H2020-ICT-2015-688835) / Grant Agreement No. 688835”.

The user will send a copy of any document or papers that reports on research
that uses the DE-ENIGMA Database to Prof. Liz Pellicano or to
<publications@de-enigma.com>.

**6\. Academic research**

The user may only use the database for academic research.

##  7\. Warranty

The database comes without any warranty. Professor Maja Pantic and the iBUG
Group at Imperial College London, who oversee the database, cannot be held
accountable for any damage (physical, financial or otherwise) caused by the
use of the database. The iBUG Group at Imperial College London will try to
prevent any damage by keeping the database virus free.

##  8\. Misuse

If at any point, the administrators of DE-ENIGMA database and/or the Centre
for Research in Autism and Education and/or the iBUG Group at Imperial College
London have a reasonable doubt that the user does not act in accordance to
this EULA, s/he will be notified of this and will immediately be declined the
access to the database.

User: ___________________________________________

User’s Affiliation: _________________________________

User’s address: ____________________________________

User’s e-mail: _____________________________________

Additional Researcher 1______________________________

Additional Researcher 2______________________________

Additional Researcher 3______________________________

Additional Researcher 4______________________________ Additional Researcher
5______________________________

##  Signature: Date/place

_______________________