DOI: 10.5061/dryad.v6wwpzgsj
title: Human activities with videos, inertial units and ambient sensors

description: Worldwide demographic projections point to a progressively older
 population. This fact has fostered research on Ambient Assisted Living,
 which includes developments on smart homes and social robots. To endow
 such environments with truly autonomous behaviours, algorithms must
 extract semantically meaningful information from whichever sensor data is
 available. Human activity recognition is one of the most active fields of
 research within this context. Proposed approaches vary according to the
 input modality and the environments considered. Different from others,
 this paper addresses the problem of recognising heterogeneous activities
 of daily living centred in home environments considering simultaneously
 data from videos, wearable IMUs and ambient sensors. For this, two
 contributions are presented. The first is the creation of the Heriot-Watt
 University/University of Sao Paulo (HWU-USP) activities dataset, which was
 recorded at the Robotic Assisted Living Testbed at Heriot-Watt University.
 This dataset differs from other multimodal datasets due to the fact that
 it consists of daily living activities with either periodical patterns or
 long-term dependencies, which are captured in a very rich and
 heterogeneous sensing environment. In particular, this dataset combines
 data from a humanoid robot’s RGBD (RGB + depth) camera, with inertial
 sensors from wearable devices, and ambient sensors from a smart home. The
 second contribution is the proposal of a Deep Learning (DL) framework,
 which provides multimodal activity recognition based on videos, inertial
 sensors and ambient sensors from the smart home, on their own or fused to
 each other. The classification DL framework has also validated on our
 dataset and on the University of Texas at Dallas Multimodal Human
 Activities Dataset (UTD-MHAD), a widely used benchmark for activity
 recognition based on videos and inertial sensors, providing a comparative
 analysis between the results on the two datasets considered. Results
 demonstrate that the introduction of data from ambient sensors
 expressively improved the accuracy results.
description: The data collection was based on multimodal
 data from individuals performing activities of daily living. It considered
 inertial data from wearable devices, RGB and depth videos, as well as data
 from environmental sensors. All participants were adults without
 incapacitant physical or cognitive disabilities. The experiments were
 performed at the Robotic Assisted Living Testbed (RALT), Heriot-Watt
 University, Edinburgh Campus. A TIAGo robot, manufactured by Pal
 Robotics, was placed at the corner of the kitchen of the smart home,
 and recorded RGB and depth videos, while inertial sensors were placed at
 the participant's wrist and waist to record its movements. The
 recordings also included ambient sensors, i.e., switches at the cupboards
 and drawers, current measurements and presence detectors. All data was
 synchronised, in order to allow experiments on multimodal human activity
 recognition. The activities considered were "making a cup of
 tea", "making a sandwich", "making a bowl of
 cereals", "setting the table", "using a laptop",
 "using a phone", "reading a newspaper", "cleaning
 the dishes", and "tidying the kitchen."