DOI: 10.5061/dryad.tb2rbp078
title: Real-world human-robot interaction data with robotic pets in user homes in the United States and South Korea

description: Socially-assistive robots (SARs) hold significant potential to transform
 the management of chronic healthcare conditions (e.g. diabetes,
 Alzheimer’s, dementia) outside the clinic walls. However doing so entails
 embedding such autonomous robots into people’s daily lives and home living
 environments, which are deeply shaped by the cultural and geographic
 locations within which they are situated. That begs the question of
 whether we can design autonomous interactive behaviors between SARs and
 humans based on universal machine learning (ML) and deep learning (DL)
 models of robotic sensor data that would work across such diverse
 environments. To investigate this, we conducted a long-term user study
 with 26 participants across two diverse locations (the United States and
 South Korea) with SARs deployed in each user’s home for several weeks. We
 collected robotic sensor data every second of every day, combined with
 sophisticated ecological momentary assessment (EMA) sampling techniques,
 to generate a large-scale dataset of over 270 million data points
 representing 173 hours of randomly-sampled naturalistic interaction data
 between the human and robot pet. Interaction behaviors included activities
 like playing, petting, talking, cooking, etc.
description: Data was collected from robot sensors during deployment of the
 robots in user homes over a period of 3 weeks, using a sampling technique
 called ecological momentary assessment (EMA) in order to generate
 realistic real-word interaction data.
description: # Real-World Human-Robot Interaction Data with Robotic Pets in User Homes
 in the United States and South Korea
 The study included 26
 participants, 13 from South Korea and 13 from the United States. The
 participants were drawn from the general population aged 20-35 and living
 alone, approximately 70% of the sample was female. The robot included
 sensors that could detect light, sound, movement, indoor air quality, and
 other environmental health data in the vicinity of the robot (please refer
 to associated published papers for details). While sensor data was
 collected via the collars, self-reported interaction behavior modalities
 were collected simultaneously using the Expiwell EMA mobile app
 (). ## Description of the data and file
 structure Sensor data from the robot ("feature" data) was
 collected roughly 9 times per second, every minute of every day, across
 the three- week deployment period.  Meanwhile, the interaction modality
 data ("target" data) was collected via the EMA app randomly
 during each day about ongoing interactions during the preceding 15 minute
 window. The dataset thus includes the target interaction modality for
 prediction, as well as a series of columns of various sensor features. A
 detailed description of each individual column in the datasets is provided
 in the table below. Note that participants sometimes reported multiple
 modalities occurring during the same interaction period (on average
 roughly 2 modality types per interaction), which requires the data to be
 pre-processed appropriately. In our case, we chose to turn the dataset
 into a series of binary classification problems (e.g. petting vs
 not-petting), so that was a non-issue. Essentially this is time-series
 data, where all the rows from the 15 minute interaction can be used to
 predict the interaction modality of that interaction. There are two
 separate files in the dataset, one for participants from South Korea and
 one for participants for the United States. The data was collected using
 the same protocol in both locations. However, due to some software changes
 in between the two phases, the sampling frequency for the sensor data may
 be slightly higher for the US data than Korean data. As such 5 seconds
 worth of data equates to roughly 80 rows (US) but roughly 50 rows (Korea).
 We addressed that by sub-sampling the US data during pre-processing in the
 Python code, which is mentioned in the "Code/Software" section
 below. | Column | Description | | ------------------- |
 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | RowID | Unique ID for each row (XX\_XX\_XX), where the 1st XX indicates nationality, the 2nd XX indicates participant ID, and the 3rd XX indicates the interaction ID for each 15 minute interaction | | Real\_time | Timestamp indicating when sensors were read, down to the second level | | accX | Motion amount from accelerometer in X (lateral) direction (meters/sec<sup>2</sup>) | | accY | Motion amount from accelerometer in Y (lateral) direction (meters/sec<sup>2</sup>) | | accZ | Motion amount from accelerometer in Z (up/down) direction (meters/sec<sup>2</sup>) | | chord | Absolute amount of rotational motion since previous reading (calculated from X/Y/Z) | | arc | Absolute amount of rotational motion since previous reading (calculated from X/Y/Z) | | orientation | The orientation of the robot at a given time, using accelerometer manufacturer-specified thresholds. These numerical codes are equivalent to the orientation category labels below | | orientation\_cat | Specific orientation categories detected (labels), using accelerometer manufacturer-specified thresholds | | ir | Infrared reading (ambient light value), scaled from completely dark (0) to blinding light (1024) | | full | Alternate ambient light value (equivalent to ir), same scale as ir column | | iaq | Indoor air quality reading (scaled), measured in ppm | | staticIaq | Indoor air quality estimate (unscaled), scaled IAQ value is probably better, measured in ppm | | pressure | Air pressure reading, measured in millibars | | gasResistance | Electrical flow resistance in the presence of specific target gases. They are widely used in a variety of applications, such as environmental monitoring, industrial safety, and air quality control. Measured as pascal per m/s | | temperature | Indoor air temperature (celsius) | | humidity | Air humidity reading (relative percent) | | co2Equivalent | Indoor CO2 levels (ppm) | | breathVocEquivalent | Volatile organic compounds (ppm), such as those released by cooking, hairspray, etc. | | audioLevel | Ambient sound levels (dBV, typically 0-100 scale in our device setup) | | sound\_cat | Discretized audio levels into categories (quiet, moderate, loud) | | Modality\_cat | Interaction behavior modality (numerical code), used as the Target variable prediction | | Modality | Interaction behavior modality (label), same as Modality\_cat but text label rather than numerical code | ## Sharing/Access information If you make use of this data, please cite: **Bennett C.C., Sabanovic S., Stanojevic C., Henkel Z., Kim S., Lee J., Baugus K., Piatt J.A., Yu J., Oh j., Collins S., and Bethel C. (2023). Enabling robotic pets to autonomously adapt their own behaviors to enhance therapeutic effects: A data-driven approach. *IEEE International Symposium on Robot and Human interactive Communication (RO-MAN)*, pp.1-8.** ## Code/Software Machine Learning & Deep Learning was performed using custom code written in Python. That code also includes the necessary data pre-processing steps as well. The code is included with the datasets.