Intelligent monitoring of human activity in cyber-physical-human systems: from smart medication bottles to occupant activity detection in buildings
Creators: name: Aldeer, Murtadha
Publisher: name: No Publisher Supplied
Year: 2023

description: Cyber-Physical-Human Systems (CPHS), serve as the underlying framework fora range of novel applications such as home assistance, healthcare, smart cities, smart manufacturing, and human-robot interactions. As a sophisticated subset of the Internet of Things (IoT) and Cyber-Physical Systems (CPS), CPHS places a strong emphasis on creating devices and environments that are easy for humans to use and interact with, ultimately helping people to achieve their goals. Although CPHS has opened up new avenues for scene and human behavior understanding, it brings a significant level of complexity to the system design and implementation. This complexity arises from the need to accommodate a variety of open behavior states, diverse device compositions, as well as privacy concerns.Desired applications are provided to the user as the smart, sensor-equipped object induces his physical interaction. It should be noted that the physical construction of the smart object is one factor that may affect its instrumentation as well as the user interaction with it. For example, a recent wave of research in the healthcare sector included the development of pill-logging systems in the form of pill bottles, that monitor when a pill is extracted. Automatic pill logging systems are an important technology for medication adherence, medication error prevention, and tracing. Nonetheless, a pill bottle with a turnable cap differs from that with a flippable cap. Hence, design difference forces the user to interact differently across these domains. Similarly, at a larger scale, Human Activity Recognition (HAR) in smart spaces is enabled by using sensing data, where sensor-based systems in the environment capture occupant activities and appliance usage. Captured environmental context may reveal characteristic properties that enable various applications like occupancy detection and activity monitoring, and essential applications for energy consumption optimization. However, the performed activity can be highly affected by the physical layout of the smart space. This research focuses on developing sensing systems and machine learning models targeting human behavior interaction change through physical interaction at different scales and lay under different research themes { smart health and intelligent buildings.The first three parts of the dissertation focus on human activity sensing and identification at the personal use scale through smart health devices. In the first part, we develop PatientSense, a system that investigates the identification of persons taking medication using a sensor-equipped pill bottle. PatientSense is an in-bottle patient discrimination system that is accurate, low-energy, and unobtrusive and uses off-the-shelf motion sensors. We believe this is the first smart medication-bottle system that accurately identifies the person using it. We evaluated PatientSense using in-house datasets, and the results showed that we can achieve high accuracy and recall in identifying a patient among a group of people (i.e. one versus all) or identifying the exact patient from that group.The second part of the dissertation introduces a system called Is That You Again?, which tackles how multi-bottle settings or long-term bottle use can affect their ability to log accurately, forcing frequent re-training/recalibration. To address these shortcomings, we apply domain adaptation across two different generations of 3D-printed smart bottles. We adopt a canonical domain adaptation approach, where we reweigh samples for a model trained on one bottle, and then apply it to unlabeled samples obtained from a new bottle. We also consider the use of deep learning-based models. More specifically, we apply a technique we call "input-type casting", whereby we recast the time-series input into the visual domain by generating a Recurrence Plot (RP). Then, we use a pre-trained image classification deep neural network and retrain it to identify user samples. We show that this approach not only outperforms the domain adaptation-based approach but also maintains a sustained level of performance for many users, providing a much higher model capacity for differentiation.In the third part of the dissertation, we address the problem of medication noncompliance, which occurs when a legitimate user intentionally deceives the system. To address this challenge, we propose systems based on smart pill bottles and wireless earbuds embedding an Inertial Measurement Unit (IMU). The information from smart pill bottles can tell if the bottle was opened if a pill was retrieved, and who retrieved it. Additional information from the smart earable can detect when a pill is swallowed a few seconds after opening the cap by using IMU data and machine learning algorithms.The fourth part of this dissertation focuses on human activity sensing in smart indoor environments at the building scale. One objective of monitoring human activity in buildings is to balance occupant comfort and energy consumption. Additionally, sensor data is required to enable desired services. However, privacy concerns remain significant for future buildings with deeply sensed environments. Therefore, a framework for data collection and active labeling is proposed to reduce the number of labels required to maximize event coverage and continuously learn the underlying distribution of various events. To dive into these studies, an IoT sensor and testbed are needed. For that reason, Maestro, an ambient sensing platform that enables smart environment applications, is developed. It integrates multiple low-level sensors for general-purpose sensing and activity recognition. Experiments were conducted to investigate the effectiveness of Maestro in enabling IoT applications while preserving user privacy and reducing labeling efforts by using active learning. Additionally, we conducted an empirical study of daily event recognition in smart space, investigating the associativity of events with sensor types and sensor positions.