Group Activity Recognition Using Wearable Sensing Devices

One goal of ubiquitous and pervasive computing is to enable devices to adapt to the contexts and activities of the individuals interacting with them through a process called context or activity recognition. However, humans are social and spend most of their time in groups, and the behavior of the group is fundamentally different than the behavior of each individual in it, referred to as “emergent behavior” in social psychology. As the number of devices and users increases, using infrastructure and server-side processing for recognition becomes infeasible due to state space explosion and the “curse of dimensionality. ” As a result, peerto-peer (P2P) methods for detecting this behavior within the network of sensing devices are needed. Machine learning and classification are demonstrably effective for recognizing the behavior of individuals based on wearable sensing devices, especially accelerometers. However, systems which recognize contexts and activities of groups of individuals are required in order to support the needs of the group. The mobile devices we carry present an optimal platform, but aggregating data from the multitude of embedded sensors which observe behavior can congest networks. P2P approaches are attractive but challenging, since the behavior of the group cannot be observed at any single location. The challenges are (1) to understand the data which is required for recognition, (2) to detect different groups who may be in the same environment, and (3) to recognize the physical behavior or activities of the group, all in a P2P fashion. Furthermore, all of this must be done while (4) respecting the limited resources and primary functions of the sensing devices, e.g. mobile phones. The contribution of this dissertation consists of the following: • A formal definition of group activity recognition and differentiation from multi-user activity recognition. • Methods for using vibration sensors to reduce the power consumption of physical activity sensing. • Methods for reducing the consumption of the recognition toolchain using prediction for dynamic sensor selection. • An analysis of different abstraction levels for sensor observations with respect to group activity recognition. • Algorithms for detection group affiliation in P2P networks of mobile, wearable devices. • Algorithms for inferring emergent group behavior in P2P networks of mobile, wearable devices. This dissertation presents novel methods for reducing the power consumption of the machine learning tool-chain for recognition of human behavior in distributed systems. First a vibrational sensor is investigated with respect to its potential for activity recognition. The sensor proves useful for recognizing activities with highfrequency vibrational components such as riding a bicycle, as well as activities with impacts or concussions, such as walking or jogging. The approach consumes 50 times less than an accelerometer and samples high motion frequency information (3− 8 kHz) which the accelerometer can not. Methods for online sensor selection using the predictability of human behavior are explored, which can greatly reduce the energy footprint of activity sensing without sacrificing the accuracy of recognition. By predicting likely and unlikely activities for the immediate future, sensors required to differentiate unlikely activities can be turned off to conserve energy. Thereby energy savings of around 85% 90% where achieved in turn for a loss of 1.5 3 percentage points of recognition accuracy To approach the problem of distributed recognition of emergent group activities, the level of abstraction at which to fuse individual behavior information into group information is evaluated. Here power consumption and accuracy of inference is explored for using local features, local behavior classes, as well as unsupervised soft and hard clustering as the basis for inference of group activities. Using single-user activities saves 40% of power consumed by communication, but causes a 47% loss in recognition effectiveness due to technical issues Unsupervised clustering has a high potential, reducing energy consumption by 36%, while only causing 2.8% reduction in recognition rates. The unsupervised clustering abstraction level is therefore used as the abstraction level of choice for this dissertation. Using unsupervised clustering of behavioral observations, statements can be made about the behavior of a group, based on distributed observations of individual constituents. A method using the Jeffrey’s divergence of behavioral clusters and P2P communication called divergence-based affiliation detection (DBAD) is introduced, whereby group affiliations can be detected in multi group environments. When compared to a centralized approach, the DBAD reduces power consumption by up to 43% while maintaining affiliation detection rates and at the same eliminating the need for centralized resources. A method for using distributed probabilistic inference with loopy belief propagation (DPI-LBP) is presented which allows emergent group behavior to be recognized by the distributed network, further reducing energy consumption without the need for infrastructure. Again it is shown that performance is comparable to monolithic approaches, while reducing overall impact on the power consumption of the devices by up to a factor of 40. Local processing and memory usage increase slightly, but are well within tolerable boundaries.