Global Priors of Place and Activity Tags.

This paper describes an approach for creating detailed fullcoverage labellings of human activity. Our goal is to create global maps of physical positions labelled with a distribution over the most likely place name and most likely activity. We ground our ontology of labels as: the term that a person would want to display to someone before they initiate a communication. Rather than compiling a canonical list of possible labels, we piggyback the label data collection in a situated communicative exchange. Using ideas inspired by image segmentation and extended to support our goals we propose machine learning techniques for smoothing distributions across gaps in existing data. Introduction and Related Work The consumerization of sensor-laden platforms such as mobile phones, laptops, and vehicles, that also provide access to the collected data, is enabling data sharing and activity reasoning to scale to new levels. A difficult challenge which remains is to understand how the sensors from different users, ostensibly deployed for specific different reasons, can be principally combined and aggregated for new types of uses. In this paper we introduce a data collection approach and inference mechanism by which it may be possible to create dense geographic maps that are identified with place and activity labels. We obtain information about place and activity labelling in a supervised manner by piggybacking onto the communication practices of instant messaging and cellphone users. Related work on plan recognition (Kautz and Allen 1986; Wilensky 1983; Colbry, Peintner, and Pollack 2002), visionbased activity recognition (Jebara and Pentland 1999; Boger et al. 2006; Shi et al. 2004), and object-interaction-based activity recognition (Perkowitz et al. 2004; Philipose et al. 2004) formulates activities as collections of stereotypical sequential actions. When these approaches are applied to real-world data, they tend to be successful in modelling low-level behaviors in controlled environments or in very domain-specific applications (e.g., hand-washing). We believe that when applied to global scale environments these Copyright c © 2009, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. approaches will be of limited success. The wide variety of valid approaches to achieving a goal (executing an activity) and correspondingly, the wide variety of reasons why such an activity might want to be recognized, make a global statebased recognition system a daunting pursuit. For example, successfully recognizing that someone has “made tea” has important modelling implications if the application automatically calls social partners to the table as opposed to ordering new tea bags or monitoring caffeine intake. An alternative approach in the literature has been to treat activity recognition as a classification problem with less strict interpretation, if any at all, of the steps involved. This includes unimodal evaluations of activity and social context from audio (Choudhury and Pentland 2003; Stager et al. 2003; Stager, Lukowicz, and Troster 2004), video (Fitzpatrick and Kemp 2003), accelerometers (K. Van Laerhoven and Gellersen 2004) and RFID (Patterson et al. 2005). Although in this work we also approach the activity recognition task as one of classification, we emphasize multi-modal sensor evaluations like (Kern et al. 2004) and (Choudhury, Lester, and Borriello 2005) in order to avoid biasing our effectiveness on activities that are easily discriminated by one sensor (e.g., choosing activities such as “hammering” or “grinding coffee” because sensing is done with a microphone). In this paper we describe a system for collecting place and activity data from users. We characterize the existing data and propose two models for smoothing the estimations of place and activity names using loopy belief propagation. Additionally we suggest some relevant extensions that are necessary in order to account for application specific effects. Our goal is to use our data to create a dense map of priors over place and activity labels for every location on earth.