Real-Time Classification of Everyday Fitness Activities on Windows Mobile By

Human activity recognition enables ubiquitous computing applications to leverage knowledge of people's context. The ability to distinguish a small set of activities has already shown the capacity to enable a wide range of important applications that integrate with fitness/healthcare [1], environmental sustainability [2], and entertainment [5]. However, the state-of-the-art in activity recognition is custom hardware that can be expensive to build, difficult to maintain, and an additional item for people to wear or carry around. Mobile phones provide an opportunity to augment an existing device that people already carry with them throughout their daily lives. In our work, we have implemented an activity classification system on the Windows Mobile OS paired with the HTC Touch Diamond mobile phone. Our system uses a threeaxis accelerometer to classify human activities including running, walking, bicycling, and sitting. It achieves considerable accuracy and a day's battery life. Introduction / Motivations Computing is growing ever more ubiquitous and context-aware. Systems falling into this category are those that are capable of communicating among each other and that are readily aware of their surroundings, including any individuals that may be present within these surroundings. One example is a system that interprets an individual’s geographic location during call initiation and connects the line to the appropriate phone number (whether work, mobile, or landline) given personalized settings [3]. An important point to observe is that although we may have once considered phone communication and GPS as two distinct processes, this line of separation between these two processes is becoming increasingly blurred and information is becoming more readily accessible among many distributed devices. Human activity classification is an important advancement in these efforts. Such systems provide the capability to interpret a user’s activities with the use of sensors, which are commonly distributed as a set of several sensors of various types (accelerometer, pressure, GPS, RFID, etc). Undoubtedly, the future of intelligent computing will need to rely, at least Baghe r i Ga r akan i | 3 partially, on a system’s ability to correctly classify a user’s activity and surroundings. Consider a system that is capable of adjusting room lighting based solely on a user’s activities that it interprets – perhaps upon sitting at ones desk the light would center around the desk, upon sleeping the lights would turn off, upon waking up the lights would gradually turn on, and upon dancing the lights would begin to strobe with different colors. Once provided with the correct activity, such lighting operations could easily be set to perform in response to each of the activities just described. The means of detecting these activities (and ideally a much larger set of common activities), however, remains a difficult problem to tackle. A preliminary step in advancing such systems is considering a limited domain of possible activities, perhaps composed of only a couple distinct values. This is an area that has thus far seen much more success and, despite its domain-limited capabilities, it can be a part of many important applications when integrated with areas such as: fitness/healthcare where individuals are encouraged to engage in fitness activities [1], environmental sustainability where individuals are encouraged to reduce use of public transportation [2], as well as for entertainment purposes such as the Mario Fit application that allows gameplay based on physical activity in lieu of a handheld controller [4]. This lattermost application of entertainment can also be viewed as encouragement of the same fitness/health benefits due to its interactivity [5]. Although certain custom hardware devices are made specifically for inferring activities, they are typically expensive to build, difficult to maintain, and an additional item for people to wear or carry around. It is thus no surprise that such efforts have not reached a widespread audience, despite their proven feasibility and obvious potential. Better hardware must be realized for activity inference that remedies the inadequacies of these custom-made devices. In the case of activity inference, and especially for long-term applications, it is further reasonable to consider a device that can at most times be assumed to be carried around by the user. As such, mobile phones appear to be a plausible solution for the choice of hardware. The internal accelerometer and GPS devices, which are Baghe r i Ga r akan i | 4 increasingly common as built-in features of these phones, appropriately provide a plausible means to infer activity. This paper discusses the implementation of real-time activity inference on the Windows Mobile platform using the HTC Touch Diamond mobile phone. Our system, which is based on similar findings and implementation on the iPhone [4], is capable of accurately classifying trained human activities, which can (as will be the case within this paper) consist of running, walking, sitting, and biking. Activity models are learned using a collection of features that are calculated based on samples from the 3-axis accelerometer. These models then serve as a means for recognizing new activities based on data that is collected in real-time. I will begin by discussing some background work that was the basis of our motivation and by addressing the closely related solution developed for the iPhone. I will then briefly overview the three stages of our approach, which include logging data, creating models, and classifying activities. Afterwards, I will discuss in greater detail our design choices and architecture, as well as some challenges we faced. I will then provide results and a discussion of the accuracy and robustness of our solution based on several experiments. Finally, I will briefly discuss some possible paths for future work. Background / Related Work As previously mentioned, activity inference systems that distinguish between a few discrete values have already been successful in the past, although with specialized hardware. A subset of these systems include the MSP belt-worn sensing device [6], the eWatch wrist device [7], and the Nike+iPod Sport Kit [8] seen respectively in Figures 1-3. All three systems mentioned here can classify activities similar to those which I propose within this paper – running, walking, standing, etc. Figure 1 An attachable belt-worn sensor used as part of the MSP system Baghe r i Ga r akan i | 5 Although all of these systems support activity inference, they have done so with a dependence on custom-made hardware. This is a problem because these devices are typically expensive to build and difficult to maintain. Perhaps more importantly, these are items that must be worn in addition to others that the user may be already wearing. In some cases this may also mean having to carry around a device that is awkward in appearance and/or placement. These limitations were recognized at the University of Washington and an effort to implement activity recognition on a mobile device was undertaken by T. Scott Saponas, Jonathan Lester, Jon Froehlich, James Fogarty, and James Landay. This effort was successfully implemented on the iPhone mobile device, making use of the device’s threeaxis accelerometer. Their goal was based on the idea that “utilizing commodity devices for activity inference provides researchers with access to robust, readily-available hardware and potentially large preexisting user bases” [4]. Furthermore, to reinforce this claim, they provided a set of open source tools for collecting, building, and running activity models that could be incorporated into future application development. As accelerometers become an increasingly popular component of mobile devices, it will be necessary to expand this idea to other platforms in addition to the iPhone. Although work is already being done to expand this idea to the Android platform, the issue has not yet been addressed for the Windows Mobile platform, which is used by many mobile devices on the market and is capable of supporting the very same activity inference techniques. In order to further pursue the objective of providing activity inference capabilities based on Figure 3 The ike+ iPod sensor is placed in a person’s shoe and communicates wireless to their iPod [9] Figure 2 A sensor designed to appear like a common watch as used in the eWatch system [8] readily-available hardware and researchers, it will be necessary to port this