Towards Dynamically Configurable Context Recognition Systems
Kai KunzeDavid Bannach

Graduate School of Engineering
Osaka Prefecture University
1-1 Gakuencho, Osaka, Japanfirstname.lastname@gmail.com

Embedded Systems LabUniversity Passau
Innstr. 43, Passau, Germany
firstname.lastname@uni-passau.deAbstract
General representation, abstraction and exchange definitions are crucial for dynamically configurable context recognition. However, to evaluate potential definitions, suitable standard datasets are needed. This paper
presents our effort to create and maintain large scale,multimodal standard datasets for context recognition research. We ourselves used these datasets in previous
research to deal with placement effects and presented
low-level sensor abstractions in motion based on-bodysensing.
Researchers, conducting novel data collections, can relyon the toolchain and the the low-level sensor abstractions summarized in this paper. Additionally, they candraw from our experiences developing and conducting
context recognition experiments.
Our toolchain is already a valuable rapid prototyping
tool. Still, we plan to extend it to crowd-based sensing,enabling the general public to gather context data, learnmore about their lives and contribute to context recognition research.
Applying higher level context reasoning on the gathered
context data is a obvious extension to our work.

Motivation
As of today, Context recognition systems, the core enablersof pervasive computing, are still handcrafted for specific application scenarios. To find the right sensors, features andclassifiers to recognize non-trivial activities, is sadly more anart than science. Suitable context abstraction and representation formats for context would help for better re-use and selfconfiguration. Yet, they are missing. To develop and evaluate
these formats, there’s the need for standardized, multimodal
context data collections and corresponding software tools tomanage them.
As a basis for discussions about context representation
formats, we present the following in this paper:• a description and pointers to several multimodal context
datasets recorded with our toolchain for rapid prototyping.
• a short overview of our toolchain for recording, handlingand using large context data sets.
Copyright c 2012, Association for the Advancement of Artificial
Intelligence (www.aaai.org). All rights reserved.

• a placement categorization and low level sensor abstraction for motion based, on-body context recognition (evaluated on the datasets introduced).
• the proposal towards a crowd-sourced context data collection.
All tools introduced are open-sourced. The datasets are in
the process of being published.

Sensor-rich context data collection
Compared to the computer vision or speech recognition
fields, context recognition still lacks standardized datasets.
This makes it very difficult to compare recognition systems,
algorithms, models and the usefulness of higher level abstractions. However, compared to other research disciplines
recording context recognition datasets proves to be difficult
due to the following:
Diverse Sensing Modalities – Often a multitude of different sensors needs to be managed. One needs to deal withthe differing physical properties, sampling rate and otherpeculiarities. .If the application area is relatively unexplored, it is difficult to determine which sensing modality
will work best.
User/Environment Augmentation – From a research perspective, the more and the diverse the sensors are that areused in a setup the better. On the other hand, the moresensors the more complex the recordings and the moreburden is on the user.
Synchronization – Of course, the sensor streams also need
to be synchronized. Activity class assignments get especially tricky if the sensing devices do not supply a steadysampling rate.
Broad Application Scenarios – The user’s ”Context” a system is to recognize depends highly on a given application
scenario. As there are wide application areas for pervasive computing technology, it is not feasible to provide adataset for every use case.
To tackle a part of these problems and manage the complexity of the recordings better, we developed an integrated
toolchain for development, testing and deployment of context recognition systems.

TimeGetting)
UpBreakfast)
preparation

BreakfastGet)SalamiGet)KnifeModesOf)
lokomotion

Walking

Manipulative
Gestures

Open)DrawerReach)
DrawerMove)
DrawerLying)down...

Get)PlateGet)BreadRelease)
DrawerCleaning)
upStanding

WalkingClose)DrawerMove)BreadIIGet)GlassFetch)BreadReach)
BreadIReach)
DrawerMove)
DrawerRelease)
DrawerIIIIV

Figure 1: Experimental Setup for the ”Everyday living” dataset. On the left, the on-body sensing setup is shown. The picturein the middle depicts the sensors in the environment. On the right, you see a part of the hierarchical activities performed by thetest subjects. The experiments were recorded for the Opportunity EU Project. (Roggen, Calatroni, and Rossi 2010)

Toolchain for recording, post-processing and online
classification
The toolchain is composed of three parts, as depicted infig. 3. The first part deals with data collection. We can monitor all sensors, detect failures and signal degradation during this part. After collection, the datasets are stored in adatabase (an Apache CouchDB in our case). In the data management and annotation step, sensor signals can be plotted,
labelled and arranged (channel synchronizations can be adjusted using offsets etc.). Training data traces can be generated in this step. These traces are then used to train machinelearning classifiers and enable online context classificationin the last step of our toolchain (Bannach and Lukowicz
2011). Core components are explained subsequently:
Context Recognition Network Toolbox (CRNT) – TheCRN Toolbox allows for a fast implementation of context
recognition systems. It can be also used for synchronizedrecordings of data streams. It comes with software components provinding a broad range of online signal processing filters and machine learning algorithm. It supports
a broad range of mobile and wearable devices.
MASS (Monitoring Application for Sensor Systems) –MASS helps to monitor experiments. It features graphical and tabular views for visualizing sensor up-times anddynamic plots of live sensor signals for quick checking ofsignal quality.
Labelling Tool– This tool enables us to review and synchronize different sensor streams. We can also assign labels to given sensor signal regions and export data fortraining classifiers.
Fig. 2 shows some of the software tools used during thethree part process.

Available datasets
Even though we developed this context recognition software
stack, recording a dataset is still a lot of work. With presenting some of the datasets we recorded we show the usefulnessof our software stack and give other researchers the possibility to utilize our already recorded data for their hypotheses.

In the following, we introduce four larger datasets recordedby collaborating researchers and our group utilizing our integrated toolchain:
Everyday living – This is by far the largest data set. It wasrecorded as part of the Opportunity EU Project. The activities are from everyday living and include ”making asandwich”, ”pouring coffee”, ”eating” etc. 12 users repeated the setup 5 times each with a total of 72 sensors (in the environment, on the body and on objects), 10modalities and 25 hours of data (Roggen, Calatroni, andRossi 2010). Part of the data is already published under
http://www.opportunity-project.eu/challenge/.
Drink and Work –This data set contains mostly sitting activities, working on a computer and taking in food anddrinks. In total it includes 6 subjects with 4 reparations
each; one experimental run is around 30-40 min. The sensor setup includes motion, capacitive and force resistive
sensors on the body of the user (Cheng, Amft, and Lukowicz 2010).
Bicycle Repair – The experimental setup includes repair
activities on a bike (attaching a tire, opening screws etc.)with 6 test subjects. Again motion sensors and ultrasonicsensors on the body and in the environment were used(see (Stiefmeier, Ogris, and Junker 2006)).
Car Maintenance – This dataset contains car maintenanceactivities. 5 subjects are recorded with motion, force resistive sensors, an ultra wide band positioning system andRFID tags (Stiefmeier et al. 2008).
We are currently post-processing these datasets and arepreparing them for a public release. Interested researchers
can also contact us directly to gain early access to them.

Towards self-configuration and abstraction
Leveraging the multimodal experiments described, we could
develop a set of categorizations and low level sensor abstractions, focusing on motion based on-body sensing.

Dealing with on-body placement effects
The vast majority of context recognition research assumes
well defined, fixed sensor locations. In every day situations

Figure 2: The Context Recognition Network toolbox concept on the left shows a repository of combinable components. The
monitoring tool in the middle helps to keep track of sensor failures etc. during recordings. The labeling tool on the right helps
to assign context labels and readjust sensor data in post-processing. All software is made available as open source (Bannachand Lukowicz 2011).
sensor placement is often dictated by practicability, usability, and user acceptance constraints rather by the requirements of the recognition system. On-body device placement
may also change during deployment. To make context recognition more self-configurable, we developed the following
methods (Kunze 2011):• to detect if a device is carried on the body or placed at a
specific location in the environment (on a wooden table,in a closed metal compartment etc.),• to infer the coarse on-body location (on the wrist, on thetorso, ...) of a device, solely based on rotation and acceleration signals of the device.
• to utilize heuristics that significantly increase the robustness of motion sensor-based activity recognition with respect to sensor displacement.
• to detect the orientation of a device related to the users
body.
All methods have been empirically evaluated on the elaborate, realistic experimental setups presented above. These
inference methods can be used as a starting point for a context categorization of on-body devices. After detecting its
on-body placement and orientation, a device is able to decide which type of contextual inference it can provide. Forexample, a device on the wrist is able to recognize specificarm movements, whereas a device in the trousers pocket isbetter to infer modes of locomotion.

Sensor abstractions
We also started to investigate low level encapsulations ofsensors based on the large scale experimental setups. For example, angular velocity information derived from magnetic
field sensors can approximate and replace gyroscope signals(Kunze et al. 2010; Bahle, Kunze, and Lukowicz 2010). We
showed to which extend magnetic field sensors and gyroscopes are interchangeable for getting angular velocity (seefig. 5 for a signal example). This is a crucial first step towards dealing with changing sensor configurations and failures.
Although abstractions on the signal level are already valuable, common abstractions on higher levels are crucial for abroader adoption of context recognition. Take the conceptof location on mobile devices. The software development
kits from Android and iPhone encapsulate the complexity
for the developers. The developer does not need to choose if
he wants to use cell-tower triangulation, wifi or GPS for localization. The modality is chosen by the SDK depending on
the developer’s needs. The developer just receives location
coordinates.
We need similar encapsulations for other sensing modalities. For example, a straight forward abstraction to introduce
for human motion would be a ”modes of locomotion” sensor, that can recognize walking, standing etc. If this inference is achieved over a ball switch, gyro, accelerometer ormagnetic field sensor should be handled transparently.
Low level primitives, like modes of locomotion, should
be straight forward to define given the sensing modalities.
However, higher level abstraction primitives will be moredifficult, as they might depend on the application scenario,
the current circumstances or even the cultural background of
the user.

Crowdsourcing context data
In an attempt to get an idea what these higher level primitives might be and to streamline context data recording, westarted to develop a version of our toolchain for non-expert
use. As a next step, we are remodeling parts of our recoding and labeling tools. We provide a sensor logger for theAndroid and iOS platforms available in the corresponding
application stores for everybody to download. Smartphones
today come with a multitude of sensors very similar in quality to the dedicated devices used in research. A prototype,stripped-down version of our labeling tool already runs in abrowser (based on html and javascript) and can access thedata from the phone loggers.
Of course, we cannot hope that the toolchain, as it isnow, will directly be used by ordinary people. Although
there might be some very interested users, especially in thequantified-self community, the average user might feel overwhelmed and alienated by raw sensor signals. Therefore,
there are some obvious issues to discuss:• To engage regular people to collect sensor data, we needto provide additional value to them. This can be done inIs the device on the body?
EnvironmentalPlacement:
Active Sampling no

yes

On-body
Placement Recognition

Can we recognize the body part?
Can we determine
its placementin the environment?

Did the orientation
change?
yesinfer orientation

Is it displaced?

noyes

no

use as trained

Orientation Recognition
while walking

compensate

Heuristics
for Displacement

Figure 4: Categorization and contributions dealing with onbody placement effects of single sensing devices (Kunze2011).
1086420−2−4

Figure 3: Overview of the three step process of our integrated toolchain (Bannach and Lukowicz 2011)0246810samples

12141618
4x 10giving the users access to self-tracking information andsome rudimentary inference on their collected data.

Figure 5: Signal level estimation of angular velocity using
the magnetic field sensor vs. the gyroscope. To magnetic
field sensor estimation is shifted by six units to make it easier to compare the two signals.

• Some self-configuration and abstraction primitives need
to be introduced, to make handling of the tools easy andenable usable data collections.

• The use of our tools and expertise to design experiments
for evaluation of context representations.

• The user interfaces right now are geared towards expert
users and require a steep learning curve.

• Collaborations to setup and maintain a crowdsourced context sensing architecture and tools.

• A lot of users will just have smart phones. Is it possible
to provide the complete toolchain on smart phone like devices?

Acknowledgements• We cannot expect the same precision and quality in sensor data and labeling from the crowd-sourced collections.
On the other hand, there is a plentitude of additional data
sources on the phone to tap into (e.g. a lot of people use
twitter on their mobiles).

Summary
In this paper we presented our efforts towards creating standardized datasets (and tools to record them) for developing
context representations and abstractions. Extensions to ourwork would include:• The evaluation of high level context representations onthe datasets provided (e.g. testing ontology based contextmodels).

This work is partially supported by the EU-funded OPPORTUNITY project (http://www.opportunity- project.eu) andthe WearIT@Work project (http://www.wearitatwork.com).

References
Bahle, G.; Kunze, K.; and Lukowicz, P. 2010. On the use ofmagnetic field disturbances as features for activity recognition with on body sensors. In EuroSSC’10: Proceedings of
the 5th European conference on Smart sensing and context.
Passau, Germany: Springer-Verlag.
Bannach, D., and Lukowicz, P. 2011. Integrated Tool Chainfor Recording, Handling, and Utilizing Large, Multimodal
Context Data Sets for Context Recognition Systems. Proceedings of the 2nd Workshop on Context-Systems Design,Evaluation and Optimisation.

Cheng, J.; Amft, O.; and Lukowicz, P. 2010. Active capacitive sensing: Exploring a new wearable sensing modality for
activity recognition. Pervasive Computing.
Kunze, K.; Bahle, G.; Lukowicz, P.; and Partridge, K. 2010.
Can magnetic field sensors replace gyroscopes in wearable
sensing applications? In ISWC ’10: Proceedings of the 2010
11th IEEE International Symposium on Wearable Computers. Seoul, South Korea: IEEE.
Kunze, K. 2011. Compensating for On-Body Placement Effects in Activity Recognition. Ph.D. Dissertation, University
Passau.
Roggen, D.; Calatroni, A.; and Rossi, M. 2010. Walkthrough the OPPORTUNITY dataset for activity recognition in sensor rich environments. Adjunct Proceedings of theEighth International Conference on Pervasive Computing.
Stiefmeier, T.; Roggen, D.; Ogris, G.; Lukowicz, P.; andTroster, G. 2008. Wearable Activity Tracking in Car Manufacturing. IEEE Pervasive Computing 7(2).
Stiefmeier, T.; Ogris, G.; and Junker, H. 2006. Combiningmotion sensors and ultrasonic hands tracking for continuous
activity recognition in a maintenance scenario. Proceedings of the 2006 IEEE International Symposium on Wearable Computers.
l