Integrated Tool Chain for Recording and Handling Large,Multimodal Context Recognition Data Sets
David Bannach, Kai Kunze, Jens Weppner, and Paul Lukowicz
Embedded Systems Lab, University of Passau, {name.surname}@uni-passau.deas part of the EU ALLOW project. It is currently being
made available through the iPhone AppStore.

ABSTRACTThe demo will present a tool chain for recording, monitoring,labeling, and manipulation of complex multimodal data sets
for activity recognition. The tool chain is comprehensive(going from logging, through labeling, monitoring to postprocessing and managing the data), integrated (with all tools
being able to cooperate on joint data sets), and build around
comfortable graphical user interfaces.

CRN Toolbox

For data recording in general, e.g. with wearable or
stationary sensors that are not integrated into a mobile
computing platform, we use the Context Recognition
Network (CRN) Toolbox [8]. The CRN Toolbox allows fora fast implementation of activity and context recognition
systems and for simple recording of data streams. It utilizes
parameterizable and reusable software components andprovides a broad set of online algorithms for multimodal
sensor input, signal processing, and pattern recognition. Itfeatures mechanisms for distributed processing and support
for mobile and wearable devices.

Author Keywords Activity Recognition, Context, Sensing,Toolsets
ACM Classification Keywords J.0 General; D.0 General
General Terms Design, Documentation, Experimentation
INTRODUCTION

Activity and context recognition research is increasingly
moving away from small-scale lab setups towards large,
complex and often long-term experiments in unstructured
real life environments [1,2]. Recording and labeling thecorresponding data sets can be a significant challenge. So
can be the post processing (re-labeling, identifying missing
data, synchronizing different signals etc.) and working withit (e.g. selecting streams for recognition and training,testing). Based on our experience with a number of suchrecordings [3,4,5] we have developed an integrated tool
chain that supports the above functionalities. It consists of amultiplatform set of data loggers (with extensive labelingfunctionality) that feed into an open source database(CouchDB), combined with comfortable, GUI based
software for real time monitoring complex multimodal data
streams, and a graphical data labeling tool.

Figure 1: CRN Toolbox Concept: (1) a repository ofparameterizable software components including I/O device
readers and writers, filtering- and classification algorithms,components for splitting, merging, and synchronizing datastreams, (2) a graphical editor for specifying data flow andconfiguring components, (3) the runtime environment for
online executing the configured software components, (4)arbitrary external tools, communicating with the runtime, e.g.
live data stream plotting or another Toolbox (local or remote).

RECORDING OF SENSOR DATA STREAMS
Context Logger

As it is a growing trend to develop context sensitive
applications for mobile devices [6,7], we decided toimplement a data recording tool for the iPhone and theGoogle Android platforms. The application allows for aneasy recording and labeling of all sensors on the device(accelerometer (40-100 Hz), GPS (position, altitude, speed,course, accuracy), compass (magnetic- and true heading,geomagnetic data), sound, WiFi information (name, MAC
address, signal strength, protection, visibility) and the stateof the iPhone's proximity sensor. This Context Logger has
been used in a broad range of student projects and a longterm (one month) real life recording in a geriatric hospital

The concept of the CRN Toolbox (see Fig. 2) is derived
from the observation that most systems for online context
recognition are built from a relatively small set ofalgorithms
selectedfromsliding-window
signalpartitioning, standard time and frequency domain features,classifiers, and time series or event-based modelingalgorithms. Often such systems differ only in theirparameterization and combination of the algorithms.
Context Database

For the purpose of organizing unified data access we store
the recordings in a database. We use the Apache CouchDBto implement the Context Database because of its

Copyright is held by the author/owner(s).
UbiComp’10, September 26–29, 2010, Copenhagen, Denmark.
ACM 978-1-60558-843-8/10/09.

357

scalability and its support for replication. Users may easily
replicate parts of the database locally, e.g. for performancereasons, and still continue to work on exactly the sameinterface. The CRN Toolbox and Context Logger canupload data recordings directly to the Context Database.

in parallel, (3) assign labels to selected sections, (4)configure number and type of label tracks, and (5) generate
data traces by following a configurable scenario ofappearing and disappearing sensors. It integrates with thetools described above using the Context Database.

LIVE MONITORING OF RECORDINGS

During recordings with larger number of sensors it is
crucial to monitor the state of each sensor and also to
document the recording sessions. Sensor failures can occur
often (for various reasons) and would make whole multihours of recordings unusable if not detected early enough.
For this purpose we developed MASS (Monitoring
Application for Sensor Systems). MASS is a software toolthat helps monitoring and documenting experiments withmultimodal sensor setups. It features graphical and tabular
views for visualizing sensor uptimes and dynamic plots oflive sensor signals for quick checking of signal quality (seeFig. 3). Users may visually place sensor status icons on
images and maps to document sensor positioning and to get
an intuitive, location-based overview of sensor states. All
modifications and events during a recording are logged fordocumentation purposes. MASS integrates seamlessly with
the CRN Toolbox and Context Logger. The recording tools
advertise their service through Multicast-DNS such that
MASS can find them and connect to them. MASS receives
sensor status updates via UDP multicast messages from therecording tools and can monitor the sample rate to detect
anomalies. For displaying live sensor signals MASS
dynamically opens a TCP connection to the recording tool.

Figure 3: Labeling Tool screenshot.
REFERENCES
1. S. S. Intille, K. Larson, E. Munguia Tapia, J. Beaudin, P.
Kaushik, J. Nawyn, and R. Rockinson, Using a live-inlaboratory for ubiquitous computing research, inProceedings of PERVASIVE 2006, 349-365.
2. D. Cook, M. Schmitter-Edgecombe, A. Crandall, C. Sanders,B. Thomas. Collecting and disseminating smart home
sensor data in the CASAS project. Proc. of the CHI
Workshop on Developing Shared Home Behavior Datasets toAdvance HCI and Ubiquitous Computing Research, 2009.
3. Stiefmeier, T., Roggen, D., Ogris, G., Lukowicz, P.,
Tröster,G., Wearable Activity Tracking in Car
Manufacturin, IEEE Pervasive Computing, Special issue onActivity-Based Computing Vol. 7:2, 2008, 42-50
4. P. Lukowicz, et al. Recording a complex, multi modal
activity data set for context recogntion in: Workshop on
Context-Systems Design, Evaluation and Optimisation atARCS, 2010, Hannover, Germany, 2010
5. Stiefmeier, T., Ogris, G., Junker, H., Lukowicz, P., Tröster,G.: Combining motion sensors and ultrasonic hands
tracking for continuous activity recognition in a
maintenance scenario. In: Proc. 10th IEEE Int. Symposium
on Wearable Computers. ISWC 2006.
6. Yan, T., Marzilli, M., Holmes, R., Ganesan, D., Corner, M.: :mCrowd - A Platform for Mobile Crowdsourcing. In Proc.
SenSys’09, 2009.

Figure 2: MASS screenshot.
REVIEWING AND LABELING RECORDED DATA

7. Saponas, T.S., Lester, J., Froehlich, J., Fogarty, J. Landay, J.:
iLearn on the iPhone: Real-Time Human Activity
Classification on Commodity Mobile Phones. In Proc. ofthe 27th international conference on Human factors incomputing systems, 2008, 1043-1052.

Once a dataset is recorded it has to be prepared forutilization in training and evaluation of context recognition
algorithms. For that purpose all sensor streams must be
synchronized properly and the segments of interest must be
determined and labeled. We developed the Labeling Toolfor exactly this purpose. It synchronously displays several
video- and sensor data streams in parallel alongside withseveral label tracks (see Fig. 4). It allows the user to, (1)synchronize video- and sensor data streams by aligningthem in the timeline view, (2) playback and seek all streams

8. Bannach, D. Amft, O., Lukowicz, P.: Rapid Prototyping ofActivity Recognition Applications. IEEE Pervasive
Computing, Special issue on Activity-Based Computing Vol.
7:2, 2008, 22-31

358