UBICOMP '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USA

Kraken.me – Multi-Device User
Tracking Suite
Abstract

Immanuel Schweizer
TU Darmstadt Telecooperation
Hochschulstrasse 1064289 Darmstadt, Germany
schweizer@tk.informatik.tudarmstadt.de

An in-depths understanding of human activity is arelevant contribution to the design of interactive systems
to support human activity. This is of explicit relevance forassistance systems building on prediction andrecommendation.
However, the understanding of human activity is limited.
Albeit the omnispresence of smart phones and computers,
the actual execution of complex activities with thosedevices in relation to context factors is not completely
understood. One possible reason is the limited amount ofactivity related data to perform actual research.

Benedikt Schmidt
TU Darmstadt Telecooperation
Hochschulstrasse 1064289 Darmstadt, Germany
benedikt.schmidt@tk.informatik.tudarmstadt.dePermission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are notmade or distributed for profit or commercial advantage and that copies bear
this notice and the full citation on the first page. Copyrights for componentsof this work owned by others than ACM must be honored. Abstracting withcredit is permitted. To copy otherwise, or republish, to post on servers or toredistribute to lists, requires prior specific permission and/or a fee. Request
permissions from permissions@acm.org.
UbiComp ’14 , September 13 - 17 2014, Seattle, WA, USA
Copyright c 2014 ACM 978-1-4503-3047-3/14/09...$15.00.
http://dx.doi.org/10.1145/2638728.2641307

853

In this paper, we present the Kraken.me framework toaddress this lack of information. Kraken.me is the first
tracking suite to offer integrated tools for mobile, social,and desktop tracking. It is also, to our knowledge, thefirst tool to emphasize the collection of data from both
physical and soft sensors. In this paper, we will introduce
the overall architecture, system components, and future
research ideas for Kraken.me.

Author Keywords
User tracking suite, Crowd Sensing, User-centric Sensing

ACM Classification Keywords
H.1.2 [Models and Principles]: User/Machine Systems

UBICOMP '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USA

IntroductionMore and more applications and services support users byeliciting preferences, habits and plans. Consider
recommender systems or systems for personal assistance,
e.g, Google Now, Microsoft’s Cortana or Apple’s Siri. To
address the resulting requirements, knowledge abouthuman activity is crucial. Therefore an in-depths
understanding of human activity, short term as well as
long term is an ongoing research topic for science andindustry. For this research activity data is required. Data
which emerges in the daily business of many companies
such as Google and Facebook whose business includes thecollection of information about individuals and their world.
Activity research in the industry is beneficial as activity
knowledge is a business value resulting in better services.
However, due to the business value the gained knowledge
is only shared partially.
For research this poses a challenge. How to provide
researchers with access to data that is as rich or evenricher than what companies collect from millions andbillions of users? For example, one of our current research
directions is intention prediction and personal assistancesystems. But, whatever type of prediction one wants toresearch, the initial requirement is always the same. We
require a suitable data set for the topic at hand.
In the past people have collected very focused data sets
for individual research questions [6]. These data sets are
great to answer these specific questions. However,research domains like intention prediction will benefit fromdata sets which capture many different characteristics andattributes to enable exploratory studies and enable thecomparison of approaches which build on different
attributes of the data. Even the data sets available forcompanies will focus characteristics and attributes fromthe core services offered by the respective company. We
envision a data set with a broader scope. This is where
research is in a unique position as it can be vendor
agnostic. And this is why we build Kraken.me1 . Kraken is
the first suite to integrate tracking tools for mobile, social,and desktop. But we cannot only track users in a
multi-device environment. We also emphasize theintegration of both physical (e.g., location, acceleration,etc.) and soft (e.g., software interaction, contacts,
calender, etc.) sensors.
To give you an overview about the depth collected by
Kraken.me, we will shortly specify the challenge tackled
by Kraken.me, scope and design principles. Then we will
introduce the overall system architecture of Kraken.me.
Afterwards, we will discuss every Kraken.me tool andwhat information is collected. Last but not least, we want
to share and discuss what we see as the overall challenge
and possible contributions of the Kraken.me system down
the road.

Challenge
The purpose of Kraken.me is the collection of data about
activities in terms of interaction histories and user relateddata.
We know that interaction histories are an invaluable
resource for research in the context of human activity anduser support. An interaction history provides a detailed
log about the interaction of a user with a device orsystem. The relevance of this data for the analysis ofhuman activity has been recognized early [1]. However,interaction histories have always been created forindividual research projects and generally focused on avery specific kind of interaction information.
1 https://kraken.me854

WORKSHOP: HASCA

Figure 1: Architecture of Kraken.me

To actually understand interaction and respective
activities a more holistic approach is required. Consider
the way people interact with devices. The windows user
who collects information from a PDF file and a browser,
pastes the data into a spreadsheet, asks a colleague in a
messaging application for details and finally sends thecomplete spreadsheet to a mailing list. Consider themobile user who quickly switches between multiple apps
to plan a subway trip while coordinating a meeting with afriend via phone while accessing the named spreadsheet.
These examples illustrate that interaction goes beyond
application and even device boundaries. Individuals
develop complex interaction processes to use information
processing and developing tools in an optimized way torealize complex activities and pursue respective goals. A
connection between interaction, activity and goal which
has not yet completely been understood.
For research on user activity, we see a demand for crossapplication and cross device data about users: a tool suiteto create extensive and rich interaction histories.

855

Furthermore, interaction is closely related to theinteracting subject, a social individual. Therefore, we
argue for an integration of information from social
networks with interaction histories.

Scope and Design Principles
Primarily, Kraken.me is a platform for the creation ofextensive and rich interaction histories across multiple
devices combined with information about users. Theinteraction histories can be connected to data from social
networks for more in-depth information.
Furthermore, Kraken.me is a platform for the rapid
prototyping of research results in terms of user support
functions like recommender and prediction algorithms.
Based on a modular architecture and online update
mechanisms the development and delivery of suchfunctions is simple.
The design of Kraken.me was guided by the following
basic - partially contradictory - design principles:

UBICOMP '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USA• Greediness: Kraken.me collects all available data.
Exception to this rule is information required toaccess services and transactions (e.g. passwords,
user names, etc.)• Openness: Users are informed about the collected
data, users can access their data and they can
trigger the deletion of their data.
• Self-determination: Users can decide which data isactually collected by the system. While thegreediness results in the integration of as many
sensors as possible the self-determination regulates
the actual collection. Kraken.me can be set into adeaf mode without collection of data and considers
indication of privacy like the inPrivate mode
provided by most web browsers.
• Secureness: The data collected by Kraken.me needs
to be stored and transmitted securely.
• Unobtrusiveness: The data collection process of
Kraken.me is intended to be as unobtrusive as
possible. This means that no actual user interaction
is required during the data collection process, accessof the system is simple and battery life and transfer
data volume are conserved.
• Usefulness: Kraken.me design considers the use of
the data to actually help the user by providing
information and services based on the collected
data.

Kraken.me Architecture
Kraken.me aims at tracking users across devices. The goalis quite literally to use each and every sensor available oneach of those devices. The overall architecture of
Kraken.me is depicted in Figure 1.

856

Currently, Kraken.me offers access to three main sourcesof information. A mobile application available for Androidthat offers both data collection from soft and physicalsensors and a first visualization. A desktop monitor for
Windows that runs in the background and collects datafrom soft sensors (e.g., application interaction, accessed
documents, mails, browser history, etc.). Social networks
are the third source of information. Users can connect
their social logins (e.g., Facebook, Google+, Twitter,etc.), which, given the user’s permission, allows us to
crawl the respective API.
We decided to go for a server architecture hosted on
Amazon Web Services for scalability as the amount ofdata collected per user is significant. The central server
has a number of tasks. The most crucial being data
storage. All data is stored using flat tables to retain theraw data in almost original structure. These flat tables arelater enriched into more valuable higher level structures,e.g., graph databases. The raw data is not anonymized as
it is not feasible given the amount of personal information
stored on the system.
The next sections will introduce all Kraken.me tools andtheir current data collection profile.

Kraken.me Mobile Monitor
Mobile phones are unique tools in that they provide accessto a host of physical sensors (e.g., accelerometer,
location, temperature, heart rate etc.) and a host of soft
sensors (e.g., mails, calendar, contacts, etc.). Hence,
mobile phones are at the core of most comprehensive
personal tracking suites [2]. Kraken.me is not different,
offering an Android application to track both physical andsoft sensors. In designing Kraken.me, we drawn heavily
from our experience developing otherWORKSHOP: HASCA

Figure 2: Kraken.me Mobile

applications [11, 10]. Table 1 illustrates the data collected
if the users opts for the full tracking profile.

Data

Description

Sound pressure

Most sensors are easily available using the API or somestandard procedure. However, since we want to dig deeper
into the user’s interaction with the device we used a trickthat makes tracking on Android even more powerful. For
Kraken.me to work the user is asked to enable access tothe accessibility framework. The accessibility framework is
for example meant to enable applications for the visually
impaired, to navigate the operating system. Once anapplication is registered with the framework every
interaction with the device is sent to the application. In
theory, this allows for unfiltered tracking of every userinteraction. This is also the main reason Kraken.me isavailable for Android only at the moment. This depth ofdata cannot yet be collected on iOS.

Ring modedBSPL value captured withthe microphone
Is the phone on silent, vibration, etc.
Readings from the accelerometer
Activity Recognition provided by the Android API
Lux as provided by the Android API
The active and all available
network connections
All visited web sites excluding inPrivate mode
All saved contacts
All saved calendar eventsand reminders
Location as provided by theAndroid API
The call history
Time span apps are in foreground
Statistics about key strokes

With this information, Kraken.me is able to identify andvisualize (cmp. Fig. 2) means of interaction with thedevice, i.e. foreground app and interaction with therespective app, and relate this information to the locationand physical surrounding of the user. In the future wewant to extend both physical sensors and logging of userinteractions. Especially health related data, e.g., step
count, or environmental data, e.g. temperature andhumidity, are not collected currently.

Acceleration
Activity
IlluminanceNetwork Connectivity
Browsing History
Contacts
Calendar
Location
Call Log
Foreground apps
Keyboard

Table 1: Data collected by Kraken.me Mobile

What Kraken.me mobile brings to the table is the
combination of physical sensor with an interesting take onsoft sensors and interaction data.

Kraken.me Desktop Monitor
Kraken.me mobile is already an interesting stand alone
tool for personal data collection. However, with theaddition of desktop data it is even more valuable. WIMP
style interfaces have a high relevance for the execution of857

UBICOMP '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USA

work with computers. The Kraken.me desktop monitor
focuses the interaction of the user with the Windows
operating system. The monitor captures the foregroundprocesses and the interaction with the respective
processes. For some applications additional information
like the accessed information objects (e.g., text
documents, websites) is captured. Table 2 provides an
overview of the information monitored by the desktop
monitor if the broadest collection profile is chosen. The
desktop monitor is based upon our prior work [9] andrelies heavily on functionalities provided by the UIAutomation framework and available process diagnostics.

files. This means we can extract tracking data for themost common file types (cmp. Fig. 3).
Data

Description

Foreground Process

Program in interaction focus
Clicks and focus ofuser interface elements, e.g. buttons, menus, etc.
with labels
The information
objects accessedwith the officesuite(Microsoft
Outlook,Word,
Excel,
Powerpoint) and Adobe
Acrobat
URL, title andfulltext of website
accessedwith
Chrome, Internet
Explorer, Firefox
Content in the clipboard
Indicator for accessed information
objects

Interaction with User Interface

Office Applications

Web Browsers

Clipboard

Figure 3: Kraken.me Desktop Monitor

Kraken.me Desktop has a lot of overlap with the mobile
monitor when it comes to soft sensors, e.g., emails,
contacts. It augments some sensors, e.g., the browsing
and location history. As people might only be using thedesktop monitor this is already crucial.
The desktop monitor goes above and beyond the mobile
monitor by tracking the information objects, i.e. thedocuments people work on. This tracking of information
objects is enabled for PDF, Word, Excel, and Powerpoint

858

Files locked by process

Table 2: Data collected by the Kraken.me Desktop Monitor

Kraken.me Social Connector
The Social Connector is used to connect Kraken.me toexisting social networks. To use the social connector,
users login through the Kraken.me website to explicitlyWORKSHOP: HASCA

connect their Kraken.me account with social networks.
Based on collection profiles users can influence theamount of data actually collected from the connected
networks. Once the connection is established, therespective API of social networks is accessed. As long as
the connection exists, the connected social networks foreach account are crawled once a week and new orchanged information is persisted. The initial version of
Kraken.me comprises connectors for Facebook, LinkedIn,
Google+, Xing, Foursquare, and Microsoft Live.

App is persisted in a flat, unfiltered form. Such a directinterface is useful for the creation of basic data sets to beprocessed with machine learning techniques. Second, ahigh level perspective is created using graphs. Researchers
can create models which provide different perspectives on
the data based on so called aggregators. An aggregatorcreates a Neo4J Database instance for a specific subset ofthe data. The high level perspective is of specific use forthe human interpretation of the data and for application
development.

Since each network offers different data, Table 3 provides
an overview of the information which is collected for
Facebook. Facebook offers the richest data set, hence, it
is given as a reference point.

In the future we plan to investigate offline and online
analysis of the raw data layer as well as finding meaningful
representations on the graph layer.

Kraken.me webpage
Data

Description

ProfileActivity Stream
Location
PhotosProfile information
Recent network activities
GPS position
Photos of the user and photos
accessible by user
List of friends
Metadata of exchanged messages without message text

Friends
Internal Messages

Currently, Kraken.me (cmp. Fig. 4) offers links todownload the desktop and mobile monitor. A user can log
into his profile and connect different social accounts andsee some basic visualizations. Over time we hope that
Kraken.me can morph into a showcase for our analysis,
can offer personal assistance, and a rich visualization ofinteraction data.
For the time being, however, the most important sites areprobably Download and Privacy.

Table 3: Data collected through the Facebook API

Initial Directions
Most people are active in at least one social network.
Thus, a lot of social data is readily available.

Data storage
Data collected by any of the aforementioned tools of
Kraken.me is stored following two different strategies.
First, a strategy of unfiltered, complete data sets. The
raw data provided by the Kraken.me applications, thesocial connector, the Desktop monitor and the Android

859

A variety of research directions benefit from the datacollected by Kraken.me. In the following, we discuss ourcurrent research directions with first topics.
Data collection and processing We can already ask
interesting questions before any meaningful analysis.
How can data collection be as unobtrusive andenergy efficient as possible? Which data can be

UBICOMP '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USA

Figure 4: Frontpage of Kraken.me

processed locally? For example, can we reliably
implement Named Entity Recognition on mobile
phones and not transfer full text? These are only
some of the questions we want to answer.
Sociology and interaction research The nature ofhuman activity and relationships between
interaction, activity and goal is an important topic.
It is the foundation for the development of future
recommender and predictive systems. Our first860

activity is the combination of a diary study with the
Kraken.me data set to analyze the relationshipbetween interaction, activity and goal. More specificwe will ask people about their plans for the future
and use their diary and our tracking to understandthe difference between planning and execution.
Mnemonic support, Awareness and Quantified self
Interaction data helps people to recall what they
have done in the past, thus, getting an awareness of

WORKSHOP: HASCA

their past. The data collected with Kraken.me willbe the foundation for the development of searchservices on historic data (answering questions like”what did I do yesterday morning?”) and to
perform analytics on the individual history (relevant
topics, trending topics, time spent with tasks, etc.).
These applications are of specific relevance for thequantified self community which is also in the focus
of our participant groups.
Recommendation, Predication and Assistance The
historic data contains clues for predicting andrecommending future actions. We want tounderstand and predict the intention of users. This
will allow for much more powerful personal
assistance. Going beyond hard coded use cases intoa general intention theory.

Soft sensors for interaction tracking are software programs
which collect information about the interaction withapplications. Soft sensors have been used early for the
analysis of user activities with shells [1]. Other
applications of soft sensors comprise the clustering ofrelated user activities [8], use of the data for activity
recognition based on machine learning [7] or for learning
analytics in the domain of technology enhanced learning[3]. Different examples show the success of crawlers forthe collection of data from social networks [4].
The overview shows that the main contribution of
Kraken.me in comparison to the existing work is theintegration of several existing data collection techniques.
The combination of soft and physical sensors with thecrawling of social networks to create interaction dataenriched by social information is unique.

Conclusion

Related Work
The Kraken.me tools realize a crowd sensing application
which combines interaction monitors and crawlers. In thefollowing we give a brief overview of related work on those
three aspects.
Crowd sensing leverages crowds for the collection of data.
From historic examples of spotting the route of monarchbutterflies following newspaper requests the idea and theterm especially stands for the use of mobile devices for thecollection of data within a community [5]. Existing crowd
sensing applications generally focus a very specificproblem, like the collection of data about air pollution,
creation of traffic information etc. [6]. Kraken.me applies
the idea of data collection with a specific focus on theindividual, considering data from physical sensors andfrom soft sensors for interaction tracking and crawling.

861

In this paper we introduced Kraken.me. Kraken.me
provides tracking tools for mobile, desktop, and social.
The additional integration of soft and physical sensors
enables fine-grained user profiles.
The groundwork has been laid and we hope to collect acomprehensive data set over the next months. We alsoplan on sharing some anonymized subsets with theresearch community. Another obvious addition for the
future are mobile and desktop monitors for other platforms
to include even more possible users. Also, as more data iscollected, we will increase the quantity and quality ofvisualizations, and offer more active support to users.
All of this will help answering some of the questions
outlined above, some questions we might not even thinkof today, and maybe even advance our understand of
human behavior at large.

UBICOMP '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USA

References[1] Liam Bannon, Allen Cypher, Steven Greenspan, and
M.L. Monty. Evaluation and analysis of users’
activity organization. In Proceedings of the SIGCHI
conference on Human Factors in Computing
Systems, pages 54–57. ACM Press, 1983.
[2] Ning Ding, Daniel Wagner, Xiaomeng Chen,Y Charlie Hu, and Andrew Rice. Characterizing andmodeling the impact of wireless signal strength on
smartphone battery drain. In Proceedings of the
ACM SIGMETRICS/International conference on
Measurement and modeling of computer systems,
2013.
[3] Erik Duval. Attention Please ! Learning Analytics for
Visualization and Recommendation. In Proceedingsof LAK11: 1st International Conference on Learning
Analytics and Knowledge. ACM Press, 2011.
[4] M Faloutsos, T Karagiannis, and S Moon. Practical
Recommendations on Crawling Online Social
Networks. IEEE Journal on Selected Areas inCommunications, 29(9):1872–1892, 2010.
[5] RK Ganti, Fan Ye, and Hui Lei. Mobile
crowdsensing: Current state and future challenges.
IEEE Communications Magazine, (Section 2), 2011.
[6] ND Lane, Emiliano Miluzzo, and Hong Lu. A surveyof mobile phone sensing. IEEE Communications
Magazine, (September):140–150, 2010.
[7] Andreas S Rath and Stefanie N Lindstaedt. UICO:
An ontology-based user interaction context model for
Automatic Task Detection on the Computer
Desktop. In Proceedings of the 1st Workshop on
Context, Information and Ontologies, number Ciao.
ACM Press, 2009.
[8] Tye Rattenbury and John Canny. CAAD: anautomatic task support system. In Proceedings of the
SIGCHI conference on Human factors in computing

862

systems, pages 696–706. ACM Press, 2007.
[9] Benedikt Schmidt and Eicke Godehardt. Interaction
Data Management. In Knowlege-Based andIntelligent Information and Engineering Systems.
Springer, 2011.
[10] Immanuel Schweizer. MoSeS: Mobile Sensing
System. In Informing Future Design via Large-Scale
Research Methods and Big Data, 2013.
[11] Immanuel Schweizer, Roman Bärtl, Axel Schulz,
Florian Probst, and Max Mühlhäuser. NoiseMap Real-time participatory noise maps. In Second
International Workshop on Sensing Applications on
Mobile Phones, 2011.