UBICOMP '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USA

Shiny – An Activity Logging
Platform for Google Glass
Shoya Ishimaru
Jens Weppner
Osaka Prefecture University
DFKI
Sakai, Japan
Kaiserslautern, Germany
ishimaru@m.cs.osakafu-u.ac.jp jens.weppner@dfki.de

Andreas Poxrucker
DFKI
Kaiserslautern, Germany
andreas.poxrucker@dfki.de

Kai Kunze
Osaka Prefecture University
Sakai, Japan
kai.kunze@gmail.com

Abstract
We describe an activity logging platform for Google Glass
based on our previous work. We introduce newmulti-modal methods for quick non-disturbing interactions
for activity logging control and real time ground truthlabeling, consisting of swipe gesture, head gesture andlaser pointer tagging methods. The methods are evaluatedin user studies towards estimating their effectiveness.

Author Keywords
Activity Recognition; Platform; Google Glass;
Ground-truth LabelingPaul Lukowicz
DFKI
Kaiserslautern, Germany
paul.lukowicz@dfki.de

Koichi Kise
Osaka Prefecture University
Sakai, Japan
kise@cs.osakafu-u.ac.jp

ACM Classification Keywords
H.5.2 [Information Interfaces and Presentation]: UserInterfaces, Graphical User Interfaces

IntroductionPermission to make digital or hard copies of part or all of this work for personalor classroom use is granted without fee provided that copies are not made or
distributed for profit or commercial advantage and that copies bear this noticeand the full citation on the first page. Copyrights for third-party componentsof this work must be honored. For all other uses, contact the Owner/Author.
Copyright is held by the owner/author(s).
UbiComp’14 Adjunct, September 13-17, 2014, Seattle, WA, USA
ACM 978-1-4503-3047-3/14/09.
http://dx.doi.org/10.1145/2638728.2638798

283

Moving towards standardized datasets, one of the major
problems in activity recognition is ground-truth labeling.
Traditionally, this is done offline from the researcher andrequires a lot of effort even for controlled datasets withfew participants [1, 3, 4]. It is not feasible to do minutelabeling for large scale, real-life recordings. We believethat commercially available wearable computers withhead-mounted display will enable us to better track andlabel activities during everyday life. In addition, there is aUBICOMP '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USAgrowing community of people who perform self-tracking
using spread sheets and smart phone applications torecord what they are doing during the day [2]. Smartphone applications, like reporter 1 , pop up reminders 5-10
times a day and ask the user questions about theiractions. The granularity of these loggers is too low and it
is too disruptive for a lot of people to use theseapplications on a mobile phone. In turn, we evaluate if it’s
possible to create a wearable computing interface forHMDs (in our case Google Glass) that gives the user thepossibility to quickly select his current activity from agiven set without disturbing him too much.
Figure 1: The diagram displays
the reaction time vs thecorrectness of the input. Eachsymbol depicts one experimentrun (40 samples). The starsymbol reflects the swipe method
and the circle symbol reflects theclassic input method.

Swipe Gestures
Google Glass and similar devices target micro-interaction
which can be performed fast and with minimal
interference with other cognitive or physical activity,
making them a perfect platform for ground-truth labelingand life-logging. For many types of tasks the default
Google Glass interface works very well. An exception isthe selection of items from larger lists, in particular when
voice interaction is not desired and the user is moving .
The default interface involves hitting the right item
through a controlled analogue motion (depending on how
fast/far the finger moves) on the touchpad which are
difficult to perform exactly when the user is moving andrequire a degree of concentration. As an alternative wepropose a ”digital” interface that codes the item selection
through hierarchical combinations of multitouch gestures(number of fingers, number of swipes, direction of swipe)
supported by appropriate representation of the choice on
the screen. It makes one dimension of the target size
infinite. The user does not need to scroll to the ”right”
card, but makes his selection over number of fingers andswipe direction.
1 http://www.reporter-app.com

284

1-finger gesture

2-finger gesture

3-finger gesture

Figure 2: Google Glass swipe gesture input method
screenshots. The upper screen displays the categorical overview
with hints how to access them (one dot = one finger, ...). As
soon as one, two or three fingers touch the touchpad on theright side of Google Glass the corresponding label selection
matrix appears. To select a label the user keeps his fingers on
the touch pad and swipes to the front, back, up or down.

Evaluation
For our initial experiment, we evaluated 11 participants,
using both input methods subsequently. We studied thehierarchical list based input method and ours referred tothe multi-touch swipe gesture input. We considered themain target of Google Glass: Performing quick look ups
and input while underway: The participant was instructedto continuously walk during the experiment to achieve anatural environment with distractions i.e. watching theirsteps to avoid obstacles . While the participant was inmotion the experiment observer communicated one label
after another. We repeated this 40 times for each methodwith random labels. Figure 1 displays the reaction time vsthe correctness of the input. For all participants thereaction time is decreased significantly (p-value of 0.04).
The reaction time is reduced by 1.7 seconds on average.
Yet, the mean accuracy decreases to 75% for the swipe
interface versus 95% on average for the classic interface.

SESSION: UBICOMP DEMOSHead Gestures
Over%25°%

Move%le,/right%

Tilt%le,/right%

Figure 3: Head movements
considered for gestures:moving/tilting left/right.

!me/ms&For hands-free interactions with the logger we
implemented head gestures. We implemented two gesturesets using Google Glass. The first gesture set (referred toas ”move” in the following) consists of turning the headleft or right to browse through the cards on the HMD.
The second gesture set (referred to as ”tilt” later on) uses
tilting the head over 25 ◦ in one direction for browsing.
For both sets a double ”nod down” is used for selection.
The recognition is done using the motion sensor
abstraction in Android, simple thresholding on the rotation
vector aggregated from the accelerometer, gyroscope andmagnetic field sensor over the Android Sensor API. Basedon successful initial tests with participants we consideredcalibration or user dependent classification as unnecessary.
Evaluation
We conducted an initial study to investigate the suitabilityof the designed head gestures. We compared the standard
touch pad provided by Glass (using a hand), the tilt
gestures and the move gestures. We measured the time it
took after the user left the start screen until he selected
the right card and counted how many errors happened
during the input (e.g. card interface moving left instead ofright). 3 male and 2 female participants (age: 23 - 60)
were asked to select 1 out of 11 cards shown on Glass(one start screen and 10 cards with numbers 1 - 10 inorder). Participants were asked to select each of the cards
10 times in a randomized order. We disabled Google
Glass’ standard activation head gesture.

285

standard"

1lt"

move"

1800"
1600"
1400"
1200"
1000"
800"
600"
400"
200"
0"
u1""

u2"

u3"

u4"

u5"

Figure 4: Average time to switch between live cards on theGoogle Glass display for the different interaction modalities

As seen from the average time to switch between cards inFigure 4, users are not surprisingly fastest on the standard
Glass interface with around 710 ms using one hand forselection, the ”tilt” interface is with a mean of 782 ms,the ”move” interface is worse with 1092 ms averageswitch time.

Interactive Tagging of Real World Objects
In many situations, the ability to select real world objects
can be an important feature of an activity logging
platform. We investigated two different live video tagging
approaches and implemented prototype versions of themon Glass for an early evaluation. In the first approach, theuser selects an object of interest by circling it with anoff-the-shelf laser pointer. The software tracks its curve ina sequence of video frames, and recognizes circularmovement. The user receives immediate feedback by
playing a sound as soon as the spot is detected andtracked. Having recognized the circling gesture, the sub
image enclosing the tracking curve is extracted. Figure 5
summarizes the selection procedure. The second interface
is based on the touchpad of Glass. As shown in figure 6,
the user subsequently adjusts the dimensions of arectangle overlaid to the video stream to fit the object toselect.

UBICOMP '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USA

Live camera
streamSelection bycirculation

Extracted image
preview

Evaluation
We evaluated the functionality, usability, and usefulness ofboth interfaces in a study involving 11 test persons. All ofthem were between 20 and 40 years old. Seven did nothave a background in computer science, the remaining
four persons were from our research department. The
participants had to tag a number of objects listed in
Figure 7 which were of different size and had to be
selected under different illumination conditions. During
the experiment, we counted the number of tries as well as
the time it took to select an object successfully with eachsystem. Figure 7 summarizes the time/tries of bothinterfaces for each object averaged over all people.

Figure 5: Laser pointer based
image selection.

2.5
GLaser pointerTouchpad

Mean number of attempts

KHA1na, d, e, g, h, n, o0b, c, f, i, j, l, m

0

2

4

6

8

10

12

14

16

18

20

Mean time(s)
Figure 7: Mean time/meantries
per person for bothinterfaces. Each point corresponds to a different object
averaged over all participants. Letters denote objects asfollows: (A, a) Socket, (B, b) Lightswitch (hallway), (C, c)
Smoke alarm, (D, d) Door plate, (E, e) Lightswitch (foyer), (F,f) Router (foyer), (G, g) Painting, (H, h) Lamp, (I, i) Router
foyer 2, (J, j) Fire alarm, (K, k) Poster logo, (L, l) Router(hallway), (M, m) Handle, (N, n) First aid box, (O, o)
Cupboard lock
J

Extracted image
preview

Figure 6: Touchpad based image
selection.

Conclusion
In this paper we extended our work on activity recognition
towards Google Glass. We presented a new input methodfor quick micro interaction with Google Glass touchpad
swipe gestures, a method to navigate through theContextLogger interface by head gestures, and a method
for tagging and picturing of real world objects with a laser
pointer.

References
C, E, M, O

1.5

0.5

Selection byrectangle
adjustment

The closest to our framework is Openshades.com, an open
framework for Google Glass. Yet, Openshades focuses
mostly on computer vision and human computerinteraction, not on activity recognition and activity
logging.

B, D, F, I, J, L, N

2Live camera
streamRelated Work

Sensor Data and Backend Support
The system is compatible to the data format of the CRNT
Toolbox (using CSV and json as export format) and thedata can also be loaded into our open-sourced labeling
tool for offline adjustments.

286[1] Bannach, D., Kunze, K., Weppner, J., and Lukowicz,
P. Integrated tool chain for recording and handlinglarge, multimodal context recognition data sets. InProc. of Ubicomp, ACM (2010), 357–358.
[2] Gouveia, R., and Karapanos, E. Footprint tracker:
supporting diary studies with lifelogging. In Proc. ofthe SIGCHI, ACM (2013), 2921–2930.
[3] Ishimaru, S., Kunze, K., Kise, K., Weppner, J.,
Dengel, A., Lukowicz, P., and Bulling, A. In the blinkof an eye - activity recognition with google glass. In
Proc. of AH, ACM (2014), 150–153.
[4] Lester, J., Choudhury, T., and Borriello, G. A
practical approach to recognizing physical activities.
In Pervasive Computing. Springer, 2006, 1–16.