Memory Specs–An Annotation System on
Google Glass using Document Image
Retrieval
Abstract
Katsuma Tanaka

Motoi Iwata

Graduate School of Engineering
Osaka Prefecture University
1-1 Gakuen-cho, Naka, Sakai,
Osaka, Japanfirstname.lastname@gmail.com

Graduate School of Engineering
Osaka Prefecture University
1-1 Gakuen-cho, Naka, Sakai,
Osaka, Japan
Katsuma@m.cs.osakafu-u.ac.jp

Kai Kunze

Koichi Kise

Graduate School of Engineering
Osaka Prefecture University
1-1 Gakuen-cho, Naka, Sakai,
Osaka, Japanmasa@cs.osakafu-u.ac.jp

Graduate School of Engineering
Osaka Prefecture University
1-1 Gakuen-cho, Naka, Sakai,Osaka, Japan
kise@cs.osakafu-u.ac.jp

We present a system working on a wearable computerthat can annotate and retrieve annotations for signs,posters, public displays etc. The only limitation of thesystem: the annotated objects need to contain at least
3-5 lines of text with fixed layout. We evaluate our system
in a poster session scenario with 5 use cases: retrieving
annotations, adding notes, taking pictures and recording
audio and attaching them to posters. As a ubiquitousnote taking and annotation application, we believe thesystem can bridge online and offline discussions and givethe user a way to reflect on seen information by providingthem a web interface to all the annotations they took
together with the source material (in our case poster andpaper publications), helping to organize their thoughts
and improving their recall.

Author Keywords
Annotations; memory; document image retrieval;
wearable; google glass; interactive paper
Permission to make digital or hard copies of part or all of this work for personalor classroom use is granted without fee provided that copies are not made or
distributed for profit or commercial advantage and that copies bear this noticeand the full citation on the first page. Copyrights for third-party componentsof this work must be honored. For all other uses, contact the Owner/Author.
Copyright is held by the owner/author(s).
UbiComp’14 Adjunct, September 13-17, 2014, Seattle, WA, USA
ACM 978-1-4503-3047-3/14/09.
http://dx.doi.org/10.1145/2638728.2638775

ACM Classification Keywords
H.5.2 [Information Interfaces and Presentation]: User
Interfaces Graphical User Interfaces;; H.5.m [Informationinterfaces and presentation (e.g., HCI)]: Miscellaneous.

Introduction

Related WorkWith the concepts like geo tagging and more general,augmented/mediated reality, slowly virtual and physicalworlds are more and more connected. Researchers are
more and more focusing on how to combine offline/online
topic discussions and content [6].

There are a lot of researchers focusing on combining
digital and real-world using annotation technologies,
especially focusing on Augmented Reality [7, 6]. Most ofthem use mobile devices, e.g. smart phones. The works
we are aware of use SIFT or similar features, for indexing
and retrieval and thus can deal only with a small numberof objects/texts.

To this end, we present a wearable application forannotations on ambient text. Our system can be used toannotate and retrieve annotations for any poster or otherpublic sign that contains more than 3-6 lines of text. This
text needs to be registered in a database beforehand(using a digital version of the text) or on the fly by taking
a picture.
Figure 1: Pictures from thesystem evaluation: Annotating aposter with a voice note andretrieving a video annotation
associated to a text region of aposter.

The contributions of this paper are two fold. First, weevaluate a Google Glass prototype of our system for aposter session scenario: Retrieving information, attachingnotes to a poster, attaching images to a poster, attachingaudio notes. A record of the annotations can be later
retrieved using a web interface. We present insights fromusing the prototype, user opinion and an expert analysis.
Second, we provide open system libraries for the use ofannotations with documents, signs, posters for mobile
platforms (iOS, Android and Google Glass) as well as abackend server for registering and retrieving annotations
for a given document.
We believe that our system can help to merge offline andonline topic discussions and enable more and more people
to take part in them. Additionally, as a history ofannotations and the source material can be retrieved
using a web interface on the annotation server, the system
can help the individual to organize their thoughts around
topics and to better remember discussions andconversations.

Researchers already explore annotations to support
knowledge acquisition and memory recall mostly ineducation and work environments [5]. We build on thisresearch in providing an viable annotation system runningon a wearable computer for large scale deployments.
Other work focuses on document annotations withdifferent application goals: interfaces for multimedia
control or retrieval of travel information [2]. Additionally,researchers in AR tend to use QR codes or similar opticalmarkers [1]. We find that for our use case these are too
restrictive as it requires to prepare specific print media andlimits the accuracy and place you can put annotations.
We build on our work for active reading support for
documents using smart phones [4, 8].

Annotations for the Real World
Currently we focus on annotations on ambient text
displayed on signs, displays, posters etc. for mainly tworeasons. First, as we get a lot of information fromambient texts, we believe that this makes a good starting
point for fostering offline/online discussions. Second, thetechnology of document image retrieval matured enoughso this can be used in large scale and has not the troubleof other approaches (needing special markers, expensive
feature calculation).

We decided to use a document image retrieval method
based on ”Locally Likely Arrangement Hashing”(LLAH)[4]. LLAH is very fast. The feature extraction
method can be implemented on todays mobile devices andrun in perceived realtime. Smaller databases (a couple ofdocuments/ambient texts) can be searched on a powerful
wearable computer (e.g. Google Glass) or smart phone aswell. For larger document databases we need a client
server infrastructure.

Usage Scenario – Poster Annotations
To evaluate our idea of a wearable annotation system, weimplement a prototype using Google Glass focusing on aposter session scenario at a conference. Poster sessions atcomputer science conferences are a perfect first test of
our wearable annotation system, as there is a lot of
ambient text and most likely also a large population of
Google Glass users. In the following we describe the usage
scenarios we support for our first prototype.
Retrieving information – When the poster is registeredwith our system. The presenter of a poster can for
example link videos to their poster text. Registration of
additional information can be done using Google Glass ora mobile phone or desktop client. For ease of use werecommend using a digital version of the poster and adesktop client.
Attaching notes to a poster – A user can take a pictureusing our Google Glass prototype and use the Google
Glass voice recognition system to leave a text note to theposter.
Figure 2: Screenshots from theGoogle Glass prototype:uploading an image from theposter text region, adding aaudio note and retrieving paperand video from the poster.

Attaching images to a poster – A user can attach images.
We think this could make it easier to remember the faceof an author or presenter for a given poster or give thepossibility to add interesting prototype pictures to thedigital poster document.
Attaching Audio Notes – A user can also record and add
audio files to a poster, so he’s able to remember and recall
the content of discussions related to the poster topic.

Implementation
As proof of concept, we implemented a first prototype on
Google Glass supporting all use cases presented
beforehand (sample screenshots of the application inFigure 2. We have working application level libraries for
iOS, Android, Linux and Windows. As mentioned before,
the application libraries can be used in stand-alone orclient-server mode. The Google Glass prototype usescurrently the client-server model.
We also show a Web interface, allowing the user to see allposters he annotated and access his annotations as well as
the public annotations of his friends (see Fig. 3). So
people not attending a given poster session or event canvirtually take part in discussions and the user can later on
have a record of his/her conversations to better recall
ideas and thoughts.

Evaluation
System Test
If we run our system in client server mode a large numberof documents can be supported (We used 100 Million text
regions for demonstration purposes with an average
retrieval time of 178 ms). Unfortunately the wait time forthe user on our first prototype is longer (around 2 sec.).
See the discussion later. As seen from previous

publications, scaling our system to a large size is not anissue [3]. The bigger limitation is the necessity of 3-6 linesof text ( at least 4-5 words per line). We are currently
working on integration of other computer vision features
to integrate them without loosing the scalability andperformance.

Shoya Ishimaru, Kai Kunze, Yuzuko Utsumi, Masakazu Iwamura, Koichi Kise

Motivation

Graduate School of Engineering Osaka Prefecture University
1-1 Gakuen-cho, Naka, Sakai, Osaka, Japan kai.kunze@gmail.com

As reading is an integral part of our knowledge lives, we should know more about our reading activities.
TheiPad demo
application
looks
nice!phone and tablets that aims at giving user more
This poster
introduces
a readingapplication
for smart

Eyegaze estimation
As commodity tablets and phones get more and more
powerful and are mostly equipped with front facing
cameras, we wonder if it is possible to use them to detectthe eye gaze of the user while he is looking on the screenof the device.
Since eye gaze is related to knowledge and expertise, it isvaluable to record and review it to improve our reading
activity.
To see if we are able to provide users with enoughSuccessful eye gaze estimation on iPad and iPhone

we implemented a prototype application on the iOSplatform.

Towards using Eyegaze

Why do you use model-based approach?

To improve reading detection
we want to use robust eyetracking.
So far, a hybrid approach using a combination of existingtechniques works best. We implemented a shape-based
shape models, detecting eye corners and iris shape by

Initial Experiments

Usability Impressions
For an initial test, we had 2 users with Google Glassinteract with the system annotating 3 posters andretrieving information (see Figure 1). We also performed
an expert evaluation with one UI researcher. In thefollowing, we summarize the results.
Popular Features Audio notes and Video Retrieval– The
two users especially liked to add quickly audio notes to aposter while discussing about the poster content. The
second popular feature was retrieving a video about theposter topic was

There are 5 test subjects with two experimental runs per
participant: One for training and the other for testing. To
establish a reliable performance baseline, we used a portable
watching a dot moving on the screen. The dot moves from left toright. It jumped to next line and moved from left to right again.
We record the coordinates of the dot and images from front
camera. Since subjects kept watching the dot, its coordinates onscreen are very close to the actual gaze point.

Conclusion and Future WorkI’d explore
like to furtheremail how
him tolater
about reading
this research.
We will
implement
detection and a more stable eye tracking for reading
detection. Most importantly, the next experiments will focus on more natural reading and tablet usage
scenarios, using a stationary eye tracker for ground truth.
contact: kai.kunze@gmail.com

website: kaikunze.deFigure 3: Screenshot from theweb page, showing all annotatedposters and a poster with text
annotations.

Quick Capture – From the evaluation, we saw the need forquick capture. The main problem for annotations rightnow is an average wait time of 2 seconds after taking thepicture, this is due to some programming limitation on
Google Glass (the image should first be saved before it’ssend to the server). The users also raised the wish to justcapture a poster picture so they can later on read thepaper or watch the video on the desktop client (notadding annotations). We will add this use case (quickcapture) in the next prototype.

Conclusion and Future Work
We present a framework to annotate ambient text
digitally. We implemented a prototype system on Google
Glass, evaluating its functionality on a poster session
scenario in a small scale. As a next step, we would like to

use the system in a real conference scenario to gather
feedback from more users identifying use cases andimproving user interface and overall usability.

References[1] Chen, N.-S., Teng, D. C.-E., Lee, C.-H., and Kinshuk.
Augmenting paper-based reading activity with directaccess to digital materials and scaffolded questioning.
Comput. Educ. (2011), 1705–1715.
[2] Erol, B., Antúnez, E., and Hull, J. J. Hotpaper:
multimedia interaction with paper using mobilephones. In Proc of MM 2008, MM ’08 (2008).
[3] Iwamura, M., Nakai, T., and Kise, K. Improvement ofretrieval speed and required amount of memory forgeometric hashing by combining local invariants.
Proc. of BMVC2007 (2007), 1010–1019.
[4] Kunze, K., Tanaka, K., Iwamura, M., and Kise, K.
Annotate me: Supporting active reading using
real-time document image retrieval on mobile devices.
UbiComp ’13 Adjunct (2013).
[5] Mackay, W. E., and laure Fayard, A. Designinginteractive paper: Lessons from three augmented
reality projects. In In Proc. of IWAR98, A K Peters,Ltd (1999), 81–90.
[6] Rekimoto, J., and Nagao, K. The world through thecomputer: Computer augmented interaction with realworld environments. In Proc. of UIST, ACM (1995),29–36.
[7] Rose, E., Breen, D., Ahlers, K. H., Crampton, C.,
Tuceryan, M., Whitaker, R., and Greer, D. Annotating
real-world objects using augmented reality. Computergraphics, Academic Press Ltd. (1995).
[8] Tanaka, K., Iwata, M., Kunze, K., Iwamura, M., andKise, K. Share me-a digital annotation sharing service
for paper documents with multiple clients support. In(ACPR, IEEE (2013), 779–782.