Ubiquitous Memories:
Wearable Interface for Computational Augmentation of
Human Memory based on Real World Objects
Tatsuyuki Kawamura
Graduate School of Information Science, Nara Institute of Science and Technology, 89165, Takayama-cho, Ikoma, Nara 630-0101, JAPAN. email: tatsu-k@is.aist-nara.ac.jp
Tomohiro Fukuhara
Graduate School of Information Science, Nara Institute of Science and Technology, 89165, Takayama-cho, Ikoma, Nara 630-0101, JAPAN. email: tomohi-f@is.aist-nara.ac.jp
Hideaki Takeda
The National Institute of Informatics, 2-1-2, Hitotsubashi, Chiyoda-ku, Tokyo 101-8430,
JAPAN. email: takeda@nii.ac.jp
Yasuyuki Kono
Graduate School of Information Science, Nara Institute of Science and Technology, 89165, Takayama-cho, Ikoma, Nara 630-0101, JAPAN. email: kono@is.aist-nara.ac.jp
Masatsugu Kidode
Graduate School of Information Science, Nara Institute of Science and Technology, 89165, Takayama-cho, Ikoma, Nara 630-0101, JAPAN. email: kidode@is.aist-nara.ac.jp
Abstract
This paper presents a novel computational human memory augmentation concept, Ubiquitous Memories, used to support everyday information related activities. Memory aid architecture has been used to annotate augmented video data as wearable information. We havedeveloped a prototype system, Ubiquitous Memories, composed of a head-mounted display, awearable camera and a Radio Frequency Identiﬁcation (RFID) device, which attaches to anindividual user, as well as a set of RFID tags and web servers. These RFID tags are set on
their corresponding real world objects. A user might link a desired video data with a certain
real world object by means of a RFID tag on the object. We have adopted touching as thenatural operation of the system. Two experiments have been conducted to evaluate the eﬀectiveness of the system and the results have shown that this system eﬀectively supports memory
recollection of past events.

1

Introduction

We propose a real-world oriented augmented memory concept, Ubiquitous Memories.
Augmentation of human memory using computational technologies has been studied
extensively in recent years. Rhodes (1997) termed the augmentation of human memory“augmented memory.” A computational memory aid system such as Ubiquitous Memories is crucial for the augmentation of daily life because psychological results show thatthe human brain causes mistakes in the encoding process, the storing process or the
retrieval process. Nickerson et al. (1979), for example, showed that most Americantest subjects could not draw even 50% of the design of a 1-cent coin when asked toremember the coin visually. Wrewer et al. (1981) investigated what strategy people canuse to memorize environmental contexts in a room.
In the ﬁeld of memory aids, two representative works on ubiquitous computing andwearable computing exist. Lamming et al. (1994) developed Forget-me-not, a prototype
system for a portable episodic memory aid. Their system recorded a user’s action history
using sensors that were implanted in a laboratory and Active Badges worn by users.
The user can refer to their history and easily recall past events to a PDA. The study
is based on the concept of “ubiquitous computing” proposed by Weiser (1991). Mann
(1998) described a user who wears a CCD camera and sensors to record the user’s daily
life. The system allows the user to get information anytime and anywhere the user
wants. The human-centered computer technology is called “wearable computing.” The
wearable computing technology should be made aware of the wearer’s internal (desire,emotion, health, action, etc.) and external (goings-on, temperature, other people, etc.)state at any time.
In the ﬁeld of supporting human memory on wearable computers in daily life, Jebara et al. (1998) proposed the video replay system, DyPERS, which stores a user’s
visual and auditory scenes. The system can retrieve a video clip using a signal thatwas explicitly registered by the user by pushing a button while the user looked at aninteresting scene. Jennifer et al. (1998) developed the StartleCam, which records avideo data triggered by skin conductivity from a startled response from a user. Aizawa
et al. (2001) proposed a system that summarizes a video from a wearable camera using
brain waves. The above two systems automatically start recording when the user showsa noticeable interest. The VAM system by Farrigdon et al. (2000) detects a humanface, which was recorded at previously, and displays information of the retrieved person. These researchers have developed human-centered computing systems that focucedon recognizing the user’s interests by representing augmented memory. Remembrance
strategies, however, are not only operated with interests or information in the humanbrain, but these strategies are also involved with the relationship between events andreal world objects.
The concept of Ubiquitous Memories has the strong ability to reduce a user’s memory overload, because the user is able to recall experience by simply touching the object
linked with the experience. Realizing Ubiquitous Memories, physical objects are implanted or attached micro-miniature type tags. We use a Radio Frequency Identiﬁcation(RFID) device and RFID tags because the tag does not need battery. The tag is somestudies to utilize real world objects in human support by computers. Rekimoto et al.
(1998) proposed a system for browsing information that included the physical context
of a user’s current situation. Their aim is to develop interactive technologies in physicalenvironments. However, neither a computational augmentation of human memory nor adevelopment for operating a memory aid system was considered. Michael et al. (1997)
proposed artefact-centerd computing, called MediaCups. This system needs electric
power to use digital artefacts.
In this paper, we deﬁne augmented memory as video data recorded by a camera wornby a user. Our concept provides the user wearing the Ubiquitous Memories system two
abilities: i.e., the ability to link augmented memories to real world objects, and theability to recollect augmented memories by simply touching the objects. The overall
aim of our concept is the realization of a digital nostalgia based on real world objects as
an augmentation of human memory in daily life. Digital nostalgia creates an autobio-

graphic history by linking a human experience with a real world object contained in theexperience. The user wearing the Ubiquitous Memories system can be easily reminded
of a certain past event linked to the object as if the user were virtually there.
In the research ﬁeld of memory aids, a proposal for the concept and the developmentof a system are not only important points, but an investigation into the eﬀectivenessof the system must also be evaluated. We therefore did two experiments for a single
user. Our experimental results show that the system eﬀectively performs in retaining
the user’s past events. First, we investigated the inﬂuences when a particular context
in a video can give the user. Second, we examined how many events the user can retain
with our system.
In the next section, we brieﬂy discuss an overview of the Ubiquitous Memories
system. In section 3, we present the experiments and results to show how the Ubiquitous
Memories system can support a user. The discussion section in section 4 evaluates theavailability of Ubiquitous Memories. The ﬁnal section, section 5, concludes this paper.

2

Overview of the Ubiquitous Memories system

We have developed a prototype system of Ubiquitous Memories to realize a real-worldoriented augmented memory concept. Figure 1 illustrates an overview of the Ubiquitous
Memories system.
It is crucial for such a system to provide “natural” operations. In everyday life, auser picks up a real world object when the user is interested in that object or needs
that object. “Touching” is a natural operation used to establish a link between theuser’s external activities and an operational object. So, we decided that the operational
object is selected by the ‘touching” operation in the Ubiquitous Memories system.

(a) An operation procedure of the system(b) An appearance of the user

Figure 1: Overview of the Ubiquitous Memories system
The current system hardware consists of a Head-mounted Display (HMD), a wearable camera, which is attached at the center of the HMD, and a Radio Frequency
Identiﬁcation (RFID) device for a wearable system, RFID tags for objects, and web
servers. The RFID device is incorporated into a gloved hand. The RFID device canimmediately read the RFID tag data when the device approaches the tag within 3cm
distance. The entire system connects to the World Wide Web via a wireless LAN.
We show a basic procedure of the system’s operation in Figure 1. The system
state has a base state i.e., MEMORIZE and REMEMBER. The state of the system isnormally at REMEMBER. The user can select and execute one of the deﬁned operationsof the system by touching a RFID operation tag with a special ID associated with theoperation, called an “operation tag.” Operation tags are worn on the opposite wristof the user’s gloved hand. The Ubiquitous Memories system allows the user to use thefollowing the two operations:
MEMORIZE The state of the system is changed from REMEMBER to MEMORIZE
when the user touches the operation tag MEMORIZE (I). The user then approaches
the gloved hand near a physical object (II). The system then records a video fromthe head-mounted camera (III). The system connects with the web server to store
the video linked with the RFID tag ID that is attached to the object (IV).
REMEMBER The user just touches a RFID tag attached to an object the user
chooses to remember (shown in Figure 2 (a)) (V). The system retrieves the video
data from the web server (VI). The retrieved video data is then replayed in the
top-left area in the screen of the HMD (shown in Figure 2 (b)) (VII).

(a) Selecting the object(b) Replaying the video

Figure 2: An example of operation (REMEMBER)

3

Experiments and Results

3.1

Inﬂuence of a Context in a Video to Human Memory

In the experiment, we investigated how a context in a video can give inﬂuence to a userof the Ubiquitous Memories system. It is important to ﬁnd eﬀective operation rules
about memorization, as well as what kind of context should be recorded in a video
and how such memories should be linked with a physical object for memory aid andidentiﬁcation.
3.1.1

Experimental Methods

This experiment was conducted at the Nara Institute of Science and Technology (NAIST)in Nara, Japan, among graduate students of the Information Science Department. 17
Japanese test subjects were participated in the experiment.
The experiment used pairs of an object image and a video in a trial (Figure 3). We
set a notebook PC under laboratory conditions. A pair of an object image and a video
was displayed on a PC monitor. The linked video is replayed automatically when thesubject clicks the displayed image. The experiment contains 20 trials. In the 10 trials,

Figure 3: Reference display for experiement 1

each object in the displayed image is contained in the linked video. In the other 10
trials, each object and the video have no semantic relationship. In the experiment, eachtrial was alternately performed. Test subjects could watch a video 2 times in a trial,
and then, the subject could proceed the next trial immediately.
After the subject ﬁnished all trials, there was a 3-minute delay period. In the
period, the Japanese test subject had to read an scientiﬁc international journal paper
and translate outloud each and every sentence from English into Japanese. The subject
then answered a questionnaire. In addition, after a 1-week delay period, the test subjects
had to answer the same questionnaire again. 11 questionnaires were returned.
The questionnaire contained 20 recall questions in Japanese. Each question showed
an object image used in the experiment. The subjects ﬁlled in as many answers as they
could in less than 10 minutes. It should be noted that the sequence of questions was
diﬀerent from the sequence of trials. The test subjects could answer these questionnaires
in a random order.
3.1.2

Results

This section analyzes the results of the 17 (3-minute delay) and 11 (1-week delay)
questionnaires that we collected from the Japanese graduate students of the Information
Science Department at the NAIST in Nara, Japan.
Figure 4 shows the average number of correct answers. The left side of the ﬁgureshows the average number of the correct answers in the 3-minute delay questionnaires.
The right side illustrates the average number of correct answers in the 1-week delay
questionnaires. Black dots result of the questions for images with videos containing
images themselves. White dots illustrate for images, which were not contained in thevideo.
In the 3-minute delay questionnaires, the average number of black dots is approximately 1.4 times greater than the average number of white dots. This result shows asigniﬁcant diﬀerence (p<0.0005). In the 1-week delay questionnaires, a diﬀerence between the average number of black dots and the average number of white dots widens
to approximately 2.3 times. This result shows a signiﬁcant diﬀerence (p<0.0005).

Figure 4: Context eﬀect

3.2

Recollection Eﬃciency of a Memorization Strategy

The purpose of the experiment was to examine how many events a user can retain using
a quantitative analysis. We adopted a comparative experiment with 4 memorization
strategies.
3.2.1

Experimental Methods

This experiment was conducted at the NAIST in Nara, Japan, among graduate students of the Information Science Department. 20 test subjects were included in theexperiment.
In the experiment, we used 10 physical objects, 10 portraits of unfamiliar people,
and two sets of 10 playing cards composed of number 1 through 10. We conducted theexperiment under laboratory conditions. One experiment was composed of a memory
test and a recall test. The memory test asked the subject to memorize 10 trials. The
recall test meant that the subject answered a questionnaire.
In a trial of the memory test, the subject was shown a pair consisting of an object
and a portrait. The subject then selected one of the four corners of the portrait. Finally,
the subject was shown the predetermined pair of cards. The subject looked at thesenumbers within 30 seconds. The subject had to memorize the object and portrait pair,the corner of the portrait and two card numbers. The subject continuously tried tomemorize all trials. All subjects had to do two experiments within the following 4
conditions.
C1: use human memory
C2: use only facial characteristics (record with a paper and a pen)C3: refer to portraits that were used in a memory trial
C4: use the Ubiquitous Memories system
Test subjects were divided into four groups. Group 1 did two experiments using conditions C1 through C3. Group 2 experimented using in conditions C3 through C1.
Group 3 experimented two times using conditions C2 through C4. Group 4 did two
experiments using in conditions C4 through C2.

In the recall test, a questionnaire contained 10 recall questions. The subject wasgiven one object image in each question. There were three empty boxes (portrait,
corner and card numbers) in a question. Next, the subject selected a portrait id from alist of 40 portraits, marked a corner (Left-Top, Left-Bottom, Right-Top, Right-Bottom),and wrote down two card numbers. The subject was given a list of 10 portraits used in
the memory test only in condition C3. All subjects ﬁlled in some or all answers within
10 minutes. The sequence of questions was changed from the sequence of trials in thememory test. All subjects could answer the questions in a random order.
3.2.2

Results

This section describes the result of the 20 questionnaires collected from the graduate
students of the Information Science Department in NAIST.
Table 11 illustrates recall rates from 20 questionnaires. In this paper, we deﬁned N,

Table 1: Recall rate
C１
C２
C３
N**
24.0% 31.0% 10.0%
PB’F’ 11.0% 8.0% 19.0%
P’BF’ 12.0% 9.0% 5.0%
PBF’ 23.0% 20.0% 32.0%
P’B’F 8.0% 4.0% 3.0%
PB’F
4.0% 9.0% 11.0%
P’BF
5.0% 1.0% 3.0%
PBF
13.0% 18.0% 17.0%P**
51.0% 55.0% 79.0%
B
53.0% 48.0% 57.0%F
30.0% 32.0% 34.0%

C４
2.0%
19.0%
3.0%
31.0%
1.0%
19.0%
0.0%
25.0%
94.0%
59.0%
45.0%

P, B, F and ’. N, which is a percentage of errors, is represented as follows: no answers
of a portrait, a card and card numbers were correct answers on several questions. Pshows that the answer of a portrait was correct. B and F have the same relation asthat between P and the answer of a portrait. B shows that the answer of a corner wascorrect. F represent that the answer of card numbers was correct. X’ (X is either P, B
or F) represents the answer of a question related to X that was not correct.
In Table 1, N and P show a signiﬁcant diﬀerence among the 4 test conditions(p<0.001). P by C4 is, however, not 100%. Ideally, C4 must be 100% because oursystem gives a directly related portrait. In the sum of P’BF’, P’B’F and P’BF, we cansee the inﬂuence on the diﬀerence among the test conditions (C1: 25.0%, C2: 14.0%,
C3: 11.0%, C4: 4.0%, p<0.001). A sum of PBF’ and PBF shows the transparency on
the diﬀerence of test conditions (C1: 36.0%, C2: 38.0%, C3: 49.0%, C4: 56.0%, p>0.1).
Figure 5 shows an F-ﬂow model to explain dependency among P, B, and F in Table 1.
We can investigate the test subject’s remembrance strategies for the experiment. We
introduce O that represents an object as a question in the questionnaire. We envisioned
a ﬂow rate of a ﬂow path for the model. A ﬂow pattern in the model corresponds with1**: p<0.001, *: p<0.05, †: 0.05<p<0.1

Figure 5: F-ﬂow model

Table 2: F-ﬂow path ratios
C１
C２
C３
C４
of
17.9% 9.5% 7.1% 2.5%
op** 74.7% 88.9% 89.5% 97.5%
ob
7.3%
1.6% 3.4% 0.0%
pf
44.7% 50.9% 40.5% 38.5%
pb
30.0% 38.0% 49.0% 59.0%bf
37.3% 39.6% 52.4% 59.0%

an associative pattern from O to F. The outﬂow of recall from O and the inﬂow to F are
both 100% as a total. Table 2 shows the representation of the F ﬂow in the experiment.
The path of op, ob, of, pb, pf and bf show the percentages for the ﬂow rates.
In an analysis of the variance to the result of the experiment, op shows signiﬁcantdiﬀerences (p<0.0005).

4

Discussions

We discuss how a computational augmentation of human memory based on physicalobjects can aid a user’s recollection of past events in daily life. The goal of our investigation using the two experiments is to evaluate the eﬀectiveness of the Ubiquitous
Memories system for personal use.
First, we should examine what kind of context in a video is adequate for supporting
human memory. Figure 4 illustrated how a video, which is semantically related to aphysical object, had a strong and long-term eﬀect for human memory. A person retains
a relationship between physical objects and his/her memory associated with them, fora long time. Therefore, it is easy to remember a past event when just looking at aphysical object.
Second, we should investigate what kind of memory aid method showed the highest
performance. Table 1 showed that our proposed system had the most eﬀective test
conditions. There were 4 conditions (C1: use human memory, C2: use only facialcharacteristics, C3: refer to partraits that were used in a memory trial, C4: use theUbiquitous Memories system) in the experiment. In the result of PBF (C1: 13.0%, C2:
18.0%, C3: 17.0%, C4: 25.0%) and P (C1: 51.0%, C2: 55.0%, C3: 79.0%, C4: 94.0%),
C4 showed the strongest relationship of an object linked to a portrait. Each condition inthe experiment supported test subjects only in the F-ﬂow path op. The result not only
showed the inﬂuence of the supporting memorization in the path op, but also showedinﬂuences in the path pb and bf. In Table 2, we observed that the C4-ﬂow showed themost well-knit ﬂow path in op (C1: 74.7%, C2: 88.9%, C3: 89.5%, C4: 97.5%), pb (C1:
30.0%, C2: 38.0%, C3: 49.0%, C4: 59.0%) and bf (C1: 37.3%, C2: 39.6%, C3: 52.4%,
C4: 59.0%).
In the two experiments, we show that computationally augmented Ubiquitous Memories is capable of supporting human memory. Videos linked with physical objects in
the real world can give a key to the user’s recollection of a past event with the single
user. In particular, the system shows considerable eﬀect when a physical object related
to an event was recorded in a video.
The experimental results, additionally, show that Ubiquitous Memories has the potential for sharing human memories among people. Takatori (1980) showed that cooperative recall is superior to individual recall in the memory prosess. Human memory
cannot be shared explicitly in the real world. People can only trade real world experiences with external explanations. We believe that video helps facilitate memory recall
in the real world. We have developed the next version of the Ubiquitous Memories
system for sharing augmented memories among multi-users. This development enables
us to conduct further experiments on how multi-users can operate the system to shareaugmented memories.

5

Concluding Remarks

This paper proposed a novel concept of Ubiquitous Memories for memory aid. A prototype system of the Ubiquitous Memories system has been developed on a wearable
camera, with a RFID tag reader and a RFID tag attached to a physical object. We
examined the eﬀectiveness of Ubiquitous Memories for the augmentation of humanmemory. We found that the relation between a physical object and a video containing
the object is capable of facilitating a memory of the object that will be retained by theuser. We proved that Ubiquitous Memories gives a clear memorizing strategy to theuser.
The shortcomings of this study include the necessity for developing a multi-users
version of the Ubiquitous Memories for sharing augmented memories, as noted in thepaper. Finally, the current system only linked one video on a physical object. We are
increasing the number of videos each physical object.
In a related study, an integration problem is given for recollection strategies. Kawamura et al. (2001) proposed a memory aid system for supporting location-based memory
recollection of past events. In the future, we will integrate the Ubiquitous Memories
system with the location-based memory recollection support system.
We believe that the two experiments discussed in this paper provide evidence for the
adoption of the Ubiquitous Memories system for memory aid using the computational
augmentation of a user’s memory. Ubiquitous Memories supports the eﬀective andinteractive application of wearable computing and ubiquitous computing.
This research is supported by Core Research for Evolutional Science and Technology(CREST) Program “Advanced Media Technology for Everyday Living” of Japan Scienceand Technology Corporation (JST).

References[1] AIZAWA,K. & ISHIJIMA,K. & SHINA,M. (2001) Automatic summarization of
wearable video-indexing sujective interest. In Proceedings of The Second IEEE
Pacific-Rim Conference on Multimendia (PCM2001), Springer LNCS2195, pp.16–
23.
[2] BREWER,W.F. & TREYENS,J.C. (1981) Role of schemata in memory for places.
Cognitive Phychology, 13, pp.207–230.
[3] FARRIGDON,J. & ONI,V. (2000) Visual augmented memory (VAM). In Proceedings of The Fourth International Symposium on Wearable Computers (ISWC 2000),
pp.167–168.
[4] JEBARA,T. & SCHIELE,B. & OLIVER,N. & PENTLAND,A. (1998) DyPERS:
dynamic personal enhanced reality system. MIT Media Laboratory, Perceptual
Computing Technical Report #463.
[5] KAWAMURA,T. & KONO,Y. & KIDODE,M. (2001) A novel retrieval to support
a user’s recollection of past events aiming for wearable information playing. In Proceedings of The Second IEEE Pacific-Rim Conference on Multimendia (PCM2001),
Springer LNCS2195, pp.24–32.
[6] JENNIFER,H. & ROSALLIND,W. (1998) StartleCam: a cybernetic wearable camera. In Pcoceedings of The Second International Symposium on Wearable Computers (ISWC’98), pp.42–49.
[7] LAMMING,M. & FLYNN,M. (1994) Forget-me-not: intimate computing in support of human memory. In FRIENDS21: International Symposium on Next Generation Human Interface, pp.125–128.
[8] MICHAEL,B, & HANS,W.G. & ALBRECHT,S. (2001) MediaCups: experience
with design and use of computer augmented everyday artefacts. Computer Networks, 35 (4), pp.401–409.
[9] NICKERSON,R.S. & ADAMS,M.J. (1979) Long-term memory for a common object. Cognitive Psychology, 11, pp.283–307.
[10] REKIMOTO,J. & SAITOH,M (1998) Augment-able reality: situated communication through digital and physical spaces. In Proceedings of The Second International
Symposium on Wearable Computers (ISWC’98), pp.68–75.
[11] RHODES,B. (1997) The wearable remembrance agent: a system for augmented
memory. In Proceedings of The First International Symposium on Wearable Computers (ISWC’97), pp.123–128.
[12] TAKATORI,K. (1980) The role of communication in the memory process -a comparative study of individual recall and cooperative recall-. Educational Psychlogy
Vol XXVIII, No. 2, pp.108–113. (in Japanese)[13] WEISER,M. (1991) The computer for the 21th century. Scientific American(September), pp.66–75.