Wearable Computers as Intelligent AgentsThad Starner
GVU Center, College of Computing
Georgia Institute of Technology
Atlanta, GA 30332-0280 USA
Forty years ago, pioneers such as J.C.R. Licklider and Douglas Englebart championed the idea of interactive
computing as a way of creating a “man-machine symbiosis” where mankind would be enabled to think inways that were previously impossible [3, 5]. Unfortunately, in those times, such a mental coupling was
limited to sitting in front of a terminal and keying in requests. Today, wearable computers, through theirsmall size, proximity to the body, and usability in almost any situation, may enable a more intimate form ofcognition suggested by these early visions. Specifically, wearable computers may begin to act as intelligent
agents during everyday life, assisting in a variety of tasks depending on the user’s context.

1 Wearable computing

Figure 2: Wearable computing platforms worn at
MIT and Georgia Tech.

Figure 1: MicroOptical Corporation’s head-up display embedded in prescription eyeglasses.

Wearable computers often perform background tasks
such as providing reminders, capturing information
or experience, and retrieving time-sensitive information in support of the user. In order to perform suchtasks, the interface must take on a form of “symbiosis” between the user and the computational assistant.
J.C.R. Licklider, an early Advanced Research
Projects Agency (ARPA) director for the UnitedStates whose research agenda is often credited forlaying the foundations of the Internet and interactive
computing, hinted at such a symbiosis in his writingsin 1960 [5]:Wearable computers are generally equated withhead-up, wearable displays, one-handed keyboards,
and custom computers worn in satchels or belt packs(see Figures 1-2). Often, the public focuses moreon the gadgetry involved in wearable computing
than on the field’s goals. Wearable computing may
be described as the pursuit of an interface ideal –that of a continuously-worn intelligent assistant thataugments the wearer’s memory, intellect, creativity,communication skills, physical senses, and physicalabilities. Unlike desktop applications such as spreadsheets or graphics drawing programs which monopolize the user’s attention, wearable computer applications should be designed to take a secondary role.

“Man-computer symbiosis” is a sub1

class of man-machine systems. There aremany man-machine systems. At present
however, there are no man-computer symbioses. ... The hope is that, in not too
many years, human brains and computing machines will be coupled together very
tightly and that the resulting partnershipwill think as no human brain has everthought and process data in a way not approached by the information-handling machines we know today.
To achieve the tight partnership suggested by
Licklider in daily life, the computer must be a constant companion for the user. The computer must
share in the experience of the user’s life, drawing
input from the user’s environment to learn how theuser reasons and communicates about the world. As
it learns, the computer can provide increasingly useful and graceful assistance. Ideally, a wearable computer
1. Persists and provides constant access to information services: Designed for everyday andcontinuous use, the wearable can interact withthe user at any given time, interrupting whennecessary and appropriate. Correspondingly,
the wearable can be accessed by the user
quickly and with very little effort.
2. Senses and models context: In order to provide
the best cognitive support for the user, the wearable must try to observe and model the user’s
environment, the user’s physical and mental
state, and the wearable’s own internal state.
3. Adapts interaction modalities: The wearable
should adapt its input and output modalities automatically to those which are most appropriate and socially graceful at the time. In many
instances, the computer interface will be secondary to the user’s primary task and shouldtake the minimal necessary amount of the user’s
attention. In addition, the interface should guarantee privacy when appropriate, adapt to its userover time, and encourage personalization.
4. Augments and mediates interactions with theenvironment: The wearable should provide universal information support for the user in boththe physical and virtual realms. The wearable
should mediate between automation or computation in the environment and the user to present
a interface consistent with the user’s preferences and abilities. In addition, the wearable
should manage potential interruptions, such as
phone calls or e-mail, to best serve its user.

Obviously, this is an ambitious list of desired attributes, requiring many years of research. However,
the effort may provide insights into human intelligence and everyday human-computer interfaces.

2 Why use wearable computers?
2.1 Portable consumer electronics
Some people wear too many computers already. A
businessman might be seen carrying a personal digital assistant (PDA) to keep his contacts, a cellular
telephone to make calls, a pager to help screen incoming phone calls and messages, a laptop for business applications, an electronic translator for practicing a new language, and a calculator wristwatch.
These devices contain very similar components: amicroprocessor, memory, screen, keyboard or input
stylus, battery, and, in the case of the first three, awireless modem [12].
Many other portable consumer electronic devices
are evolving to have similar components as well:
the compact disc player is becoming the computerdriven MP3 player, the portable dictaphone is being replaced by solid state audio digitizers, digital
cameras are beginning to supplant the film camera
markets, and digital camcorders have outmoded theiranalog counterparts. These devices share many of
the same components; their main distinctions are determined by their sensors, interfaces, and application
software. Why not take advantage of commonality
in components to eliminate cost, weight, and redundancy and improve connectivity and services?
One could imagine a box the size of a deck ofcards enclosing a powerful yet energy-conserving
CPU and a large capacity data storage device. This
pocket-sized wearable computer has one output display – an LED to indicate that it is on and that its
wireless network is functioning. The wireless network connects peripherals to the wearable computerin a radius of about two to three meters centered at
the body.
The user defines the functionality of the wearable computer by the peripherals chosen. For example, if the user likes to listen to music, wireless earphones allow access to the MP3’s stored on the wearable computer’s hard drive. Adding a walnut-sizedcamera and the appropriate software transforms thewearable computer into a camcorder. With an Internet modem, the wearable becomes a pager, cellular
phone, web-browser, and e-mail reader. Connecting
medical sensors to the wearable concentrates many
diagnostic and recording devices into one unit.
Thus, with wearable computing and a wirelessbody-centered network, redundancy in components

can be reduced. In addition, functionality can beincreased as derivative services are created leveraging the convergence of media on the body. The
portable consumer electronics market will become
fertile ground for rapid innovation. Companies can
respond to a new need or niche market quickly with
an appropriate peripheral or software without having
to re-design all the subsystems on each iteration. Sophisticated portable electronics will become cheaperand more powerful for the consumer, and the computer industry will get a new, attractive upgrade path
to pursue.

IR LEDs
IR Pass Filter

Reflected light

Board Camera
Hand

IR Light

2.2 Mediating interactions in the world
The introduction of the windows, icon, menu, andpointer (WIMP) desktop metaphor in graphical userinterfaces provided a means by which a user interface could mediate between applications and theuser. Similarly, wearable computers will help
provide a consistent interface to computationallyaugmented objects in the physical world. For example, the Gesture Pendant (Figures reffig:pendant24) is a computer vision-based gesture recognizer
worn as a piece of jewelry (the design is similar tothe desk-based Toshiba “Motion Processor” project[15]). The device acts as a universal remote control to stereos, televisions, VCRs, home thermostats,and lighting systems [14]. Remote controls may
be difficult for older adults to use due to dexterity,
vision, or cognitive impairments; instead, the Gesture Pendant recognizers broad gestures to performthe same task. Since many functions are similar for
many devices, one gesture may provide an intuitivecommand across the devices. For example, raisingor lowering a finger in front of the pendant mightraise or lower the volume of a stereo, increase or decrease the temperature setting on a thermostat, andbrighten or dim the lights depending on which object
the wearer faces. Thus, the user may define or learn
one interface and then apply it repeatedly to manydifferent devices depending on the context. One
can imagine a more general purpose wearable computer that, through software, adapts itself to provide
a consistent interface to whatever electronics or computer systems are available in the environment. No
matter where in the world the wearable user travels,
the wearable provides an appropriate interface in theuser’s native language to home automation that may
be available in the user’s environment. Such infrastructure will be a boon to the handicapped. For example, imagine a tactile display for a blind wearable
user that renders simple graphics on the user’s back[2, 7]. When provided with a wireless interface andthe appropriate software hooks, the wearable computer can translate images from graphical systems,

Figure 3: The Gesture Pendant’s infrared light emitting diodes invisibly illuminate the area in front of
the user’s body so that the camera embedded in thependant can recognize the user’s gestures. A filter
that blocks visible light helps the vision system ignore lights in the environment
such as those used on interactive map kiosks, into
tactile images with which the user can interact. Already wearable computer systems are being investigated for assisting blind users crossing the wide,busy streets of Atlanta. These systems ”watch” forinfrared beacons embedded in walk street signals andtranslate this into a tactile wayfinding system for theuser [10].

2.3 Aiding human to human communication
The wearable can also assist in human to humancommunication. For example, Telesign, an ongoing research project at Georgia Tech, uses a hatmounted, downward looking camera in an effort torecognize sign language (Figure 5). A goal of Telesign is to demonstrate a limited working prototype ofa cellular phone for the Deaf as shown in Figure 6.
Such a system could also be used as part of an interpreting system when a deaf person needs to communicate with a hearing person who does not sign. In
carefully controlled laboratory conditions with user
dependent training, the recognition system performs
sentence-level American Sign Language recognitionat 97% accuracy with signs chosen from a forty word
vocabulary. Currently, we are trying to design computer vision and interface techniques that will allow
the creation of a more mobile system.
Similar systems might be used to translate between spoken languages. Carnegie Mellon researchers have already demonstrated a English to
Croatian speech translator for use in the BosnianHello Joe

TDD
CameraWe need ...

Computer and call
control

Wireless Video
Display

Gesture PendantPC
Control Devices
Slink-e
X10

SIGN TRANSLATIONSERVICE (STS):

Control Signals to Devices

Caller 1: Hello Joe
TDD 1: We need some help
on the contract
STS: OK guys. You caught
me walking to my next
appointmnet. Let me find someplace to stand and sign ...

Television

Stereo System

Lava Lamp

Figure 4: The Gesture Pendant allows users to control appliances in their home through gesture.
peace-keeping mission [13].

Figure 6: Telesign: A cellular phone system for the
hearing-impaired.

camera to look at a particular object. Correspondingly, the wearable could mediate reality [6] for theuser by displaying what it is seeing with its cameraon the user’s head-up display. Such a system may
be valuable for augmenting what the user could normally perceive, similar to providing range information while using a telephoto lense. This mediation
could also provide protection against distracting or
dangerous visual input, such as directly looking at
the sun while scanning the sky.

Figure 5: The hat-mounted camera, pointed downward to the hands and the corresponding view.
World/
Joint Task Context

External
Stimuli

Symbiotic System

2.4 Providing context sensitive reminders and intelligent assistantsContext sensitive interfaces may become very powerful with wearable computers. The wearable might
liste to the wearer’s conversations and provide reminders as appropriate to the current topic [8, 16].
Such reminders might range from the text of the laste-mail message exchanged between the participantsof an ongoing conversation to recommendations oflocal restaurants that both participants would enjoy.
2.4.1 Perception and symbiosis
In one approach to creating intelligent assistants, thewearable computer may be a first-class participant in
the user’s world (see Figure 7). The wearable senses
the world from the same “first-person” perspective
as the human. Both the wearable and the humanhave the potential to directly influence the sensorsof each other. For example, the human, through hishead movement, could direct the wearable’s video

Interaction
BridgeHuman

Wearable

Goals

EffectFigure 7: Wearable/user symbiosis.
In addition, both the wearable and the user canmanipulate the environment. In most cases, thewearable will effect the environment by communicating with other pieces of automation, such as theuser’s cellular phone, vehicle, or personal robotics,
while the user will effect the environment by performing a physical act, such as a making gesture orutterance. Both the wearable and the human have theability to observe each other’s manipulations. The
wearable exploits its on-body cameras and microphones to observe the user’s gestures and speech,and the user may use the wearable interface itself
to monitor the traffic history between the wearable
and other automation. In addition, the user shouldbe able to query the wearable as to the reasoning foreach action.
One of the strengths of the symbiotic system is the
cross-monitoring of the agents’ (human and wearable) actions. The wearable continuously tries to create and update a user model of the human’s goals
and abilities. The wearable can passively monitor
the human’s reactions to external stimuli and learn
to recognize meaningful events in the human’s life[1]. One can imagine this behavioristic approachbeing used as research direction for artificial intelligence; such an approach would try to emulate notjust human behavior, but a particular human’s behavior in a reasonable manner given novel stimuli. Fromcognitive science we know that such a “user-model
complete” approach has fundamental limits due to alack of knowledge of the internal state of the user;
however, a wearable intelligent assistant can take amore active approach. The wearable assistant may
gain some insight into the user’s goals by how the
wearer uses the computer. By maintaining a recordof all its interactions and the corresponding external conditions during those interactions, the assistant might learn to associate a certain set of external
stimuli with user desires. For example, if the user
keys “Joe Snyder” while alone in a shopping mall,the wearable might learn to bring up a map showing
Joe’s current position on a map relative to the user.
However, if the user keys “Joe...” while Joe is in theuser’s immediate vicinity, the wearable displays thelast e-mail sent by Joe so that the users may proceedwith an informed conversation.
In addition to learning through interaction, thewearable may form a user model by directly asking
the user for help. The assistant may associate a certain user behavior (for example, beginning a computer application for inventory management) with agiven context (like exiting a building at a geographical location indicating a supply warehouse). The assistant may then alert the user that it believes it canrecognize this behavior, ask whether or not it is a significant event, what the event should be called, andif it should begin the computer application automatically when it next observes the event.
Of course, such intelligent assistants based onreal-world, first-person perception are very difficult
to create. Most computer gesture and speech recognition systems are designed for use in a laboratory
and observe the user from sensors mounted in theenvironment instead of on the body. In most cases,
the variable noise conditions of mobility pose a significant barrier to the adoption of such systems.

2.4.2 Just-in-time Information: The Remembrance Agent (RA)Computers perform well at storing data and executing repetitive functions quickly. Humans, on theother hand, make intuitive leaps and recognize patterns and structure, even when passive. Thus, aninterface in which the wearable computer helps theuser remember and access information seems profitable. While word processing may comprise the majority of computer use, it requires only a small fraction of the computer’s processing power. Instead of
wasting the remaining power, an information agent
can use the time to search the user’s personal text
database for information relevant to the current task.
The names and short excerpts of the closest matching files can then be displayed. If the search engine isfast enough, a continuously changing list of matches
can be maintained, increasing the probability that auseful piece of information will be recovered. Thus,
the agent can act as a memory aid. Even if the user
mostly ignores the agent, he will still tend to glance
at it whenever there is a short break in his work. In
order to explore such a work environment, the Remembrance Agent [9] was created.
The benefits of the Remembrance Agent (RA) are
many. First, the RA provides timely information. If
the user is writing a paper, the RA might suggest relevant references. If the user is reading email andscheduling an appointment, the RA may happen tosuggest relevant constraints. If the user is holding
a conversation with a colleague at a conference, theRA might bring up relevant associations based on thenotes the user is taking. Since the RA “thinks” differently than its user, it often suggests combinations
that the user might never assemble himself. Thus, theRA can act as a constant “brain-storming” system.
The Remembrance Agent can also help with personal organization. As new information arrives, theRA, by its nature, suggests files with similar information. Thus, the user gets suggestions on whereto store the new information, avoiding the common
phenomenon of multiple files with similar content(e.g. archives-linux and linux-archives). The firsttrial of the prototype RA revealed many such inconsistencies within the sample database and even suggested a new research project by its groupings.
As a user collects a large database of private
knowledge, his RA becomes an expert on that knowledge through constant re-training. A goal of the RA
is to allow co-workers to access the “public” portions of this database conveniently without interrupting the user. Thus, if a colleague wants to know
about augmented reality, he simply sends a message to the user’s Remembrance Agent (e.g. thadra@cc.gatech.edu). The RA can then return its best

guess at an appropriate file. Thus, the user is notinterrupted by the query, and he never has to format
his knowledge explicitly, as with HTML. Knowledgetransfer may occur in a similar fashion. When an engineer trains his successor, he can also transfer his
RA’s database of knowledge on the subject so that
his replacement may continually receive the benefitof his experience even after he has left. Finally, if alarge collective of people use Remembrance Agents,
queries can be sent to communities, not just individuals. This allows questions of the form “How do I
reboot a Sun workstation?” to be sent to 1000 coworkers whose systems, in their spare cycles, may
send a response. The questioner’s RA, who knows
how the user “thinks,” can then organize the responses into a top 10 list for convenience.
The Remembrance Agent (RA) is comprised of
two parts, the user interface and the search engine.
The user interface continuously watches what theuser types and reads, and it sends this information to
the search engine. The search engine finds old email,notes files, and on-line documents that are relevant tothe user’s context. This information is then unobtrusively displayed in the user’s field of view.
The Remembrance Agent displays one-line suggestions at the bottom of the emacs display buffer,along with a numeric rating indicating the relevancyof the document. These items contain just enoughinformation to represent the contents of the full document being suggested. For example, the suggestion
line for a piece of email includes the subject line, thesender, and the date received. The suggestion line
for a notes file contains the file name, owner, datelast modified, and a summary the file. With a simple
key combination, the user can display the full textof a suggested document. The current information
retrieval implementation uses a term frequency, inverse document frequency algorithm for comparing
documents [11].
Figure 8 shows the output of the Remembrance
Agent (in the bottom buffer) while editing a document (top buffer). The reference database for the
RA was e-mail archives. Personal email and notes
files are good sources for an RA to reference, because these files are already personalized and automatically change with the user’s interests.
The Remembrance Agent can also be a method
for casual, low-overhead collaboration. Current everyday wearable computer users take copious notes
during their normal day to assist in their work andrecreation. As an informal experiment, three wearable computer users from the same laboratory combined their personal databases used in conjunction
with the RA so that they could gain the benefit ofeach other’s past experiences. Interestingly, RA sug-gestions from a colleague’s database can be quite
disquieting. The user recognizes the significance ofthe suggestion and can almost claim the memory as
his own due to the similarity with his own experiences, but he knows that it isn’t his entry. These“shadow memories” provide an asynchronous formof collaboration where the collaborators are passive
and their wearable agents share information as a matter of the normal use of the wearable.
While the Remembrance Agent gains most of its
contextual information from typed text, wearable
computers have the potential to provide a wealth ofcontextual features [4]. Additional sources of user
context information may include time of day, location, biosignals, face recognition, and object recognition. Currently the RA is being coupled with voicerecognition to provide information support during
conversations. Thus, the RA begins to show the advantages of wearable, context-driven augmented reality. However, with an even more comprehensive
agent, the wearable computer may be able to uncover trends in the user’s everyday life, predict theuser’s needs, and pre-emptively gather resources forupcoming tasks [1].

Figure 8: The Remembrance Agent in use whilewriting a paper. The bottom buffer shows the RA’s
suggestions for files relevant to the user’s currentwork.

3 Conclusion
Wearable computing enables significant new research opportunities in interface, artificial intelligence, and perception. As research into perception
and user modeling through devices carried on the
body progresses, new intelligent interfaces will result that will reduce work and complexity and leadto new capabilities.

4 Acknowledgements
This paper reflects the hard work of many membersof the Georgia Tech Contextual Computing Groupand the MIT Media Laboratory Wearable Computing
Project. In particular, Brad Rhodes and Brad Singletary have influenced the discussions herein.

References[1] B. Clarkson and A. Pentland. Unsupervised clustering of ambulatory audio and video. In ICASSP,
1999.
[2] C. Collins, L. Scadden, and A. Alden. Mobility studies with a tactile imaging device. In Fourth Conf. on
Systems and Devices for the disabled, Seattle, WA,
June 1977.
[3] D. Engelbart. Augmenting human intellect: A conceptual framework. Technical Report AFOSR-3233,
Stanford Research Institute, Menlo Park, CA, October 1962.
[4] M. Lamming and M. Flynn. Forget-me-not: Intimate computing in support of human memory.
In FRIEND21: Inter. Symp. on Next Generation
Human Interface, pages 125–128, Meguro Gajoen,
Japan, 1994.
[5] J. Licklider. Man-computer symbiosis. IRE Transactions of Human Factors in Electronics, HFE-1(1):4–
11, March 1960.
[6] S. Mann. On the bandwagon or beyond wearable
computing. Personal Technologies, 1(4), 1997.
[7] J. Nissen. Cupid and tactile exploration of virtual
maps. In IEEE VRAIS98 Workshop on Interfaces for
Wearable Computers, March 1998.
[8] B. Rhodes. Just-In-Time Information Retrieval. PhD
thesis, MIT Media Laboratory, Cambridge, MA,
May 2000.
[9] B. Rhodes and T. Starner. Remembrance agent:
A continuously running automated information retreival system. In Proc. of Pract. App. of Intelligent Agents and Multi-Agent Tech. (PAAM), London,
April 1996.
[10] D. Ross and B. Blasch. Wearable interfaces for orientation and wayfinding. In ACM conference on Assistive Technologies, pages 193–200, 2000.
[11] G. Salton. The SMART Retrieval System - experiments in automatic document processing. PernticeHall, Inc., Englewood Cliffs, NJ, 1971.
[12] O. Shivers. Bodytalk and the bodynet: A personal
information infrastructure. Technical report, MIT
Laboratory for Computer Science, Personal Information Architecture Note 1, Cambridge, MA, 1993.
[13] A. Smailagic, D. Siewiorek, R. Martin, and
D. Reilly. Cmu wearable computers for real-time
speech translation. In IEEE Intl. Symp. on Wearable
Computers, San Francisco, CA, 1999.
[14] T. Starner, J. Auxier, D. Ashbrook, and M. Gandy.
The gesture pendant: A self-illuminating, wearable,
infrared computer vision system for home automation control and medical monitoring. In IEEE Intl.
Symp. on Wearable Computers, Atlanta, GA, 2000.
[15] Toshiba.
Toshiba’s motion processor recognizes gestures in real time.
Available at:
http://www.toshiba.com/news/980715.htm,
July
1998.

[16] B. Wong and T. Starner. Conversational speech
recognition for creating intelligent agents on wearables. In Human Computer Interface International(HCII), New Orleans, LA, 2001.