Special Feature: New Interface Technologies Creating

Revolutionizing Mobile Communications
through New Interface Technologies
Hirotaka Nakano, Toshiaki Sugimura†, Shoji Kurakake,and Atsushi Takeshita
Abstract
NTT DoCoMo Multimedia Laboratories are researching user interfaces that will enable you to use
devices, objects, and information naturally and with ease as if they were part of your body. This article
describes associated technical issues and introduces the concept of Human Centered Communication
and Computing (HC3) based on the technologies described.

1. What is HC3?
Reflecting its corporate philosophy to “create newcommunication services that will revolutionize theway we communicate and establish a new global
communications culture,” NTT DoCoMo proposes
the concept of Human Centered Communication andComputing (HC3). The company is expending muchresearch effort toward practical implementation ofthis concept.
The aim of HC3 is to enrich communication andenhance human capabilities through interfaces thatcan convey a user’s ideas, sensations, sensitivity, andintent. These interfaces are not about operating
equipment with clear intent of accomplishing sometask. They are rather technologies that will enablepeople to enjoy the benefits of information technology in a free and natural way, that will facilitate bodyoriented communication, and that will represent andsupport people both physically and intellectually.
Specifically, HC3 means creating new informationcommunication technologies taking into account thefeasibility and means of achieving a wide variety ofrequirements. These include the abilities to “usesomething in the way one wishes,” “share information and feelings as desired,” “support and complement a user by inferring his or her thoughts or con† NTT DoCoMo Multimedia Laboratories
Yokosuka-shi, 239-8536 Japan
E-mail: sugi@mml.yrp.nttdocomo.co.jp

60

text,” and “use things with ease in a natural, humanoriented manner.”
This introductory article surveys the needs of society and the role of technology, presents a future
image of communications, and describes issues andspecific research themes that must be addressed andpursued to satisfy technical requirements. The otherrelated articles in this issue describe specific research
activities and their current state of progress.
2. Needs of society and role of technology
The 21st century has come to be called the “age of
people”, the “age of environmental management”,
and the “information age” [1]. Here, we discuss theneeds of this diversified age in terms of what future
mobile communications should aim for and describe
technical requirements and the role of technology tothis end.
In the “age of people”, a human-centered system isessential if people are to live a truly human life, and acommunication and computing system that is especially oriented to human needs should play an
extremely important role here.
In this regard, there is much anticipation for true
multimode communication that will enable people toreach even higher levels of mutual understanding andempathy. In addition to conventional forms of communication such as text, speech, and video, multimode communication will be able to convey information to all five senses and to exhibit embodiment charNTT Technical Review

Innovations in Mobile Communication

acteristics (by using a physical entity to convey gestures and facial expressions more expressively).
It is also important here to provide high-reality
forms of communication to make a deep impression
on users and hold their attention and to provide content-based services that are both emotionally andphysically impressive. Interfaces are also of great
importance. For example, there is a need for an interface that can perform operations and give instructions
by inputting and conveying what the user is thinking
about or what the user wants to do, i.e., user intent,
without requiring the user to learn how to operate it.
The need is also felt for a user interface that can manage devices naturally and at will just as if they werepart of one’s body through simple operations andinstructions using ordinary physical actions.
In addition, to enhance a person’s information-processing ability and power of movement and to enrichthat person’s activities, there is a need for an environment that can support and complement that person inaccordance with surrounding people and objects.
Furthermore, in support of the aging society, there isa need for technologies that can support and complement internal organs, motor functions, and the like topreserve a healthy mind and body and promote a
sense of fulfillment and satisfaction.
The “age of environmental management” demands
technologies that can ease the load on the global environment. These include technologies for making
environmental monitoring functions habitual andomnipresent through the five senses as a useful meansof protecting and preserving the environment, andtechnologies for reducing traffic by providing alternatives to traveling such as videoconferencing, by
optimizing traffic routes, etc.
Much attention is now being focused on a ubiquitous information-communications environment
toward an “information age” in which the information world and real world that have traditionally been
separate will merge to form a “super environment”[2]. Specifically, it will be possible to merge theinformation world within communication networks
and the real world of human activities by making it
easy to acquire real-world information and to use that
information to communicate, and by replacing someendeavors in the real world with telecommunications.
Here, within a world having a huge volume of information and a great number of devices in many forms,
it is important that we have technologies that can support people or act for them in locating and handling
optimal information and devices. This super environment will therefore be convenient and comfortable,
Vol. 1 No. 8 Nov. 2003

but it will also raise concerns about invasion of privacy and increase in crime, for example. Under thesecircumstances, technologies for handling and supporting private information in public spaces, fordetecting crime-related information and events, andfor implementing encryption and security protocol
will take on added importance.
3. Classification of technical issues by humancharacteristics
Table 1 lists technical requirements originatingfrom social needs and technical issues that must be
addressed to satisfy them. Horizontally, the table isdivided into three sections according to human characteristics in relation to the input, processing, andoutput of information. These are “sensory organ system” in relation to information input; “brain andintelligent-processing system” in relation to information processing centered about the brain; and “motorand behavior system” in relation to the expression
and presentation of information, emotions, andintent. Each of these sections is further divided into
innate and acquired functions [3]. Vertically, the table
is divided into social needs and technical requirements. Each table entry describes technical issues
that must be addressed to satisfy the requirements forthat row within the system for that column.
The sensory organ system consists of innate functions that govern the five senses, and assisting andextending these sensory functions are technicalissues. The brain and intelligent-processing system
consists of both innate and acquired functions, withthe former consisting of memory, recognition, inference, and learning and the latter consisting of preferences, value judgments, comfort, and contentment.
For both types of functions, researching and elucidating cognitive characteristics and extracting information-processing mechanisms as intelligent technologies are common issues with respect to the various
technical requirements listed.
The motor and behavior system is where the resultsof internal human activities such as emotions andthoughts come to be manifested. This system
involves a variety of technical issues. First, there is aneed for explanations and models of the mechanisms
behind information-based behavior and movement in
humans. Next, there is a need for technologies forconveying and sharing emotions and intent throughvarious forms of expressive behavior, gestures, andhand movements in accordance with human norms ofcommunication. Finally, technologies are needed for61Special Feature

Table 1. Main technical
Human characteristics

Input (sensory organ system)
Innate functions

Needs of society® Technical requirements• Enable communication
conducive to mutual
understanding and empathy

Five senses: hearing, sight, touch, smell, taste• Input, convey, reproduce
as new dimensions in media
Expand range of conveying
and sharing sensations
and sensitivities(communication and content
based on the five senses)• Raise magnitude of
media quality
Conveyultra reality®

®

Age of people• Provide impressive andmoving content• Input, convey, and reproduce directly what your body feels as
sensation information
Direct five-sense interface• Input and reproduce by sensation-based devices andenable media conversion between senses
Indirect five-senseinterface (by either an interface attached to your body or one
attached to a remote robot)• Sense and convey internal physical functions and motor functions
by fulltime-wear devices
Interface for monitoring your physicalcondition®

• Provide an easy-to-use
human interface®

®

Age of environmental
management• Enhance human capabilities(including health support)• Collect and use information
useful for environmental
management®

• Reduce load on theenvironment• Improve information selectivity
taking individual characteristics
and context into accountInformation age• Achieve sensors (compact, low-power, zero-installation, cooperative
operation) corresponding to the five senses and install, operate, andcollect data according to objectives
Information platform using aubiquitous sensor network (using sensors embedded in buildings,
roads, vehicles, people, etc.)• Improve information
adaptability taking individual
characteristics and context
into account• Sense and utilize information on your five senses as a means of
extracting your individuality and as a means of determining yourcurrent context
Understanding individuality and context via thefive senses• Sense and utilize five-sense information about your environment
using wearable devices and those located nearby as a means ofdetermining surrounding conditions
Understanding surrounding
conditions via the five senses®

®

• Improve environmental
adaptability for enjoyable
use of services

62

NTT Technical Review

Special Feature

issues for achieving HC3.
Processing (brain and intelligent-processing system)
Innate functions

Acquired functions

Memory, recognition,
inference, learning

Preferences, value judgments,
comfort and contentment• Improve translation functions→ Overcome language barrier〈Outside the scope of thisresearch〉• Search out agreeable
conversation partners →Support and provide
enhanced selectivity for
individual use• Search out desirable content→ Support and provide
enhanced selectivity for
individual use• Directly input, convey, and• Deal and respond flexibly withsearching/adjusting requirements reproduce what one is thinking,what one wants to do (intent)
and conditions mainly based on
language and images (cope with without having to manipulate
devices → Ultimate thoughta variety of expressions; adapt
interface
to insufficient information withappropriate questions, etc.)→ Intelligent agents• Record one’ s own behavior
automatically, and select anduse desired information at will
→ Intelligent memory support• Automatically record dataobtained by monitoring behavior
and physical conditions, detectchanges, and issue alerts→ Intelligent health monitoring

Output (motor and behavior system)
Innate functions

Acquired functions

Physical form/functions

Speaking, expressing intent,interactive behavior• Create embodied expressions (gestures, hand movements, facial
expressions, etc.) → Integrate emotions, intent, and atmosphere(form a collaborative space where context can be shared)• Satisfy “immediacy” in which functions can be used just as soon
as one thinks about them; “wearability” in which devices are always
worn on one’s body in a comfortable manner, and “usability” inwhich input can be performed instantly without impeding the thoughtprocess. → Fulltime-wear Interface ready to use• Manipulate information by manipulating physical entities → Intuitive
interface through real entities• Use ordinary physical actions to manipulate and instruct devices
naturally and easily → Use part of the human body as an interface
mechanism• Sense physical functions and motor functions and physicallysupport and complement those functions → Physical function
support• Project physical sensations and emotions and manipulate at will aremotely located robot sensitive to your five senses → Pseudoteleportation (for security, nursing, remote operations, exploring
undeveloped space, etc.)• Efficiently store large amounts of collected data → Ultralarge-scale, • Use results of human behavioral prediction and natural phenomena
real-world database based on hierarchically distributed andprediction to select optimal and alternative means of human activity
centralized role-sharing management→ Reduce and control energy consumption optimally by using• Filter vast amounts of data to obtain required information
information on a world-wide scale
and perform ultralarge-scale data mining to discover new
relationships → Support and provide ultralarge-scale analysisof real-world information• Store five-sense information and behavioral information collectedon the individual and learn and infer individual characteristics(preferences, comfort and contentment) → Support and perform
learning and inference of individual characteristics• Infer current context from five-sense information and behavioral
information collected on the individual → Support and perform
inference of individual context• Filter out optimal information or process information adaptively forthe individual based on learned individual characteristics and on
five-sense information, behavioral information, and surrounding
conditions currently being collected → Support and perform
adaptive information selection taking individual characteristics
and current context into accountVol. 1 No. 8 Nov. 2003

• Control methods of expressing, uttering, and presenting information
based on individual characteristics and context and current
surrounding conditions → Support methods of expressing andpresenting information taking individual characteristics andcontext into account• Construct an optimal service-usage environment for the individualby selecting and combining resources from the peripheral resourcegroups using individual characteristics and context and current
surrounding conditions → Support and perform optimization ofservice-usage environments taking individual characteristics andcontext into account63

Special Feature

supporting and enhancing human information-processing ability and power of movement by enabling
devices to be operated as an integral part of one’s
body and by enabling the body itself to be used as aninterface.
4. Future image of HC3
Figure 1 shows the future image of HC3 after theabove technical problems have been solved. Here, wehave the birth of a “new-century man” (i-borg) withenhanced senses and no limit to the amount of knowledge that he can acquire. This is made possible
through fulltime-wear information accessories and a
24-hour network connection that enables the newcentury man to interact with a wide range of information-communication devices located throughout theworld. In other words, HC3 is communication andcomputing through interfaces that make it feel as if
information, devices, and the network are all an integral part of one’s body and that enable such resources
to be used naturally and with ease. With HC3, a per-son can operate fulltime-wear devices or nearby
devices naturally just as if they were part of one’s
body without having to be particularly conscious ofthem. In this way, a person can convey and deeply
share at will not only information but also sensations
and emotions, and will be able to access the vast
amount of information on the network as desired foruse as personal knowledge. In addition, HC3 will feature personal agents that become familiar with one’s
preferences, way of thinking, and other characteristics to support and even represent a person in someintellectual activities. Finally, HC3 will enable a person to project his or her physical movements, sensations, emotions, and intent to a remotely located robot
and interact with and manipulate that robot as one’s
alter ego. This will, in effect, extend one’s physicalpresence to somewhere else in the world.
5. Main research themes
The main research themes in HC3 are described
below.

HC3: Human centered communication and computing featuring interfaces that enable
anything to be used naturally and with ease just as if it was a part of one’s body.
Brain-enhanced man (i-borg)
Evolution toward “new-century man” with five enhanced
senses and unlimited knowledge made possible throughinformation accessories worn on the body, a continuous
network connection, and cooperation with granular
computers scattered throughout the world.

Head: Unlimited knowledge and advanced powers of judgment• Brainwave-input equipment

Acquisition of knowledge in the world
Network• Agents• Information
retrieval• Remote robot(alter-ego interface)

Gritters(Gritters:
granular computers)• Public gateway(network connection point)• Object
information

Ultra-small computers
having individual IDs
and sensors that are
affixed to all objects
and scattered everywhereon the earth.

• Internal datatransmission• Internal
electric-powertransmission

Short-range wirelesstransmission
Personal gateway(connection tooutside world)

Eyes: See invisible things• Retina-projection display(augmented reality display)
Ears: Hear inaudible sounds• Ear-hole-inserted receiver
Nose: Recognize by smelling• Olfactory sensors and smell
synthesis
Mouth: Talk without attracting attention• Voiceless speaking device
Body: Always healthy (health management); individual authentications• Vital-signs sensors
ACCESSors(ACCESSors:
information accessories)
Ultra-small biological interface
devices always worn on the bodywithout interfering with daily life.
Hands: Manipulate at will
• Wrist-worn commander• EMG input equipment• Environmental data
• Location beacon
IDs and information-communication
functions are given to all objects(and scattered throughout the world)

Legs & foot: Simple and accurate movement• Location sensor• Power generation by walking• Power generation by body heatFig. 1. Future image of information communications in the 21st century toward HC3.

64

NTT Technical Review

Special Feature

5.1 Mechanisms of human biological information
Interface technology that can directly input, transfer, and reproduce biological information shouldmake it possible to use touch, smell, and other senses
as new forms of media and bring about radical
changes in the way people communicate. An essential element of this approach is research that attempts
to clarify the mechanisms of human biological information. To this end, we have adopted the superconducting quantum interference device (SQUID) as a
biological magnetic-measurement system. This system, which is capable of extremely high temporal andspatial resolutions, can make non-contact measurements of the small magnetic fields generated by electric currents that accompany brain, nerve, and muscle
activities. The system can also be used as an interface. At present, research and development of a technique for analyzing the accuracy of SQUID output
results are proceeding in parallel with research on themechanisms behind human biological information.

5.4 Alter-ego interface technology
The target here is a “real interface” that can convey
one’s posture, actions, and behavior by projecting
one’s self via a robot entity (alter ego), a function notpossible with conventional audio and video media.
Such an interface will play an important role in theevolution of the cellular telephone from “word-based
communication to emotion-based communication”.

5.2 Biological interface technology
Interfaces that measure, analyze, and use biologicalinformation are being researched with the aim of creating new media that can apply the five senses andoperation-by-movement to communication. The firststep in this research is to explain the basic mechanisms of the peripheral nervous system with the aimof developing application technologies. We are therefore targeting technologies for sensing, analyzing andusing biological information related to muscle movement and technologies for sensing, analyzing, reproducing, and using surface sensations related to thesense of touch.

5.6 Ultra-real 3D video communications
This research aims to develop 3D presentation technology that can reduce the restrictions associatedwith wearable devices and enable a user to enjoy genuine 3D video in a mobile environment. This technology will feature both full and motion parallax andaccommodation (focal adjustment).

5.3 Fulltime-wear interface technology
In the society of the near future where high-speed
wireless networks are commonplace, there will be noneed to attach computers that store information onto
each and every human body. The minimum requirement for equipment that a human must carry will
instead be interfaces (devices) that convey the intent
of the user to the network side and, conversely, that
present the user with various kinds of information
obtained from the network. The aim here is to achievea “fulltime-wear interface” that satisfies three key
conditions simultaneously. These are “immediacy”,
or immediate access to desired information, “wearability”, or continuous wear to provide this immediacy
without hindering daily life, and “usability”, or rapid
input and output of information without hindering
one’s flow of thought.
Vol. 1 No. 8 Nov. 2003

5.5 Ultra-real 3D speech/audio communications
This research theme focuses on technology that cangenerate a 3D sound field sensitive to space in a
mobile environment using, for example, sensors that
detect body movements. Included in this research isaugmented reality in which a user can experienceeven more than what exists in reality. Another direction of this research is the generation of artificial 3D
sound fields such as in three-way calling, where it
would appear that the voice of one speaker is comingfrom the left and that of another from the right.

5.7 Service selectivity in a ubiquitous environment
The technology targeted here will employ the direct
use of objects (such as tools, appliances, vehicles,
books, buildings, and places) in the environment thata person inhabits as triggers for finding related services without hindering that person’s activities in thereal world. Also, with the aim of acquiring personal
preferences and circumstances associated with theselection of an optimal service from among several
candidate services found in this way, research here
will also aim for technology that can continuously
store information on all kinds of daily activities in aform conducive to automated processing so thatbehavioral patterns can be extracted.
5.8 Adaptive construction of service-usage
environment
This research theme aims to develop technologythat can determine what information-communication
resource groups are available for a person moving ina ubiquitous environment given personal situations
and policies as well as current conditions in the information-communications environment. Using the65Special Feature

resources so determined, this technology will be able
to construct an optimal information-communications
environment for the user at all times. A user will
therefore be able to use services under optimal conditions without interruption no matter where he or she
may be.
In a ubiquitous society providing many and diversified services, it is essential that desired services befound efficiently according to current conditions.
Studies will therefore be performed on state-dependent and context-dependent service discovery andservice-customizing methods to enhance this processeven further.
6. HC3 is our vision
We have described, from a human-centered point ofview, the technical issues and research surroundinguser interfaces that enable information, devices, andjust about anything to be used freely and with easejust as if they were part of one’s body. We have proposed a future image of human centered communication and computing (HC3) based on these interfaces,
and have introduced the issues that must be addressedto achieve HC3 and approaches to solving them.
References
[1]
[2]

[3]
[4]

66

“Information Communications Technology in 2015,” NTT Publishing Co., Ltd., 2001 (in Japanese).
K. Sakamura, “The Ubiquitous Computer Revolution: The World
Standard of the Next Generation Society,” Kadokawa Shoten Publishing, 2003 (in Japanese).
“Next-generation Telecom: Current Possibilities,” NTT PublishingCo., Ltd., 1992 (in Japanese).
G. Salomon, “Distributed Cognitions,” Cambridge University Press,
1993.

Hirotaka Nakano
Executive Manager, NTT DoCoMo Multimedia Laboratories.
He received the B.S, M.S, and Ph. D. degrees
in electric and electronic engineering from the
University of Tokyo, Tokyo in 1972, 1974 and1977, respectively. Since joining NTT in 1977,
he has been researching image/video retrieval
systems (CAPTAIN, VOD, etc.) and multimedia
communication systems. Currently, he isengaged in research on mobile multimedia. He isa member of the Institute of Electronics, Information and Communication Engineers (IEICE),
the Institute of Image Information and Television
Engineers, and the Institute of Image Electronics
Engineers of Japan.

Toshiaki Sugimura
Senior Executive Research Engineer, NTT
DoCoMo Multimedia Laboratories.
He received the B.S. and M.S. degrees in computer science from Tokyo Institute of Technology, Tokyo in 1978 and 1980, respectively. In
1980, he joined Nippon Telegraph and Telephone Public Corporation (now NTT). He moved
to NTT DoCoMo in 1999. He is engaged inresearch on natural language processing, intelligent processors, mobile multimedia, ubiquitous
information-communications environments, andother subjects. He is a member of IEICE, theInformation Processing Society of Japan (IPSJ),
and the Association for Computing Machinery(ACM).

Shoji Kurakake
Executive Research Engineer, NTT DoCoMo
Multimedia Laboratories.
He received the B.S. and M.S. degrees in mathematical engineering and information physics
from the University of Tokyo, Tokyo in 1983 and1985, respectively. Since joining NTT in 1985,
he has been researching character recognition,
image recognition, media handling, mobile
agents, and ubiquitous services. He is a memberof IEICE, IEEE, and ACM.

Atsushi Takeshita
Executive Research Engineer, NTT DoCoMo
Multimedia Laboratories.
He received the B.S. and M.S. degrees in information science from Kyoto University, Kyoto in1986 and 1988, respectively. He joined NTT in1988 and is currently researching multimedia
information searching and mobile internets. He isa member of ACM and IPSJ.

NTT Technical Review