Intelligent Cooperative Systems
Professor HIROSE, Michitaka

The domain of expanded virtual reality
extends from space to time

Interviewers: Jun Yamashita, research associate / Ryoko Ueoka, adjunct research fellow

̶̶Could you start by introducing the main research theme offrom the conventional computer technologies at that time. I

your laboratory?

think that VR served as a driving force, at least in part, for thedevelopment of computers in the 1990s.

In a word, it is virtual reality (VR). VR started as screen
technology and, initially, most of what you heard about VR

At the beginning, the virtual world was a closed world

was how viewers would feel as if they were in the screen

inside a box called computer. But the world inside the box

space. More recently, however, the concept of VR has beensoon reached its limitations, and hence, the first conceptual

gradually expanded to include, for instance, wearable

expansion occurred. That is, VR came out of the box. One

computers, which Ms. Ueoka has taken up as her researchof the new concepts thus born is mixed reality (MR), a

theme, and virtual experience in ordinary space, which istechnology that has been developed to fulfill the desire to

being studied by Mr. Yamashita. In this regard, I would say

watch the real world and a virtual world simultaneously. In

that "expanded VR" is the research theme of our laboratory.

practical term, you are now able to see through the special
goggles you wear, thus, you can see both the real world and̶̶Let us delve further into concrete details. In short,a virtual world. This technology started from superposing thewhat you have just said is that your research theme hasreal and virtual images.

expanded into what may be described as the boundary area
between technology and people, in which people use virtual

But simply superposing the real and virtual images inside atechnology as an expanded technology component to enjoy

small room is not interesting enough and you come to wantsomething in the real space. Am I correct?

to get out of the room in search of somewhere more spacious.
This desire led to another major direction that is to broaden

It is just around the end of the 1980s that this technology

the space by combining the MR technology with mobile

called VR emerged. At that time, VR was a kind of technology

computing. So, in the initial stage, MR was nothing more thanthat enables you to interact with a computer graphics (CG)the superposition of double images, namely, a virtual world

object right in front of you on a real time basis, wearing

and the real world. But when MR was combined with otherspecial goggles and gloves called data gloves. That is how VR

technologies such as wearable computing, the scope of virtual

came into our life. In other words, VR ushered you into theworld expanded into the real world, or I should say that theworld of CG, which was indeed a very novel concept.

technology itself changed in nature.

This served as an anchoring point from which the VR

Another interesting point of the VR technology concerns thetechnology has evolved in various directions. At ﬁrst, VR was"five senses." For instance, when people see an extremely

thought to be little more than a poor imitation of a computer

close-to-real CG image, they definitely try to touch it. This isgame which would go out of fashion in a year or so. As it

because every human being, by nature, is not fully satisfied

turned out, however, the boom did not end there. Looking

simply with watching things move. Instead, he or she wants toback now, we can see that VR is a technology quite different

touch, smell, or taste them as well. That is, every human being1

has a desire to conﬁrm with his or her senses in perceiving the

They are linked with each other but branching out in somewhat

world.

different directions. Thats the kind of image I have. And

once branches grow out, each of them gets more space toProfessor (Takeshi) Yoro of the Faculty of Medicine has set out

expand freely, thus, a new domain begins to grow there. Our

the deﬁnition of "mono (goods)." According to Professor Yoro,

VR research project, too, was once keen on CABIN and fromwhat can be recognized through a single sense ‒ for instance,

there new research themes, such as the one undertaken bya rainbow ‒ is "gensho (phenomenon)" while the term "monoMs. Ueoka (life log), were born.

(goods)" can be used only for what can be recognized throughall the ﬁve senses. This is a very philosophical argument but
it would be interesting if we could virtually synthesize all theﬁve senses. Philosophers would be in trouble, though.

Content that conveys the e xcitement oftechnology

Up until now, computer science has hardly paid attention
to the physical movements of human being as you can see
from how a computer keyboard or a display is structured.

̶̶I think that there are broadly two types of VR researchers,

That is, the reach of the senses of human being has not been

namely, those engaged in basic research and those working

utilized. However, when virtual elements come into being inon the development of concrete systems. Professor Hirose,a large space, it opens up a new world of physical senses andyou seem to belong to the second group.

computers.
In the field of computer science, you used to be admiredNow, let me summarize what I have said up to this point.

for creating something conceptual in the past. Conceptual

Evolution is taking place in two directions. One is thethings are things that present a new formula, for instance, acombination of mobile and wearable technologies while thesystem in which you wear a special goggle and catch a three-other is related to the ﬁve senses of human being. It may bedimensional object right in front of you or the embedding

the case that these two directions are connected underneathof tags. That is, let

the surface.

instance. As a conceptual work, it is ﬁne and meaningful evens say you have invented an airplane, for

if the airplane you have built is still in a very premature stage. It
is how a conceptual thing was supposed to be in the past and"Mobiles and wearables" and the "ﬁve senses"
Two directions for VR evolution

has been accepted as such. But when it comes to something
like the development of computer screen technology, it is
not enough to create a conceptual work. For instance, even if
you invent a new form of media that supersede movies, you
cannot convey the excitement of the new technology simply

̶̶Within a pillar called VR, there are two sub-pillars ‒by presenting conceptual test patterns.

"mobiles and wearables" and the "five senses" ‒ and these
sub-pillars may be linked with each other. That is where you

̶̶Do you mean that we need certain content in order toare trying to further the research.

convey the uniqueness and excitement of the technology?

Let me explain what is happening to the mainstream research

In developing information technology, we also need to

on VR, which is to let people experience an artificial world

develop content that is, to a certain extent, appealing andwith an extremely high realistic sensation. As you know,

digestible. Otherwise, people would not be able to tell

we have a big projection display system called CABIN at thewhether the new technology is interesting or not. This may

Hongo Campus of the University of Tokyo. Using gigantic

be a challenge that we will continue to face in developing

screens such as those you see at movie theaters, the system

information technology. As information technology further

let you feel as if you were stepping into the world inside thematures, we will inevitably move into an era of content. Even

screens. So, the mainstream of VR is moving in the direction

if what you are to create is meant to be a mere test pattern,of heavier and larger systems. At the same time, however,

you must elevate it to a certain level in quality or you will

there emerged another direction of research in which, as I justnot be able to convince people. We must achieve a very highmentioned, we use extremely small and compact computers

quality level, that is, to an extent comparable to that of CABIN.

such as wearables. These two streams of research are

We must show what wonderful things can be done with theseemingly unrelated with each other and pointed to different

technology. Being a faculty member of a university, I ﬁnd thisdirections. But I think they are actually the two sides of thetask quite tough and challenging, though.

same coin. A VR world you can experience using a wearable
computer is far greater in scale than the size of CABIN because,̶̶So, as a test case, you have been courageously undertaking

for instance, the whole campus or the streets outside could

various projects. For instance, as part of an exhibition

be turned into a virtual world. Even though the size of thefeaturing the Maya Civilization held at the National Science

computer is small, we can probably say that the whole system

Museum, you presented a virtual image space to introduce

is large in scale.

the concept of image technology and content to the generalpublic.

̶̶The idea that these two directions are actually same is avery unique viewpoint, which I guess is of your origination.

Generally speaking, people doing research at university arequite imaginative. So, explain the concept of your research

2and prepare a special goggle and data gloves. Then, they

̶̶As an example of research that we have been able towould roughly grasp the whole picture of a world you are

undertake thanks to being a member of the Research Center

trying to build. But when you deal with ordinary people, you

for Advanced Science and Technology (RCAST), we can cite aneed to present your idea in an easier-to-understand manner.

radio frequency identiﬁcation (RFID) tag project.

If you simply say a mixture of virtual and real, people would
not understand what you are talking about. So, if we want to

Only now, we are beginning to hear people talking about

truly inﬂuence the society with engineering, we must properly

embedding RFID tags in various places. We were quite

create and incorporate content to go with it. I guess this is theahead of time when we embedded such tags in the floors

way we need to be.

around No. 4 building of RCAST. And after we actually hadthem embedded, we came to realize various problems. For

̶̶Now, let me ask you about the concrete substance of yourinstance, embedding tags has tuned out to be considerable

research.

construction work. Also, they were submerged when we had
extremely heavy rainfall some years ago. At the time of theWe have just completed an exhibition project in which Mr.

game exhibition, we had to install 400 sensors in the ceilings.

Yamashita has been engaged. The project was for the "Video

Although we are trying to create a virtual space, we end up

Game and Digital Science" exhibition held at the National

doing lots of physical work just like what we would do for

Science Museum. As part of this game exhibition, we created aconstructing a real, physical space. In this regard, I remember

new way of exhibition thereby enabling visitors to enjoy boththat you, Mr. Yamashita, said, "Ubiquitous means matching ofreal and virtual exhibits simultaneously as they walk around.

physical objects."

During the past three months or so, we have been carryingout experiments for this project.

Engineering technology has a pragmatic aspect. In order torealize certain concept, we must solve various problems that

What was very good about this project is that we madecome along with that concept. Only when we ﬁnd solutions

the whole system compact and were able to present very

to all these problems and have them incorporated into theinteresting exhibits. Also, we let people interact withconcept, we can call it technology. Simply having a concept iscomputer by means other than the sense of sight. I wonder

not enough. We must try it out to ﬁnd problems and solutions

why we had never done before. People would not watch ato the problems. At RCAST, research evolves quickly because

display while moving around. Up until now we were trying

we are able to start experimentation without wasting time.

to present images on a display but that was wrong. Because

That is probably one of interesting features of RCAST.

people are moving around, we had to let them recognize
objects by means of non-visual senses such as auditory andtactile senses. Another notable thing about this project is that
we created very sophisticated content. For instance, we asked
professional voice actors to do the voices. No one would listenCollect information with a wearable computer"Lifelog" that started from a simple idea

to the mumbling voice of a student at our laboratory. But
professional voice actors have a special skill and keep people
tuned to their voice. In addition to developing an interesting̶̶Professor Hirose, about what you earlier mentioned about

concept, we must make its content interesting. Only then, we

VR, that it did not end just as a transient phenomenon as

can convey the excitement of technology. We need both of

initially thought, isnthem, concept and content. Otherwise, it would be no good.

easily linked to various aspects of human?

This argument I have just made is somewhat different from

The recently emerging concept of transdisciplinary approach

the so-called content problem we have been hearing lately.

is to combine and mix things that are alien to each other. But

We are not seeking to create content in a way to excel inin VR, things are mixed from the very beginning. VR is not apresentation within the given framework of technology

technical term originally and reality is something that exists in

as is the case with movies. What we are trying to do is toyour head.

t it because this particular technology iscreate interesting content while changing the technologicalframework at the same time. In this sense, we are not

For instance, in the case of a fusion between science and thethinking of content as a separate object. Instead, we think ofhumanities, we are currently working with Professor Mikuriya

technology and content as one united system. That is why

and others to launch a new project. About our research on

we asked Professor (Toshio) Iwai to join us under special"lifelog," a tool for keeping the whole record of your life, we

appointment.

were simply thinking about keeping records of experience.
But when Professor Mikuriya comes into play, this evolves into

This means that there are many things that we cannot

a very lively story. I thought this is the power of the humanities

convey simply by establishing a technological platformscience. It is impressive how things broaden and evolve.

and developing new concept. Probably what is important

Sometimes, they turn out to be nothing short of contents. Ms.

is to show the feasibility, i.e. what we can do based on thisUeoka, you probably did not expect any association between

platform. Information technology is now coming to a turning

lifelog and "oral history," did you?

point. It is no longer about simply finding applications. We
are entering a stage where we need to focus on what sort of̶̶The Lifelog project started from what I wondered whileinteresting content we can create with new technology.

doing research on wearable computers. That is, if an extremely
small information device can be worn by a human, I thought3

we can use the device not simply as a means to provide

us, researchers specialized in science, is that the way we see

information but as a tool to collect various sorts of personal

things is too objective to capture stories behind the things. But

information. When I tried this out, my own experience in theunless we capture both the form and story, we cannot keep

reality has manifested itself in a way quite different from whatcomplete records. We have yet to know how we can keep

I had expected.

records on them and how the records can be synthesized. But
it will be interesting to think these things.

That is interesting, isnt it? In the ﬁeld of VR, the mainstream

is research on computer displays although, in terms

Not only looking back to the past, we can also look to theof interaction, some research on technology to take infuture. Here comes in the concept of "time machine," which

information into computers has been conducted by necessity.

is so to speak a technology for traveling through time. After

But in research on wearable computers, this relationship isall, the 20th century has been an era of a spatial concept, inbeing reversed. When computers become further smaller

which our spatial ability has improved very much but we havein size and installed in various places of the world as if they

developed little time concept. I think this concept of time iswere sensory organs, information siphoned through theseabout the only major research theme left for us to work on incomputers becomes far more important and informative.

the 21st century. Currently, we are working on a "virtual timemachine" project which was jointly launched with the National

I think Ms. Ueokas research must have been derived fromsuch context. All the experience input into a computer isInstitute of Advanced Industrial Science and Technology (AIST).
The speciﬁc content of this project is coming soon (laughter).

recorded and will stay there unless you erase the relevant
data. Furthermore, the data can be viewed by others. Once

Toward the past, for instance in case of lifelog, we trace thetaken into the media, experience of yourself or others can bedata back. Toward the future, we should be able to use theshared if so wished.

simulation function of computers. We are now in the era
when we can somewhat travel to the future. To put it other

Originally, experience record is something of personal interest

way round, letand for keeping record on everyday life of individuals, i.e.

certain future point of time by using a simulator. Then, when

information that normally would not surface outwardly. In thisthat moment actually comes, what we are doing is nothingsense, this seems to be rather remote from business. But what

but confirming the predicted situation. Even today, we canofﬁcials of Dentsu Inc. said after taking a look at this researchpredict certain future events. For instance, checking with theis that this can be used as a marketing tool. Indeed, while"eki-navi (station navigation)" Internet service, you would

questionnaire surveys have been used as a means to findsay, "In 30 minutes from now, I will be getting there late"out an average picture of things, lifelog enables us to traceor "However hard I try I cannot get there on time." So, in aeverything in someone

sense, we are already entering a new system where we uses life. This is a very delicate way ofcolleting data that walks on a thin line between infringements say we predict a situation we will be in at asomething resembling a time machine in life.

and non-infringement of privacy. But this is very interestingand it may be possible to use this method in examining

It may seem that things are moving in direction unrelated

the trend of TV viewers

to VR. But the world of information is a virtual world, isnpreference and checking livingthabits that lead to lifestyle-related diseases. There is lot of

it? The world of information is free to move backward or

information which is spreading so thinly across our everyday

forward. What is interesting is that the real world is beginning

life that we have been unable to use it. So, the question would

to become virtualized with many virtual things coming into

be whether we can collect and give new life to such hitherto

the real world and that such a thing is becoming possible inunused information by using this new mechanism called

our world. An increase in the degree of freedom of the world

lifelog.

is kind of dangerous. But it is not totally a bad thing. If we free
up our imagination, we can develop many interesting things

What we are trying to do in the lifelog project is to use afrom here.

computer as a recording device, which is in a sense not at all
strange or surprising. The reason why this kind of research(October 22, 2004)

has become very realistic is that we are now using e-mails indoing work whereby certain segments of our life go throughcomputers. That is, we are living in a world where we can,if we try to, capture these segments of life at a certain pointof the process of transmitting information. For instance,
when people use an IC card such as SUICA, it is possible tokeep records of who passed where at what time. And suchinformation can be organized and synthesized from various
viewpoints whereby a certain image will emerge. Such a thing
will become possible with lifelog.
What we are trying to do with Professor Mikuriya is to create
something like a photo album so that we can etch the shapeof past systems. For the moment, we are trying to work out
how we can create such an album. A typical shortcoming of4