5-4 Perceptual and Cognitive Mechanisms ofPresence and its Evaluation Technology
ANDO Hiroshi, Akiko Callan, Norberto Eiji Nawa, NISHINO Yurie, Juan Liu,
WADA Atsushi, and SAKANO Yuichi
This paper first describes the evaluation indexes of presence, i.e., the components and factors of presence, and the measurement and evaluation methods of presence. Second, as a presence-evaluation technique, we describe the development of evaluation technology which canmeasure human brain activities when large-field 3D images are shown. We also describe thequantitative evaluation methods of human sense of surface quality, especially glossiness, and theresults of psychophysical experiments. Finally, future issues of perceptual and cognitive mechanisms of presence and its evaluation techniques are discussed.

Keywords
Presence, Evaluation methods, Brain activity imaging, Psychophysics, Glossiness

1 Introduction
In order to construct an “ultra-realistic
communication system” which causes people
to feel a strong sense of presence, it is necessary to analyze in depth what the nature ofpresence as felt by people. In human beings,
information received from the outside world
through the sensory receptors of their five senses is transmitted to their brains, the information
then undergoes complex perceptualand cognitive information processing to be felt as presence. As such, in order to clarify the sense ofpresence felt by humans, we need to investigate the perceptual and cognitive mechanismsof the brain and neural system that allow humans to accurately interpret the realities of theexternal world.
If the mechanisms of the sense of presence
that humans feel were able to be scientificallyclarified and presence accurately measured
and evaluated, it would have a huge value inconstructing an “ultra-realistic communication
system”. The first is the significance of the safe-

ty of an ultra-realistic communication system.
An ultra-realistic communication system is an
artificial system which can convey to humans
information for which there is no actual substance as if there were, and there is a possibility
that such systems could harm the brain or neural system of a human unnoticed. In order tosafely convey presence without causing physical or mental discomfort or fatigue to humans,
the presence felt by humans must be accurately
and quantitatively evaluated. Another meaningof the clarification and evaluation of presenceis that it may lead to the discovery of delivering presence to humans. For example, by using the photoreceptors equipped in the humaneye, which have the sensitivity characteristics
to sense the three colors (red, green and blue),
video displays can cause a person to sense all
possible colors through combinations of theaforementioned three colors. If the perceptualand cognitive mechanisms of all five senses
were able to be clarified in this way, people
could be made to feel presence in an efficientand effective manner without faithfully recreat-

ANDO Hiroshi et al.

159

ing all physical properties. The “ultra” in ultrarealistic communication system can be thoughtof as implying that the aim is to construct a new
realism system that uses human perceptual andcognitive mechanisms to surpass simply faithfully recreating physical information.
This paper first describes the evaluation indexes of presence felt in section 2, and theevaluation methods of presence. Second, as apresence-evaluation technique in section 3 we
describe the development of evaluation technology which can measure human brain activities when large-field 3D images are shown. In
4, we also describe the development of quantitative evaluation methods of human sense of
surface quality, and the results of psychophysical experiments. Then in 5, future issues of perceptual and cognitive mechanisms of presence
and its evaluation techniques are discussed.
And finally in 6 this paper’s conclusions arepresented.

2 Presence evaluation indexes
and evaluation techniques
2.1 Presence components and factors
Presence felt by humans can be said to be“feeling as if they were in the actual place”,
however if one more deeply delves into theconcept of presence, one will find that rather
than being a single sense, presence is actuallycomposed of multiple sensory components.
Therefore as shown in Fig. 1, when analyzingpresence, we analyze it using 3 components ofspatial components, temporal components andphysical components as evaluation indexes [1].
First, the senses of 3D, surface quality andsurroundness can be used as spatial components for presence. The sense of 3D is a sense
related to the depth (distance from the observer) and shape of an object existing in a space.
The sense of surface quality is a sense related
to the roughness of the surface of an object(texture), an object’s hardness/softness (elasticity), glossiness (surface reflectivity), transparency and temperature (heat transfer coefficient), and from this information humans infer
the material (metal, wood, rubber, etc.) of an

object. Surroundness is a sense where one
senses the spatial extent of the area around oneself, the sense of immersion in that space and in
addition the sense of the atmosphere of a place
can also be considered one form of a sense of
surroundness.
Next, sense of movement, sense of causality and sense of simultaneity can be used as
temporal components for presence. The senseof movement is a sense which takes in temporal changes in the environment, which allows
humans to quickly estimate the movement(speed/direction) of objects in the external
word. Causality is a sense that feels that somephenomenon is the cause of another and gives
rise to a result. Simultaneity is a sense that allows one to feel that separate phenomena areoccurring simultaneously. For example, an image of two items touching and a sound of collision would be felt as being simultaneous andhumans feel that these would be the result of
the same physical factors.
And finally, sense of self existence, senseof interactivity and affection can be used as
body-related components for presence. Evenif a person accurately takes in the environment spatially and temporally, this does notmean that they necessarily sense themselves.
When an individual sense their own physicality together with the environment, they sense
presence more strongly. Sense of self existence
is a sense where a person senses the condition (position, direction, movement) of their
entire body or part of their body (arms, legs,head, etc.). Interactivity is a sense of interaction where one feels they can obtain a specified
reaction by acting on objects or other people
within the environment in a given manner. Affection is a pleasurable or unpleasurable sensation felt by the body in response to a subject,and people sense presence more strongly whenexperiencing these types of emotion.
Humans integrate various sensory components described above to sense presence. Moreover, these sensory components are not necessarily always independent of each other, but
reciprocally related and matched, and it can beassumed when presence sensory components

160 Journal of the National Institute of Information and Communications Technology Vol.57 Nos.1/2 2010

Fig.1 Presence indexes (components and factors)

are provided in a complex manner synergistic
effects can be obtained. On the other hand, if
several of these sensory components are missing, or inconsistent with each other, this canalso sometimes lead to the creation of a uniquetype of presence. For example, if a person observes or experiences a virtual stereoscopic image, which has no tactile response even if
touched with the hand, this leads to the individual feeling surprise. This can be thought ofas providing a type of presence to people which
is a virtual duality of “there is something therethat shouldn’t be”.
Next, let’s consider where presence comesfrom and the factors involved. It is necessary toconsider external factors and internal factors asfactors that cause presence sensory components (Fig. 1). Presence resulting from external
factors is presence which is based on physicalinformation from the external world which isdetected by visual, auditory, somatosensory(haptic), olfactory, gustatory and other sensory
organs. On the other hand, presence resulting
from internal factors is presence that is generated in the brain based on memories of senses
accumulated in the brain through past experiences and learning. For example, if an individual hears the sound of a wave, they may see thescenery of the ocean, however this is a sensory

image recalled from past experiences. In addition, people can also recall vast amounts of images from text information as when an individual reads stories or poetry.
It can be thought that not only do people
analyze information from the external worldand store it in their brains for recreation in thismanner, but that they can also use information
accumulated from learning and experiences tointerpret and supplement information from theexternal world to create that sense of existence.
This shows just how important it is to refer to
how to increase presence by drawing on information stored in the brainwhen constructing an
ultra-realistic communication system.
2.2 Presence evaluation methods
How can accurately measuring and evaluating presence, which is composed of a variety of
sensory components, be accomplished? There
are five methods for measuring presence as felt
by people, 1) subjective evaluation (impression
rating), 2) psychophysical evaluation, 3) brain
activity imaging, 4) biosignal measurement and5) behavioral measurement. The characteristics
and merits/demerits of each of these evaluation
methods are shown in Table 1.
Subjective evaluation is normally a method
whereby subjects self assess their impressions

ANDO Hiroshi et al.

161

Table 1 Presence measurement and evaluation methods

in response to presented words (adjective pairs,etc.). Impression rating can be easily carriedout using only questionnaires and no special
measuring equipment. However, it is not necessarily easy for one to accurately introspect(consciously make) impressions that they areperceiving. Therefore, it is necessary for subjects to be strict in their selection of words inpreliminary tests, etc. and data must be statistically analyzed to obtain the highest reliability
possible. In addition, when carrying out precise
sensory evaluation, practice and training is also
important to ensure that stable reactions are
displayed when subjects are faced with specificstimuli.
In contrast, psychophysical evaluation is amethod whereby people’s response characteristics to physical stimuli are quantitatively measured. For example, “point of subjective equality”, where a subject displays the same feelings
under differing presented conditions and “justnoticeable differences” where a subject begins
to display different feelings can be found asphysical numerical values through carrying out
psychophysical experiments. When using psychophysical evaluation, it is also possible todeepen subconscious processes, and obtain
quantitative and highly reliable results by link-ing with the physical dimension. However, it isdifficult to manage this physical link with higher order cognitive senses through the psychophysics method.
On the other hand, a method based on thebrain activity imaging method can also be considered as a method where presence is directly
measured. In recent years non-invasive brain
activity measurement technologies have developed quickly. For non-invasive brain activity
imaging there are fMRI (functional magnetic
resonance imaging) and NIRS (near-infrared
spectroscopy), which handle changes in the
blood flow (metabolism) of the brain, and EEG(electroencephalogram) and MEG (magnetoencephalogram) which handle neural activity
as magnetic changes. In addition, biosignal(heartbeat, galvanic skin response, breathing,pupil diameter, etc.) and behavioral (eye movement, accommodation, body sway, bodily action, etc.) measurement methods can also be
useful in order to handle unconscious changes
that occur in a person’s body.
As summarized in Table 1, each of the presence evaluation methods has their own characteristic advantages and limitations, and there isno perfect measurement method at present. As
such, it is necessary to select these methods as

162 Journal of the National Institute of Information and Communications Technology Vol.57 Nos.1/2 2010

appropriate to the evaluation targets and objectives, and integrate multiple methods in orderto improve measurement and evaluation reliability.

3 Large-field 3D image brain
activity evaluation technologies
There is a desire for the development ofevaluation technologies that objectively andquantitatively evaluate what effect 3D images(binocular disparity images) have on humanbeings. The authors are developing technologies for evaluating the effect of large-field 3D
images on humans using fMRI brain activity
imaging. In general, if images are presented ina wide visual field, presence greatly improves,and senses of surroundness and immersion are
invoked. On the other hand, watching images
with very sharp movements on a large-field canincrease a feeling of discomfort called visuallyinduced-motion-sickness.
In order to evaluate the positive and negative effects created by these large-field 3D images, a device for presenting large-field 3D images to subjects is necessary. However, on MRI
equipment up until now, it was only possible topresent images of a narrow horizontal visual
field of about 20 degrees. This made the areaaround the head of the subject extremely tightand this is because there was not yet any tech-

nology for generating large-field 3D images.
Generally, watching high definition (HDTV)
images at a standard distance (3 times theheight of the display) provides a horizontal visual field of view of 33 degrees. As such, it was
not possible to recreate the viewing environment of a standard television inside of an MRI
machine up until now. Further, for giant screen
theaters (IMAX®) where viewers can enjoy
very powerful images, the horizontal viewing
angle from the center seats is approximately 70
degrees. The recommended horizontal viewing
angle for the images for the super high vision
currently being developed by the NHK Science& Technical Research Laboratories is 100 degrees for the most effective viewing. Presenting this type of large-field image to a subject
and carrying out fMRI brain activity imaging
has been impossible up until now.
In response, the authors of this paper havedeveloped the technology of a super wide angle
ocular lens which will allow viewing of images
inside of an MRI machine at a horizontal visual
field of 100 degrees (Fig. 2)[2]. In general, MRI
generates a high magnetic field, so LCD panelsand other magnetic bodies cannot be taken inside an MRI unit. However, even if the areaaround the head is a high magnetic field (3 Tesla which is 70,000 times the standard geomagnetism rate), the magnetic field drops drastically in corners of the room (to about 0.0005

Fig.2 Large-visual-field 3D image evaluation equipment

ANDO Hiroshi et al.

163

Fig.3 Brain regions activated by large-visual-field 3D images

Tesla). Here we places 2 projectors, one for theleft eye and one for the right, in the corners of
the room in magnetically shielded boxes, andused a super telephoto lens to project the images onto a small screen (13cm width) inside
the MRI machine. Having the subject view thisscreen through the super wide angle ocular lensprovides them with a horizontal visual field of
100 degrees. The lens materials were carefully
selected so as not to cause any noise in thebrain activity images. In addition, this equipment allows for obtaining the eye movement of
the subject from both eyes. Measurement ofeye movement was accomplished through reflection of several small mirrors by 2 CCD
cameras installed behind the subject’s head andanalyzing this movement in real-time.
The results of the fMRI brain activity imaging on the effects of the 100 degree horizontal
viewing angle on the brain of using this largefield 3D image evaluation equipment are shownin Fig. 3. For this experiment, a video of multiple randomly placed dots moving back andforth was used for visual stimulation. The leftof Fig. 3 shows the regions of the brain activated by the 3D viewing conditions vs. the conditions with no image shown. As the horizontal
field of view of the image was increased from25 degrees, to 50 and 100, the visual cortex(each region of V1, V2, V3 and V3A) in theoccipital region of the brain was seen to be activated in a larger range. This is because thevisual cortex neurons have a receptive field
relative to the position of the retinas. The rightof Fig. 3 shows comparison between the activated brain regions when 2D images and 3D
images were shown (with a horizontal visual
field of 100 degrees for both). These results
show that when viewing 3D images, the region
from the occipital to parietal regions of thebrain was strongly activated. The results of thisexperiment show that the brain regions activated when viewing a large-field 3D can be accurately measured using fMRI brain activity imaging with the equipment we have developed.
Hereafter, we plan to clarify the details of
the positive (the sense of 3D, immersion, etc.)
and negative (discomfort, visually-inducedmotion-sickness, etc.) effects on the brain bylarge-field 3D images through researching thecorresponding relationships between brain activity and psychological reactions when watching large-field 3D images.

4 Quantitative and objective
evaluation of presence
We have explored the perceptual and cognitive mechanisms created by presence fromvarious sensory information using quantitativeand objective evaluation methods; for example,
psychophysical experiments[3] and brain activity imaging[4] on perceived surface quality(glossiness), neural correlates of imagery in-

164 Journal of the National Institute of Information and Communications Technology Vol.57 Nos.1/2 2010

（A）

（B）

（C）

（D）

Fig.4 Psychophysical experiments on glossiness perception: (a) Experimental apparatus, (B) Stimuli, (C)Experiment investigating dynamic effects (Experiment 1), (D) Experiment investigating 3D effects(Experiment 2) (Extracted from the reference [3])duced by ambient sound[5], brain activity imaging related to dynamic stereophonic sound[6],brain activity imaging related to naturalness
and unnaturalness of images[7], psychophysical experiments[8] and brain activity imaging[9]on visual-haptic integration, and psychophysical research on the effects of contact sound andsmell on haptics[10][11].
An overview of psychophysical experiments on perception of glossiness is given below as an example of presence evaluation.
In general, when a 3D image is viewed,
people perceive not only the depth and shape as
more realistic, but also the surface quality ofobjects. This can be because the images supplied to the right and left eyes are slightly different, and the perception of the glittering androughness of the surface are improved. In addition the perceived surface quality of a 3D image using a multi-view method which allows
viewing of 3D images with the naked eye by
presenting large numbers of parallax images isdifferent from a two-view method where specialized 3D glasses are used. This is because
for the two-view method the viewer receives
image information from only one viewpoint

even if looking at the 3D image from different
viewpoints and positions, whereas for themulti-view method the image can be viewed
from different viewpoints by moving one’s
head (for example, an image looked at from theleft of an object and an image looked at fromthe right). Actually when moving one’s head
and looking at an object’s surface, the reflected
light from the surface changes, and one may
feel that the glossiness has increased.
The authors quantitatively measured in thepsychophysical experiments to what degree theglossiness of an object’s surface changed when
an image of the object was dynamicallychanged and when different information wasprovided to both eyes[3]. In this experiment a
two-view 3D display (time-division system using shutter glasses) was used, and a multi-view
condition was imitated by having the subject
move their heads with tracking their head
movements in real-time (Fig. 4 (A)). As shownin Fig. 4 (B), a CG image of a textured flat object was used as visual stimuli. In experiment
1, the difference in glossiness for monocularvision was compared for dynamic conditions
where the changes to the viewed image were

ANDO Hiroshi et al.

165

Fig.5 Results of psychophysical experiment on glossiness perception (Extracted from the reference [3])simulated when viewpoint changes and static
conditions where the object image doesn’tchange (Fig. 4 (C)). In experiment 2, glossiness
differences were compared under binoculardisparity conditions (stereo conditions) andconditions where the same image was shown toboth eyes (flat surface conditions). In experiment 3, compared to the monocular view conditions in experiment 1,dynamic and static conditions in the case of binocular view werecompared. (Results of the experiment 4 andother controlled conditions are omitted here.)
The glossiness perceived by the subjects
was measured using two types of psychophysical methods, the magnitude estimation and adjustment methods. In the magnitude estimation
method, we had subjects evaluate glossiness ofthe test condition with glossiness of the reference condition set at 10. In the adjustment
method, we had the subjects adjust the glossiness parameter (the specular reflection rate) sothat perceived glossiness of the reference condition and test condition were the same.
Results of the psychophysical experiments
are shown in Fig. 5. The top row shows the results of the magnitude estimation method (therating task) and the bottom row shows the results of the adjustment method (the matchingtask) . The results of experiment 1 show thatcompared to the conditions of looking at a dynamic image, in the conditions of looking at a
static image the evaluation of glossiness was
lower and the same glossiness could not be obtained without raising the specular reflection
rate. The results of experiment 2 show thatcompared to the conditions of looking at an image with binocular disparity, in the conditionsof looking at a flat image the rating value ofglossiness was lower and the same perceptionof glossiness could not be obtained without
raising the specular reflection rate. The resultsof experiment 3 show that even with the added
binocular disparity, compared to the conditionsof looking at a dynamic image, in the conditions for looking at a static image the rating
value of glossiness was lower and the same
perception of glossiness could not be obtained
without raising the specular reflection rate.
To summarize these results, the movementof an object increases perception of glossiness(Experiment 1), the binocular disparity increases perception of glossiness (Experiment 2), andthat if a object surface with binocular disparity
moves, perception of glossiness increases evenmore (Experiment 3). This means that the perceived glossiness of an object surface on a twoview 3D display increases from the perceived
glossiness on a flat display and that the per-

166 Journal of the National Institute of Information and Communications Technology Vol.57 Nos.1/2 2010

ceived glossiness with changing the viewpoint
on a multi-view increases even more. As shown
above, the difference between perception of
surface quality under different presentation
methods can be quantitatively measured
through psychophysical experiments.

generating, associating and predicting sensory
information based on knowledge and memory.
We plan to continue to pursue research fromthe above three viewpoints hereafter.

5 Future issues

This paper first described the evaluationindexes, i.e., the components and factors, andthe evaluation methods of presence. Second,
as a presence-evaluation technique, we described the development of evaluation technology which can measure human brain activities
when large-visual field 3D images are shown.
We also describe the quantitative evaluation
methods of human sense of surface quality, especially glossiness, and the results of experiments. Hereafter, we will continuously try to
clarify human perceptual and cognitive mechanisms of presence, and both develop more objective and quantitative evaluation technologies as well as apply the obtained resultsto thedevelopment of ultra-realistic communication
systems.

The following three items can be put forth
as future issues on perceptual and cognitive
mechanisms of presence and its evaluation
techniques. First, we need to further clarify themeaning of brain activities by measuring brain
activity and the psychophysical and behavioral
responses when presented with given sensory
information and analyzing the corresponding
relationships. This process will lead to the establishment of methods that allow for evaluation of improved presence and increased discomfort through measurement of brain activity.
Second, we need to find conditions for improving perception of presence through quantitativeevaluation of presence using psychophysicalexperiments. It can be thought that these typesof quantified technology requirement will be
extremely significant for the future development of ultra-realistic communication systems.
Finally, we need to seek to exceed the presence
provided by physical information through utilizing human perceptual and cognitive mechanisms of presence. We believe this can beachieved by effectively using functions for

6 Conclusions

Acknowledgements
Development of the large-visual-field 3D
image evaluation system was carried out aspart of the “Research and Development on
Glasses Free 3D Image Technologies” commissioned by the Ministry of Internal Affairs
and Communications in FY2009.

References
1 H. Ando, “The components and factors of presence(in Japanese),” Virtual Reality 6.2.2. , edited by the
Virtual Reality Society of Japan, Kogyo Chosakai Publishing Co., Ltd., 2010.

2 “Development of the evaluation system for 3D image using brain activity imaging: towards the quantitativeevaluation for the presence and safety of wide visual field 3D image (in Japanese),” Press Release on theofficial site of NICT, Nov. 1, 2010. http://www2.nict.go.jp/pub/whatsnew/press/h22/101101-02/101101-02.
html

3 Y. Sakano and H. Ando, “Effects of head motion and stereo viewing on perceived glossiness,” Journal of
Vision, 10 (9): 15, 1–14, 2010.

4 A. Wada, Y. Sakano, and H. Ando, “Identifying human brain areas activated by attention toglossiness,”
17th Annual Cognitive Neuroscience Society Meeting (CNS 2010),Ⅰ106, Apr. 2010.

ANDO Hiroshi et al.

167

5 A. Callan and H. Ando, “Neural correlates of imagery induced by the ambient sound,” Proceedings of
10th Annual Int. Workshop on Presence, pp. 73–77, Nov. 2007.

6 A. Callan and H. Ando, “Neural correlates of realistic auditory motion perception,” ExcitingBiologies 2008:
Biology of Cognition, No. 8, Oct. 2008.

7 N. E. Nawa and H. Ando, “Distinguishing activity elicited by intrinsic salience in the frontoparietal network
using fMRI,” Cognitive Neuroscience Society, p. 231, Apr. 2010.

8 J. Liu and H. Ando, “Effects of visible contact point on visual-haptic integration in 3D shape perception,”
33rd European Conference on Visual Perception (ECVP2010), Aug. 2010.

9 Y. Nishino and H. Ando, “Crossmodal interactions between visual and haptic information in 3D object
learning and top-down processing: an fMRI study,” 30th European Conference on Visual Perception(ECVP2007), Perception, Vol. 36, supplement, pp. 209–210, Aug. 2007.

10 J. Liu and H. Ando, “Contact sound can influence haptic stiffness perception,” International Multisensory
Research Forum 2010 (IMRF 2010), Jun. 2010.

11 Y. Nishino, D. W. Kim, J. Liu, and H. Ando, “Interaction between olfactory and somatosensory perception:
Do odors influence stiffness and roughness perception? (in Japanese),” JASTS 2010(the 44th conferenceof The Japanese Association for the Study of Taste and Smell), Sep. 2010.
(Accepted Sept. 9, 2010)

ANDO Hiroshi, Ph.D.
Group Leader, Multimodal
Communication Group, Universal
Media Research Center
Brain and Cognitive Sciences,
Multisensory Cognition Mechanisms,
Multisensory Interfaces

Akiko Callan
Expert Researcher, Multimodal
Communication Group, Universal
Media Research Center
Cognitive Neuroscience, Auditory
Cogntive Science

Norberto Eiji Nawa, Ph.D.
Expert Researcher, Multimodal
Communication Group, Universal
Media Research Center
Cognitive Neuroscience,
Psychophysiology

NISHINO Yurie
Expert Researcher, Multimodal
Communication Group, Universal
Media Research Center
Cognitive Psychology, Cognitive
Neuroscience

Juan Liu, Ph.D
Expert Researcher, Multimodal
Communication Group, Universal
Media Research Center
Multisensory Cognition Mechanisms,
Multisensory Interfaces

WADA Atsushi, Ph.D.
Expert Researcher, Multimodal
Communication Group, Universal
Media Research Center
Cognitive Neuroscience, Functional
Neuroimaging, Machine Learning

SAKANO Yuichi, Ph.D.
Expert Researcher, Multimodal
Communication Group, Universal
Media Research Center
Visual Psychophysics, Cognitive
Neuroscience

168 Journal of the National Institute of Information and Communications Technology Vol.57 Nos.1/2 2010