Research and Development for Assessment of Cognitive Function
with Immersive Virtual Reality System
Shingo Katoh 1, Akinori Nagano 1, and Zhiwei Luo 1
11

Department of Computational Science, Graduate School of System Informatics, Kobe University, Japan

Background

Cognitive function is very important for our social life.
Cognitive dysfunction often disturbs healthy life of patients. In
addition, it is difficult to predict the problems patients face intheir actual daily life. Therefore, assessment of cognitive
function is required in this ageing society. Such assessments aimto evaluate negative influences caused by a variety of cognitive
dysfunctions. Conventionally, such assessments have been
performed primarily based on neuropsychological tests. The
effectiveness of these assessments has been proved to date,nonetheless, it is also known that these tests cannot directlypredict the problems patients face in actual daily life. This isbecause the tests are performed on the desk. Also, the executionof the tests often burden the patients.
Recently, it has been proposed that virtual reality (VR)technology could be utilized for the assessment of cognitivefunction, as it is possible to perform the assessment under more
realistic environment than conventional paper-and-pencil based
assessments. There are some preceding studies that reported
assessments of cognitive function using VR technology. For
example, a virtual cooking system [1], a virtual shopping system[2] and a virtual system of supermarket [3] have been reported.
Since 2008, a 2-D virtual shopping test (VST) system has beendeveloped at Kobe University, Japan [4] [5] [6]. In the VST, theuser executes shopping and some other tasks. This system uses
the virtual shopping mall created with computer graphics. VST
adopts a touch panel screen as its user-friendly interface (Fig. 1).
This user interface helps the user (mostly elderly people) tooperate the system easily (Fig. 2). It has been revealed that thissystem is effective for assessment of higher cognitive function[7].
In this study, we developed a prototype of 3-D VST, which
is a development of the past 2-D system. We selected immersive
display system named “CAVE” as equipment for presenting 3-D
images. There are two reasons why we selected a 3-Denvironment. First, reality is one of the most important elementsof the assessment of cognitive functions with VR. We assume
that the more realistic VST becomes, the better test we canexecute. Second, we try to test if CAVE is useful for theassessment of cognitive function. In order to utilize CAVE as adevice presenting VR, we need a new user interface instead of atouch screen. User interfaces used in such assessment systems
should be intuitive and easy to use. Therefore we put emphasison the user interface in this development.

Fig. 1. 2-D VST with a touch screen

Fig. 2. Flowchart of VST
2Method

Immersive Display System “CAVE”
CAVE is one of the most immersive display systems which
was developed at Illinois University, USA in 1991. Standard
CAVE is composed of a cube-shaped room whose walls are 3-Dscreens. The number of screens depends on each CAVE system.
We used a four-screen CAVE system for this research (Fig. 3).
Four means front, left and right walls and floor. User stands inthe room composed of 3-D screens wearing stereoscopic glasses
and holding a controller called “wand” (Fig. 4). Wand has somebuttons and a joystick. Tracking cameras set around the ceiling
capture tracking markers attached on glasses and wand. API for
CAVE named “CAVELib” automatically computes theirposition and direction in real time. Using the data, stereoscopic
image for the user’s eyes is projected on the wall of screens.
2.1

Fig. 3. Four-screen CAVE

3Fig. 4. The stereoscopic glasses and the wand2.2

Outline of the Prototype of VST
On 3-D VST, users perform shopping in virtual shopping
mall projected on screens (Fig. 5). Because the scene of theshopping mall is projected on the walls surrounding the user,
he/she has a feeling as if being in a real shopping mall. There
are some differences from the 2-D VST except for the form ofpresentation. While insides of the shop and products are
represented simply by pictures in the 2-D VST, users can enterthe virtual shops in the 3-D VST. In the virtual shops, users cangrab products as if they do in real shopping. Finishing tasks in ashop, users go out from the exit.

Results

We had two male subjects in their twenties experience thissystem and asked their opinions. Both of them could utilize theuser interface without problems. However, according to theiropinions about the use of interface, below issues were found.
It takes time to get used to the CAVE system and theuser interface.
Touching the balls to navigate might not be intuitive.
Sometimes unexpected touch on the navigating balloccurs.
From these comments, when aged persons or patients ofcognitive impairment use the system, the possibility is high thatthey cannot operate the system as they want, and this problem
could hinder a proper assessment.
4Conclusion

We developed the prototype of 3-D VST. This system does
not yet contain the following functions which are in the 2-DVST.
Output a log of tests
Play sounds
Set the difficulty of task
3-D VST would be effective because it includes shopping
function of core system of VST. In addition, 3-D VST has somestrength such as the details of inside of the shops and carrying ofthe products. So, the reality is considered to be largely improved.
However, some users point out the difficulty of userinterface. Therefore, it is important to aim further developmentof useful user interface. We also have to add more functions andevaluate the effectiveness for practical use.
Acknowledgement

Fig. 5. A scene of 3-D VST
2.3

User Interface
Expanding 2-D system into 3-D system needs changes of its
user interface. For patients, it is difficult to operate the systemwith buttons and a joystick of the wand. Therefore, wedeveloped a new user interface using tracking systems of CAVE.
Users can control their action in the virtual shopping mall withthe wand’s position. On the floor, a red circle is shown. While
standing on the circle, yellow balls appear around the user.
These balls are navigating balls. The user touches navigatingballs by the wand to move in the VR space. Unique actions are
assigned to each ball. For example, the ball in front of usercorresponds to going straight on and the ball on the right of usercorresponds to turning right. When the user wants to pick up aproduct, he can do it that by directly touching the product withthe wand. At the entrance and exit of shops, bright red balls areshown. The user can go into and out of shops by touching theseballs. The data of the position of user’s head is obtained fromthe tracking markers attached on the glasses. These data areused for distinction of standing in the red circle or not. The same
type of data of the wand is used for distinction of touching
objects. This interface is considered to enable more intuitive
operation which helps users (Fig. 6).

Fig. 6. The red circle and the navigating balls

We would like to acknowledge Dr. Kei ko Seki, Dr. Rumi
Tanemura, Dr. Hiromi Sakai, Ms. Sayaka Okahashi, Ms.
Maki Kojima and Ms. Hiromi Endo for their help in thisstudy.

References
[1]

[2]

[3]

[4]

[5]

[6]

[7]

L. Zhang, “A Virtual Reality Environment for Evaluation of aDaily Living Skill in Brain Injury Rehabilitation: Reliability andValidity”, Arch Phys Med Rehabil, Vol. 84, pp1118-1124, (2003)
N. Titov, R. G. Knight, “A Computer-Based Procedure forAssessing Functional Cognitive Skills in Patients with
Neurological Injuries: The Virtual Street” , Brain Injury, Vol. 19,
pp315-322, (2005)
Y. J. Kang, J. Ku, K. Han, S.I. Kim, T. W. Yu, J. H. Lee, C.I. Park,“Development and Clinical Trial of Virtual Reality-Based
Cognitive Assessment in People with Stroke: Preliminary Study”,
Cyber Psychology & Behavior, Vol. 11, pp329-339, (2008)
Y. Yamanaka, “Development and Application of Virtual Reality
System for Realistic Assessment of Cognitive Functions”, A
Master’s Thesis of Department of Computer and System
Engineering, Kobe University, Japan, (2008)
T. Oshiumi, “Realistic Assessment of Cognitive Functions”, A
Graduate Thesis of Department of Computer and System
Engineering, Kobe University, Japan, (2008)
H. Dong, T. Oshiumi, A. Nagano, Z. Luo, “Development of a 3D
Interactive Virtual Market System with Adaptive TreadmillControl”, Intelligent Robots and Systems, pp5238-5244, (2010)
S. Okahashi, K. Seki, A. Nagano, R. Tanemura, M. Kotsuta,
Z. W. Luo, “Development of a Virtual Shopping Test for theEvaluation of Cognitive Function in Brain-Damaged Patiens”,
Higher Brain Function Research, Vol. 32(2), (2012)