# Exploration in 3D Virtual Worlds with Haptic-Audio Support for Nonvisual Spatial Recognition

Ying Ying Huang

<figure><img src="image_1.png" /></figure>

## To cite this version:

Ying Ying Huang. Exploration in 3D Virtual Worlds with Haptic-Audio Support for Nonvisual Spatial Recognition. Second IFIP TC 13 Symposium on Human-Computer Interaction (HCIS)/ Held as Part of World Computer Congress (WCC), Sep 2010, Brisbane, Australia. pp.269-272, 10.1007/978-3-642-15231-3_27. hal-01061034

HAL Id: hal-01061034

https://inria.hal.science/hal-01061034

Submitted on 5 Sep 2014

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>
# Exploration in 3D Virtual Worlds with Haptic-Audio Support for Nonvisual Spatial Recognition

Ying ying Huang

HCI Dept. School of Computer Science and Communication, Royal Institute of Technology SE-10044, Stockholm, Sweden
yingying@csc.kth.se

**Abstract.** Mobility indoors or outdoors is a problem for visually impaired people because of lacking the visual channel. Appropriate learning tools about spatial information could be used as a preparation for navigation before going to the real place. Current accessibility technologies, such as screen readers, do not well convey spatial layout or structure. In this paper, an experimental study on accessing non-visual spatial information and on support for efficient navigation and orientation with haptic and audio cues is presented. A 3D virtual simulation prototype of a real world environment was created for this purpose. Different navigation tools were designed in the prototype with haptic and audio cues. The main results of the study reported are: (a) the development of a virtual 3D environment helping visually impaired users to learn about real space where they are required to navigate (e.g. in schools, work places, public buildings), (b) whether the spatial information for establishing the mental mapping of the space could be acquired by using compensatory sensorial channels (e.g., touch and hearing), as an alternative to the visual channel; (c) how haptic and audio cues can enable the efficient navigation, mobility and orientation in such 3D virtual environments. Results from qualitative analysis regarding learning process and actual performance in the 3D virtual world are presented.

**Keywords:** Haptic; Audio; Virtual Reality; Navigation; Cognitive Mapping;

## INTRODUCTION

Spatial information presented visually is not easily accessible for visually impaired users. Current accessibility technologies, such as screen readers, do not well convey spatial layout or structure. Virtual reality has been a popular paradigm in simulation based training, in the gaming and entertainment industries. It has also been used for rehabilitation and learning environments for people with sensory, physical, mental, and learning disabilities [9][10]. Particularly haptic technology offers alternative means of presenting spatial information to visually disabled users. Research on the
implementation of haptic technologies within Virtual Environments (VEs) has
reported on potential for supporting development of cognitive models of navigation
and spatial knowledge with sighted people [1][2][3][4] and blind people [5][6].
However, users normally interact with the virtual world using a single "pen" probe or
stylus. It is not very easy for them to interact with 3D models in the absence of visual
cues because of the point-contact nature of the haptic device. There are two main
problems as mentioned in the study by Walker et al. [8] One is that it can be hard to
locate the data initially. The second is that it will be difficult to maintain contact with
it after locating an object. As Sjostrom [3] reports: "For a blind person, locating an
object with a point probe can be as hard as finding a needle in a haystack". Colwell et
al. [7] describe users as becoming "lost in haptic space" when they are unfamiliar
with this type of interaction, as they frequently lose contact with an object they are
trying to explore.

A 3D haptic and audio virtual simulation environment of Kulturhuset (House of Culture) in Stockholm was developed for this experimental study. Such touch-and-hearing tools could potentially be used by visually impaired users to learn about physical spaces that they are required to navigate (e.g., school, work place, public buildings). The prototype allows users to explore the virtual Kulturhuset building to find different locations. The Research questions of this study were:

(a) the development of a virtual 3D environment helping visually impaired users to learn about real space where they are required to navigate (e.g. in schools, work places, public buildings), (b) whether the spatial information for establishing the mental mapping of the space could be acquired by using compensatory sensorial channels (e.g., touch and hearing), as an alternative to the visual channel; (c) how haptic and audio cues can enable the efficient navigation, mobility and orientation in such 3D virtual environments. Results from qualitative analysis regarding learning process and actual performance in the 3D virtual world are presented.

THE STUDY

Five visually impaired participants took part in the study, one female 23 years old, males 28, 35, 40 and 52 years old. The participants tried the Kulturhuset prototype (Fig 1, Fig 2) individually. Users were selected by the following criteria: computer users in daily life, interest in computer games and, interest in technologies that assist in acquiring information through other senses, such as touch and audio. Participants had to identify both the specific objects and the structure (spatial layout) of the virtual environment under three conditions: audio; haptic; audio/haptic. Participants completed two tasks (objects identification) under each condition in a random order. We keep the visual channel available for experimenter use. In all conditions there are magnetic forces designed for each object in order to attract the user when they move closer than 5 meters.
<figure><img src="image_3.png" /><figcaption>Fig. 1. A virtual version of Kulturhuset.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Fig. 2 Escalator.</figcaption></figure>

In the experiment, a Dell Precision Laptop with two dual core processors is used. One PHANTOM Omni, the haptic device, is connected serially to Laptop. For developing the application H3DAPI 2.0, X3D, Python as well as Visual Studio .NET 2003, are employed. H3DAPI can be used both as an API to implement X3D applications, and as a loader of applications built in X3D and Python. A video camera is used for recording purpose and Camstudio for screen capturing during the experiments.

<figure><img src="image_5.png" /><figcaption>Fig. 3. User interacting with the haptic virtual environment using a PHANTOM Omni force feedback device (source: SensAble Technologies Inc)</figcaption></figure>

A video analysis was performed after the test sessions in this study. The interview was semi-structured and lasted for about 20 minutes. Finally, the results from all users were compared in order to obtain general findings and interesting patterns as results.

## RESULT AND DISCUSSION

This work presents an effort to explore the impacts of haptic and audio cues on non-visual spatial recognition for visually impaired users. Four issues were analyzed based on the study, they are: accessibility and Usability; Object Recognition; Spatial Recognition and the impacts of Haptic and Audio Aids on Navigation, Mobility and Orientation.

The most important conclusions that can be drawn for the design of such environments are listed below:

* It is possible to access the virtual world of a real world by visually impaired users with haptic and audio cues;

* Both objects and spatial structure recognition will be faster, more accurate, and easier when both types of cues are available.

* Haptic cues alone will aid structure recognition, but audio cues alone will not.
* Once a node has been located, audio cues will provide more efficient means (speech or non-speech) of identifying it than haptic cues. Therefore, participants will rate haptic cues as more useful for identifying structure and audio cues as more useful for identifying objects.

* Integration of haptic and audio feedback provides more efficient navigation, mobility and orientation that either of them alone.

* It is better to have larger surfaces in all objects.

* The design of spatial architecture, fixed reference points like walls, floor, fixed objects and sensitive texture with haptic feedback are important to support orientation.

We hope that the study could be an investigation on the accessibility and usability for blind and visually impaired people to use haptic and audio virtual reality software for journey planning before they move into the real world. It could also serve as an input for visually impaired users to access online 3D virtual reality environments like Second Life with haptic and audio cues.

REFERENCE

1. HURMUZLU, Y., EPHANOV, A. STOIANOVICI, D. Effect of a Pneumatically Driven Haptic Interface on the Perceptual Capabilities of Human Operators. Presence: Teleoperators and Virtual Environments. 7(3), 290-307. (1998)

2. KUBER, R., YU, W., MCALLISTER, G. Towards Developing Assistive Haptic Feedback for Visually Impaired Internet Users, In Proceedings of the 2007 Conference on Human Factors in Computing Systems, San Jose, California, USA, 1525 – 1534. (2007)

3. SJOSTROM, C. Designing Haptic Computer Interfaces for Blind People. Proc. ISSPA, 1, 68-71. (2001)

4. BERGAMASCO, M. AVIZZANO, C. DI PETRI, G., BARBAGLI, F. AND FRISOLI, A. The museum of pure form: system architecture, In Proceedings of 10th IEEE International Workshop on Robot and Human Interactive Communication, Piscataway, NY: IEEE Press, 112-117. (2001)

5. BERGAMASCO, M. AND PRISCO, G. Design of an anthropomorphic haptic interface for the human arm, In Y. Shirai and S. Hircse (Eds.), Robotic Research, the Eight International Symposia, 278-289. London: Springer. (1998)

6. FRISOLI, A., SIMONCINI, F. AND BERGAMASCO, M. Mechanical design of a haptic interface for the hand, In Proceedings of 2002 ASME DETC 27th Biennial Mechanisms and Robotics Conference, Montreal, Canada: ASME Press, 25-32. (2002)

7. Colwell, C., Petrie, H., Kornbrot, D. Haptic virtual reality for blind computer users. Paper presented at the Assets '98 Conference, Los Angeles, CA. Available in: /http://phoenix.herts.ac.uk/sdru/pubs/VE/colwell.htmlS. (1998)

8. B. N. Walker, J. Lindsay, 2006. Navigation performance with a virtual auditory display: Effects of beacon sound, capture radius, and practice. In Human Factors, volume 48(2), pages 265-278.

9. Schultheis, M.T., Rizzo, A.A., 2001. The application of virtual reality technology for rehabilitation. Rehabilitation Psychology 46 (3), 296 – 311.

10. Standen, P.J., Brown, D.J., Cromby, J.J., 2001. The effective use of virtual environments in the education and rehabilitation of students with intellectual disabilities. British Journal of Education Technology 32 (3), 289 – 299.