# Cooking in the Dark: Exploring Spatial Audio as MR Assistive Technology for the Visually Impaired

Renan Guarese, Franklin Bastidas, João Becker, Mariane Giambastiani, Yhonatan Iquiapaza, Lennon Macedo, Luciana Nedel, Anderson Maciel, Fabio Zambetta, Ron Van Schyndel

<figure><img src="image_1.png" /></figure>

## To cite this version:

Renan Guarese, Franklin Bastidas, João Becker, Mariane Giambastiani, Yhonatan Iquiapaza, et al. "Cooking in the Dark: Exploring Spatial Audio as MR Assistive Technology for the Visually Impaired." 18th IFIP Conference on Human-Computer Interaction (INTERACT), Aug 2021, Bari, Italy. pp.318-322, 10.1007/978-3-030-85607-6_29. hal-04291191

HAL Id: hal-04291191

https://inria.hal.science/hal-04291191v1

Submitted on 17 Nov 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
Cooking in the dark: exploring spatial audio as
MR assistive technology for the visually
impaired

Renan Guarese<fnref n="1" />[0000-0003-1206-5701], Franklin Bastidas<fnref n="2" />[0000-0002-2241-5181],
João Becker<fnref n="2" />, Mariane Giambastiani<fnref n="2" />, Yhonatan Iquiapaza<fnref n="2" />, Lennon
Macedo<fnref n="2" />[0000-0002-3686-0550], Luciana Nedel<fnref n="2" />[0000-0002-2390-1392], Anderson
Maciel<fnref n="2" />[0000-0002-0780-6555], Fabio Zambetta<fnref n="1" />[0000-0003-4133-7913], and Ron
van Schyndel<fnref n="1" />[0000-0002-5837-1539]

<figure><img src="image_4.png" /><figcaption>**Fig. 1.** Left: blindfolded user putting on the HMD. Right: user interacting with the game via voice command and receiving spatialized sound feedback.</figcaption></figure>

**Abstract.** In the context of raising awareness to assistive technologies, we propose a gaming experience that allows users to embody having a visual impairment. By occluding the user’s vision and providing spatialized audio and passive haptic feedback, allied with a speech recognition digital assistant, our goal is to present a multi-sensory experience that offers the user a sense of embodiment inside a mixed reality blindness simulation. Inside the game environment, the player is required to cook a meal completely in the dark. Being aided solely by their remaining senses and a digital assistant with spatialized audio capabilities.

**Keywords:** Mixed and Augmented Reality, Assistive Technologies, Accessibility, Embodiment, Spatial audio, Passive Haptics

<footnote n="1">¹ Royal Melbourne Institute of Technology, Melbourne VIC, Australia
renan.martins.guarese@student.rmit.edu.au, {fabio.zambetta,
ron.vanschyndel}@rmit.edu.au</footnote>

<footnote n="2">² Federal University of Rio Grande do Sul, Porto Alegre, RS, Brazil
lennon.macedo@ufrgs.br, {fgbcuya, jcbecker, mtgiambastiani, yjiccama,
nedel, amaciel}@inf.ufrgs.br</footnote>
# 1 Introduction

Recently, researchers have claimed that the responsibility of addressing disability should be placed on everyone collectively, including technology designers [11, 1]. In that line of thought, works have focused XR (augmented and virtual reality) applications towards the Blind and Visually Impaired (BVI) [4, 2, 3]. Specifically working with audio, these rely on the notion that current spatialized audio technologies allow for a decent accuracy on estimating a sound source, especially when head movement is being tracked [2]. Considering assistive technologies in particular, domotics [8, 9] and virtual assistants [10, 12] have also shown their potential to help day-to-day activities of people with disabilities.

# 2 Related work

Exploring the spatiality of indoor environments, Iravantchi et al. introduced the concept of Digital Ventriloquism [6]. Everyday objects are given a voice via sound projections in a highly directional pan-tilt ultrasonic array. In a different project particularly focusing on BVI users, Ferrand et al. [2, 3] developed a navigation device to guide users in everyday activities with sounds using binaural techniques, which proved to be viable for walking and even roller-skating. Among their approaches, specifically Head-related transfer functions (HRTF) and Inertial Measurement Unit (IMU) sensors were similarly used in the current project for the synthesis of spatial sound beacons positioned according to everyday objects in the game, guiding players towards them.

# 3 Game scenario and tasks

The game portrays a smart home kitchen environment. The player has a dinner date planned with a guest and the lights go out. In order to complete the game, a meal needs to be prepared and served before the guest arrives, completely in the dark. A digital voice assistant guides the user throughout the experience, reminding them of the recipe, cooking procedures, time constraints and how to find each required ingredient and utensil. The smart home enables sound to be projected from different objects, in a digital ventriloquism manner [6], aiding the user to find the necessary resources without any visual cues. The voice assistant is capable of speech recognition, understanding and replying to player questions related to the tasks.

All user tasks require a sense of micro-navigation by the user, i.e. sensing of the immediate environment for obstacles and hazards [13], which is provided by the smart home via an assistive technology manner. Most tasks are simply based on finding and reallocating objects, such as putting spaghetti into a pot. The difficulty relies on the lack of vision: resources will need to be found solely on sound and tactile guidance. The majority of objects represented in the game are real, such as the food, water and kitchen utensils, improving the player's sense of presence via passive haptics [5].
# 4 Materials and methods

Being supervised by a Visually Impaired researcher in its development, the game is meant to explore the different senses BVI people rely on. Considering the total darkness in the game scenario, a regular blindfold is used to occlude any real or virtual visual aspects, as depicted in Fig. 1-left. Besides that, several tools were used to convey the player with audio and haptic experiences.

**Spatial audio:** As users are required to locate objects solely on sound and passive tactile feedback, a reliable spatial audio system is necessary. This navigation method relies on the natural ability of humans to localize directions of a sound. As to virtually synthesize these binaural sounds, transfer functions that characterize how ears receive sound from different points in space can be used [2]. These are known as HRTFs, and they're based on physical models and measurements of human head, torso, and ear shapes. The human brain responds to these differences to provide perceived direction in sound. As a device that already implements an approximation of these based on the current user's inter-pupillary distance coupled to inside-out head-tracking, the Microsoft HoloLens was chosen as the main platform for the game.

**Passive haptics:** Given that people might not be confident enough to make precise judgements on object locations based solely on sound, players are also expected to reach out their arms in order to understand the environment in front of them. By having real objects and props placed in the game, the user's sense of presence may be elevated [5]. Feeling the textures of, weighing and smelling ingredients are typical actions expected from cooking a meal. Thus, using the passive haptics of real kitchen utensils, wind from a fan, water from a sink and even real food [7] might arguably help the player in embodying the game protagonist.

**Digital assistant:** As a means to interface the smart-home elements of the game towards the virtually blind main character, a digital voice assistant was implemented. According to recent studies, people with disabilities - particularly the BVI - have been widely benefited by the use of commercially available virtual assistants in daily tasks [10]. Even specifically within MR experiences, preliminary results indicated that the presence of a speech-based virtual assistant improved user performance in the execution of MR activities [12]. By taking advantage of the speech input features of the HoloLens, voice commands related to the game tasks were implemented. Allying that to text-to-speech synthesis, a conversational agent named Axela was created, being able to respond to users' demands and inquiries.

# 5 Conclusions and future work

This work demonstrates an innovative way of raising awareness to assistive technologies for the BVI through a game. Along with the passive haptics, narrative,
and voice interface of the virtual assistant, it was possible to create a game in which a protagonist is simulated with a total loss of vision. For further works, we aim to compare the performances of BVI and sighted users within the simulation. We hypothesize that BVI users are likely to have behaviors in place facilitating themselves in the kitchen. By observing these behaviors, we aim to optimize our micro-guidance methods for BVI users, so that these can be used in real-life scenarios, besides gaming environments.

## References

1. Branham, S.M., Kane, S.K.: Collaborative accessibility: How blind and sighted companions co-create accessible home spaces. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. CHI '15 (2015)

2. Ferrand, S., Alouges, F., Aussal, M.: Binaural spatialization methods for indoor navigation. In: Audio Engineering Society Convention 142. (2017)

3. Ferrand, S., Alouges, F., Aussal, M.: An augmented reality audio device helping blind people navigation. In: Computers Helping People with Special Needs (2018)

4. Grayson, M., Thieme, A., Marques, R., Massiceti, D., Cutrell, E., Morrison, C.: A dynamic ai system for extending the capabilities of blind people. In: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. CHI EA '20 (2020)

5. Insko, B.E.: Passive Haptics Significantly Enhances Virtual Environments. Ph.D. thesis (2001)

6. Iravantchi, Y., Goel, M., Harrison, C.: Digital ventriloquism: Giving voice to everyday objects. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. CHI '20 (2020)

7. Khot, R.A., Mueller, F.: Human-food interaction. Foundations and Trends in Human-Computer Interaction (2019)

8. Navarro-Tuch, S.A., Bustamante-Bello, M.R., Avila-Vazquez, R., Izquierdo-Reyes, J., Ramirez-Mendoza, R., Gutierrez-Martinez, Y., Pablos Hach, J.L.: Emotional domotics: Inhabitable space variable control for the emotions modulation. In: Proceedings of SAI Intelligent Systems Conference 2016 (2018)

9. de Oliveira, G.A.A., de Bettio, R.W., Freire, A.P.: Accessibility of the smart home for users with visual disabilities: An evaluation of open source mobile applications for home automation. In: Proceedings of the 15th Brazilian Symposium on Human Factors in Computing Systems. IHC '16 (2016)

10. Storer, K.M., Judge, T.K., Branham, S.M.: "all in the same boat": Tradeoffs of voice assistant ownership for mixed-visual-ability families (2020)

11. Thieme, A., Bennett, C.L., Morrison, C., Cutrell, E., Taylor, A.S.: "i can do everything but see!" - how people with vision impairments negotiate their abilities in social contexts (2018)

12. Vona, F., Torelli, E., Beccaluva, E., Garzotto, F.: Exploring the potential of speech-based virtual assistants in mixed reality applications for people with cognitive disabilities (2020)

13. Zöllner, M., Huber, S., Jetter, H.C., Reiterer, H.: Navi – a proof-of-concept of a mobile navigational aid for visually impaired based on the microsoft kinect. In: Human-Computer Interaction – INTERACT 2011 (2011)