<figure><img src="image_1.png" /></figure>

HAL
open science

# Multisense Blind Shooter: Auditory mobile application to assist blind and visually impaired people in target shooting

Florian Apavou, Tifanie Bouchara Bouchara, Jacques Shoetter

► To cite this version:

Florian Apavou, Tifanie Bouchara Bouchara, Jacques Shoetter. Multisense Blind Shooter: Auditory mobile application to assist blind and visually impaired people in target shooting. Disab2022, Workshop "Engineering Interactive Computing Systems for People with Disabilities". 14th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS2022)., Jun 2022, Nice, France. [hal-04429146](https://hal.science/hal-04429146v1)

HAL Id: hal-04429146

https://hal.science/hal-04429146v1

Submitted on 31 Jan 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Multisense Blind Shooter: Auditory mobile application to assist blind and visually impaired people in target shooting

Florian Apavou<fnref n="1" />, Tifanie Bouchara<fnref n="1" />,* and Jacques Schoetter<fnref n="2" />

**Abstract**

The Multisense project aims at making sport, and in particular modern pentathlon, include blind and visually impaired individuals (BVI) in ordinary club. With that perspective in mind, a first prototype of a mobile application, called Multisense Blind Shooter, was developed to make laser-run, especially target shooting, accessible to BVI. The smartphone has to be attached on an ordinary gun thanks to a 3D printed fixation. The application is based on auditory feedbacks to inform the shooters about the gesture correction they have to apply to hit the target. Four designs of auditory feedback paradigm are proposed.

**Keywords**

Human-Machine Interface, Visual Impairment, Assistive Technology, Accessibility, Auditory Display, Auditory Guidance, Sports and Video Games

## 1. Introduction

The Multisense project, carried out in partnership with an association for awareness of visual impairment, MixHandi-Cap sur la Vie (http://www.mcv-slb.fr/) aims to help the inclusion of visually impaired people in sport, particularly in modern Pentathlon. The demonstration we propose here concerns more specifically the laser-run discipline and more precisely the target shooting task. In this task, the constraints are both precision and time as the shooter must hit the target, using a laser weapon (without any projectile), 5 times in less than 50 seconds. Although adapted weapons already exist in other sports shooting practices [1], their cost do not allow ordinary clubs to buy them and the target shooting practice therefore remains reserved for adapted sports clubs poorly distributed over the territory. Furthermore this forces blind and visually impaired people (BVI) to keep among themselves instead of practicing with able-bodied individuals in ordinary clubs. In addition, the sound substitution used in these weapons is very difficult for beginners to access because it relies on memorizing a reference musical note and sound cues that only give access to a single dimension (distance to target) without indicating in

Disab'22: Disab2022, Workshop Engineering Interactive Computing Systems for People with Disabilities". 14th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS2022). June, 2022, Nice, France

*Corresponding author.

florian.apavou@universite-paris-saclay.fr (F. Apavou); tifanie.bouchara@universite-paris-saclay.fr (T. Bouchara); jacques.schoetter@orange.fr (J. Schoetter)

https://perso.limsi.fr/bouchara/ (T. Bouchara)

<figure><img src="image_3.png" /></figure>

© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).

<figure><img src="image_4.png" /></figure>

CEUR Workshop Proceedings (CEUR-WS.org)

<footnote n="1">¹Université Paris-Saclay, LISN, Orsay, France</footnote>

<footnote n="2">²MixHandi-Cap sur la Vie, Breuillet, France</footnote>
which direction to redirect the weapon. As a solution, the Multisense project seeks to provide
digital assistance based on auditory feedbacks to make target shooting more accessible to people
with visual disabilities. This will enable the practice of Laser Run in ordinary sports clubs more
inclusive, mixing able-bodied people and people with disabilities and thus will participate in
the well-being and social integration of people "differently competent" while promoting the
evolution of representations, mentalities and learning to solidarity with respect for diversity
for able-bodied people. Our proof of concept, named Multisense Blind Shooter, is based on
sonification, i.e. representing non audio information through non verbal sounds (equivalent of
visualisation for graphical representation of data). Our main question remains in the selection
of the optimal auditory parameters to make shooting more accessible to beginners and still
allow them to progress from recreational practice to competition. To answer this question,
Multisense Blind Shooter currently proposed four different sonification approaches. This will
allow us to later compare the actual sonified version of the target shooting assistance currently
used in international sports shooting competition (i.e. a single dimension, only based on pitch
and the reaching of a reference note pitch) with three other new methods. The first one is an
adaptative method which aims to improve the accuracy of target acquisition in the zone close
to the center of the target. The second one relies on 2D or even 3D sound guidance methods
which have been developed very recently for other use contexts such as surgery [2]. Finally, we
propose to include a sound spatialization system which has already shown its effectiveness in
auditory guidance here again in another context of use, namely racing video games accessible
to the blind [3].

## 2. System Overview

The Multisense Blind Shooter mobile application was developed with ease of access in mind. Four sound renderings are currently available on the application until they can be compared and eventually mixed and optimized. The smartphone fits into a 3D printed holder that is attached to a conventional shooting weapon (1). This simplifies the equipment needed for clubs by allowing visually impaired people to use their own phones.

<figure><img src="image_5.png" /><figcaption>**Figure 1:** A visually impaired participant using the Multisense Blind Shooter system : an ordinary gun combined with a mobile phone rendering audio feedback.</figcaption></figure>
## 2.1. Apparatus

Our mobile application is developed on Unity 3D, allowing us to access the phone's sensors to retrieve yaw, pitch and roll. To calibrate the center of the target (align the virtual target that appears on the phone and the real physical target), a sighted person will need to aim at the real target once to set the center in the application. Once done, the user will be able to shoot with the laser gun on the real target. The sensors data are used to calculate the rotation of the aim in the application. By projecting a ray from the Unity's scene camera on a plane, the coordinates of the sight are obtained in relation to the virtual target center. The distance information in abscissa and ordinate are then transformed in sound through one of the sound feedback algorithms before sent to the user in real time through headphones (required for spatialization). For our future experiments purposes, we are also able to shoot on the virtual target directly by touching the screen to retrieve the coordinates of the sight.

## 2.2. Sonification methods

Sonification is possible thanks to the many properties of sound that can be modified to obtain different returns. Based on former research on auditory guidance [4, 5, 2], the sonification methods integrated in Multisense rely on spatialization, pitch and the acoustic phenomenon of beats and roughness. Spatialization refers to the localisation of the sound in space that can be simulated by various audio techniques like stereo for 2D renderings and binaural for 3D renderings. Pitch defines the psycho-acoustical parameter that allows the difference between low and high sounds. It is directly related to the perception of the fundamental frequency of the sound and therefore to the perception of musical note played (example note A4 correspond to 440Hz, the lower the frequency the lower the pitch). This property is for example used in the Shepard tones which is made by adding sinusoidal frequencies separated by an octave creating an auditory illusion of a sound that constantly rises or falls. Beat occurs when two signals of close frequency are superimposed, a periodic modulation of the volume is then perceived. This modulation becomes more and more rapid when the frequencies are distant until a limit of approximately 15 Hz where the beat is so rapid that the sound becomes buzzing, it is the roughness [6].

**1D guidance through Pitch [1].** The first method of sonification is inspired by the systems currently in place in world competitions of adapted shooting sports. It uses only the variation in pitch. The closer the user gets to the center, the higher the sound becomes. A reference note informs the shooter that they reached the center of the target. This method does not allow to know exactly where you are on the target but only the distance from the center. It requires a lot of training to learn the reference note and is therefore not dedicated to novices. This is why we want to compare its performance with other existing sonification strategies.

**2D guidance through Pitch, Roughness and Beatings [5, 2].** This sonification, inspired by the work of Ziemer et al., uses multiple properties of sound to guide the user to a given point in surgery. This method is interesting since it sonifies each part of the target (top, bottom, right and left) in a different way, giving a more precise idea of the direction of the weapon. The x-axis is sonified in such a way that the user hears a range of Shepard's tones rising more
slowly as they approach the target from the left and falling more rapidly as they move away
from the center to the right. The y-axis uses the beat and roughness of the sound so that when
approaching the target from above, the beat becomes slower and slower. While when one
approaches from below, the sound becomes less and less rough. When the user is in the center,
the volume as well as the height of the sound remain constant.

**Adaptative 2D guidance through pitch and Beatings.** This is a 2-dimensional sonification that uses beats as well as pitch to transmit the position of the user according to the center of the target. On the x-axis, the closer you get to the center, the faster the beat, on the y-axis, the closer you get to the center, the higher the pitch is. In the center of the target, a distinct sound called white noise is created to distinguish it from the rest of the target. This method is therefore inspired by the two previous ones. However, this system uses a logarithmic dichotomy algorithm to obtain a more and more precise sonification as the user approaches the center of the target. Indeed, the range of sound modulation adapts according to the distance to the center. When the user travels half the distance to the target, the range is remapped so that this new distance becomes the new maximum distance. This method is the only one that have been pre-tested. Unfortunately pre-tests shown that the remapping is difficult to handle for the non-initiated. This is why we decided to add another sound parameter to our rendering : spatialization, as described as follow.

**Spatialized Sonifications.** Our application takes the 3 previous sonification methods and allows to add a binaural spatialization so the target appears localized in space. The users have then to try to place the sound in front of them to hit the center of the target. This methods is inspired from race gaming accessibility method [3] where the sound of the car engine had to stay in the center to avoid barriers and other obstacles.

**3. Conclusion and Future work**

Our next step will be to test our different audio feedbacks to ensure their usability and to
determine their limits in terms of accuracy and rapidity of shooting. Another interesting
question will be to determine the learning curve, i.e. the evolution of shooting performance
after different types and duration of training, for each sonification strategy. Actually, we may
even voluntarily choose to mix the different sonification strategies and personalize the audio
rendering to better address preferences and needs of each user. These sonification methods
can also be tested in the context of video games accessibility as the task of target acquisition
is similar in both contexts of First Person Shooter games and target shooting sports except
for their constraints. Indeed, video games do not require a harsh accuracy compared to sport
performances, while on the contrary the target can be moving and shooting back in video games
but will stay fix in sport.

We would like to thank Tim Ziemer for having shared his own code from [2] so we can
integrate it directly in our Unity app. We are also thankfull to Matthieu Aussal and Morgane
Besnier for their technical help on other sonification methods and in the project in general.
References

[1] B. B. Sport, A guide to visually impaired friendly sport, 2018.

[2] T. Ziemer, N. Nuchprayoon, H. Schultheis, Psychoacoustic sonification as user interface for human-machine interaction, CoRR abs/1912.08609 (2019). URL: http://arxiv.org/abs/1912.08609. arXiv:1912.08609.

[3] B. A. Smith, S. K. Nayar, The rad: Making racing games equivalently accessible to people who are blind, in: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI '18, Association for Computing Machinery, New York, NY, USA, 2018, p. 1–12. URL: https://doi.org/10.1145/3173574.3174090. doi:10.1145/3173574.3174090.

[4] G. Parseihian, M. Aramaki, S. Ystad, R. Kronland-Martinet, Exploration of Sonification Strategies for Guidance in a Blind Driving Game, in: Technology with Swing, Revised papers of the CMMR2017, 2018. URL: https://hal.archives-ouvertes.fr/hal-01933559.

[5] T. Ziemer, H. Schultheis, Psychoacoustic auditory display for navigation: an auditory assistance system for spatial orientation tasks, Journal on Multimodal User Interfaces (2019) 205–218. URL: https://doi.org/10.1007/s12193-018-0282-2. doi:10.1007/s12193-018-0282-2.

[6] E. Zwicker, H. Fastl, Psychoacoustics. Facts and Models, Springer Series in Information Sciences, 2nd. ed., Springer Berlin, Heidelberg, 1999.