# Guiding sightless people one dimension at a time: how the strategy used in spontaneous voice instructions benefits sonification-based guidance

Coline Fons, Allan Henry, Sylvain Huet, Denis Pellerin, Solange Rossato, Christian Graff

<figure><img src="image_1.png" /></figure>

## To cite this version:

Coline Fons, Allan Henry, Sylvain Huet, Denis Pellerin, Solange Rossato, et al.. Guiding sightless people one dimension at a time: how the strategy used in spontaneous voice instructions benefits sonification-based guidance. 2025. hal-05165934

HAL Id: hal-05165934

https://hal.science/hal-05165934v1

Preprint submitted on 16 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Guiding sightless people one dimension at a time: how the strategy used in spontaneous voice instructions benefits sonification-based guidance

Coline Fons<fnref n="1" />,<fnref n="2" />*, Allan Henry<fnref n="1" />,<fnref n="2" />,<fnref n="3" />, Sylvain Huet<fnref n="2" />, Denis Pellerin<fnref n="2" />, Solange Rossato<fnref n="3" />, Christian Graff<fnref n="1" />

## Abstract

Sensory substitution devices (SSDs) aim to assist visually impaired people by converting spatial information into non-visual modalities, such as sound. Among existing guidance methods, voice instructions are the most intuitive, while sonification offers continuous and precise feedback. In this study, we explore a sequential sonification approach (which conveys instructions one dimension at a time, one after the other), inspired by voice guidance, and compare its effectiveness against parallel sonification (which conveys all spatial dimensions simultaneously) and voice guidance provided by a naive human guide.

Two experiments were conducted using an invisible target-reaching task in a 3D virtual space. The first experiment was carried out on 20 sighted participants (eyes closed) and one visually impaired participant. This experiment compared sequential and parallel sonification, demonstrating that sequential sonification enables faster target acquisition and is preferred by participants due to its lower cognitive load. The second experiment involved 10 participants and 10 guides. It compared sequential sonification

<footnote n="1">¹ LPNC, CNRS, Université Grenoble Alpes, Université Savoie Mont-Blanc, Grenoble, France</footnote>

<footnote n="2">² GIPSA-lab, Grenoble INP, CNRS, Université Grenoble Alpes, Grenoble, France</footnote>

<footnote n="3">³ LIG, Grenoble INP, CNRS, Université Grenoble Alpes, Grenoble, France</footnote>

<footnote>* Corresponding author : coline.fons@univ-grenoble-alpes.fr</footnote>
with voice guidance, showing that while voice instructions are natural and do not require a long learning phase, they suffer from latency and discontinuity, making them less efficient than sonification. Furthermore, our analysis of naive guides’ voice commands assessed that they indeed naturally provide instructions one spatial dimension at a time, reinforcing the relevance of sequential sonification.

These findings highlight the advantages of our sequential sonification, which combines the precision of non-spatialized sonification, the intuitiveness of voice guidance, and the natural cues of spatialized sonification (stereo). By balancing efficiency and cognitive ease, this approach represents a promising solution for auditory guidance systems designed for visually impaired users.

Introduction

Since the early days of sensory substitution, which aimed to replace the eye and provide
a complete but overwhelming vision experience for blind and visually impaired people,
sensory substitution has then been applied to assist blind and visually impaired people
with specific tasks. Sensory substitution devices (SSDs) capture information normally
accessible by sight, and encode it in another functional sensory modality, generally
hearing or touch. A large number of studies on these devices are being carried out, but
their transfer to practical use is fraught with challenges, one of the biggest limitation
being learning how to use and master the tool [1,2].

We are interested here in SSDs designed for a specific task, in this case reaching a
3-D located object with the hand, guided by sound. Far from wanting to completely
replace sight, we are seeking to develop an effective system to assist a person deprived
of sight in this specific but generic task. We want to create a system precise enough to
perform the task quickly, but without creating cognitive overload. Our work shows the
advantage of using sequential commands. This strategy is compared with the ‘parallel
strategy’ in sonification: The parallel strategy combines all dimensions needed to reach
the target in a single signal; The sequential strategy breaks down this composite vector
into elementary (one-dimensional) vectors delivered in a series of signals. By making all
the information needed available in each signal, the parallel strategy is theoretically
faster, but it appears more demanding in terms of cognitive workload. We also show
that guidance by sonification is faster than voice guidance by a naive human guide.

## Sensory substitution

Sensory substitution replaces a deficient sensory modality with another. A sensory substitution device (SSD) consists of a sensor (to extract visual or spatial data), a transcoder (to encode it into another modality), and an effector (to transmit it to the user). Early research, pioneered by Paul Bach-y-Rita, focused on visual-tactile substitution [3-5], using a matrix of vibrators to project visual data captured by a camera onto the skin. Despite claims of vision-like perception [6], learning is slow, blind users taking about fifteen minutes to recognize new objects after more than 40 hours of training.

The first Auditory-Vision Substitution Device (or AVSS, Audio-Vision Substitution System) was the vOICe, developed by Meijer [7]. This device converts an image captured by a camera into a soundscape. Each soundscape is a sweep from left to right of the image. Frequency represents the vertical axis: a pixel at a higher position will be converted to a higher frequency. Loudness represents brightness: a pixel with higher brightness will be converted to higher loudness. Other AVSS work in a similar way, for example the system developed by Cronly-Dillon, Persaud and Gregory [8], the EyeMusic [9], the PVSA [10], the Vibe [11, 12] or the MeloSee [13]. The EyeMusic [9], unlike other systems, converts the colors of the pixel into musical instruments. The MeloSee [13] was tested on a specific task, a maze navigation task.

All these sensory substitution devices have the same ambition of seeking to replace the eye, and therefore by providing blind and visually impaired people with all the cognitive and behavioural experiences that accompany vision. However, although some users are able to report a detailed visual phenomenology after several months of intensive use of the device [14], the difficulty of learning these systems seems to hinder their use. [15] propose prerequisites to be taken into account when designing an SSD. Among these are cognitive load and ease of learning, which in the systems presented seem to be overlooked in favor of resolution and quantity of information.

The SSDs adopted by blind and partially-sighted people are those that are easy to use and inexpensive. In most cases, rather than completely replacing the eye, these
systems assist them with one task at a time. Many assistive applications are now
available on smartphones. The most widely used in the United States, for example, are
Color Picker, which will indicate the colour of an object from the camera, and screen
readers such as TalkBack [16, 17].

In the literature, sensory substitution is widely used in audio guidance systems. We
will review the existing system that use audio to guide people to reach targets.

Guidance SSDs

An SSD guiding a person towards a target will collect spatial data via a sensor (usually a camera). The deviation from the target is calculated from this data. The reference frame can be the head [18–20] or the hand [21–23] depending on the position of the camera. In some studies, guidance shifts from a head-centred referential for navigation to the target location to a hand-centred referential for precise guidance of the hand to the target [24, 25].

Once the deviations have been calculated, they are converted into sound or vibration
and transmitted to the user. We will concentrate here on systems that use sound as a
guidance method. This is because sounds offer more possibilities, as the ear's
information capacity is greater than that of the skin, as summarised by [26].

Three types of sound feedback can be used to guide a person: voice guidance, also
known as verbal instructions, spatialized sonification and non-spatialized sonification.

**Voice guidance** in a target-reaching task can correspond to commands such as ‘go right’, ‘go forward’, ‘top left’, etc.

**Sonification** involves using non-verbal sounds to convey perceptual data [27].

**Spatialized sonification** uses the auditory system's ability to locate a sound
source by virtually recreating its position. To do this, either stereo or HRTF
(head-related transfer function) is used. HRTF uses a mathematical transfer function to
simulate the transformations made to sound waves by a listener's body, which in
conjunction with the interaural time (ITD) and level (ILD) difference, enables humans
to locate a sound source.

**Non-spatialized sonification** uses the physical properties of sounds such as
volume, frequency, tempo, etc. For example, the distance to the target can be converted into tempo.

Each of these three methods has its advantages and disadvantages.

Voice guidance is considered natural and intuitive, and does not require extensive training. However, it can be considered as slow, because it takes time to hear and process each instruction, and there is a delay before giving the next instruction. In contrast, non-spatialized sonification can continuously vary sound parameters like frequency, tempo, or intensity in real time, instantly reflecting the user's movements and allowing faster guidance.

Spatialized sonification has the advantage of being natural and intuitive, but it is not very precise because it is limited by the auditory system's spatial resolution [28–30].

Non-spatialized sonification is less intuitive and natural than the other two methods, as the association between spatial metrics and sound is arbitrary, and requires a longer learning process to master the tool. However, it provides more precise guidance.

Indeed, spatialized sonification is inherently limited by the human auditory system's spatial resolution. On the horizontal plane, humans can typically distinguish sound sources separated by about 1° ± 3° [29]. In contrast, the auditory system can discriminate between a much greater number of frequency steps (from 640 to over 4000 distinguishable frequencies across the audible spectrum [31]). This means that when conveying deviation from target, encoding it through changes in frequency allows for much finer granularity than spatialized sound. As a result, non-spatialized sonification can provide users with more precise guidance. A study showed that sound localization was better when combining non-spatialized sonification to spatialized sonification [32]. In addition, it is possible to exploit the fact that certain associations between spatial and sound parameters are more natural than others to create a system using non-spatialized sonification that is intuitive and simple to use. For example, a high pitch and tempo are conceptually associated with greater proximity, as well as a sense of urgency [24]. We tend to associate high pitch with a high position in space [33]. Finally, intensity is naturally associated with proximity, given that the amplitude of the signal increases as the sound source approaches.

These methods have been used in systems for guiding people to targets.

In [20, 21, 34], participants had to reach targets, guided only by voice commands
("right", "left", "up", "down", etc). Many studies combine verbal instructions and
sonification [22, 35] or verbal instructions and vibrations [25]. Studies that use
sonification often combine spatialized and non-spatialized sonification [19, 24, 36–38].
In [39], the authors compared spatialized audio with verbal instructions, vibrations and
non-spatialized sonification (that of Meijer [7], the vOICe), in a target-reaching task.

Studies comparing different types of feedback indicate better performance in finding
a target using a combination of non-spatialized sonification and spatialized sonification,
compared with spatialized sonification alone [24, 36]. In [39], the authors found better
performance for spatialized sound than for non-spatialized sonification (the vOICe), but
the participants actually failed to complete the task with the vOICe, as they were
unable to distinguish the cursor from the target. This confirms the need to adapt the
sound feedback to the needs of the task. The participant does not need to have their
absolute position and the absolute position of the target (as it is the case with the
vOICe), but to know the distance between the two and in which direction they should
move to reduce this distance.

## Choices of sonification parameters

To create an accurate but simple and intuitive guidance system, we can combine
non-spatialized sonification, which offers accuracy and speed, with spatialized
sonification, which is more natural and intuitive but not very precise. It is the case in a
number of studies [19, 24, 36–38]. In most of the studies using sonification, information
concerning several dimensions (the horizontal position of the target, its vertical position
and the distance to the target) is given simultaneously [19, 36–39].

However, the results of our previous studies [40, 41] show that in a target-reaching
task with the hand, participants are faster with a non-spatialized sonification that
dissociates the vertical and horizontal dimensions, by varying one sound parameter per
dimension, rather than a sonification that varies one sound parameter as a function of a
3D spatial metric. Instead of coding the overall angle difference between the direction
pointed and the target direction by pitch, it's more efficient to code the azimuthal
orientation (horizontal difference) by pitch, and use a timbre change to indicate the
correct height (vertical difference). The participants' description of their target search
strategy indicated that they naturally searched for the target on one spatial dimension 142
at a time. Furthermore, a previous study showed that users found sequential 143
sonification to be easier to use than parallel sonification on a target-reaching task where 144
the target was within reach, although performances were the same in both conditions 145
and only vertical and horizontal deviations were sequential whereas distance was 146
presented at all time [42]. Finally, studies using voice guidance overwhelmingly use 147
sequential verbal instructions, with commands such as ‘right’, ‘left’, ‘up’, 148
‘down’ [21, 22, 25, 34, 35, 39], and very few use commands such as ‘top right’, ‘bottom 149
left’, etc. [20] or indicate the exact position of the target on all dimensions at once [35]. 150

Sequentiality of commands confronts some advices in the literature, that 151
recommends that each dimension be integrated into a single auditory stream and that 152
they be interpretable simultaneously [29, 43–46]. However, the sonification they are 153
developing has a medical purpose, intended for surgery [47]. We believe that finding an 154
object with the hand does not require as much precision, and that users can gain in 155
comfort and ease of use by progressing toward the target on one dimension at a time. It 156
would therefore be interesting to combine the speed and precision of sonification with 157
the intuitiveness of voice commands, by creating a sequential sonification mode, which 158
also corresponds to the sequential search strategy used by participants in our previous 159
studies, even when all dimensions were available at the same time [40, 41]. 160

To demonstrate the advantage of sequential sonification over parallel sonification, 161
the sonifications must differ in their temporal presentation only, and have the same 162
sound metaphors (conversions of spatial metrics into sound) for each dimension. The 163
choice of these metaphors was made following the advice given in the literature to 164
create the most accurate sonification possible, and following the results of a previous 165
study that compared multiple sonifications for transcoding the vertical and the 166
horizontal position of a target [42]. 167

The metaphors we used can be presented either sequentially (one dimension at a 168
time) or in parallel (all at the same time). The metaphors are as follow: 169

**Azimuth:** the direction pointed in relation to the direction of the target is 170
converted into frequency on a continuous scale. The closer the direction pointed is to 171
the direction of the target, the higher the sound. The further away from the correct 172
direction, the lower the sound. Frequency is the sound parameter most frequently used 173
in sonification [48]. Stereo completes this metaphor, and is a form of spatialized
sonification. Although more natural and intuitive than non-spatialized sonification, it is
subject to the precision limitations of the auditory system. We have therefore used a
simplified, discrete form of spatialization, with sounds only in the right ear when the
target is on the right, and sounds only in the left ear when the target is on the left.
Sounds were presented in both ears when pointing in the direction of the target.

**Depth:** the distance between the finger and the target is continuously converted into tempo. At a distance, the tempo is slow, and the closer the finger is to the vertical plane where the target is located, the faster it becomes. Intensity is often used to convert distance, with which it is naturally associated, as the amplitude of a signal increases as the sound source gets closer. However, tempo is also associated with distance, being used to signal the proximity of an obstacle, for example on a car's reversing radar. A high tempo is conceptually associated with proximity and a sense of urgency [24]. It provides more precise indications than intensity [27], which is why we have chosen it here.

**Elevation:** the vowel [a] is generated when the target is below the participant's hand. The vowel [o] is generated when the target is above the participant's hand. This metaphor takes into account the natural association between these vowels and the French words 'haut' (high, up) pronounced [o] and 'bas' (low, down) pronounced [ba]. We are getting closer to verbal instructions here, which are more natural and intuitive than sonification, but at the same time these vowels should be quicker to interpret than full words, and they are easier to combine with pitch changes for the parallel condition.

In sequential mode, these metaphors are presented one after the other. The azimuthal direction is given first, then distance, once the participant is in the right direction, then target height, once the participant has corrected both direction and distance. Of course, if the participant deviates from the right direction, this information will be given again. The same applies to distance.

## Our contribution

In this study, we explore a sequential sonification approach (which convey spatial information step-by-step), inspired by voice guidance and results of previous studies,
and compare its effectiveness against parallel sonification (which convey all spatial information at the same time) and voice guidance. The guide is naïve (no precise instructions on how to guide) and sees the target in a virtual representation of the scene on a screen. Commands are therefore spontaneous, the idea being to favor intuitive and natural verbal instructions. Recording the voice statements used in natural spontaneous voice guidance may provide new insights of the natural functioning of human interaction to inspire further advances in SSD guidance.

In a first experiment, we compare ‘parallel’ sonification guidance, providing the participant with all the information simultaneously, with ‘sequential’ sonification guidance, converting one spatial dimension at a time. In a second experiment, we show the advantage of this sequential sonification mode over voice guidance, and demonstrate that the commands used spontaneously by a human guide are indeed sequential, even when they are not constrained by a list of voice commands that impose it.

We tested our different types of guidance on a 3D target-reaching task. Participants moved in a motion capture space with their eyes closed, to reach an invisible virtual target with their hand. As discussed in [49], immersion in a 3D environment allows us to dissociate the sensor (cameras) from the frame of reference from which deviations from the target are calculated, which can be placed anywhere we want on the body. Here the participant’s pointing finger and middle of torso positions were captured by the cameras, and deviations of the hand direction and position with respect to the target were converted into sound in the sonification guidance conditions. In the voice guidance condition, a guide gave the participant instructions on how to reach the target. The guidance conditions were evaluated by the time taken to reach the target and by a questionnaire on the ergonomic qualities of the system.

These two experiments enabled us to test four hypotheses, which we will address at the start of the discussion:

* First experiment: A sequential sonification mode will allow faster target-reaching than a parallel sonification mode.

* First experiment: The sequential mode will be preferred by participants over the parallel mode.

* Second experiment: Sonification guidance will be more effective than voice
guidance by a human guide.

* Second experiment: Analysis of the guides' voice commands will confirm that they spontaneously use sequential commands, i.e. they direct participants along one spatial dimension at a time.

This experiment will also provide an overview of the diversity of statements, of various lengths, used spontaneously by the guides.

# Materials and methods

This work was authorized by the ethical committee CER Grenoble Alpes (Avis-2018-06-19-1).

# Engineering

In this study, we dissociate the sensor (motion capture system) from the spatial reference frame (user's body). Participants wore objects equipped with reflectors that were located by a Vicon® optical motion capture system. Reflector coordinates were used to locate the position of two anatomical points: the middle of the torso (point O) and the tip of the index finger (point P). The position of these anatomical points was transmitted from the acquisition computer running Vicon Tracker® software to the pilot computer through the Virtual Reality Peripheral Network (VRPN) protocol. On the pilot computer, our C++ control software performed the following tasks: 1. It immersed the participant in a virtual environment containing the target, utilizing the OpenScene Graph (OSG) 3D toolkit; 2. It computed the spatial metrics required for sound conversions; 3. It transmitted these metrics to the PureData sound system, running on the same computer, which synthesized the corresponding sounds and delivered them to the participant; 4. It managed the experimental protocol using a finite-state machine.

The cameras of the Vicon optical motion capture system are positioned on a rectangular support measuring 5.8 m * 5 m at 2.8 m from the ground, giving us a capture area of 4.8 m * 4.5 m * 2 m. The targets used here are spheres with a radius r
= 7.5 cm. They can be positioned at 27 positions on a 3*3*3 grid of 124 cm in x, 135 cm in y, and at three different heights in z: 70 cm, 110 cm, 150 cm.

262

263

## Participant's equipment

264

During the experiment, the participant was harnessed with two sets of reflectors to track the position of the dominant-hand index fingertip and its reaching direction with respect to the middle of the torso. They also wore a bone-conduction stereo headset (Aftershokz Sportz3) connected to a bodypack receiver (Mipro MI909R) to hear the sound feedback. They held a miniature keyboard (iclever Rii 2.4 GHz L * 1 * h = 151 * 59 * 12.5 mm) in their nondominant hand to remotely launch each trial.

265

266

267

268

269

270

## Spatial metrics and sonification

271

The spatial features extracted from the scene (see Fig 1) were inferred from point P the tip of the index finger, point O the inner middle of the torso, and point T the center of the spherical target. The vector OP defines the participant's pointing direction, to be compared to the direction to the target OT; distance PT defines the proximity to the reach. The three main spatial metrics calculated were:

272

273

274

275

276

* The azimuth angle $\theta_h = POT_h$, i.e., the projection of the angle formed by the lines OP and OT onto the horizontal plane parallel to the ground (angular deviation).

277

278

* The depth distance d, i.e., the projection of [PT] onto the horizontal plane parallel to the ground (distance deviation).

279

280

* The height difference $\Delta Z = ZP - ZT$, i.e., the projection of the distance [PT] onto the vertical axis Z (distance deviation).

281

282

These spatial metrics were then converted into sound, for the two sonification guidance modes, either sequential or parallel. The metaphors used were the same in the two sonification modes. In the parallel mode, the three main spatial metrics were expressed simultaneously in a compound sound. In the sequential mode, only one metric was expressed at a time, the choice of the spatial metric depending on the magnitude of the deviation from the target for each measurement: first, the azimuth deviation was transmitted, then when the directional angle was reduced below a certain threshold, the

283

284

285

286

287

288

289
<figure><img src="image_3.png" /><figcaption>Fig 1. Representation of target positions (spheres) and spatial metrics converted to sound.</figcaption></figure>

Here, the focus target (in red) is positioned at the lowest height, and in the middle of one smaller outer side of the grid.

distance to the target was given, and finally the target height, once the direction and 290
distance were both below a certain threshold. Conversions of spatial metrics into sound 291
were as follow: 292

**Azimuth direction** $\theta_h$: The sound pitch varied as a function of $\theta_h$ on a continuum ranging from $fmin = 110$ Hz for $|\theta_h| = 90^\circ$, to $fmax = 330$ Hz for $|\theta_h| = 0^\circ$. In sequential mode, the sound was a sinusoid that was generated only when $|\theta_h| > 3^\circ$. The conversion of $\theta_h$ into pitch follows Stevens’ [50] psychophysical law, a power law function. 293
294
295
296
297

**Distance d**: The tempo of the sound varied as a function of d, on a continuum ranging from $Tmin = 200$ BPM (for $dmax = 6$ m) to $Tmax = 600$ BPM (for $dmin = 0$ m). In sequential mode, the sound indicating distance was a sinusoid that was generated only when $|\theta_h| < 3^\circ$ and $d > r(targetradius) + 0.05$ m or when $|\theta_h| < 3^\circ$ and $d < r + 0.05$ m and $|\Delta Z| < r$ (meaning when the direction was correct and the distance to the target was greater than 5 cm, or when the direction was correct and the distance to the target was less than 5 cm and the finger was at target height). 298
299
300
301
302
303
304
**Height** |ΔZ|. If ΔZ < 0 then the vowel [o] was generated. If ΔZ > 0, then the vowel [a] was generated. In sequential mode, the vowel was generated at a constant pitch of 110 Hz when |θ<sub>h</sub>| < 3° and d < (r + 0.05 m) and |ΔZ| > r.

305

306

307

In parallel mode, the sound was a vowel ([o] or [a] depending on ΔZ), generated by
beats whose tempo depends on d, with a pitch varying depending on θh.

308

309

The r + 0.05 was introduced after pre-tests had shown that it was complicated for
participants to find the correct target height in sequential mode when vowels were given
only when the finger was in the column containing the target, as they tended to
overshoot or undershoot it. As a result, they often lost height information and wasted
time finding it again. With these settings, in sequential mode, participants found the
right direction, moved forward to within 5 cm of the column containing the target,
found the right height and then moved forward until they reached the target.

310

311

312

313

314

315

316

Additional sound metaphors were used for both sonification modes: When the fingertip point P overshot the target T (|OP| > |OT|), a buzzer sound (square signal) replaced every other sound. When point P entered the target, a mild white noise was generated and the main audio stopped. A distinct tinkle announced the trial's start, another one when the target has been reached, and a third one announced when the time limit had been reached.

317

318

319

320

321

322

Protocol

323

The experiment lasted approximately one hour. It took place in the motion capture space described above. Participants had to find and reach with their fingertip several targets presented in succession.

324

325

326

**Familiarization phase:** Before they started the trials, a familiarization phase allowed participants to explore sound metaphors around a visible target. The target, a virtual 30-cm diameter sphere, was physically materialized by a real sphere made of wooden rings, superimposed over the position of the virtual target. Participants were equipped and could explore the sound metaphors, and experience how the sound feedback was related to their movements and position around and within the target. They were given instructions by the experimenter ensuring that all the sound metaphors had been heard and understood before testing.

327

328

329

330

331

332

333

334
**Target-reaching task:** Participants, eyes closed, then performed the target-reaching task, where they had to reach invisible virtual targets. The task comprised a training phase followed by a test phase. The two phases were similar, since in both cases the participants completed blocks of 6 trials, i.e. 6 consecutive targets to be reached, positioned in a semi-random way: between two successive trials, the target position changed along each of the three grid coordinates, so that the participant had to cover a minimum distance between two targets (about 1.88 m). The sequence of a block will be described below.

Participants started by doing as many training blocks as they deemed necessary to feel comfortable with each guidance mode, before starting the test blocks. For each guidance condition, participants completed two test blocks of 6 trials. Only test blocks were used in the analyses. The two guidance conditions alternated from one test block to the next, and their order of passage was counterbalanced between the participants. Participants could do additional training blocks before each test block if they deemed it necessary.

For each block (training or test), participants started at the centre of the room, with their eyes closed. When the experimenter indicated that the block had started, they could press the start button on the mini-keyboard. The start-up tinkle indicated the beginning of the trial, then the sound feedback was emitted, changing according to either sonification mode assigned to the block, and according to the participant's movement and position as described above. The target was considered reached when the point P (fingertip) remained within the target (indicated by the white noise sound feedback) for 400 ms, which triggered the victory tinkle and ended the trial. When the participant had not reached the target within a 120-second limit, another tinkle for "defeat" indicated the end of the trial. After each trial, the participant stayed on the spot, from where they could launch the next trial using the mini-keyboard when ready. This continued until all six trials in the test block were completed.

**Ergonomics questionnaire** After the target-reaching task, participants completed a questionnaire about the experience. They were asked for their general opinion about the experiment, the challenges they encountered, and to rate each of the ergonomic qualities of the two guidance modes, using a seven steps Likert scale, on three aspects: efficiency,
ease and comfort. The questionnaire also comprised a space of open comments, that
were also discussed with the experimenter.

Analyses

Analyses were performed using RStudio (RStudio 2023.12.1+402).

The participants' objective performance score was the target-reaching time, which
has an asymmetric distribution, so we used a Cox analysis to compare between
conditions. A Cox model analyzes repeated measures without averaging the data for
each participant, taking into account intra- and inter-participant variability [51]. We
used the coxph function in the Survival package of the RStudio software.

Participants provided subjective ratings on three criteria (efficiency, ease and
comfort) for each guidance condition. Each rating is an ordinal categorical variable
ranging from 1 to 7. Since the scores for the three criteria are correlated within each
condition, we included all ratings in a single ordinal regression model rather than
analyzing each criterion separately. This approach allows us to evaluate the overall
effect of the guidance condition on subjective ratings without aggregating them into a
single score. The analysis was conducted using an ordinal regression model with the
clm() function from the ordinal package in RStudio.

Sequential vs. Parallel sonified guiding tests

The aim of this first experiment is to demonstrate an increase in performance and
judgement with a sonification that dissociates the dimensions spatially but also
temporally (sequential mode) compared with a sonification that dissociates the
dimensions spatially but provides them all at the same time to the participant (parallel
mode).

Materials and methods

Participants

20 sighted people (22 ± 6 years old) participated in the experiment. They were
psychology students and received a bonus for their grade on an experimental psychology
topic course in exchange for their participation. One 18 year old visually impaired student also took part in the experiment. All participants gave their informed written consent prior to participating in the study. This work was authorized by the ethical committee CER Grenoble Alpes (Avis-2018-06-19-1).

Participants were recruited between September 30, 2024, and December 19, 2024.

## Protocol

Prior to conducting the tests, participants read a detailed description of the experimental procedure, including the goal of the experiment, the material used, the sound metaphors and the different phases of the experiment. This information was then explained orally by the experimenter.

The rest of the protocol is as described in the above Materials and Methods section common to both experiments. Participants did test blocks in the two sonification guidance conditions: sequential and parallel mode.

## Analyses

Before running the Cox analysis, data were cleaned by removing target-reaching times greater than the mean plus two standard deviations. This represented 4.0% of the sighted participants' data (the blind participant's data is not included in the analyses, but described and compared with that of the sighted participants).

## Results

**Target-reaching times:** sighted participants were significantly faster with the sequential mode ($M = 17.82$ s; $SD = 8.38$ s) than with the parallel mode ($M = 19.57$ s; $SD = 9.23$ s) ($z = 2.00$, $p = 0.045$) (see Fig 2). The visually impaired participant had a similar pattern of results, with a mean target-reaching time of 27.09 s ($SD = 11.66$ s) for the parallel mode, and 24.43 s ($SD = 13.27$ s) for the sequential mode. His performance is also represented in Fig 2 (participant 21).

**Ergonomics:** Results of the ordinal regression reveals that participants generally preferred the sequential mode of sonification ($z = -2.37$, $p = 0.018$). Scores given by participants on each condition are represented in Fig 3. The visually impaired
<figure><img src="image_4.png" /><figcaption>Fig 2. Box plots of target-reaching times (log10 scale) for each participant under the two sonification modalities.</figcaption></figure>

Each box represents the median (horizontal line), the interquartile range (box), and the extreme outlier values (whiskers). Participants 1 to 20 are sighted participants, which performed the experiment with their eyes closed. Participant 21 is visually impaired (VI).
<figure><img src="image_5.png" /><figcaption>Fig 3. Bar chart showing the distribution of participant ratings (1 to 7) across the two sonification conditions.</figcaption></figure>

All three criteria are included (efficiency, ease, comfort). The color gradient represents score levels, with red for 2 and dark green for 7 (no participant gave a score of 1), so that greener areas represent more positive judgments. Asterisks represent scores given by the visually impaired participant for each of the three criteria.

participant also expressed a preference for the sequential mode, the scores he gave for each criteria are represented in Fig 3.

Discussion

We compared sonification that dissociated spatial dimensions both spatially and temporally (sequential mode) with sonification that dissociated dimensions spatially only, the dimensions being presented simultaneously (parallel mode). The results of the tests on a target-reaching task with the hand showed a faster reaching of the target with the sequential mode compared with the parallel mode. Analysis of the subjective scores given to the two sonification modes by the participants also showed a preference for the sequential mode over the parallel mode, in accordance with objective measures.

These results demonstrate the advantage of a strategy that delivers one dimension at a time: the sequential mode is both more effective (faster) and judged more efficient, comfortable and easy to use by participants. It reduces the amount of information to be interpreted at one time, thereby lightening the cognitive load and enabling the target to be reached more quickly. The participants found this sonification more comfortable and easier to use, as there was less information to interpret simultaneously. Hearing
variations in a single sound parameter for a single dimension made it easier to react more efficiently to the variations. This mode was therefore easier to understand and learn.

Although the overall results show an advantage and a preference for sequential mode, out of 20 sighted participants, 5 indicated a preference for parallel mode. These participants found the parallel mode more precise and comfortable than the sequential mode. One participant even found that with parallel mode she had the feeling that she was seeing through her eyes. Parallel mode gives you access to all the information simultaneously. It does not restrict the strategy for reaching the target. Thus, the participants who appreciated this sonification mode could be those who were able to take advantage of the benefits of this metaphor, which enables them to move towards the target with a direct trajectory if they manage to interpret everything at the same time, or to correct one deviation at a time as with sequential mode, but in an order decided by the participant, which may be different from that imposed by sequential mode.

However, of these 5 participants, only 2 were actually faster in parallel mode than in sequential mode. This may mean that the participants did not succeed in moving in a straight line towards the target but that they naturally adopted a strategy similar to that imposed by the sequential mode. This proposal seems plausible, when we have a look at the answers to the questionnaire.

The questionnaire asked participants to describe the strategy used to find the target with the two sonification modes. The responses show that of the 21 participants, 16 (including the blind participant) indicated that they adopted a similar strategy for both modes, i.e. focusing on one dimension at a time. Only 2 participants (including one who preferred parallel mode) indicated that they tried to interpret all the information at once in parallel mode. The other three participants gave unclear answers or did not answer the question. This means that of 21 participants, only one preferred the parallel mode AND performed better with this mode by reaching the target by the shortest trajectory.
## Conclusion

This first experiment showed that sequential sonification, presenting information one dimension at a time, is more effective and ergonomic than parallel sonification, which provides all dimensions simultaneously. The participants not only reached the targets more quickly, but also expressed a preference for this approach, emphasizing its simplicity and ease of interpretation. Since the temporal sequencing of the dimensions was partially inspired by informal observation of voice guidance, we conducted a second experiment in a second experiment in an identical target reaching task, first to compare sonification guidance with voice guidance, and second to assess the sequentiality spontaneously used by guides.

## Sonification vs. Voice guidance tests

Our perspective is to create an accurate and easy-to-use SSD with sound guidance. The previous experiment shows the advantage of a sequential sonification mode, giving one piece of information at a time, over a parallel sonification mode, which allows all the dimensions to be interpreted at once. One of the inspirations for this sequential mode is the fact that instructions on one dimension at a time would be the way we naturally guide a person by voice. At least, in guidance SSDs using verbal instructions, sequential instructions such as 'go right', 'go forward' [21, 22, 25, 34, 35, 39] are more often used than instructions including several dimensions such as 'top right' [20]. But in these studies, an algorithm defines which instructions are given, not a human guide. Regarding the voice commands of drones [52–54], predefined and limited sets of commands are used, typically involving either a single direction or no direction at all. For instance, while [55] attempts to introduce a more structured grammar to modulate instructions, the resulting commands still produce unidirectional movements.

Two questions then arise: is natural, human guidance spontaneously sequential? If so, would a blind person still be better guided by sequential sonification rather than by someone giving spontaneous verbal instructions?

This experiment aimed to answer these two questions. In a target-reaching task with the hand in a 3D virtual space, identical to the previous one, participants tested two types of guidance: guidance by sonification in sequential mode (identical to the previous
experiment), and voice guidance, with instructions given by naive guides.

We expect that guides will spontaneously use sequential instructions. We recorded
the natural voice commands of people guiding a partner towards a target the latter
could not see. We then used a voice processing system to break down the voice guidance
into statements. We then listed the statements, and classified them firstly according to
their form, i.e. the words used, and their number per statement. We then classified
them according to their content, by identifying the different spatial dimensions they
each contained. We expect statements containing only one dimension at a time. We will
also take advantage of this spontaneous corpus to explore which syntax is favored and
find out whether an additional tactic or strategy might emerge when spontaneously
guiding towards a target, in order to provide more inspiration for automated systems.

Despite the advantage of the naturalness and simplicity of voice guidance, we expect
that sonification will enable the target to be reached more quickly, thanks to the
advantages of sonification over voice guidance explained above.

# Materials and methods

## Participants

Twenty sighted people participated in this experiment (22 ± 7 years old). Participants
were associated in ten dyads, in which one was the guide for the voice guidance
condition, the other was the participant having to reach the targets in both guidance
conditions. For simplicity's sake, we will call them guides and participants respectively.
Their role was determined randomly, by drawing lots.

All participants and guides were psychology students and received a bonus for their grade on an experimental psychology topic course in exchange for their participation.
All participants and guides gave their informed written consent prior to participating in the study. This work was authorized by the ethical committee CER Grenoble Alpes (Avis-2018-06-19-1).

Participants were recruited between December 9, 2024, and December 20, 2024.
<table><caption>**Fig 4. Visualization screen of the virtual environment containing the target.**
The guide visualizes the scene from the point of view of the participant's finger. The orange sphere represents the target.</caption><tr><td><strong>Engineering</strong></td><td>522</td></tr><tr><td>For the voice guidance condition, the experimental protocol was implemented using a Python program, on a second computer, which managed target generation, validation and microphone recording. An Unreal Engine program enabled the guide to visualize the virtual environment (see Fig. 4), while a C++ program recorded data from the torso and the finger position. Communication between these components was facilitated by the Open Sound Control (OSC) protocol, which transmitted information about the targets, including their positions, dimensions and acquisition status.</td><td>523<br>524<br>525<br>526<br>527<br>528<br>529</td></tr><tr><td>The guide visualized the virtual environment containing the target, as if a camera's viewpoint was fixed on the participant's finger, ensuring that the finger's rotation did not affect camera orientation. The guide operated with his back to the motion capture area, with no communication other than the microphone connected to the participant's headphones. The camera's viewing angle, with a field of view (FOV) of 82°, was determined by the angular deviation between the torso and the finger.</td><td>530<br>531<br>532<br>533<br>534<br>535<br>536<br>537</td></tr><tr><td>For the sequential sonification condition, the system was as described in the joint Material and Methods section.</td><td>538</td></tr><tr><td><strong>Guide's equipment</strong></td><td>539<br>540</td></tr><tr><td>The participant's equipment was the same as in the previous experiment in both conditions (voice and sonification)</td><td>541<br>542<br>543<br>544</td></tr><tr><td>The guide (operating in the voice guidance condition) was equipped with a microphone (Boya BY-M1) to communicate with the participant, allowing them to provide verbal instructions. The participant had no mean of responding or communicating back to the guide.</td><td>545</td></tr><tr><td><strong>Protocol</strong></td><td>546<br>547</td></tr><tr><td>The protocol was broadly similar to that of the first experiment, described in the joint Materials and Method section.</td><td>548</td></tr></table>
Instructions given to the guided participant were the same as for the previous experiment. The guide was informed that they had to direct the participant towards a 3D position in a virtual world, indicated by an orange sphere in the vizualized virtual environment. It was specified that the point of view was located at the tip of the participant's finger. It was mentioned that the target could be positioned behind the participant at the beginning of the trial.

As for the first experiment, participants did a familiarization phase, training blocks and then test blocks. For the voice guidance, the main difference for the participant was that it was the guide who launched the trial by pressing a key. All indications concerning the start of a trial, reaching the target or reaching the time limit and the end of the block were given by the guide. Apart from that, the course of the block was exactly the same.

**Ergonomics questionnaire:** As for the previous experiment, participants filled a questionnaire that required to give ergonomics scores to both guidance conditions, on three scales from 1 to 7 for efficiency, ease of use and comfort.

## Analyses

**Participants' performance:** Before running the Cox analysis, data were cleaned by removing target-reaching times greater than the mean plus two standard deviations (the voice corpus of these trials was kept). This represented 3.3% of the participants' data.

**Guides' voice commands:** we analyzed the audio recordings of the guides' voice commands, to determine whether these commands are sequential. If they are sequential, there should be a majority of commands indicating a single direction (right, left, up, down, etc.) and few commands indicating several directions at once (up to right, front to left, etc.). We used Whisper software to transcribe the audio recordings. Whisper [56] is an automatic speech recognition (ASR) system developed by OpenAI. For alignment and sentence segmentation, it dynamically segments transcriptions based on pauses, punctuation, and syntactic structure. It also removes hesitations and filler pauses. We then drew on the vocabulary present in the state of the art to list keywords that could be used to categorize commands in one (or more) direction according to the keywords they contain [52–55]. These are the direction categories, and the keywords that determine if a command belong in each category:
* forward: [avance (*move forward*), tout droit (*in front*), devant (*straight*), en face (*facing*)],

* backward: [recule (*move back*), derrière (*behind*)],

* left: [gauche (*left*)],

* right: [droite (*right*)],

* down: [bas (*down*), descend (*lower*)],

* up: [haut (*up*), monte (*go up*), lever (*raise*)]

These keywords express movements according to the 3 translations in 3D space.
Some voice commands do not correspond to any direction category: in this case, they
are categorized as “other” (commands like “a little bit more”, “stop”, etc.). We will
describe these other commands in the results section.

## Results

### Target-reaching time

Target-reaching times: participants were significantly faster in the sequential
sonification condition ($M = 24.38$ s ; $SD = 15.40$ s) than in the voice condition ($M =
34.56$ s ; $SD = 12.89$ s) ($z = -5.84, p < .001$) (see Fig 5).

### Ergonomics scores

Results of the ordinal regression revealed no preference for one or the other condition.
Scores given by the participants for each condition are represented in Fig 6.

### Voice commands

Analysis of the voice commands (see Fig. 7) shows that out of a total of 1470
statements that have been segmented and transcribed using Whisper, 53.9% comprised
keywords expressing movement in at least one of the 6 directions.

A vast majority (89.3%) of these statements involving movement instructions
comprised only one direction, confirming the sequential hypothesis even in spontaneous
<figure><img src="image_6.png" /><figcaption>Fig 5. Box plots of target-reaching times (log10 scale) for each participant under the two modalities: sonification and voice guidance.</figcaption></figure>

Each box represents the median (horizontal line), the interquartile range (box), and the extreme outlier values (whiskers).

<figure><img src="image_7.png" /><figcaption>Fig 6. Bar chart showing the distribution of participant ratings (1 to 7) across the two guidance conditions, sonification and voice guidance.</figcaption></figure>

All three criteria are included (efficiency, ease, comfort). The color gradient represents score levels, with orange for 3 and dark green for 7 (no participant gave a score below 3), so that greener areas represent more positive judgments.
<figure><img src="image_8.png" /><figcaption>Fig 7. Percentage of statements in each direction category.</figcaption></figure>

0 corresponds to statements with no directions category (e.g. "stop", "again", etc.). 1 is for statements containing only one of the six directions category (e.g. "go forward", "a little bit to the left", etc.). 2 is for statements containing two directions category (e.g. "move to the top right", etc.). The remaining part represents segmentation errors of Whisper.

guidance by naive persons. Only a few combine two directional instructions such as "to the left and down" or "at the very bottom right".

The remaining 69 statements were classified as expressing three or more movements instructions. A manual verification of all theses statements reveals that this classification was due to segmentation errors of Whisper with the aggregation of several subsequent instructions, although long silent pauses were clearly present in the recording sound. Absolutely all these 69 statements were due to errors during the speech segmentation and transcription process.

It is worth noting the large amount of statements that express no directional instruction as defined by our list of keywords. Among these 0-dimensional statements, about 40% corresponded to simple commands such as "stop". The remainder mainly consisted of reiterations (such as "again" or "more"), degree adverbs (like "a little bit"), or validation expressions (such as "ok", "perfect", or "great").

For the same classification, the formulations varied between participants. For example, for the direction category forward: "move forward", "straight ahead", "move forward a little". For validation statements: "great", "ok", "perfect", "it's fine". Fig 8 represents the distribution of statements based on their occurrence. We can see a great proportion of statements are unique: the lowest (right) part of the graph indicates that a large proportion of the different statements are said only once. The command "stop"
<figure><img src="image_9.png" /><figcaption>Fig 8. Cumulative distribution of repeated statements based on their occurrence, tagged by category of movement.</figcaption></figure>

Each point represents one statement, with its position on the y-axis corresponding to the number of times it is repeated in the dataset (10° are statements said only once). The x-axis represents the order of statements from most to least frequent (10° is the statement repeated the most times). The color of each point indicates the pre-assigned direction category of the statement. The anteroposterior axis (in red) contains the direction categories "forward" and "backward". The mediolateral axis (in blue) contains the direction categories "left" and "right". The vertical axis (in green) contains the direction categories "up" and "down". The category "other" (in grey) contains diverse statements without directions like "stop", reiterations ("again"), degree adverbs ("more"), etc. The labels contain the statement and the number of times it appears. For example: "a little to the left" appears a total of 17 times (ordinate), ranking 8th among the most frequent statements (abscissa).

was the one that was repeated the most, and it was said by all guides.

Discussion

In terms of performance, the analysis of target-reaching times shows a great advantage for sequential sonification over voice guidance by a human guide. This difference can be explained by the fact that voice guidance is much slower [24]. By the time the instruction statement is given, it may no longer be valid: for example, the time to say "go to the right", the participant is already too far.

Analysis of the guides' voice commands shows that 50% of all statements (and 97.4% of those containing directions) include only a single category of direction. This confirms
our hypothesis that a guide who has not received specific instructions on how to guide a person towards a target will spontaneously use a sequential mode of guidance, as in the vast majority of studies that use predefined verbal instructions [21, 22, 25, 34, 35, 39]. The better performance with sonification than with voice guidance is therefore not due to the fact that our sonification is sequential (in contrast to the difference in performance observed with parallel sonification), since voice guidance is sequential too.

Sonification is much faster and more accurate. The use of a continuously varying pitch allows very precise guidance due to the auditory system's great ability to discriminate between pitches [31]. It is therefore easy for the participant to determine the exact direction of the target. The fact that the frequency varies continuously allows the participant to know immediately if they have deviated from the correct direction when moving towards the target, and to correct the direction quickly if necessary. In contrast, with voice guidance, the participant does not have continuous feedback, so if they deviate from their trajectory while moving towards the target, they will not know until the next command from the guide. As detailed later, this explains the most frequent use of the commands "stop", "again", "ok stop", etc.

The use of stereo allows the participant to know immediately and intuitively whether the target is on the right or the left when the test begins. The use of discrete stereo (right, left or straight ahead) overcomes the imprecision of the auditory system in locating a sound source [28-30] that could be found using HRTF. There is therefore little chance of confusion between right and left.

Apart from the directional voice commands, the guides unanimously used the "stop" command, a quick way of announcing the end of the previous directional command and making the transition to the next command. This stop can be compared with the transition from one sound to another when changing dimension in sequential sonification, which is quicker than a "stop" voice command. "Stop" is also a sequentiality indicator. We do not say "move forward" and then "go up", which would cause you to continue moving forward while going up. Instead, we say "move forward", "stop", "go up". Participants also used a lot of degree adverbs to indicate the amount of movement to be made and thus avoid overshooting the target position on one dimension or another. In our sonification, tempo and pitch indicated the amount of movement required to correct the deviation in direction and distance respectively. Previous studies
showed that sonification was more effective at conveying the amount of movement 664
required to reach the target [34]. The height coding indicated the direction (up, down) 665
but not the distance, which is not a problem because the vowel change from one side of 666
the target to the other quickly indicates the height of the target, and makes it possible 667
to change direction quickly if the person exceeds the correct height. 668

In terms of perceived satisfaction, ordinal regression analysis of the scores did not 669
reveal a preference for voice guidance or guidance by sonification, for any of the three 670
criteria (efficiency, ease and comfort). The advantage of sonification in terms of speed of 671
target-reaching was not enough to tip the balance in its favor in terms of the 672
participants' feelings. The more natural and intuitive nature of the voice commands 673
may have earned the preference of some participants, despite their poorer performance. 674
Indeed, both the scores given by the participants and their qualitative feedback reflected 675
a preference: those who favored one condition generally gave it slightly higher scores, 676
even if the difference was not large. Among the 10 participants, 4 preferred the 677
sonification condition, 5 preferred the voice guidance condition, and only 1 gave 678
identical scores to both and did not express a preference. These preferences were 679
reflected in the discussions with the experimenter. Opinions were therefore divided. 680
Compared to the simplicity of voice commands, which require little to no learning for 681
the participant, sonification may have seemed more complex and time-consuming to 682
master to some participants, despite the better performances. 683

## Conclusion

This second experiment showed that sequential sonification is more effective than voice 684
guidance, although participants did not give better ergonomics scores to this condition 685
on average. Analysis of the voice commands assessed that spontaneous voice guidance is 686
also sequential. It is therefore not sequentiality that explains the difference in 687
performance between the two guidance conditions, but rather the speed of sonification 688
guidance thanks to its continuous and immediate feedback. 689
General conclusion

These two experiments highlighted the advantages of guidance by sequential sonification.
Compared with parallel sonification, sequential sonification enables faster, more
intuitive navigation, reducing cognitive load. Compared with the voice guidance of a
human guide, sonification proved to be more precise and effective, thanks to its
continuous and immediate feedback. The fact that the sonification condition was
sequential could have explained the difference in performance between voice guidance
and sonification guidance, since as we saw in the first experiment, sequential
sonification guidance is more effective than parallel sonification guidance. But voice
instructions were showed to also be sequential, so it does not explain the difference in
performance. Although intuitive, voice instructions proved to be limited by their
latency and lack of continuous feedback.

Our sequential sonification combines the speed and precision of non-spatialized
sonification with the intuitiveness and naturalness of spatialized sonification (stereo)
and voice guidance (sequential commands). This sonification adapts to both the needs
of the task and the cognitive constraints of users. This work paves the way for practical
applications of guidance by sonification towards a target for people with disabilities,
enabling greater autonomy than guidance by a third human party.

# 1 Data Availability Statements

References

1. Maidenbaum S, Chebat DR, Levy-Tzedek S, Amedi A. Depth-to-audio sensory substitution for increasing the accessibility of virtual environments. In: Universal Access in Human-Computer Interaction. Design and Development Methods for Universal Access: 8th International Conference, UAHCI 2014, Held as Part of HCI International 2014, Heraklion, Crete, Greece, June 22-27, 2014, Proceedings, Part I 8. Springer; 2014. p. 398–406.

2. Lenay C, Gapenne O, Hanneton S, Marque C, Genouëlle C. Chapter 16. Sensory substitution: Limits and perspectives. In: Touching for knowing: Cognitive
psychology of haptic manual perception. John Benjamins Publishing Company;
2006. p. 275–292.

3. Bach-y Rita P. Sensory plasticity: Applications to a vision substitution system. Acta Neurologica Scandinavica. 1967;43(4):417-426.
doi:10.1111/j.1600-0404.1967.tb05747.x.

4. Bach-y Rita P, Collins CC, Saunders FA, White B, Scadden L. Vision substitution by tactile image projection. Nature. 1969;221(5184):963-964.
doi:10.1038/221963a0.

5. White BW, Saunders FA, Scadden L, Bach-Y-Rita P, Collins CC. Seeing with the skin. Perception & Psychophysics. 1970;7:23-27. doi:10.3758/BF03210126.

6. Bach-y Rita P. Tactile sensory substitution studies. Annals of the New York Academy of Sciences. 2004;1013(1):83-91. doi:10.1196/annals.1305.006.

7. Meijer PB. An experimental system for auditory image representations. IEEE transactions on biomedical engineering. 1992;39(2):112-121.
doi:10.1109/10.121642.

8. Cronly-Dillon J, Persaud K, Gregory R. The perception of visual images encoded in musical form: a study in cross-modality information transfer. Proceedings of the Royal Society of London Series B: Biological Sciences.
1999;266(1436):2427-2433. doi:10.1098/rspb.1999.0942.

9. Abboud S, Hanassy S, Levy-Tzedek S, Maidenbaum S, Amedi A. EyeMusic: Introducing a “visual” colorful experience for the blind using auditory sensory substitution. Restorative neurology and neuroscience. 2014;32(2):247-257.
doi:10.3233/RNN-130338.

10. Capelle C, Trullemans C, Arno P, Veraart C. A real-time experimental prototype for enhancement of vision rehabilitation using auditory substitution. IEEE Transactions on Biomedical Engineering. 1998;45(10):1279-1293.
doi:10.1109/10.720206.
11. Auvray M, Hanneton S, Lenay C, O'Regan K. There is something out there: distal attribution in sensory substitution, twenty years later. Journal of integrative neuroscience. 2005;4(04):505-521. doi:10.1142/S0219635205001002.

12. Hanneton S, Auvray M, Durette B. The Vibe: a versatile vision-to-audition sensory substitution device. Applied Bionics and Biomechanics. 2010;7(4):269-276. doi:10.1080/11762322.2010.512734.

13. Stoll C, Palluel-Germain R, Fristot V, Pellerin D, Alleysson D, Graff C. Navigating from a depth image converted into sound. Applied bionics and biomechanics. 2015;2015(1):543492. doi:10.1155/2015/543492.

14. Ward J, Meijer P. Visual experiences in the blind induced by an auditory sensory substitution device. Consciousness and cognition. 2010;19(1):492-500. doi:10.1016/j.concog.2009.10.006.

15. Kilian J, Neugebauer A, Scherffig L, Wahl S. The unfolding space glove: A wearable spatio-visual to haptic sensory substitution device for blind people. Sensors. 2022;22(5):1859. doi:10.3390/s22051859.

16. Griffin-Shirley N, Banda DR, Ajuwon PM, Cheon J, Lee J, Park HR, et al. A survey on the use of mobile applications for people who are visually impaired. Journal of Visual Impairment & Blindness. 2017;111(4):307-323. doi:10.1177/0145482X1711100402.

17. Senjam SS, Manna S, Bascaran C. Smartphones-based assistive technology: accessibility features and apps for people with visual impairment, and its usage, challenges, and usability testing. Clinical optometry. 2021; p. 311-322. doi:10.2147/OPTO.S336361.

18. Katz BF, Kammoun S, Parseihian G, Gutierrez O, Brilhault A, Auvray M, et al. NAVIG: augmented reality guidance system for the visually impaired. Virtual Reality. 2012;16(4):253-269. doi:10.1007/s10055-012-0213-6.

19. Liu Y, Stiles NR, Meister M. Augmented reality powers a cognitive assistant for the blind. ELife. 2018;7:e37841. doi:10.7554/eLife.37841.001.
20. Thakoor K, Mante N, Zhang C, Siagian C, Weiland J, Itti L, et al. A system for assisting the visually impaired in localization and grasp of desired objects. In: European Conference on Computer Vision. Springer; 2014. p. 643–657.

21. Hild M, Cheng F. Grasping guidance for visually impaired persons based on computed visual-auditory feedback. In: 2014 International Conference on Computer Vision Theory and Applications (VISAPP). vol. 3. IEEE; 2014. p. 75–82.

22. Manduchi R, Coughlan JM. The last meter: blind visual guidance to a target. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; 2014. p. 3113–3122.

23. Shih ML, Chen YC, Tung CY, Sun C, Cheng CJ, Chan L, et al. Dlwv2: A deep learning-based wearable vision-system with vibrotactile-feedback for visually impaired people to reach objects. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE; 2018. p. 1–9.

24. May KR, Sobel B, Wilson J, Walker BN. Auditory displays to facilitate object targeting in 3D space. In: The 25th International Conference on Auditory Display (ICAD 2019). Georgia Institute of Technology; 2019. p. 155–162.

25. Zientara PA, Lee S, Smith GH, Brenner R, Itti L, Rosson MB, et al. Third eye: A shopping assistant for the visually impaired. Computer. 2017;50(2):16–24. doi:10.1109/MC.2017.36.

26. Proulx MJ, Gwinnutt J, Dell'Erba S, Levy-Tzedek S, De Sousa AA, Brown DJ. Other ways of seeing: From behavior to neural mechanisms in the online “visual” control of action with sensory substitution. Restorative neurology and neuroscience. 2015;34(1):29–44. doi:10.3233/RNN-150541.

27. Parseihian G, Gondre C, Aramaki M, Ystad S, Kronland-Martinet R. Comparison and evaluation of sonification strategies for guidance tasks. IEEE Transactions on Multimedia. 2016;18(4):674–686. doi:10.1109/TMM.2016.2531978.

28. Middlebrooks JC. Sound localization. Handbook of clinical neurology. 2015;129:99–116. doi:10.1016/B978-0-444-62630-1.00006-8.
29. Ziemer T, Schultheis H. Psychoacoustical signal processing for three-dimensional sonification. In: The 25th International Conference on Auditory Display (ICAD 2019). Georgia Institute of Technology; 2019. p. 277-284.

30. Carlini A, Bordeau C, Ambard M. Auditory localization: A comprehensive practical review. Frontiers in Psychology. 2024;15:1408073.
doi:10.3389/fpsyg.2024.1408073.

31. Zwicker E, Fastl H. Psychoacoustics: Facts and models. vol. 22. Springer Science & Business Media; 2013.

32. Song Q, Yao N, Feng Z, Chen H, Wang H. Combining spatial and non-spatial sonification for sound localization in virtual environment: a subjective study. In: CHI Conference on Human Factors in Computing Systems Extended Abstracts; 2022. p. 1-7.

33. Rusconi E, Kwan B, Giordano BL, Umilta C, Butterworth B. Spatial representation of pitch height: the SMARC effect. Cognition. 2006;99(2):113-129.
doi:10.1016/j.cognition.2005.01.004.

34. Ahmetovic D, Sato D, Oh U, Ishihara T, Kitani K, Asakawa C. Recog: Supporting blind people in recognizing personal objects. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems; 2020. p. 1-12.

35. Troncoso Aldas ND, Lee S, Lee C, Rosson MB, Carroll JM, Narayanan V. AIGuide: An Augmented Reality Hand Guidance Application for People with Visual Impairments. In: The 22nd International ACM SIGACCESS Conference on Computers and Accessibility; 2020. p. 1-13.

36. Lokki T, Grohn M. Navigation with auditory cues in a virtual environment.
IEEE MultiMedia. 2005;12(2):80-86. doi:10.1109/MMUL.2005.33.

37. Lock JC, Gilchrist ID, Gilchrist ID, Cielniak G, Bellotto N. Experimental analysis of a spatialised audio interface for people with visual impairments. ACM Transactions on Accessible Computing (TACCESS). 2020;13(4):1-21.
doi:10.1145/3412325.
38. Guarese R, Pretty E, Renata A, Polson D, Zambetta F. Exploring audio interfaces for vertical guidance in augmented reality via hand-based feedback. IEEE Transactions on Visualization and Computer Graphics. 2024;doi:10.1109/TVCG.2024.3372040.

39. Hu X, Song A, Wei Z, Zeng H. StereoPilot: A wearable target location system for blind and visually impaired using spatial audio rendering. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2022;30:1621–1630.
doi:10.1109/TNSRE.2022.3182661.

40. Fons C, Huet S, Pellerin D, Gerber S, Graff C. Moving Towards and Reaching a 3-D Target by Embodied Guidance: Parsimonious Vs Explicit Sound Metaphors. In: International Conference on Human-Computer Interaction. Springer; 2023. p. 229–243.

41. Fons C, Huet S, Pellerin D, Graff C. Using Fitts' Law to Compare Sonification Guidance Methods for Target Reaching Without Vision. In: 19th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications-HUCAPP. SCITEPRESS-Science and Technology Publications; 2024. p. 444–454.

42. Fons C, Huet S, Pellerin D, Graff C. Dissociate Dimensions to Improve Sound Metaphors Guiding the Hand Towards 3D Non-visible Targets. SN Computer Science. 2025;6(4):387.

43. Ziemer T, Schultheis H. A psychoacoustic auditory display for navigation. In: The 24th International Conference on Auditory Display (ICAD 2018). Georgia Institute of Technology; 2018. p. 136–144.

44. Ziemer T, Nuchprayoon N, Schultheis H. Psychoacoustic sonification as user interface for human-machine interaction. arXiv preprint arXiv:191208609.
2019;doi:10.48550/arXiv.1912.08609.

45. Ziemer T, Schultheis H. Three orthogonal dimensions for psychoacoustic sonification. arXiv preprint arXiv:191200766.
2019;doi:10.48550/arXiv.1912.00766.
46. Ziemer T, Schultheis H. Pampas: A psychoacoustical method for the perceptual analysis of multidimensional sonification. Frontiers in Neuroscience. 2022;16:930944.

47. Ziemer T. Three-dimensional sonification as a surgical guidance tool. Journal on Multimodal User Interfaces. 2023;17(4):253-262.

48. Dubus G, Bresin R. A systematic review of mapping strategies for the sonification of physical quantities. PloS one. 2013;8(12):e82491.
doi:10.1371/journal.pone.0082491.

49. Guezou-Philippe A, Huet S, Pellerin D, Graff C. Prototyping and evaluating sensory substitution devices by spatial immersion in virtual environments. In: VISIGRAPP 2018-13th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications. SCITEPRESS-Science and Technology Publications; 2018. p. 596-602.

50. Stevens S. On the physiological law. Psychol Rev. 1957;64:153-181.

51. Letué F, Martinez MJ, Samson A, Vilain A, Vilain C. Statistical Methodology for the Analysis of Repeated Duration Data in Behavioral Studies. Journal of Speech, Language, and Hearing Research. 2018;61(3):561-582.
doi:10.1044/2017JSLHR-S-17-0135.

52. Contreras R, Ayala A, Cruz F. Unmanned Aerial Vehicle Control through Domain-Based Automatic Speech Recognition. Computers. 2020;9(3):75.

53. Choutri K, Lagha M, Meshoul S, Batouche M, Kacel Y, Mebarkia N. A Multi-Lingual Speech Recognition-Based Framework to Human-Drone Interaction. Electronics. 2022;11(12):1829.

54. Fayjie AR, Ramezani A, Oualid D, Lee DJ. Voice enabled smart drone control. In: 2017 Ninth International Conference on Ubiquitous and Future Networks (ICUFN). Milan: IEEE; 2017. p. 119-121.

55. Poncela A, Gallardo-Estrella L. Command-Based Voice Teleoperation of a Mobile Robot via a Human-Robot Interface. Robotica. 2015;33(1):1-18.
56. Radford A, Kim JW, Xu T, Brockman G, McLeavey C, Sutskever I. Robust Speech Recognition via Large-Scale Weak Supervision. Preprint. 2023;doi:10.48550/arXiv.2212.04356.