Proceedings of ISon 2016, 5th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 2016

INTERACTIVE SONIFICATION OF THE U-DISPARITY MAPS OF 3D
SCENES
Piotr Skulimowski, Mateusz Owczarek, Andrzej Radecki, Michał Bujacz, Paweł Strumiłło
Lodz University of Technology
Lodz, Poland
piotr.skulimowski@p.lodz.pl
ABSTRACTIn this paper we propose a method for real-time, interactive
auditory representation of a 3D scene’s geometric structure bysonifying its U-disparity maps. The U-disparity is derived fromthe depth map obtained from stereovision imaging of 3D
scenes, and can be interpreted as a bird’s eye view of a scene
with highlighted scene objects. The user can interactively selectthe region of the U-disparity map for sonification. Sucha representation allows the user to effortlessly identify distanceand angular direction to potential obstacles. The prototype
application was tested by three blind users, who managed to
localize key objects in the sonified 3D indoor and outdoor
scenes.
1.

INTRODUCTION

The visually impaired people indicate limited mobility as the
major problem affecting almost all activities of daily living. The
research efforts aimed at building Electronic Travel Aids (ETA)
date back to the nineteenth century, when in 1897 Polishophthalmologist
Kazimierz
Noiszewski
constructed
Elektroftalm a device termed “electronic eye” that converted
light into sounds or vibrations by using the photoelectric
properties of Selenium cells. Although, too heavy for practicalapplication, it is considered the first electronic sonificationinterface for the visually impaired [1]. Further attempts were
pioneered by Bach-y-Rita [2], who built a number of ETA
prototypes for the blind that used tactile modality.
Dynamic development of Information and Communications
Technologies (ICT) at the turn of centuries (100 years after
seminal efforts by Noiszewski) have opened new prospects fordesigns of personal aids helping blind people in mobility (laser
and ultrasound detectors) and navigation (GPS). With regard of
the non-visual methods used for presentation of informationthese devices can be subdivided into haptic interfaces andauditory interfaces. An excellent review of wearable obstacle
avoidance ETAs is given in [3]. Due to size factor and cost, inthe majority of ETAs, the auditory displays are favoured as
opposed to haptic interfaces that require complex circuitry to
implement haptic stimulations. There are many possible
auditory representations of information than can be employed inhuman-machine interfaces (HMI were widely reviewed in [4]).
However, sonification, i.e., non-speech audio, is the method
which is predominantly used for “displaying” the environment
to the visually impaired. Quite a comprehensive review of thesonification methods devised for aiding the blind in mobility
and travel is given in [5]. Here it is worth mentioning the vOICe(www.seeingwithsound.com), a widely popularized method forsonifying monochrome images. The sonification method is
simple, however, not too intuitive and requires many weeks oftraining. The vertical coordinate of every pixel corresponds to a

specific pure-tone frequency in the range of 500 Hz (bottom
image pixels) to 5 kHz (top image pixels), whereas, loudness ofthe frequency is reflecting the local brightness of the image.
Such a sonification code is used in a repetitive, one second long,auditory representation of the image that is scanned from left toright. Such a sonification scheme is non-interactive and difficult
for the user to control.
In a recent decade an important subfield of sonification has
emerged, namely: interactive sonification [6]. In such an
approach to human-computer auditory interface the user iscapable of interacting with the sonification process, e.g. define
an image region to be sonified or tune sonification parameters
to individual requirements.
In this paper we demonstrate how the technique of interactive
sonification can be applied in a simple interface for the blindwith an aim to represent spatial 3D geometry of theenvironment. We use the so-called depth images and their
histograms termed “U-disparity” representation of the disparity
map obtained from a stereovision system sensing of theenvironment. The U-disparity maps are locally sonified in
response to the user’s tactile exploration of such an image
model of the environment.
2.

STEREOVISION BASICS

2.1. Disparity map calculation
Stereovision is a passive 3D reconstruction method fromtwo (or more) images of the same scene captured from different
locations in space. The main advantage of this approach is thatit does not need any active lighting and efficient algorithms forreconstructing 3D scene geometry are well developed. After
applying a proper calibration procedure of the stereovisionsystem, the depth map representing a 3D structure of the imaged
scenes can be reliably calculated. For a calibrated stereovisionsystem the disparity (parallax between left and right image) can
be computed as d = xl–xr, where xl and xr are the coordinates ofthe pixels in the left and right stereovision image being the
projections of the same point in space. Having calculated thedisparity values for the entire image, depth of scene points canbe calculated as:
𝐵𝑓

𝑍= 𝑑

(1)

where B is the distance between optical centres of the cameras, f
is the focal length of the camera. Fig. 1 shows distance as afunction of disparity for the selected cameras. Fig. 2 shows theleft image captured by the Bumblebee stereovision camera. The
corresponding depth map is shown in pseudo-colours (the closer
the scene point to the camera the warmer the colour) and agreyscale representation is used for displaying the U-disparity
map (explained in the next section).

18
ISon2016-1

Proceedings of ISon 2016, 5th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 2016

2.2. “U-disparity” representation

Figure 1. Distance Z(d) as a function of disparity for
different stereo-cameras featuring different image
resolution and focal lengths: BB-Bumblebee stereo
camera, DUO-Duo MLX stereo camera.

Literature review shows that the U-disparity representationof the environment from stereovision can be very effective in
obstacle detection for automotive and autonomous robots
applications [7,8,9]. The U-disparity representation is built fromthe depth map by computing histograms of consecutive columnsof the depth map. Let the reference image (and disparity image)has pixel resolution w×h. The size of the U-disparity map is
w×dmax, where dmax is the maximum allowed disparity value.
Thus the value of each point u(x, d) in the U-disparity map isthe number of scene points at x-coordinate assuming disparity d.
The U-disparity map turns out to be a very efficient
representation for localizing scene obstacles (provided thestereovision camera base is parallel to the ground plane [10]).
An obstacle located at a well localized distance usually features
very many pixels in the disparity map of the same value, which
results in high pixel values in the U-disparity map. An exampleof the U-disparity map for the outdoor scene is shown in thebottom image of Fig. 2. Note that key obstacles, the pole on theleft and high grass on the right and left are clearly highlighted in
the U-disparity map.
The justification for using the U-disparity map for scene
sonification is the representation in which vertical direction
denotes depth of scene objects and horizontal direction theazimuth angle of the objects. The U-disparity map is mucheasier for tactile exploration for the visually impaired user thanthe depth map, in which vertical direction of the map cannot be
associated with depth of scene points (due to different possible
positioning of the stereovision camera versus the environment).
3.

SYSTEM ARCHITECTURE

A simple schematic of the proposed interactive sonification
system is shown in Fig. 3.

Figure 3. A simplified architecture of an electronicsystem for interactive sonification of 3D scenes.
Stereovision camera is mounted on a user’s head and isconnected to the embedded platform via a WiFi router (Fig. 4).

Figure 2. An image recorded by the Bumblebee
stereovision camera (top), the corresponding depth map
shown in pseudo-colours (centre) and thecorresponding U disparity map (bottom).

Figure 4. Duo MLX stereovision system on a custommade “glasses”

19
ISon2016-2

Proceedings of ISon 2016, 5th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 2016

The embedded platform (NVIDIA Jetson TK1) is battery
powered and attached to a special belt. Image preprocessing
procedures and disparity map calculation algorithms are runningon the embedded platform. An Android phone is connected tothe acquisition system via a Wi-Fi network and the depth map
data is transmitted to the mobile phone. The U-disparity map iscalculated on the Android-based platform. The mobile phone
can be hidden in the user’s pocket. The user touches the screen
and the selected region of the U-disparity map is sonified. The
sonification output stream comes from the mobile phonethrough the speaker or stereo headphones.
4.

INTERACTIVE SONIFICATION
OF THE U-DISPARITY MAPS

5.

IMPLEMENTATION AND TESTS

5.1. Implementation details on the Android platform

The blind user can select a scene area for sonification bytouching the mobile phone screen on the panel displaying the
U-disparity map (see Fig. 5). Let x denote a column of the map
indicated by the user. The depth map is displayed in the current
version of the application for verification purposes. In therelease version of the application the U-disparity map will bescaled-up to the full screen width of the mobile device.
Touching the centre of the map gives information about the
obstacles in front of the user. The sound sonifying the scene
depends on the content of the U-disparity map. The indicatedcolumn x of the map controls left-right panning of the generated
sound:
𝑎𝐿𝑖 = 1 − 𝑥/𝑤

d_off+N, i.e. it corresponds to the all closest objects (please see
the example in Fig. 6 and region A indicated by asterisk).
Constants d_off and N in the proposed method depend on thegeometric parameters of the selected camera (see Fig. 1) so thata proper control of the sonified depth range can be specified.
The f_min and f_max values were selected to match technicallimitation of the speakers built-in into mobile devices that wereused for tests.

(2)

𝑎𝑅𝑖 = 𝑥/𝑤

An Android application implementing the describedsonification concept acts as a remote control to the system. The
application enables to set selected reconstruction parameterslike type of disparity calculation algorithm and its parameterssuch as window size, LED brightness level or confidence
parameters for disparity map calculations. Fig. 5 shows thesetting panel for the Duo MLX stereovision camera.
For the test purposes the current application also enables torecord image sequences with corresponding timestamps for later
re-play and analysis. Data transfer between the embedded
platform and the mobile phone is provided by the WebSocket
protocol, featuring full-duplex communication and a TCP
connection. The advantages of using the mobile instead of a
dedicated device is its relatively low price and its common use
among blind persons.

where aLi, aRi are amplitudes (volumes) of the output left andright channels of the i sound component.
It is worth noting that such a sonification method of the
obstacle horizontal position (left/right panning) is very
simplified and it is related to the depth values instead of world
coordinates.
The row in the U-disparity map (i.e. the disparity value) in
the sonified range determines the sound frequency that codes
the depth information (the higher the pitch the closer thesonified object). The sound signal generated by the system is asum of sinusoids:st§i N 1§¦ a sin ¨¨ 2S ¨© f
i

i 0

©

min

i

f max  f min · ·¸t ¸N  1 ¹ ¸¹

(3)Each sinusoid frequency represents the selected distance
range. Distance ranges are linked to the discreet values of thedisparity map. It was decided to sonify only objects which
distance (Z coordinate) from the camera is below a predefinedvalue Z_max(d_off) corresponding to disparity d_off. We define
N as the number of different sound frequencies, f_min is thefrequency of sound with index 0, and f_max is a frequency ofsound N-1 which corresponds to the closest objects. Then theamplitude of each sinusoid is calculated as:𝑎𝑖 =𝑢[𝑥,𝑑𝑜𝑓𝑓+1+𝑖]ℎfor 𝑖 < 𝑁 − 1𝑑𝑚𝑎𝑥 −𝑑𝑜𝑓𝑓𝑎𝑁−1 =∑𝑗=𝑁(4)
𝑢[𝑥,𝑑𝑜𝑓𝑓 +𝑗]ℎwhere h is the number of rows of the disparity map. It can benoticed that the highest frequency sound source f_max
represents all objects for which disparity d is larger or equal to

Figure 5. Depth map of the corridor scene with two
obstacles (top-left panel) recorded with the Duo MLX
stereovision camera and the corresponding U-disparity
map (bottom-left). The depth map is coded using
pseudo-colours. The control panel (right) allows toadjust both camera settings and disparity calculation
parameters.
5.2. Tests of the proposed sonification methods
First tests with the blind users, to ensure the repeatability,
have been carried out using the pre-recorded test sequences.
The sequences were recorded by the DUO MLX stereovision
camera, which is used as the main camera in the systemprototype. For the development purposes we used the Point
Grey colour stereovision camera which is less portable due toits size, but produces much better quality of the images and wasused in the tests of the proposed sonification method. The tests
were carried out with 3 blind and 2 sighted persons on thechosen outdoor image sequences. All testers were familiar with20
ISon2016-3

Proceedings of ISon 2016, 5th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 2016

the mobile phones with touch screens and they were instructed
how the U-disparity map is generated and acquainted with thesonification method of the U-disparity data.

finding and identifying the location of the obstacles and indescribing spatial features of the sonified scene. For short time
sessions the generated sounds were acceptable for the users, butthey noted that for longer sessions listening to such sounds
would be tiring.

Figure 6. Offline tests with the use of Android OS tablet.
The test scene was captured using the Bumblebee
camera. Original reference image is shown in Fig. 7

Figure 7. Outdoor scene used in tests with the blindusers captured using the Bumblebee camera.
Fig. 6 illustrates how the trials were conducted. The users
were asked to verbally describe the scene based on the soundgenerated by the mobile device. All users correctly found the
obstacles and were able to state, which object is closer to thecamera. They were also able to find and indicate directions
corresponding to scene spaces devoid of obstacles. It must be
noted here that the type of obstacles (e.g. tree, bench, lamp)
cannot be communicated to the blind user by means of thisscene representation scheme. The user, however, can determine
the size, distance and direction of an obstacle. These obstaclefeatures are important for safe mobility in a sonified scene.
Fig. 8 shows spectra of the sounds generated for theselected columns of the U-disparity map. The sounds wererecorded from the mobile device using the phone output. It canbe noticed that frequency components correspond to the
obstacles’ disparity values. For region A, a dominant single
frequency results from the fact that objects which are very close
to the camera (i.e. their disparity values are greater or equal tod_off+N) are coded using a single frequency component (fmax),which corresponds to the closest object. Please visit
http://eletel.p.lodz.pl/pskul/ison2016 to watch a short video ofthe proposed method.
The users were asked to express their opinion about the
proposed sonification method. They reported no problems in

Figure 8. Fourier spectra of the sounds generated forregions A, B, C of the U-disparity map (see Fig. 6). The
sounds were recorded from the phone output of themobile device.
The suggested application is not a typical ETA to be usedwhile walking, but as a “look-around” tool that allows a user tointeractively study the environment layout. Another possible
direction for improving the acceptance of the device by theusers is to remove the non-obstacle area from the U-disparity
map (see the grey region in the central part of the map in thebottom image of Fig. 2) to reduce the complexity of thegenerated sounds.
Although, the presented sonification method is very simple,
it can serve as a useful tool for aiding the visually impaired inspace orientation tasks. Its main disadvantage is that the sceneneeds to be explored by touch in a serial manner. This may takesome time before the user fully comprehends the image content
and spatial layout. Our current work is concentrated on
developing a sonification system that will combine interactive21
ISon2016-4

Proceedings of ISon 2016, 5th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 2016

sonification and classic audio description. Such an approach istermed sonic description. It introduces a complete set of soundswith unique time-frequency characteristics that describe animage sonically in a parallel form after a prior segmentation andrecognition of scene elements. This will allow, after a propermachine learning process, a quick interpretation of the observed
environment.
6.

CONCLUSIONS

An original interactive sonification technique for the
purpose of 3D scene representation for the visually impaired
people was devised and implemented. The method does not
sonify the information represented by the recorded images of
3D scenes directly (as is the case of the vOICe) but employs theprocessed depth images termed the U-disparity maps. Suchmaps allow the blind user to interactively explore depthazimuth space so facilitating search and localization of
obstacles. First trials of such an interactive sonification schemewith three blind volunteers shows a potential use of the system
as a spatial orientation aid for the visually impaired.
Acknowledgements: This work was partially supported by
the National Science Centre of Poland under grant no
2015/17/B/ST7/03884 in years 2016-2018 and by the European
Union’s Horizon 2020 Research and Innovation Programme
under grant agreement No 643636 “Sound of Vision.”
7.

REFERENCES[2] P. Bach-y-Rita, Brain Mechanisms in Sensory Substitution.
Academic Press, 1972.
[3] D. Dakopoulos, N.G Bourbakis. “Wearable obstacle
avoidance electronic travel aids for blind: a survey”, inIEEE Transactions on Systems Man and Cybernetics –Part C: Applications and Reviews, 2010, 40(1), 25–35.
[4] A. Csapo, G. Wersenyi, “Overview of auditory
representations in human-machine interfaces” in ACM
Computing Surveys, 2013, 46(2), 19:1–19:23.
[5] M. Bujacz, P. Strumillo, “Sonification: review of auditory
display solutions in electronic travel aids for the blind”, inArchives of Acoustics, 2016, 41(3), 401–414.
[6] T. Hermann, A. Hunt, “An Introduction to Interactive
Sonification” in IEEE Multimedia, April–June 2005, vol.
12, no. 2, pp. 20–24.
[7] I. Benacer, A. Hamissi, A. Khouas , “A novel stereovisionalgorithm for obstacles detection based on U-V-disparity
approach” in International Symposium on Circuits andSystems, 2015.
[8] Y. Lin, F. Guo, S. Li, “Road Obstacle Detection in Stereo
Vision Based on UV-disparity”, Journal of Information &Computational Science, 2014, 11, (4), pp. 1137–1144.
[9] R. Labayrade, D. Aubert, “In-vehicle obstacles detection
and characterization by stereovision”, in Proceedings the
1st International Workshop on In-Vehicle Cognitive
Computer Vision Systems, 2003, pp.13–19.
[10] V. Azevedo, A. Souza, L. de Paula Veronese, C. Badue,
M. Berger, “Real-time Road Surface Mapping Using
Stereo Matching, V-Disparity and Machine Learning” inInternational Joint Conference on Neural Networks, 2013.

[1] S. Maidenbaum, S. Abboud, A. Amedi, “Sensory
substitution: Closing the gap between basic research andwidespreadpractical
visualrehabilitation,”in
Neuroscience and Biobehavioral Reviews, 2014, 14, 3–15.

22
ISon2016-5