Proceedings of ISon 2016, 5th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 2016

INTERACTIVE SONIFICATION OF COLOR IMAGES ON MOBILE
DEVICES FOR BLIND PERSONS - PRELIMINARY CONCEPTS AND
FIRST TESTS
Andrzej Radecki, Michał Bujacz*, Piotr Skulimowski, , Paweł Strumiłło
Lodz Univesity of TechnologyLodz, Poland*bujaczm@p.lodz.plwas the Optophone that turned black and white print into chords
that allowed to distinguish letters [2].
More extensive reviews of sonification and interactive
sonification can be found in [10] and [11]. Probably the best
known algorithm for non-interactive sonification for the blindis The vOICe (Meijer 1992)[12]. The algorithm is basedon automated cyclic reading of pixel columns from a grayscale
image and translating them directly into the momentary soundspectrum. It has recently become available as an app available
for most mobile operating systems.
Due to the growing capability of mobile electronic devices
to process images, touch and synthesize audio, the subjectof interactive sonification for the blind has gained much interestin recent years [13][14][15].

ABSTRACTThe paper presents a proposed sonification algorithm
for presenting images on mobile devices for blind children
using an interactive auditory display. The sonification uses HSV
color images and the software is being written for Android
devices with touchscreens.
A brief discussion of the basic methods of sonication used in
the presentation of images and the history of interactive
sonification is presented in the review section.
The authors proposed several methods for interactive
sonification in which the user indicates with touch the areaof the image to sonify and depending on the image content andimage processing filters a real-time additive synthesis of several
sound buffers is prepared. The method presented in the paper
sonifies just one pixel directly under the finger, using its HSV
values as input to an additive synthesis transform.
The focus was to create a sonfication algorithm that was realtime and not computationally complex, while allowing toclearly distinguish not only brightness, but also the saturationand hue components determining color.

1.

INTRODUCTION

Attempts to utilize modern technologies to aid the blind using
acoustic signals reach as far back as the XIX century with thePolish invention called Electroftalm [1] or the early XX century
Optophone [2]. The term sonification itself has been formallydefined in the 90s as "the use of non-speech audio to convey
information" [3] or “data-dependent generation of sound in a
way that reflects objective properties of the input data” [4]. An
important organization responsible for discussion anddissemination of sonification-related research is theInternational Community for Auditory Display (ICAD,www.icad.org).
Interactive sonification is a subtopic of sonification
concerned with the creation of an interactive control loop with ahuman listener altering the properties of the synthesized audio[5][6].
Although sonification can be used for a variety of data
sources [7], the authors’ interest lies specifically in aiding
visually disabled persons. Our previous efforts were focused onpresentation of 3D scenes to the blind [8] and aiding in
navigation [9], while the current project is aimed at interactive
sonification of images on mobile devices.

Figure 1. The block diagram of the proposed interactive
sonification control loop.
2.

PROPOSED INTERACTIVE SONIFICATIONALGORITHM FOR HSV IMAGES

The proposed method for image sonification aims to hand overfull control over the sonification algorithms to the blind user.
Real-time interaction is possible through the feedback loop thatstarts with the user’s touch gestures, combines them with image
information, synthesizes the sounds that the user hears and candecide to touch the image differently. This allows for fullyinteractive sonification, i.e. the blind user decides what regionof the image to sonify and how to process the image
information (e.g. by filtering). The block diagram of theinteractive feedback loop is given in Figure 1.

2.1. The concept of the sonic space and the proposed image
to sound transform

1.1. Sonification of images to the blind

The proposed synthesis method consists of combining snm
parallel sound buffers which make up the sonic space S. The S

Attempts to interactively sonify text or images reach as earlyas the 1920s. The first interactive sonification tool for the blind68
ISon2016-1

Proceedings of ISon 2016, 5th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 2016

space has n dimensions and each of its elements is defined by
the m-th buffer contents having unique timbres, the frequency agiven buffer will be sampled with fnb and its amplitude scaling
An. This gives the S space with s elements that can be describedas follows:s ( s11 ( f1b , A1 ), ... , snm ( f nb , An ))­ 2  x for x d 0.5
g amp ( x) ®
¯2  (1  x) for x ! 0.5

The mod1 operation is used to wrap around from 1.0 to 0.0
by cutting off any integer part of the number. The nature of
the equation (3) is different than (2). In (3) all sound buffers
except for s3, which contains the base frequency all others are
scaled relative to, are played with frequency dependent only on
the saturation component. The perception of the frequencychange along with hue value was achieved by an amplitudeenvelope (5) with is dependent of H component and soundbuffer frequency localization (described by c31 to c35
coefficients). In transform (3) the synthesized sound is the samefor H component approaching both values of 0 and 1, which
follows the circular nature of hue. Although the synthesizedsound might be more complex than for transform (1), due to theamplitude restraints in function gamp, the complexity is lessaudible.
One should also notice the influence of the saturation
component S, which reduces the complexity of the sound,the closer it is to 0. This means the proposed sonification
approach may make greyscale images more easily interpreted
than full color ones.

(1)

The basic sonification scheme presented in the paper aims totransform the HSV color space to the sonic space S according to
the gHSV function which is described as:

g HSV 1 : HSV o S , g HSV 1 ( H , S , V )( s11 ( fb  H  c1 ) /( S  c2 ), V ),(2)s12 ( fb  H  c1 ), V ),(4)s31 ( fb  H  c1 )  ( S  c2 ),V ))where: H,S,V !c1, c2 – normalizing coefficients,
fb R+– audio buffer playback frequency
Transformation (2) assumes generation of three independent
sounds of predefined timbres, spaced apart in frequency.
The spacing is dependent on the color saturation S, while
the hue H shifts the base frequency. The brightness Vdetermines the loudness for all the components.
The HSV model shows high promise for the use insonification of images for the blind [15]; however, the cyclic
nature of the HSV model (Figure 2) could be problematic intranslation to frequency, as a point of discontinuity would need
to be introduced.

2.2. Spectral properties of the synthesized sounds
The use of buffers containing pure tones was initiallyconsidered, but produced very monotonous and irritating
sounds, that quickly tired the listener. Instead, a number of moreaesthetically pleasing buffers were used that had constanttimbres deemed subjectively pleasant by several experimenters.
The spectrogram of the sound buffer sn1 is shown in Figure 3.
Figures 4-6 show the audio spectrograms of several sample
color bars. Sharp changes in color are transformed into suddenchanges in the timbre of the sound (Figure 4), while smooth huetransitions are turned into smooth changes of sound (Figure 5),
with a closed loop (i.e. the end of the bar produces the same
sound as the start). When a color simultaneously changes its
brightness and intensity (Figure 6), the sound increases inamplitude, while becoming spectrally less complex due todecreased saturation.

Figure 2. The hue value of the HSV color model is circular.
The proposed transform gHSV2 that is taking into
consideration the cyclic nature of the H component is described
as:g HSV 2 : HSV o S, g HSV 2 ( H , S , V )( s11 ( f b /(1  S  c21 ),V  g amp (( H  c31 ) mod1)),
s12 ( f b /(1  S  c22 ),V  g amp (( H  c32 ) mod1)),s ( f b , V  g amp (( H  c33 ) mod1)),
13

Figure 3. Spectrogram of the sound buffer used in the additive
synthesis algorithm for sonification(3)s14 ( f b  (1  S  c22 ),V  g amp (( H  c34 ) mod1)),s51 ( f b  (1  S  c21 ),V  g amp (( H  c35 ) mod1)))
where c21=1c22=0.5 c31=0 c32=0.2c33=0.4c34=0.6, c33=0.8 –are normalizing coefficients chosen so that a consonant chord isheard at full saturation and69
ISon2016-2

Proceedings of ISon 2016, 5th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 2016

The main tool allows to synthesize a sound in real timedepending on a touched pixel’s HSV values and also log thepath of a user’s finger.
For the purpose of the study the experimenters trained
themselves and one volunteer to discern two colors (white andred) and a brighter region (0.8 V value) from a “rainbow”
background of a full H spectrum. This was done using thetraining images in Figure 7. All five participants were sighted,
but blindfolded for the testing (though not the training). After ashort training session, the task was to track a path with a fingeralong differently colored trajectories as shown in Fig 8, startingfrom one of the black regions and ending in the second one.
Sample results of the test are shown in Figures 9 to 12. In eachcase the trajectory of the user’s finger was tracked and thespectrogram of the sound was displayed. All the sample sounds
canbe
foundontheproject
websiteat
http://ztchs.p.lodz.pl/~radecki/ISON2016/The APK file for theimage sonification software will be made available there aswell.
As seen from the trajectories, the direction of the finger
movement changes only after leaving the intended path. To
provide additional information if the edge of a path was near,
the paths were locally blurred and the task was repeated. The
results are seen in Figures 12-14. The paths are evidently
tracked more precisely, although at a cost of speed. Due to theincreased complexity of the synthesized sounds, all subjects
took more time to follow the tracks whenever they were blurred.
Due to the small number of participants and no quantitative
data was gathered. The presented sonification scheme was just a
proof-of-concept for a larger study planned in the near future.

Figure 4. Spectrogram of the sonification of a discrete color
sequence, each color resulting from a combination of threebuffers scaled with amplitude and sampling frequency

Figure 5. Spectrogram of the sonification of a continuous color
sequence with a constant saturation value (S=1)

Figure 7. Training image set with a large central fieldcorresponding to the path properties in the trajectory following
tests (top) and the test images with the trajectories to be traced
by finger using only sonification (bottom).
Figure 6. Spectrogram of the sonification of a blue color hue
with a varying brightness and saturation

3.

DATA COLLECTION TOOLS AND FIRSTEXPERIMENTS

In order to verify the proposed sonification method and to judge
the possibility of differentiating between the different HSVcomponents, several software tools were prepared both forWindows and Android and a short experimental study wasperformed.

70ISon2016-3

Proceedings of ISon 2016, 5th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 2016

Figure 8. Tracking a path with minimum hue (H=0).

Figure 10. Tracking a path that has 20% less saturation thanthe background (S = 0.8).

Figure 11. Test images with blurred trajectories.

Figure 9. Tracking a path that has maximum brightness (V=1).

Figure 12. Tracking a blurred path with minimum hue (H=0).

71
ISon2016-4

Proceedings of ISon 2016, 5th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 2016school children, which will study if interactive sonification ofimages and maps could be added to their curriculum.
In the developed interactive sonification tool multiple image
filters can be used to enhance an image, examples include color
depth limitation, edge detection or Gaussian smoothing.
Currently the image processing method is chosen manually, but
an automatic adjustment basing on image content is being
considered. Images loaded into the software can also havesupplementary text information in xml files containing labels
assigned to image regions, e.g. street names for maps, written
descriptions of image content or instructions for educationaltasks.
Two main categories of sonification approaches will be tested– cyclic, which creates looped waveforms with amplitudeenvelopes changing in short 1-2 s cycles, and non-cyclic, where
the sound amplitude depends solely on the touch gestures as in
the presented paper.
The sound synthesis will be performed by summing andamplitude modulation of up to 16 modifiable sound buffers,
which can be read at different frequencies. This approachensures smooth operation on most Android mobile devices.
Additionally, Text-to-speech synthesis will be used for images
with the supplementary xml text information.
Several tools were prepared for the logging of userinteractions with the software, e.g. to track touch gestures andinteractions with the interface. This will aid in documenting
the experimental trials, once the software is tested on a larger
group of blind children and adults.
The main goal of the trials will be to determine whether
interactive sonification can be used to supplement or replacetactile materials used in classes for blind children.
To encourage reproduction of the research , the authors are
considering preparing the experiments within the SonEX
framework [16].

Figure 13. Tracking a blurred path that has maximum
brightness (V=1).

5.

CONCLUSIONS

The article presents an interactive image sonification approach
using the HSV colour model. The main strengths of theproposed approach are:- a simple synthesis algorithm (additive synthesis bycombining a small number of audio buffers)- transforming the circular nature of the hue component- simplifying the synthesized sound with the decrease of thecolor complexity (smaller S value).
So far the model has shown potential usefulness of thecontinuous transform and of image blurring for smooth soundtransitions. The prepared tools and the path tracking experimentare a proof of concept for a bigger scale test aimed at utilizing
interactive sonification for the education of blind children.

Figure 14. Tracking a blurred path that has 20% less saturation
than the background (S = 0.8).

4.

6.

ACKNOWLEDGEMENTS

The work was supported by the National Science Centre ofPoland under grant no 2015/17/B/ST7/03884 in years 20162018.

CURRENT AND FUTURE WORK

The presented sonification approach is the first of a number ofapproaches proposed in the project. The developed tools areintended to help conduct a larger study with a group of blind72
ISon2016-5

Proceedings of ISon 2016, 5th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 20167.

REFERENCES[1] M. Capp and P. Picton, “The optophone: an electronic
blind aid” in Eng. Science and Education Journal, 2000,
9(3), pp.137-143.
[2] M. Jameson,, "The Optophone: Its Beginning andDevelopment", Bulletin of prosthetics research: 25–28,
1966[3] G. Kramer, “Some Organizing Principles for Representing
Data With Sound”, In G. Kramer (Ed.), Auditory Display:Sonification, Audification and Auditory Interfaces, SFI
Studies in the Sciences of Complexity Proceedings (Vol.
XVIII), Reading, MA: Addison-Wesley Publishing
Company. 1994
[4] T. Hermann, A. Hunt and J. Neuhoff, Sonification
Handbook. Logos Publishing House, Berlin, 2000.
[5] A. Hunt, T. Hermann, and S. Pauletto. "Interacting withsonification systems: closing the loop." Information
Visualisation, 2004. IV 2004. Proceedings. EighthInternational Conference on. IEEE, 2004.
[6] N. Degara, A. Hunt, & T. Hermann,“InteractiveSonification” [Guest editors' introduction]. IEEE
MultiMedia, 22(1), 20-23 2015[7] G. Dobus and R. Bresin, “Sonification of physicalquantities throughout history: a meta-study of previous
mapping strategies” in Proceedings of the 17th
International Conference on Auditory Display, 2011.
[8] M. Bujacz, P. Skulimowski, P. Strumiłło, “Naviton - a
prototype mobility aid for auditory presentation of 3Dscenes”, Journal of the Audio Engineering Society, Vol.
60, No. 9, 2012 September, pp. 696-708[9] P. Baranski, P. Strumillo "Emphatic Trials of a
Teleassistance System for the Visually Impaired" Journalof Medical Imaging and Health Informatics 5 (8), 16401651[10] R. Sarkar, S. Bakshi and P. K’Sa, “Review on Image
Sonification: A Non-visual Scene Representation” in Int.
Conf. on Recent Advances in Information Technology,2012.
[11] G. Dubus and R. Bresin, “Sonification of physicalquantities throughout history”, The 17th International
Conference on Auditory Display, 2011.
[12] Meijer, P. “An Experimental System for Auditory Image
Representations” IEEE Transactions on BiomedicalEngineering, 39, pp. 112-121, 1992
[13] Matta, S.; Kumar, D. K.; Yu, X. & Burry, M. (2004), 'An
approach for image sonification', First International
Symposium on Control, Communications and Signal
Processing, 431-434.
[14] O'Neill, C. & Ng, K. (2008), 'Hearing Images - Interactive
Sonification Interface for Images', International
Conference on Automated solutions for Cross Media
Content and Multi-channel Distribution.
[15] S. Cavaco, “From pixels to pitches - Unveiling the worldof color for the blind”, IEEE 2nd International Conference
on Serious Games and Applications for Health, 2013.
[16] N. Degara, F. Nagel, T. Hermann, “Sonex: An Evaluation
Exchange Framework For Reproducible Sonification”.
Proc. of the 19th International Conference on Auditory
Display (ICAD-2013). Lodz, 2013, pp 167-174

73
ISon2016-6