Proceedings of ISon 2016, 5th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 2016

LOWERING THE BARRIERS TO THE CREATION OF INTERACTIVE AUDITORYDISPLAYS: AN EXPERIMENTAL INVESTIGATION
Doon MacDonald and Tony Stockman
School of Electronic Engineering and Computer Science
Queen Mary University of London
d.macdonald/t.stockman@qmul.ac.uk
ABSTRACT

describes an experimental evaluation of an approach to introducing novice interaction designers to a methodical approach to AD
creation.

What can the practice of Soundtrack Composition bring to thedesign of auditory displays? Previous research has highlighted thelack of knowledge in the wider human-computer-interaction (HCI)
community about the practice of auditory display (AD) design,
providing evidence that there is a need to capture the rationale for
AD design more effectively, disseminate good AD practice more
widely and in general lower the barriers to AD creation. This paper describes an experimental approach to address these questions.
The approach and principle aims of the method (SoundTrAD) outlined in this paper is to bring together ideas from soundtrack composition and design stages for interface creation to produce a systematic and creative approach to AD design. The instantiation ofSoundTrAD reported here takes the form of an interactive tool thatis programmed in Max/MSP, Processing and Open-Sound Control. This tool serves to guide the development of an interactive
auditory display from the point of view of an end-user. The instantiation of the tool is specific to a particular scenario to be used in
the experimental evaluation of the approach; however, in principle
the method supports the creation of a display for a wide range of
scenarios and applications and is argued to be a particularly good
match for monitoring systems. 11 novice designers took part in astudy examining their use of the tool to create an interactive auditory display. The results demonstrated that all participants wereable to complete the tasks and were therefore successful in creating interactive ADs. Qualitative results from the study indicatedthat they found the tool and the approach it embodies very usable,
engaging and enjoyable. Participants also said that their overall
understanding of ADs was significantly improved, and suggested
numerous ideas for further applications of the approach.

2. BACKGROUND
2.0.1. Methods
The lack of methods to support novice designers has been arguedby [1]. This motivated their design of PACO, a framework to support the design of ADs [2]. The main approach was to provide
access to a design pattern space for novice designers. The patterns were contrived as a result of understanding and documenting existing approaches to AD design from designers with experience in HCI. Similarly, [3] argued that approaches to AD design
tend to be ad-hoc. As a result, they formed a review of methods
that addressed the early stages of AD design namely requirements
gathering and conceptual design stages. Stephen Barrass’s TADA
method is widely reported and does utilise all stages of auditory
interface design, supporting the design from the start to the production of a display [4]. Arguably, however, this remains the only
method that supports the designer through all of the stages of interface development and there is still a lack of established and accessible methods to support a novice designer, in particular, throughall stages of the design process.
2.0.2. Soundtrack Composition
The idea of using soundtrack composition as an influence for thedesign of ADs is not new. In chapter 7 of The Sonification Handbook, Barass and Vickers, compare visual design to sonic design
by stating that, ‘where graphical visualization draws on graphic
design [it is possible to] draw on sound design for commercial
products and film sound in the next generation of ubiquitous everyday sonification’ [5, p.165]. They go on to compare composers
with film composers, claiming that whilst composers do not haveto focus on functionality and accessibility (unless they opt to doso), film composers have to be aware of the function of the audio
and how listeners perceive it.
For her Ph.D, [6] explored the use of music in the interface.
She compared the use of sound in film with its use in interfaces.
She claimed that in a film setting music provides elements that
could also prove useful within a computer interface setting. These
included continuity, which could be useful when switching between windows in a desktop, for example. Motifs for reflecting
characters, which could be useful for identifying which window
is active or as a way to introduce a particular theme, algorithm or

1. INTRODUCTION
The work described here was undertaken in the context of a wider
project which explores the question ‘what can the practice of Soundtrack composition bring to the design of ADs?’. The lack of knowledge in the wider human-computer-interaction (HCI) community
about the practice of AD design was highlighted by [1]. This
provides evidence that there is a need to capture the rationale for
AD design more effectively, disseminate good AD practice more
widely and in general lower the barriers to AD creation. It follows
from the above that it is desirable to build into courses on Interaction Design, materials that motivate the need for auditory displays
as well as approaches that are of practical value to a novice interaction designer (or designer with limited experience of applying
audio to their designs) in illustrating how they might adopt a methodical approach to the design of auditory displays. This paper

ISon2016-1

3

Proceedings of ISon 2016, 5th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 2016
principle. The use of audio to enhance action which could be used
to emphasize something on a screen.
A direct insight into how the functions of film music could
benefit computer-based design was explored by [8]. Cooley argued that, like a soundtrack, using music in interactive systems
can help expand screen space, draw attention to both on and off
screen events as well as provide characterisation and emotions in
HCI.
When referring to the use of sound to enhance learning in‘computerised instructional environments’, [9] argued that soundis somewhat under utilised. To overcome this, the authors recommend four approaches from the ‘best practices’ of film industry
sound design. The recommendations are as follows:

without audio. The following stages are not derived from one
source only, but from a reading of a range of sources on interface
development, including, [11, 12, 13, 14, 15] and [16] and represent
a summary of interface stages accounted in those references. The
different sources vary to some degree in terminology and emphasis
they place on each of the stages, however an overview is provided
here representing a broad consensus of the stages of interface development.
3.1.2. Requirements gathering
The first of the stages is the requirements gathering stage. This
stage often starts with the idea of a scenario which can be used tooutline who the users of the interface are, what their context is andwhat their tasks are.
An analysis of the scenario, in order to inform the interface
design, is known as a task analysis. The analysis of the tasks to be
performed, the users or system that performs them and where thetasks (actions) take place is important in creating a user-centred
design. Central to this is the user and their point-of-view of thesystem they are interacting with. Different users have different
preferences, goals and expectations. Analysis of the context inwhich the interface will be deployed is also important. For example, in their study of office soundscapes [17], argued that auditory
interface designers consider the different sounds in the display in
combination with the sounds that exist externally to the display
and within the context. They showed the importance of paying attention to how the different sounds combine and to the potential
masking issues that could occur between sounds when it comes tosound design choices.

1. Firstly, they propose that like sound design, the sound usedin computerised instructional materials could be used to support
storytelling in order to help learners ‘acquire, organize and synthesize’ the materials under study (p.7).
2. Secondly, they pointed out that when designing soundtracks, sound designers often begin their processes with an initial
reading of the script. From this they ‘listen out for’ objects, actions, environments, emotions and physical or dramatic tensions
that can be ‘fleshed out auditorally’ using the various sound types(which they describe as music, speech and sound effects). They
then argued that the instructions for computerised learning environments can also be identified for these key ‘storytelling elements’(p.8) through a collaborative process (between sound designer and media developer).
3. Thirdly, the authors argue that like sound designers, designers of audio for an instructional product should understand andutilise the way people listen to sounds. The authors reference themodes of listening presented by film theorist Michel Chion; reduced, causal and semantic [10].
4. Lastly, the authors argue that like sound design for films, designers should be systematic about how they incorporate sounds.
Sound designers work within a framework and literally map out
along a time line where particular sound groupings (voice, music,
sound effects) will be placed as the story unfolds. They proposethat humans learn through a process of ‘selecting, analysing andsynthesizing’ which can be seen as analogous to a film’s beginning, middle and end.

3.1.3. Conceptual design
The second stage, the conceptual design stage, involves considering the overall form of the interface including the modes of interaction to be supported and how communications between theusers and the system is to be organised. Once more this stage is informed by the context in which the system is intended to be used,the capabilities, requirements and tasks of the users. Prototyping
often plays an important role at this stage in helping to provide
users with an overall impression of the interface and what it will
be like to use, in order to obtain feedback on both its form andfunctionality.

3. SOUNDTRACK COMPOSITION MEETS AUDITORY
USER INTERFACE DESIGN

3.1.4. Detailed design

The design of our approach to AD design evaluated in this paper, called SoundTrAD, was inspired by a drawing together of approaches to user interface design and to soundtrack composition.
In order to establish the background to this, what follows in thissection is an overview of established stages employed in auditory
interface design. This is followed by a summary of approaches
to soundtrack composition. The section ends by drawing parallels
between the two, which will be used to formulate the fundamental
stages and framework of SoundTrAD.

The third stage is the physical or detailed design stage wherein
the interface is designed at a detailed level and implemented in theform it will be deployed. Lower level prototyping is often employed in this stage to obtain more detailed feedback on specificlower level features of the interface. This iterative process of detailed design, implementation and feedback through prototyping
results in a final realisation of the entire interface.
3.1.5. Summative evaluation

3.1. Interface Design

Finally, there is the summative evaluation stage, which involves
final acceptance testing of the entire interface. This should involvethe complete cycle of tasks required to be undertaken by users andshould exercise all parts of the interface. Feedback from this final
evaluation may lead to some changes being made to the interface,

3.1.1. Design Stages
Existing guidelines for interface design support several methodological stages. These have been utilised for interfaces with andISon2016-2

4

Proceedings of ISon 2016, 5th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 2016as well as lessons being learned for future interface development
projects.
The above stages provide a broad structure from which to approach the design of an interface, and were used to provide an
overall framework for the development of SoundTrAD.

to include notes on important footage or time-code numbers corresponding to the occurrence of events. As, Yewdall observed, a cue
sheet is ‘simply a road map of intent’[23, p.23].

3.2. Soundtrack composition

Once the composer has annotated their ideas for sounds, the sounds
are classified into categories of dialogue, music, sound effects (DM-Es). What is important in a soundtrack is the consideration of
how these sounds work together and how they form the bigger
soundtrack by relating to one another and to the story being told.
Within a film, the soundtrack contains not only the musical score,
but ambient sound, dialogue, sound effects, and silence, [24, p.5].
To underline the importance of considering the soundtrack andall the sounds that make it up as a whole, Lipscomb and Tolchinsky
argued for the ‘analysis of the entire soundtrack, upon which musical sound, dialogue, sound effects, silence, and some sounds thatfall in the cracks between traditional categories all exist for the purpose of enhancing the intended message of the motion picture’[24,p.5].

3.2.4. Fundamental sound classifications

The following section provides a review of the functions and principles of creating soundtracks. This review is based on [18, 19, 20,
21, 22] and [10]. The review given here draws on what is considered to be the most frequently utilised and widely held soundtrackcomposition principles. Within this, particular attention has been
paid to concepts in soundtrack composition practice that could beseen to parallel practices in interface development.
3.2.1. Functions
Sound in film is used to anchor and engage us with meaning;smooth editing, enhance and create mood and emotion, illustrate
geographical location and historical situation, focus attention on a
specific action or object, determine speed and motion and to function as a leitmotif, associated with a specific character or theme[18, 21, 22].

3.3. Bringing The Two Together
The above has very briefly outlined some of the fundamental processes in creating interfaces and soundtracks. Parallels between
these activities are proposed in table 1 which starts with the idea ofparalleling a scenario/scene. This table illustrates a parallel framework comprising the larger method stages involved in creating auditory interfaces and soundtracks. In these stages reside the steps
that were developed and explored as a result of the iterative design
approach that supported the development of SoundTrAD.
It is clear to see existing similarities between the analysis of
scenarios and film scenes. More specifically, this illustrates theimportance of identifying actions, objects, locations, as well as therelationship and transitions between events as a part of the requirements gathering and conceptual design stages. It is equally important to consider suitable audio that can map to the events alongside
the need to evaluate and iterate the design process.

3.2.2. Design Stages
Hollywood sound designer David Sonnenschein outlines a methodical approach to creating a soundtrack from the script to thefinal mix [18]. With this, Sonnenschein does not offer a technical account of suitable software, or specific hardware, or mixing
or editing techniques, but rather a creative approach to analysinga script and conceptualising, designing and delivering ideas fora soundtrack to the technical team. This includes techniques andmethods that can not only aid in the initial decision of where sounds
should go, (formerly known as ‘spotting’) but also the categorisation, mapping, placement and initial arrangement of suitable sounds.
Sonnenschein is clear to point out that the steps are subject to personal ordering and are not definitive. What is clear in this suggested approach is the importance of iteration as part of the creative process and the means by which the composer can be guided
and supported at the same time.

4. THE AIM OF SOUNDTRAD: WHAT CAN ITSUPPORT?
Below are listed the set of general aims of SoundTrAD.

3.2.3. Scenes and Spotting
Composers will analyse a scene by ‘spotting’ it for places that
could be enhanced by audio. The elements that composers could‘listen out for’ during this process have been described by Sonnenschein, who wrote that within every on-screen character, object
and action there is potential to generate a sound that can enhance
the narrative and story [18]. With the starting point of marking
the script, in order to identify key storytelling elements that canbe ‘amplified by sound’, Sonnenschein identifies the following‘voices’ to listen out for; specifically by identifying and circling
explicit words and phrases: People, objects, actions, environments,
emotions and transitions.
At this stage the idea of a cue sheet is important. Cue sheets
are the main means by which sound editors communicate the layout of their work to the human mixer [20]. They list changes
throughout a performance. Cue sheets are organized with columns

1. To lower the barrier to creating ADs in order to enablenovice designers to engage effectively in the AD design
process.
2. To support the designer in executing accountable, repeatable steps toward producing a display.
3. To enable the designer to complete a prototype/model of
their design.
4. To enable designers to document their ideas to enable themto reference and share the rationale for their designs.
The following study was performed to test whether SoundTrAD met the objectives, specifically that of lowering the barriers
to novice designers and improving their understanding of ADs.

ISon2016-3

5

Proceedings of ISon 2016, 5th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 2016
Auditory Interface Stages and Steps
Stage: Requirements Gathering - scenarios and task analysis with users/actors, events (transition between them) objects,
actions, context
Stage: Conceptual Design - thinking about interface arrangement, and what parts need sonifying and how it is laid out
Stage: Detailed Design - mapping events to audio
Stage: Evaluation

Soundtrack Stages and Steps
Stage: Spotting the scene - stories with characters, actions, objects, transitions, locations and emotional feelStage: Arranging ideas and cues, sketching, establishing and iterating ideas
Stage: Composing/Designing original music and sourcing soundsamples to map to the cues/events
Stage: Evaluation

Table 1: Parallel between audio interface design and soundtrack composition
5. SUMMARY OF THE DEVELOPMENT OF
SOUNDTRAD

scaled numbers could be mapped to any major, minor scales
and modes. The idea was that the data set is scaled to awestern scale. There were 10 options for the instrumentation from drum to strings through to tuned percussion. The
data could be represented using musical pitches or rhythmic beats and could come together to create melody and/or
rhythmic sequences as a result of the playback.

The stages of SoundTrAD were developed out of the parallels
drawn between the stages of soundtrack composition and AD design. Each stage was refined, changed, and developed as a result of
3 user studies that took place before the final study reported here.
The following provides a heavily summarised description of how
the SoundTrAD approach was developed.
Firstly, the idea of a cue sheet was introduced to help the designer map out events in a scenario by identifying actions, objects
and subsequent ideas for sound. The study utilised ideas from how
scenarios are mapped out in HCI practices [13, 12, 26] and explored how these can parallel the traditional cue sheet as used insoundtrack composition [18]. A database was created in order tostore details of sounds and sound mappings and to facilitate audition of sounds from an early stage in the design process. Ideas forthe database came from the previously referenced soundtrack literature. The database of sound examples was populated with soundsamples and supported navigation in relation to the requirements,
sound types or soundtrack composition categories. A basic timeline was created to represent the sequence of events as they unfold
during the interaction. Sound ideas were placed on the time-line
in order to start to consider event layout. As the studies developed
events on the time-line could be reordered to allow users to examine alternate interaction scenarios, check for masking and audition
early prototypes of their ADs.

2. Background: the designers were presented with 5 options
from natural to musical drones (wind, sea, calm, electricity,strings). In order to enhance the display and perception of
the data, as inspired by [29] who argued, convincingly that
drones can be used to show lengths of processes, the aim
was to map a continuous sound to a drone to clearly illustrate the length of the scenario. The parallel with soundtrack composition is that it can be used to establish place,enhance action, give sense of a size and create mood. The
designers were told what to consider when picking theirsound: who the users are, where the display is heard, how
it alters perception of the events and data.
3. The events could be mapped to Foley/SFX or music. There
were 12 options for the SFX and 7 for the motifs. Designers
were told that these sounds can be used to represent everyday things or that if events do not have a real-world association, then motifs can be chosen as an option. The motifs were presented in the form of minor and major piano
chords. In theory this could be any chords or instrument
but the idea was that the sounds should be short and canbe major (positive), minor (negative) or neutral. Motifs thathave the same instrumentation, yet are altered in key helped
narrate story and represent data points (character) or events.
From a system and design principle point of view (even if
the user is not aware), it is necessary that the chords for the
motifs, the melody for the dataset and the constant drone forbackground should all be associated with the same musicalkey.

6. THE STUDY
The aim of the study was to represent the potential of the databaseand principle behind the mapping. To demonstrate a set of mappings that draw on some principle of soundtrack composition inorder that integration of different sound types and aesthetics could
be considered. The aim was that aesthetics and sound categories
were made accessible, understood and it is clear how they relate torequirements of the AD events. Listed below are the details of the
mappings that were implemented for the study:

6.1. Participants

1. A user-entered data set was mapped to pitch to create rhythm
and melody. The designer could change the instrumentation and speed of playback in order to give them a sense ofcontrol and change the rhythm. The pitches that the data
was mapped to were scaled to a major scale as inspired byVickers [28] who argued that tonal music should be preferred over direct mappings to frequencies. The numbers
were scaled (0-10 = c/MIDI note 60, 11-20= d/MIDI note
62, 21-30, e/MIDI note 64, f/MIDI note 65, g/MIDI note
67, and so on through to MIDI note 72). In principle these11 participants took part in the evaluation. All participants had
little or no experience of creating ADs.
6.2. Tasks
The participants were presented with a hypothetical scenario/scene
involving someone using an AD to monitor footfall and staff activities in a work-place environment. They were then presented witha series of 7 tasks and help files that supported them in applying
SoundTrAD to create an AD for the scenario. If the information

ISon2016-4

6

Proceedings of ISon 2016, 5th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 2016

Figure 1: Task 1: Add footfall numbers in the number entry boxes
provided

Figure 4: Task 5: Select a backgroundFigure 2: Task 2: The cue sheet where events were documented
Figure 5: Task 6: Map events to different sound effects, Foley ormusical motifs

icon was clicked then this opened up a separate window with thehelp files. The content of each outlined further rationale behindthe mapping options. The 7 tasks are briefly summarised below:
Task 1: Enter footfall data to provide the data set for the scenario. Figure 1 illustrates the screen participants could interact
with. Participants could enter numbers between 0-100 in eachnumber box.
Task 2: View the user, system and data point events on the cue
sheet that are used to represent staff activity and significant points
in the scenario that need to be represented in audio. Enter qualitative descriptions about the events in order to help with sounddesign choices. Figure 2 illustrates the cue sheet.
Task 3: View the time-line and load the entered dataset andevent data from the cue sheet into it. Note how these are displayedin rows on the time-line, including a final row which can be used torepresent a background sound. Explore interacting with the timeline by trying to move any of the user, system or data point eventsby placing the mouse cursor over them, left-clicking and dragging.
Try pressing the play icon, notice the vertical moving line and how
it loops through the scenario every ten seconds. Figure 6 demonstrates the time-line.
Task 4: Create melody, instrumentation and rhythm. Choose
the instrument options and speed of playback that you feel best
represents the data set in the shop scenario. Figure 3 illustrates theinterface the participants could use to map their data (entered intask 1) to pitch, rhythm and speed parameters. The gauge on theright hand sound monitors the level of playback.
Task 5: Create background audio (choose a background soundthat matches the context of the given scenario) . Figure 4 highlights
the selections that participants could choose to create a backgroundfor their display.

Task 6: Add effects and music. Choose sounds from the databaseof sound effects and motifs that are good representations of theevents on the time-line. Figure 5 shows the interface participants
could use to select sound effects, Foley sounds and motifs that theyfelt best matched their design.
Task 7: Adjust and mix the sounds: Explore re-arranging theevents, adjusting the length of the time-line, the balance of thesounds, adding/removing events, adding/removing effects. See
figures 6 and 7 for the screens participants could use to auditionand design their final display.
7. FINDINGS
Participants were presented with a questionnaire pre-study to gather
their level of experience and rate their understanding of data sonification and auditory display (as outlined above). After the study,
the participants were asked to complete the questionnaire by rating
and giving their immediate feedback regarding how they felt themethod and tool had supported their understanding of data sonification and auditory display. The aim was to compare the participants reactions before and after the study.
The participants were then sent a link to an on-line survey
containing 6 questions to gather ratings and their open-ended responses regarding the usability, enjoyability and usefulness of thesystem. The ratings scaled from 1-7 and included semantic markers. For example, the question on usability ranged from 1: extremely unusable to 7; extremely usable. The survey was also
designed to gain insight into whether the participants would use
SoundTrAD again and what scenarios they thought it could caterfor.

Figure 3: Task 4: Map the footfall to different instruments andchange the speed of playback

Figure 6: Task 3 and Task 7-Time-line

ISon2016-5

7

Proceedings of ISon 2016, 5th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 2016

Figure 7: Task7-levels
Part.
1
2
3
4
5
6
7
8
9
10
11

AD
(before)Reasonable
LittleLittleLittleLittleLittleLittleLittleLittleNone
LittleSon.(before)

AD (after)

Son.(after)

LittleLittleLittleLittleLittleReasonable
LittleLittleLittleNone
LittleLittleLittleReasonable
A Lot
Reasonable
Reasonable
Reasonable
Little/Reas.
Reasonable
Reasonable
LittleA Lot
Reasonable
Reasonable
A Lot
Reasonable
Reasonable
Reasonable
Little/Reas.
Reasonable
Reasonable
LittleFigure 9: Enjoyability
7.1. Question 1: Usability
Figure 8 shows that the lowest rated usability was 5 out of 7 with 4of the 11 putting this, and the highest score from 1 participant was
7. The mode was 6 - very usable, with 6 out of the 11 participants
claiming this.
7.1.1. Summary of Question One
The lowest usability rating of 5 out of 7 came from P1, P6, P8 andP11. P1, P8 and P11 all commented on issues with the interface
and implementation as somewhat limiting its potential and usability. P6, liked the interface but compared it to the interface fromthe previous studies they have taken part in. Arguably, this could
demonstrate an improvement in the system but there was further
potential to develop it. Audio levels gave a sense of control (P2)and the instructions and layout were seen to be accessible. As P5
noted, it is “Well explained and accessible without having muchprior knowledge”.

Table 2: Understanding of Auditory Display and Sonification Pre
and Post Study

7.0.1. Question: Levels of experience and understanding preand post study
Table 2 shows the participant ratings for their level of understanding of AD and sonification before and after the study. The ratings
span from none, a little, a reasonable amount to a lot. One participant’s (P1) understanding of what an AD was went down fromreasonable to a little. Two participant’s (P2 and P11) understanding stayed the same at ‘A little’. Eight out of the eleven participants
claimed their understanding of what an AD is had increased after
the study with six responses changing from a little to reasonable,
one response from a little to a lot and one from none to reasonable.

7.2. Question 2: Enjoyability
Figure 9 shows that the lowest rated enjoyability score was 4 (neutral) from one participant and the highest was 7 out of 7 with 2
participants putting this. The mode was 6 out of 7 with 6 out of
the 11 participants stating this.

Nine out of the eleven participants claimed their understandingof what sonification is had increased after the study. Two (P1 and
P4) had their understanding change from little to a lot, six froma little to reasonable, one (P10) from none to reasonable. One
participant (P6) asserted that their understanding had stayed thesame.

7.2.1. Summary of Question 2
The lowest enjoyability score was from P6 (neutral). P2 and P3
found it slightly enjoyable. They both thought there could be lesssteps, default settings (P2) and more user control (with ability to
input their own sounds). Enjoyability was clearly related to being
able to listen to sounds and work with audio straight away, as P2
observed: “It is nice to be able to change the volume of the sounds
that associated to the events. In this way, I can make important
events more noticeable. In general, I feel in good control and isable to change all the aspects of the sound”. The sense of personalisation and control also supported enjoyability as did being able
to visualise the events and work with graphic representations.
7.3. Question 3: usefulness for designing AD’s for real world
scenarios
Figure 10 shows that the lowest rated usefulness was 5 out of 7,with 2 participants putting this, and the highest score being 7 with
4 participants putting this. The mode was 6 out of 7 with 5 out of
the 11 participants putting this.

Figure 8: Usability Rating

ISon2016-6

8

Proceedings of ISon 2016, 5th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 2016
P6: “I think it can be applied to use with any storyline/events thatneeds auditory display”
P7: “Music education, aiding system for sensory-impaired people,mobile application, animal monitoring, and so on.”P8: “Transport hubs like railways stations and airports, either for
people (queues building up, need more staff on check in desks etc)
or for the logistics of moving bags (airports) or rolling stock (sending in more carriages) or even having an alert for a problem withtrains and getting rail replacement busses.”
P9: “Digital archives. Revision aids. Teaching aids especially forstudents with special educational needs’. I would really like to
use with historical data being archived online. I think this kind ofaudio display would be great for a variety of students needs andcould make education more accessible for all. There are endlesspossibilities here I think’
P10: “To help workers ‘keep an ear’ on their work. To help supervisors monitor staff activity. To help managers review data andplan accordingly. To add auditory and notification to already existing IT systems (in vehicles/computers/classroom environments).
Hugely beneficial to people with impairments.”
P11: “in vehicles for optimum performance and monitoring. For
use in medical and sporting monitoring equipment. for monitoringenergy use in the home/business. for use in extreme locations formonitoring yourself/the environment. to aid a person with disability/medical condition. For anyone that is multitasking any situation and needs to monitor and respond to situations while engaged
in another activity”
To summarise the possible applications:Monitoring, data review and trends, real-time scenarios, multitasking, alerts and alarms, sports, medicine, health, environment,
performance, education, transport, special educational needs andbusiness.

Figure 10: Usefulness
7.3.1. Summary of question 3
2 participants scored the usefulness as 5 out of 7 (P6 and P8),
however P6 gave positive comments when stating how useful they
thought the system could be. P8 showed some confusion regarding the target users. The usefulness was related to its potential andthe wide range of scenarios it can be applied to. P9’s observation about using it for historical data is specific and related to their
specialism and so this reflects the potential for SoundTrAD to beadaptable and, as P4 stated, customizable to suit the users needs.
7.4. Question 4: Use again and Any changes?
Yes=54.55%(6)Maybe=27.27%(3)Do not know=9.09%(1)Unlikely=9.09%(1)

7.4.1. Summary of question 4
7.5.1. Question 6: Any Other Comments

It is clear that most participants would use this again given changes
or a specific need. P11 is a novice to interaction design and music composition and observed that they did not know of any othersystem that could do this. There are no commercially available
systems in the everyday world.

There were no negative comments received for this question. P9said it was “impressive stuff. P10 commented that the system has“endless potential”. P7 made suggestions for updates to interface
could be include “adding more samples”. P4, P8 and P5 said they
thought it was “enjoyable”, with P4 adding that it was “easy tolearn”.

7.5. Question 5: Other Scenarios
The responses demonstrate the wide-range of scenarios and applications that participants felt the system could be used for.
P1: “Sport events, data sonification, live data performance tool”.
P2: “I would like if it can respond to real time events. Then it
can be used in public areas such as cafeterias. The system canbe related to the play list the cafe is using. When there are morecustomers in the shop, play some pop music or music with a quick
tempo, customer might eat faster and leave quicker. When there
are only few people in the cafe, play some classic music to easethe pressure of customers.”
This response shows confusion on behalf of P2 regarding the method
and the displays it is intended to help create.
P3: “Analysis of any complex system e.g. scientific research”
P4: “For tasks or cases that need constant monitoring and needto fire alerts if changes happen, while users are occupied doing
something else. Almost like a background monitor.”
P5: “Petrol consumption. When client payments made to bank account (might be too complex but would be really helpful) Meteredwater consumption. When I need coffee!”

7.6. Summary of Findings
The aim of the study was to explore the accessibility of SoundTrAD to novice AD designers, to examine whether it improved
their understanding of ADs and sonification, and the usability ofthe tools employed to implement the approach. Out of usability,
enjoyability and usefulness it was the usefulness that received thehighest number of the maximum score 7 (extremely useful). The
next most successful in scoring highest numbers was enjoyability,
followed by usability. This outcome could reflect that the method
potential was realised for this sample of participants new to AD design, and despite some remaining issues with usability, it is clearthat given professional development that it would be a useful system. The enjoyability was also an important part of the system,in particular how this related to the use of audio throughout thedesign process. Particularly, iterating ideas, exploring options andsubsequent aesthetics of the final display.

ISon2016-7

9

Proceedings of ISon 2016, 5th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 2016
8. REFERENCES[18] D. Sonnenschein, Sound design: The expressive power ofmusic, voice and sound effects in cinema Michael Wiese Productions, 2001.

[1]

C. Frauenberger and T. Stockman, and L. Bourguet, “A survey on common practice in designing audio in the user interface,” in Proc. of the 21st British HCI Group Annual Conference on People and Computers, pp. 187-194.
[2] C. Frauenberger and T. Stockman, “Auditory display designan investigation of a design pattern approach;;. . Journal of
Human-Computer Studies 67(11), 907-922.
[3] E. Brazil, E and M. Fernstr om. M, “Subjective Experience
Methods for Early Conceptual Design of Auditory Displays,
in Proc. of the 15th int. conf. on Auditory Display (icad) pp.
18.
[4] S. Barrass “Auditory Information Design”. Unpublished doctoral dissertation, 1997.
[5] S. Barrass and P. Vickers, “Sonification Design and Aesthetics. in The sonification handbook (pp. 145-171), T. Hermann,
A. Hunt, J. G. Neuhoff (Eds.). Logos Publishing House,
Berlin, Germany.
[6] S. Grice. “The uses of audio in interface design: in particular the use of music”.Unpublished doctoral dissertation,
Dundee University, 2006.
[7] S. Monache, and P. Polotti and D. Rocchesso. “A toolkit
for explorations in sonic interaction design”. in Proc. of the
5th Audio Mostly Conference on A Conference on Interaction
with Sound 2010.
[8] M. Cooley, M.. Sound+ image in computer-based design:learning from sound in the arts. in Proc. of the 5 int. conf.
on auditory display, pp. 110, 1998
[9] M. Bishop and D. Sonnenschein, “Designing with sound toenhance learning: Four recommendations from the film industry”. The Journal of Applied Interactional Design, 2(1),pp.5-15, 2012.
[10] M. Chion, Audio Vision. Sound on Screen (C. Gorbman, Ed.).
Columbia University Press, 1994.
[11] C. Peres and V.Best and D.Brock, and B.Shinn-Cunningham
and C.Frauenberger and T.Hermann, “Audio Interfaces”. In
HCI beyond the GUI. design for Haptic, Speech, Olfactoryand other non-traditional interfaces. Kortum, P (Ed.), Morgan Kaufman Publishers, 2008[12] D. Benyon, Designing interactive systems: A comprehensive
guide to HCI and interaction design (2nd ed.). Pearson Education Ltd, 2010[13] D. Benyon and C. Macaulay, “Scenarios and the hci-se design problem”. Interacting with Computers, 14, 397-405,
2002.
[14] J. Carroll,. Making use: scenario-based design of humancomputer interactions. Cambridge, Massachusetts. London,
England: The MIT Press, 2000.
[15] J. Preece, and H. Sharp, and Y. Rogers. Interaction Design:
Beyond Human-Computer Interaction, 4th Edition, 2015.
[16] A. Dix, and S. Brewster. “Causing trouble with buttons” inProc. of hci94, 1994.
[17] G.Coleman, C. Macaulay, and A. Newell, “Sonic mapping:towards engaging the user in the design of sound for computerized artifacts” in Proc. of the 5th nordic conference onhuman-computer interaction: building bridges pp. 83-92.
ACM, 2008.

[19] R. Beauchamp, Designing Sound for Animation’. Focal
Press, 2005.
[20] T. Holman, Sound for Film and Television (Third ed.), Elsevier/Focal Press, 2010.
[21] D. Ventura, Film music in focus (2nd ed.). Rhinegold Publishing, 2010[22] K.Kalinak,. Film music: A very short introduction. Oxford
University Press, 2010.
[23] D. Yewdall, D. “Foley: The art of footsteps, props and cloth
movement”. In The practical art of motion picture sound’(chap. 16). Focal Press, 2003.
[24] S. Lipscomb, S and D. Tolchinsky. “The role of music communication in cinema”. Musical communication, pp.383404, 2005.
[25] G.Kramer, G. (Ed.). Auditory display. sonification, audification and auditory interfaces. Addison Wesley Publishing
Company,1994[26] The handbook of task analysis for human- computerinteraction. D.Diaper and N. Stanton. (Eds.), Lawrence Erlbaum Associates, 2004.
[27] K.Collins “An introduction to procedural music in video
games” Contemporary Music Review, 28(1), 5-15, 2009.
[28] P. Vickers, and J. Alty, “Musical program auralisation: astructured approach to motif design”. Interacting with Computers, 14(5), 457-485, 2002.
[29] T. Hildebrandt and S. Rinderle-Ma, “Toward a sonification
concept for business process monitoring” in Int. conf. on auditory display p. 323-330, 2013

ISon2016-8

10