Content-basedto AidCommonMusicalof

GAVIN

Visualisation
Navigation

Audio

JAMES

WOOD

Thesis submitted for the degree of Doctor of Philosophy
Department of Computer Science
University of York
December 2005

Abstract

The amount of music available for digital archival and analysis increases steadily andswiftly. As more people listen to more music it is increasingly useful to reconsider aspectsof the tools used to playback this audio.
One of these aspects is that of navigating around a music track. I propose to enhance
the user experimence of audio playback software by providing with any musical audio
track played a visual depiction of the contents in such a way that it may be utilised witha minimum of effort.
The amount of computation power for the processing of these musical audio signals isfromforimagesOn
hardware,increasing.
techniquesaudio content
creating
also
modernI
techniques.
promay realistically utilise modern signal processing and machine-learning
fromdrawingtechniques
state-of-the-art musical-audio
pose a number of novel visualisation
signal analysis techniques.
By way of proving the thesis I prototype a number of methods to do this content-based
depiction and integrate them into a common piece of software for personal music playback.
I show empirically how usage of the novel visuals differs to both typical playback softwarewith no visuals and the traditional amplitudal representation of audio.

Contents
Declaration
1

Introduction
1.1

1.2

1.3
2

1

Overview.....................................
1.1.1 Title Definitions.............................
1.1.2 Driving Factors..............................
1.1.3 Disciplines.................................
Argument.....................................
1.2.1 Aims and Objectives...........................
1.2.2 Thesis Outline..............................
1.2.3 Thesis Summary.............................
Chapter summary................................

Music

Playback

Introduction

2.2

Related Work

2.4

Audio

2
3
4
8
8
9
9
10

11

....................................

...................................
2.2.1 Navigation Bar..............................
2.2.2 Musical Audio Navigation........................
2.2.3 General Audio Browsing.........................
2.2.4 Speech Audio Browsing.........................
Analysis of the Navigation Bar.........................
2.3.1 User Study................................
2.3.2 Task Trace Analysis...........................
Conclusions....................................

Musical1

11

Navigation

2.1

2.3

3

xv

12
12
13
15
17
18
18
19
26
29

Visualisation

29

3.1

Introduction

3.2

Related Work
3.2.1...................................
Traditional Audio Visualisation

30
32

3.2.2

Self-Similarity....................................

Visualisation.....................

.......................
111

36

CONTENTS

iv

3.2.3

Visualisationfrom Segmentation40....................

44of Tonality .........................
3.2.5 Visualisation for Content-Indication..................
3.2.6 Animated Visualisations.........................
3.2.7 Scientific and Professional Applications................
3.2.8 Representation Issues..........................
3.2.9 Colour...................................
3.3 Proposed Visualisation Methods........................
3.3.1 Visual Construction Method......................
3.3.2 Signal Postprocessing..........................
3.3.3 Psychoacoustics..............................
3.3.4 Bandwise Loudness Magnitude.....................
3.3.5 Bandwise Loudness Rhythm Magnitude................
3.3.6 Novelty Score...............................
3.4 Discussion of Methods..............................
3.4.1 Test Tone.................................
3.4.2 Trip-Hop.............................
3.4.3 Rap....................................
3.4.4 Jazz....................................
3.4.5 Classical..................................
3.4.6 Downtempo/Electronic.........................
3.4.7 Pop/Rock/Metal.............................
3.4.8 Dance/Classical Fusion.........................
3.4.9 Summary.................................
3.5 Conclusions....................................
3.2.4

4

Visualisation

Chromaticity
4.1

4.2

46
49
49
50
52
52
54
54
57
66
70
72
73
74

..

75
.
77
78
79
80
81
83
83
85

Plane Trajectories

85

Introduction....................................
4.1.1 Chapter Summary............................
4.1.2 Contributions...............................
Related Work...................................
4.2.1 The Self-Organising Topographic Map

85
85
86
......

The SOM with Musical Audio .....................
4.2.3 Principle Components Analysis .....................
Proposed Visualisation Methods ........................
4.3.1 Feature Vector Processing........................
4.3.2 Colour Space...............................
4.3.3 PCA Projection.............................
4.3.4 SOM Projection.............................
4.2.2

4.3

44

...

....

....

86
87
89
90
92
94
100
103

CONTENTS

4.4

4.5
5

6

V

Discussion of Methods
4.4.1

Test Tone

4.4.2

Trip-Hop

4.4.3

Rap

4.4.4

Jazz

111

..............................

112.................................

112.................................

....................................

114....................................
4.4.5 Classical..................................
4.4.6 Downtempo/Electronic.........................
4.4.7 Pop/Rock/Metal.............................
4.4.8 Dance/Classical Fusion.........................
Conclusions....................................

115
116
117
118
119
120

Evaluation

Aidsof Navigation
5.1 Introduction....................................
5.1.1 Chapter Summary............................
5.1.2 Our Contributions............................
5.2 Notes.......................................
5.3 Boundary-Finding Tests.............................
5.3.1 Objectivity of Boundaries........................
5.3.2 Basic Methods..............................
5.3.3 CPT Projection.............................
5.3.4 Learning Rate..............................
5.4 General Task Tests................................
5.4.1 Method..................................
5.4.2 Results..................................
5.5 Conclusions....................................

123
123
123
124
124
125
125
128
133
138
142
143
144
150

Conclusions

153

6.1

153

6.2

Discussion.........
Future Directions........
....

.

A Tools Used........

155
157

B Navigation

Tasks

159

C Reference

Questionnaire

169

D Main

Task Trial

Questionnaire

171

vi

CONTENTS

List of Tables
5.1
5.2
5.3
5.4
5.5
5.6
5.7
5.8

Tracks used for determining agreement upon boundary positions.......
The five tracks chosen for the user study. ...................
The five visualisation algorithms chosen for the user study.
.........
Analysis of variance of the factors of the experiment..............
The six tracks chosen for this user study.....................
The seven conditions in the user study......................
ANOVA of task based study's collated durations................
The averages and ANOVA probabilities of non-equal means over each question in the questionnaire.

............................

vi'

126
129
129
131
134
143
145
149

viii

LIST OF TABLES

List of Figures
2.1

Three examples of the audio navigation bar in popular musical audio playback applications. The navigation bar of the interface has been highlighted
over the original image . .............................

2.2 A bar plot of the results from the questionnaire given to a number of potential usersasking what they utilise a navigation interface for. .......
2.3

Candidates were asked to find the start of the vocals in What I Miss theMost by The Aloof. (a) marks this point. ...................

14

19
20

2.4 Continuing in the same track from figure 2.3, candidates were asked how
(c)
(a),
(b)
in
total.
thetimeseach mark theandmanychorus was playedchorus sections ...................................

22

Air (Vanessa Mae)

23

2.5 Candidates were asked to find the onset of the bass instrument in Warm
2.6

. ...............................
Candidates were asked to find the start of the vocals in Money by The Easy
Star All-Stars. This point is marked (a).....................

24

2.7 Candidateswere askedto find the beginning of the outro of rock track Can't
Get Enough by Suede (a), being informed it started with and consisted ofa vocal `aaah' ....................................
3.1

3.2

3.3
3.4

26

An example of a amplitude waveform (top) and the same audio on a dB
scale (bottom). The audio for Prague Radio by Plaid was visualised. Whilethe blue area is taken up by the wave itself, the light blue areas represent
the RMS of the wave. Constructed using the Audacity sound editor. ....
An example of a spectrogram (top) and the same audio on a Phon scale(bottom). The audio for Prague Radio by Plaid was visualised. Constructed33

34

using the Geddei Nite audio analysis tool . ...................
A recurrence plot of an auto-regressive process. Original image constructedby N. Marwan, used with permission.......................
A self-similarity matrix of an excerpt from Plaid's Prague Radio. Formed
Barkfunctionband
Cosine-distancethethecriticalonsimilarity
sumusing

36

mations as feature vectors .............................

38ix

LIST OF FIGURES

X

3.5

3.6

3.7
3.8

A rhythm spectrogram of an excerpt from Plaid's Prague Radio. Formed using the Cosine-distance similarity function on the Bark critical band summations as feature vectors. Rhythm strength varies from red (highest) throughblue to green. ...................................
A simple segmentation visualisation created from What I Miss The Most by
The Aloof. Formed using the algorithm detailed by Abdallah et al. (2006).
Time runs along the x axis; segments of the same colour should represent
multiple repetitions or variations on the musical theme. ...........
Examples of Kolhoff's blooms generated according to the content of audio
and training parameters given by the user ....................
Beethoven's Moonlight Sonata (left) and Daft Punk's Superheroes (right)
with Anita Lillie's music visualisation. Reproduced with permission......

3.9 An illustration of the core visualisation method.................

3.10 The Bark critical bands and where they fall on the audio spectrum.
....
3.11 The equal-loudness curves. Each curve denotes a constant level of perceived
loudness for all frequencies. ...........................
3.12 The sone scale in terms of phon.........................
3.13 An activity flow chart of the bandwise loudness smoothed magnitude(BLSM) technique. ................................
3.14 A depiction of the track Clubbed to Death at the various stages of the BLSM
transform......................................
3.15 Altitude (Red Square Reprise) by Hybrid visualised as a basic wave (Wave)
and with bandwise loudness smoothed magnitude (BLSM).
.........
3.16 Clubbed to Death (Kurayamino Mix) by Rob D visualised as a basic wave
(Wave) and with bandwise loudness smoothed magnitude(BLSM).
.....
3.17 The track Green Onions by Booker T. and the MG's displayed with band(BLSM)
loudnesssmoothed
magnitude
compared to loudness smoothedwisemagnitude (LSM) .................................
3.18 Takl by Plaid displayed with bandwise loudness smoothed magnitude(BLSM) compared to bandwise loudness magnitude (BLM)..........
3.19 Zala by Plaid displayed with bandwise loudness smoothed magnitude(BLSM) compared to bandwise Bark smoothed magnitude (BBSM).
....

40

41
45
48

54
55
57
58
58
59
60
61

62
62
63

3.20 Moving by Supergrass displayed with bandwise loudness smoothed magni-

tude (BLSM) comparedto bandwisemel-frequencycepstral smoothed magnitude (BMSM) ..................................

displayed with bandwise loudness
smoothed magnitude (BLSM) compared to bandwise mel-frequency cepstral smoothed magnitude (BMSM) .......................

3.21 Gabriel's

65

Oboe by Ennio Morricone

65

LIST OF FIGURES

xi

3.22 An activity flow chart of the bandwiseloudnessrhythm magnitude (BLRM)
technique. .....................................
3.23 Four Tet's And They All Look Broken Hearted (top) and Unspoken (bottom) visualised as a basic wave (Wave) and with bandwise loudness rhythm
magnitude (BLRM) . ...............................
3.24 The track Sabre Dance by Aram Khachaturian visualised with bandwise
loudness rhythm magnitude (BLRM) and loudness rhythm magnitude (LRM)

66

68

.

69

3.25 The track

Without Me by Eminem visualised with bandwise loudness
69
rhythm magnitude (BLRM) and bandwise Bark rhythm magnitude (BBRM).
3.26 The track Time is the Enemy by Quantic visualised with bandwise loudness
70
rhythm magnitude (BLRM) and bandwise Bark rhythm magnitude (BBRM).
3.27 An activity flow chart of the bandwise loudness rhythm magnitude (Novelty)
technique. .....................................

71

3.28 James Bond Theme by John Barry visualised as a basic wave (Wave) andwith the novelty score (Novelty) .........................
3.29 Several visualisations of the test tone `plucks'..................
3.30 Several visualisations of the track Stem/Long Stem by DJ Shadow
.....

72
73
74

3.31 Severalvisualisations of the rap track The Force by Aim (featuring Q'n'C).

76

3.32 Several visualisations of the jazz track Take Five by Dave Brubeck.....
3.33 Several visualisations of the classical track Nachtmusik Allegro by Wolfgang

77

Amadeus Mozart

78

Rooms...........

79..................................
3.34 Several visualisations of the downtempo track Occhi Neri by The Dining. ............................

3.35 Several visualisations of the rock ballad Generator by Foo Fighters......
80
3.36 Several visualisations of the dance/classical track Clubbed to Death by Rob D. 82
4.1

An illustrationof a fully trained SOM on Finnish phonemes. Left are thelow-level spectral data, right are the phonemes represented by the data.
Reproduced from article on Scholarpedia by Kohonen (2007a), according to87

4.2

the licence. Copyright remains with the original author.
...........
The stages and intermediate data types required for the general CPT colour-

91

4.3

projection method . ................................
The full set of histograms for the track Lebanese Blonde by Thievery Cor-

93

4.4

poration. ......................................
An activity flow chart of the CPT preprocessing technique.

4.5.........
Each of 8 tracks' eigenvalues (Lebanese Blonde, classical, rock, fusion, downtempo, trip-hop, jazz, rap). The dashed red line denotes the 10% cutoff forthe sum of the eigenvalues. ...........................

94

95

LIST OF FIGURES

xii

4.6

4.7

The CIE 1931 xy chromaticity diagram, with the human-visible gamut enclosed in the pure spectral colours (the 'tongue'-shaped curve) and the lineof extraspectral purples. The ellipses correspond to the MacAdam Ellipses
but are reproduced at ten-times the actual size.................
Three simple chrominance colour planes. Note: The colour reproductionis almost certainly inaccurate and should be taken as a demonstration andnot a reference . ..................................

4.8

4.9

96

97

Three CIE-specified chrominance colour planes. Note: The colour reproduction is almost certainly inaccurate and should be taken as a demonstrationand not a reference .

98

reference .

100

Blonde by Thievery Corporation

101...............................
The proposed RGb colour plane. Note: The colour reproduction is almost
certainly inaccurate and should be taken as a demonstration and not a.....................................
4.10 The first two principle components of the spectral histograms for Lebanese........................
4.11 A demonstration of the trajectories followed by Lebanese Blonde according
to the PCA mapping; the beginning of the track is at the bottom right, withthe ending mapped to the top right. The line's width is increased for smaller
distances moved, and the colour monotonically changes through the playing
time of the track. Small diamonds are plotted every 15 seconds of time in
the music along the locus; each minute is denoted by a larger diamond.

For ease of viewing, the trajectories are smoothed by a four-sample-wide
moving average, and a small amount of Gaussian noise is added.

......
4.12 Thievery Corporation's Lebanese Blonde visualised as a basic wave (Wave)
and with PCA chroma-projection . .......................
4.13 An example of an 8x8 SOM trained on the track Lebanese Blonde.

102

103
105

.....
4.14 The first 20 principle components of the spectral histograms for Lebanese
Blonde107. ......................................
4.15 A demonstration of the trajectories followed by Lebanese Blonde according to six sizes of SOM. The line's width is increased for smaller distances
moved, and monotonically changes from black to white throughout the playing time of the track. Small diamonds are plotted every 15 seconds of timein the music along the locus; each minute is denoted by a larger diamond.
are smoothed by a four-sample-wide
moving average, and a small amount of Gaussian noise is added. ......
4.16 Thievery Corporation's Lebanese Blonde visualised as a basic wave (Wave)
For ease of viewing, the trajectories

and with a 2x2 SOM, 4x4 SOM, 8x8 SOM and 16x16 SOM..........

109

110

LIST OF FIGURES

xiii

4.17 Thievery Corporation's LebaneseBlonde in the various stages of training
learningiiterations
SOM.
denotes
the
8x8
the
of
phase. ...
ofnumberan
4.18 Several visualisations of the test tone `plucks' ..................
4.19 Several visualisations of the track Stem/Long Stem by DJ Shadow .....
4.20 Several visualisations of the rap track The Force by Aim (featuring Q'n'C).
4.21 Several visualisations of the jazz track Take Five by Dave Brubeck ......
4.22 Several visualisations of the classical track Nachtmusik Allegro by Wolfgang

111
112
113
114
115

Amadeus Mozart

116

The amaroK music player with the BLSM visualisation operational.
....
The approximate probability of any given boundary point being agreed upon

125

between a number of people

127..................................
4.23 Several visualisations of the downtempo track Occhi Neri by The Dining
117
Rooms
........................................
118
4.24 Several visualisations of the rock ballad Generator by the Foo Fighters.
..
4.25 Several visualisations of the dance/classical track Clubbed to Death by Rob D. 119
5.1
5.2
5.3

. ..........................
The box plot and score difference matrix for all music over the experiment.

5.4

The box plot and score difference matrix for electronic music over the ex-

5.5
5.6

132

periment . .....................................
The experiment and genre-wise box plots and score difference matrix for132

electronica and in general

137

operational . ....................................

138

aids are denoted by different colours

140.............................
The amaroK music player with the BLSM & Loudness Height visualisation

5.7 Scoresof every one of the 324 trials conducted. The x-axis denotesthe point
throughout the study that the trial was conducted. The different navigation
5.8
5.9.......................
The collated duration results of the task-based study . ............
The collated seek count results of the task-based study. ...........

146
147

148
5.10 The collated seek length results of the task-based study. ...........
5.11 The histogram of opinions as to the degree of representation of the tasks. . 149
5.12 The collated `useful' question on the questionnaire for the task-based study. 150
B. 1 Task 7: (a) is the original theme, (b) is the repeat . ..............
B. 2 Task 8: (a) is the repetition of the melody at the beginning..........

159
160

B. 3 Task 9: (a) is the original theme, (b) is a quieter/slower variation, (c) is the160

repetition ......................................
B. 4 Task 10: (a) is original period of playing, (b) is the second onset.......

161

B. 5 Task 11: (a) Is the 6-second break161. .......................
B. 6 Task 12: (a) is the onset of the instrument, (b) marks 1:10. .........

162

AV

LIST OF FIGURES

B. 7 Task 13: (a) is the onset of the organ. .....................
B. 8 Task 14: (a) is the onset of the drums......................
B. 9 Task 17: (a) is the first line of the second vocalist. ..............
B. 10 Task 18: (a) is the period of the second vocalist, (b) is the onset of the first
vocalist . ......................................
B. 11 Task 20: (a) is the period over which the first words are sang.........

162
163
163
164
164

B. 12 Task 21: (a), (b) and (c) all mark the verses. There is a bridge between (b)and (c) .......................................
B. 13 Task 22: (a) marks the start of the guitar solo
. ................
B. 14 Task 23: (a) marks the end of the guitar solo..................
B. 15 Task 24: (a) marks the quiet portion, (b) marks the onset of the voice.
...
B. 16 Task 25: (a) marks the end of the break, (b) marks the onset of the instrument.

165
165
166
166
167

Declaration
I declare that this thesis has been completed by myself and that, except where indicatedto the contrary, the research documented is entirely my own.
Some material presented in chapters 3 and 5 has been presented at a number of conferences.
A list of the publications in which the material has subsequently appeared is included in
the Bibliography under Wood and O'Keefe (2004,2005).

Gavin James Wood

xv

xvi

DECLARATION

Chapter1

Introduction
I don't know anything about music. In my line you don't have to. "-Elvis1.1

Presley (1935-1977)

Overview

has seen the digital representation of music, and in particular
music recordings, grow to become the most popular distribution format. From the en(1998)
Imminkit hasDoi
by,Compact
Disc
the
and
among others,
gineering novelty of
The last quarter-century(e.
MPEG-compressed
g. MP3 files) andaudioof
the growth of portable players for this media (e.g. the Apple iPod).
One difference between previous formats of media and MP3 files is, like the transition

progressed to the Internet distribution

from tape to disc for secondary computer storage, simultaneous digital random access-1 A
number of points in the music can be accessed and processed at once with no perceivable
performance degradation; this functionality has not existed in any mainstream media
before and throws open the door to a multitude of uses, one of which the present work
addresses.
With the increased popularity, fixed-function

playback hardware has been largely re(upgradable)
firmware. People arefarflexiblemore
playback software orplaced withbecoming increasingly comfortable with-andreliant upon-this software for music playback. Therefore, even modest improvements in the design of such software would have acumulatively large beneficial effect when considered in a worldwide scope.
Until recently, intra-track music navigation was limited due to the serialised nature(e.
limitedNavigation
to operations
the
was
g.
compactofplayback medium
cassette).
CD-players
fast-forwardingonand video cassetteof rewinding andstill commonplace'Compact Discs do allow a restricted kind of random accessthough only for a single read-stream andthis would typically be hidden from the user.

CHAPTER

1. INTRODUCTION

recorders or difficult random access with the needle of a gramophone player. Other contributory factors, such as a limited visual interface and minimal numerical processing power
had conspired to prevent any significant improvement in navigation facilities.
With the exception of the addition of a basic random-access interface, popular music
playback software still has a relatively simplistic interface for in-track navigation. This
thesis posits that these interfaces are sub-optimal,

and that with the current technology

superior interfaces can be devised that better allow navigation within a piece of musicalaudio.

1.1.1

Title

Definitions

I will now discuss exactly what I mean by the given title. First by defining the terms
properly then second by identifying driving factors and related fields.
Music
Discussed in the literature by for example Terhardt (1982), there are three different classesof representation of music, which indeed may be thought of as different meanings for thenoun itself. These are auditory, concerned with our perception of the phenomenon, acoustic
concerned with the objective observations that can be made from the sound and symbolic
concerned with the music theoretic ideas such as notes, idealised timing relationships andforegoingdata
forth.
Bythe
only
acousticandany meta-data such as the score
utilisingsoor annotated timeline, the present work focuses concretely on the former two aspects. In
essence, we take music to mean music recordings rather than music compositions.
Visualisation
According to the Oxford English Dictionary, `visualisation' is:To form a mental vision, image, or picture of (something notvisible or present to the sight, or of an abstraction); to make visible to themind or imagination.
"1. trans.

To form a mental picture of something not visible orpresent, or of an abstract thing, etc.; to construct a visual image or images in
2. absol. or intr.

the mind.
3. trans. To render visible. "-Simpson

et al. (1989)

Literally we will take meaning 3; "to render visible a musical audio recording". However
it is clear from the other definitions that the general meaning of visualise has to do withthe perceiver rather than the perceived-the mind's representation over any physical form.

1.1. OVERVIEW

These definitions are helpful in describing the true goal towards which this work ismerely a small step. It is an attempt, in part, to work towards a systematic method ofcreating an image which, when seen, is perceived as being the same `thing' as the soundwhen heard. In this work we wish to create this image from the pulse-code modulation
information

only. Exactly what it means for a sonic-based perception to be `the samething' as a visual perception is an interesting question in itself and is alluded to by Spence(2001).

Navigation
This thesis is concerned with how such a thing helps one to navigate around a piece ofmusical audio media. For the word `navigate', we once again defer to the OED (using theonly term not about some form of transport):"The action or process of moving around a file, file system, website, etc. "Simpson et al. (1989)
Thus, by aiding navigation, the aim is to help the action of moving around within amusical recording (generally while being played back). This being an empirical study, I
attempt to quantify the amount of help given in several ways, but typically with respect
to some particular task to be completed.
CommonThe term common navigation is used in order to distance the present work from the arenasof professional navigational aids and/or scientific visualisation. This work is concerned withaspects of navigation as used by people whilst listening to music in the most general sense.
Though this work may be of interest and use in other, more niche, areas I do not seek toaddress them explicitly. The assumptions, requirements and metrics used, especially withregard to the user interface and expected tasks, vary considerably.

1.1.2

Driving

Factors

A improvement in the navigation facilities of common music playback systems is desirable
for practical reasons alone. In addition to providing such practical benefit, this work aims
to further our understanding of musical audio information extraction by determining thecharacteristics of music that become apparent with a variety of audio processing methods.
Such an understanding may help further work not just in terms of musical audio visualisation, but also in other fields of music and audio information retrieval where it is importantto understand exactly what is being represented by the information at hand.

4

CHAPTER

1. INTRODUCTION

While the Self-Organising Map has been proposed for several music and audio-related
tasks (see section 4.2.2), it has not yet been used to provide a simple representation of asingle piece of music. As such this work provides ideas, information and empirical observations concerning the preprocessing, parameters and outcome of using such dimensionality
reductions on musical audio data, as a method of decoding the intrinsic musical information. Related areas that will therefore benefit from this work include intra- and inter-track
similarity measures.
There is a large body of work on empirical studies of navigation within audio documentsin the context of speech but almost none for musical documents. The small empirical studyof real and current user behaviour documents a basic understanding of humans' search andretrieval approaches. Furthermore the comparison of various performance metrics between
different visualisation aids helps to understand how humans react to different annotations
and exactly how these annotations manifest improved performance. Further understandingof these HCI phenomena is advantageous for continued improvements in the field.

1.1.3

Disciplines

Several disciplines are involved in the present work; I will briefly document the subset of
disciplines and how they interrelate.
Computer

Science

The discipline of computer science forms the backbone to this work. The fields of computer
(NN)
human
interaction
informationand neural networks
visualisation,
computeraudition,
to.
draws
in
important
that
the
cases
contributes
present
on
and
some
areas
all
work
are
navigation, generally focusing on speech data, is a related field; in particular the What You See Is What You Hear (WYSIWYH) system of visually annotating(1999)
inbyisHirschberg
timeline
spoken-word audio simproposal
et
al.
a
audio
a voice
Audio-content

ilar to that in the present work. Usage of content-based information on audio recordings
back
dates
the
device
to
(real
in
the
playback
user-interfaceofaworld)
provided visuallynineties, with the work of Roy and Schmandt (1996).
(HCI) is perhaps the field in which, as a whole, thehowitselfbelongs.
Concerning
we, asofunderstandingwith
ourpresent work primarily
humans, interact with machines, HCI can be seen as the field encompassing user interfaceHuman-Computer

Interaction

design engineering. Being concerned with the actions of humans to certain conditions, it
is heavily related to the field of psychology. HCI is drawn upon and contributed to in thepresent work mostly with chapter 2 where in-track navigation is considered empirically.
Neural networks (NN) may as a whole be considered a discrete discipline in the field ofbyinspiredNN,
Thoughthey
there
generallyareare many varieties ofcomputer science.
`blackisbrain.
Typically,
limitedatheconsidered
networka neuralunderstanding ofour1.1. OVERVIEW

taking some set of inputs and providing a set of more useful
outputs. Before working properly they must first go through a'training phase' whereby thecontext of the data is learned; the parameters of this phase will make the difference between
a useful and useless end result. They may be denominated into two types; supervised
box' non-linear transformation

networks where prototype mappings of inputs to outputs are given in the training phaseand unsupervised where such mappings are not provided. The present work proposes the
Map,
help
Self-Organisingtotheneuralnetwork,
unsupervised
adoption of a particular
generate helpful visualisations.
Information

Visualisation

Information visualisation, which is in essencethe present work's proposal to aid navigation,is itself a rich, inter-disciplinary field with many heavily context-based avenues. Spence(2001) argues that exploration, presentation and indeed the navigation of data are each
intrinsically linked to visualisation.
One should not ignore more philosophical streams of thought on visualisation such as
those of Wittgenstein (2004) and in particular the arguments of Biggs (2002) and Sterrett(2004) concerning it. The representation of an abstract object or event had been, untillargely a non-issue. An idea or event typically had a canonical physicalform; an example might be the symbolic score for a piece of music or the script for a play.
With the advent of a reusable, distributable physical form of representation for a piece ofWittgenstein,music, it becomes clearer to see how Wittgenstein's
One of the key insights that Wittgensteinideas came to form.

lends us, and which the present work im-

intrinsic
the
draws
is
same
article,theofplicitly
concept of multiple representations
upon,
from
the
to
in
to
other.
thattransformations
onechange
order
and
exist
"There is a general rule by means of which the musician can obtain the symderive
from
the
it
to
from
symphonythepossible
makes
phony
score, and which
the groove on the gramophone record, and using the first rule, to derive thebetweenthese
things
inner
the
That
is
similarity
constitutes
whatscore again.
"-Wittgensteindifferent
in
be
ways.
to
entirelysuchconstructedwhich seem(2004), 4.0141.
however, the present work is concerned with the transformation
between the acoustic "groove" and a second representation which may be considered more
Unlike Wittgenstein,He(1994)
thoughts
Wrightonvisualisation.
concludes
alsooffersmore
esoteric
useful.
forThethinking.
focus
for
tools
"becomeasscientificconceptualism,avisualisations...
image can `objectify' knowledge into visible form, but at the same time `situating' it withrespect to the forms of subjectivity implied in its reading". While the context of this work

CHAPTER

1. INTRODUCTION

is science and mathematics2, it nonetheless concludes that visualisation is a useful meansof dissemination of knowledge to the public.
Psychology
Psychology and psychoacoustics play an importantrole in providing systems and theories
of musical audio may be created andwith which working models of our appreciation
developed. Psychoacoustics improve our understandingof the brain's representation ofsound, and thus how it may be optimally represented for a knowledge-based or generalsignal-processing system. Psychology and (more importantly) music psychology can help
us to understand how music is different from other sounds; a key idea if the overall aim is
to be achieved.

Since the present work concentrates on mainly low-level aspects of the musical audio, psychology plays only a small role with psychoacoustics permeating through each of
the proposals. Our analysis and evaluation of the navigation aids is largely grounded instatistical tests heavily related to and used within the field of psychology.
Electronics

and Music

Technology

Signal processing and the general field of informatics plays a most important part in
extracting useful information from the incredibly dense data source of a sound-recording.
These disciplines, together with psychoacoustics, provide a pool of understanding andknowledge which is important to consider when thinking about processing sound signals.
Thus music technology, and audio engineering in general, provide a solid base from which
the present work advances.
Music

Information

Retrieval

from
field
drawing(MIR)
ispeopleretrievala very multidisciplinary
subjects as diverse as computer science and musicology together with librarians and musictechnologists. It may be viewed as being in a similar field to computer vision as it sharesMusic information

featurehigh-level
fromi.
informationits
signal,sourcepremises; e.
retrievalmany ofbeIt
forinvariantseen as a complementalso
maymeasurements.
andsearchextraction
field
to the currently
dynamic-library-searching,in
to information-retrieval
similaraandpopular topic of data-mining. Within the fields of MIR there are several on-going researchforbrowsing
includingmethods
music,
classification
segmentation ofof music,
avenueslarge music archives, and music search and retrieval.
Muchcommunity,of the related

work discussed in chapters

as indeed are portionsof the present work.

2using chaos imagery as its main example

withinthe MIR

contributes

possibly

3 and 4 is publishedAs such MIR

1.1. OVERVIEW

7

more than any other discipline to the basis of the present work. Self-similarity matrices
were presented as an MIR device (for visualisation), and much of the utilisation of a
Self-Organising Map for music visualisation is inspired by the Islands of Music work, also
largely a MIR-groundedMusic

project.

TheoryIn defining our aim as including content-based visuals, the natural source of information
for the visuals is not the music notation but rather the audio signal data that is heard by
the listener. I consider the application of theory such as that in Lerdahl and Jackendoff(1983) to such a low-level form of data out of the scope of this work. Though signal tonotation transcription systems do exist-and indeed have improved throughout the course
do not yet perform appropriately robust or accurate.
By addressing the playback of media files, the present work is concerned with performances as opposed to compositions. With western classical music, performers will typicallyof this research-they

try to play the piece exactly as the composer intended (though subtle variations in performance may be analysed and visualised with techniques such as those described by Dixon
et al., 2002a). This is in stark contrast to jazz, especially Dixieland jazz and early folk
blues music, where performances will generally attempt to create a new interpretation
the composition by varying the melody, harmonies and even time-signature.

of

Other idioms of music, for instance electronic art music and various forms of electronica(in the modern sense of the word meaning `experimental dance music') including IDM3
"specification"
definitive
the
of
only
as
and post-rock, may use a particular recording
4
impossible
being
to
from
it
This
or
simplyunnecessary
a given piece.
results either
describe it in a substantial way notationally; Middleton (1990) writes "in most Western
popular music since rock and roll ... `non-notable' parameters are of great and often
predominant importance. ". Music which relies heavily on sampling or real-time alteration(e.g. turntablism, the art of creating music through phonograph turntables, using it in the(e.
leftindeterminate
isinstrument)
undefinedg.
composition
spirit of anor whose exact
and aleatoric music) are all examples of idioms where using notation for description of abeen
introduced
haveto address some of
Other
forms
is
troublesome.
ofnotationpiecethese concerns e.g. graphical and prosaic notation. Nonetheless, the limitations of common
music notation mean that much of the music composed in the present-day is notated onlyin the form of a digital recording.
3a widely accepted, though often criticised term, being an acronym for intelligent dance music
4This may be seen as a blurring of the boundary between composition and performance. It seemsto us
a reasonable result of the culture of artists by-and-large being the only performers of their work, a trend
surely following from the widespread ownership of performance playback devices (i. e. record players andtheir modern counterparts).

8

CHAPTER

1.2

1. INTRODUCTION

Argument

I believe that inclusion of a content-based visual in popular playback navigation interfaces
speeds up certain common tasks by complementing the navigation interface. I will argue
this by showing that it is easy to implement via a modern signal processing frameworkand furthermore very fast to compute, that it does not provide a distraction and that its
appearance is not considered unattractive (a concern technically less than rigorous but
I will first demonstrate how it intuitively shows musiof great practical importance).
cally important information such as hierarchical structure and secondly conduct several
moderately sized user studies in order to test this empirically.

1.2.1

Aims and Objectives

The previous section presented a brief discussion of the motivation

for carrying out thisproject. It explained the need for the work, and where the work fits in to the generaldiscipline. The concrete objectives I aim to achieve with this work are:
" Propose a basic working theory of users' music playback navigation in terms of
purpose and strategy.
" Determine methods of automatically generating visuals from musical audio which
aid the user in determining the content of the track for navigation purposes.
" Demonstratesuch methods working on a mainstream popular music player with
commodity hardware.

To achieve these aims, the following intermediate steps were taken:"Ireviewed the current state of the art of music track navigation aids, including the
HCI technology, their current usage and the reasons for usage.

"I

reviewed the current state of the art for visualising musical audio.

"I

designed several techniques to project small portions of audio content (audio texture
windows) into a colour.

"I

designed a trial framework for determining how useful the proposed navigation aids
were.

"I

determined the degree of objectivityof certain characteristics of music in order tovalidate the trial framework.
framework,task-based
trialsthisand used statisticalanother
conducted
under
and
"I
models on the results to evaluate the techniques and make conclusions.

1.2. ARGUMENT

1.2.2

Thesis Outline

Chapter 2 concerns the nature of music track navigation in playback and its history, I
discuss exactly what it is used for, and detail a case study into the real usage a typicalnavigation system.
Chapter 3 discusses the techniques that have been in wide use for generating visualsof musical audio. I then propose several novel techniques for generating content-based
visuals, in particular techniques for transforming small blocks of audio directly to colour.
I illustrate and discuss these techniques with both real-world musical audio and portionsof audio created specifically to test certain aspects of visualisation.
Chapter 4 continues from chapter 3 by discussing the use of dimension-reduction techniques, in particular the Kohonen Self-Organising Map neural network, in order to produce
a perceptually topologically correct mapping between audio blocks and colours. I propose
a novel use of the SOM for projecting audio data into colour by mapping the audio onto a
pre-set hue plane. A similar version of the technique using principal component analysis
rather than the SOM is also proposed. The techniques are discussed as in chapter 3.
Chapter 5 discusses the trial framework devised to test the navigation aids. I propose
two methods; a simple question-answer task list and a time-limited `boundary' search.
For the second, I discuss and present empirical evidence of objective boundaries in music.
The results found by conducting the trials are then presented and discussed together withstatistically significant statements that may be made.
The thesis concludes with chapter 6 which discusses the contributions made by thisbody of work, the empirical evidence collected and statements that may be taken from it.
I go on to discuss future directions for this work in terms of HCI, visualisation and signal
processing.

1.2.3

Thesis Summary

Proposals

Time-to-space
throughmappedvisualisations

audio recordings;

of music facilitate

navigation

are automaticallygeneratedvisualisations

using a self-organising map.
Goals

" Determine how useful (if at all) a visual navigational aid is to havein common audio playback tools.
" Determine from the musical audio visualisation methods discussed
which is most useful, and the reasons why this is the case.

Contributions" Data on, and analysis of, humans' use of intra-track navigation.
" Collection of data on and analysis of actual usage characteristics ofhumans doing antra-track navigation for task completion.

10

CHAPTER

1. INTRODUCTION" Development of novel methods for creating visuals from musical audio, specifically by projecting discrete audio chunks into a colour:- RGB-mapped bandwise magnitude.
RGB-mappedbandwiserhythm magnitude.
histogram
PCA-projectedfeatures.
SOM-projected
histogramfeatures.
" Comprehensive evaluation of techniques for use by humans on tracknavigation.
Conclusions" People are able to effectively take cues from automatically-generated
visuals and utilise them effectively to summarise and navigate
thioXIg&i 8.xo

data.

" Psycho-acoustic preprocessing on the audio signal results in perceptually more meaningful visuals.
. The SOM is able to generate a visually simpler image with no loss of
performance over other more direct methods.
"Aloudness waveform envelope visual gives a less usefulrepresentation than other texture or SOM-based methods.
traditional. Rhythmic qualities of a music audio signal lend themselves less to a
useful visualisation than basic spectral surface qualities.
" Absolute meaning in terms of colour is not a prerequisite to a usefulaudio visualisation; relative meaning is enough to provide a helpful
image.

1.3

ChaptersummaryThis chapter has presented an introduction

to the work of this project. It has introduced
the principal topics of research and discussed the motivation for the investigation of theseparticular areas. The discussion of the problem resolved into a statement of the aims
and objectives of the work, and how these relate to the specific requirements to be met.
Following the presentation of the motivation and central themes of the thesis, an outlineof the thesis as a whole has been given.

Chapter

2

Music

Playback

Navigation"I see my path, but I don't know where it leads. Not knowing where I'mgoing is what inspires me to travel it. "(1837-1885)
deCastro-Rosalia

2.1

Introduction

I begin this chapter by introducing the concept of navigation within a musical audio
recording. This is important to the content of the thesis to make the reader aware of thespecific reasons for navigating at all, the variety of navigation mechanisms proposed andpopularly used and the difficulties that they present the user.
Chapter

Summary

the chapter will begin with a review of the literaturemost directly concerning musical audio track navigation. I cover the popular navigation
bar UI mechanism evident in all mainstream musical audio playback software. I review
Following the present introduction,other techniques of navigating through musical audio such as the Link Player and therhythm-metadata enabled navigation. The review is the broadened to include techniques
proposed for navigation audio generally, and then in particular spoken-word audio such as
voicemail processing.
By the end of the literature review I will have discussed the current state of the art
in musical audio navigation. I will have demonstrated that though there is a considerable
body of work dedicated to understanding user usage patterns in spoken word recording
information retrieval, little has been done specifically in usage patterns of navigation formusical audio information retrieval.

The chapter continues with two studies conducted in order to better develop understanding of common aims in musical audio recording information retrieval; I describe the11

12

CHAPTER

2. MUSIC PLAYBACKNAVIGATION

questionnaire and interviews with users of musical audio playback software conducted inorder to determine the sorts of tasks they think they require. I then analyse and discuss
the actual actions taken by users of the random-access navigation bar when attempting tocarry out a set of tasks designed around the results of the prior investigation.
By the end of the chapter the reader will have some understanding of the concept andusage of the navigation bar when used as a popular navigation aid.
Contributions
1. I present empirical data and a discussion on the uses of intra-tracknavigation formusical audio.
2.1 present in-depth traces of the listener's experience documenting actual usage characteristics for reference task completion.

2.2

RelatedWork

The mature, mainstream literature (by which is meant a full volume with commentary andreference) on HCI e.g. Dix et al. (2004) and Preece et al. (1994), when considering musicalaudio navigation interfaces, has not progressed much further than a fairly superficial comOn reflection
mentary on the video recorder metaphor (play/stop/fast-forward/rewind).
this should not be especially surprising; PC-based musical audio playback software has notbeen in common use for more than around ten years, essentially since MP3 became popularthrough its ability to be decoded on commodity hardware. Moreover such software has
only started gaining popular acceptance (i. e. outside of computer and music enthusiasts)
acceptance in the time period over which the present work was produced with the adventof services like iThnes and Napster.
Musical audio (and, largely, audio) navigation interfaces can be largely broken down
into two types; those that present an overall sequential and continuous metaphor, and those
Examplesdiscretised`hyper-linked'
metaphor.
possibly
nonlinearandwhich provide aof the former would be the common navigation bar and the classic fast-forward/rewindinterfaces found on cassette recorders; examples of the latter would include the work by
Kosonen and Eronen (2006) and ESPACE2 presented by Sawhney and Murphy (1996).

2.2.1

Navigation

Bar

An early communication by Aigrain et al. (1995) gives a reasonable definition of the termnavigation bar:
"A scrollbar with a cursor indicating the "present" position in the document
and one can directly access some relative position in the document duration

2.2. RELATED WORK

13[sic] by positioning this cursor. "-Aigrain

et al. (1995)

Though even comprehensive HCI literature such as the volume by Dix et al. (2004)
does not explicitly analyse it, it seems likely that the popular audio navigation bar stems
from two well understood user interface (UI) primitives; the scroll bar and the progressbar. The progress bar is a read-only indicator of completeness. A typical example would
be a download window in a modern Internet browser which will represent the amount of afile downloaded. The scroll bar is typically associated with some sort of windowed view; itis a control interface allowing the user to change (and query) which portion of a document
is currently being viewed. Typically they are used because the document being viewed
is larger than the limited screen space in which to view it. An example would be thescroll-bar to the right side of a web browser window, when the web page being viewed is
too big to fit on screen at once.
The navigation bar combines these two to form, depending on which UI primitive
you assume as parent, either a scroll bar which progresses automatically or a progress
bar whose progress may be reset as desired and to which no graphical view is attached.
Several examples of musical audio navigation bars may be seen in figure 2.1. One particularfunction that the navigation provides over the older video recorder inspired bars, is theability to have random access into the musical audio recording, by providing a one-to-one
mapping between points on the horizontal of the bar to moments in time of the music.
This concept is essential for the subject material of the present work, which theorises thatcertain augmentations to this basic metaphor will significantly increase the utility of the
UI. We will now discuss the navigation bar in more detail.
The navigation bar gives the user a visual cue, which may help to imagine (or visualise)
where in the music certain events are, and where they are listening currently. This is
missing from conventional playback devices such as a CD' and cassette recorder.

2.2.2

Musical

Audio

Navigation

The earliest system suggesting navigation of specifically musical audio recording is that
proposed by Aigrain et al. (1995). The interface includes several representation `strips' of
the musical audio which correspond to segmentation results. Multiple segmentations are
created from different feature sets including dynamics, frequencies, meter and stereo effects.
The presentation of these features is somewhat cryptic and is evidently designed for the
trained viewer. Navigation is performed using the familiar random access navigation-bar
metaphor.

The authors value discrete representation abovecontinuous, suggestingthat a timeline
la display of the current time through the track is not enough (on its own) to give an idea of how far
through playing it is

14

CHAPTER(a+) a rnaroNv«"r, ion

, r, I i, untu

I I. riinning

Linux

7.0.

(c) Nullsoft
Windows

Figure2.1: Three examples

version! b)

Microsoft

Windows

NAVIGATION

PlayerMediaf{ rnr Wirrri Ali, n.,,, ft \\ inriutcs

5.
. runningon Microsoft

version

2000.

XP.

bar in popularof the audio navigation

The navigation

applications.

WinA\MP

2. MUSIC PLAYBACK

bar of the interface

musical audio playback

has been highlighted

over the original

image.

of "pre-organized

selectableobjects"

are more manipulable

than a continuous

representa-

tion.
designed for content-basednavigation
proposed systems explicitlyof musical audio is the Link Player, detailed by Blackburn and DeRoure (1998). It utilises
The
found
in
databasesongs.
pointswhere
contours
aremelodic pitch-contours
with aofuser may ask to be directed to a different point in a different audio document matched
One of the earliest

accordingto the contour.

se, but rather

navigation

As such this is not navigation
betweenpointsbetweenpointsin a trackperin all tracks. They conclude that melodic pitchofwell for this task, suggesting that information

do not work especiallyaspects such as rhythm might be better suited.

contours

typically

A rhythm-metadata(2006).

Designedfor popular

music being listened
current

enabled browser was recently proposed by Kosonen and Eronen
use on mobile

devices, it allows the listenerto by means of its section-skipping

section will repeat seamlessly, utilising

The user may continue
or backward

playing

the track

into the neighbouring

segmentationfromsections,

interface.

that

replaying

to augment

theOnce in the interface,theand beat-tracking

technology.

onwards or may move forward
it until a further decision is made.

point2.2. RELATED

WORK

15

This approach is well suited to the mobile-device platform, where summary and representation methods cannot necessarily be relied upon due to the limited space for visuals.
Other possible uses aside from navigation are given; users could, for instance, automatically loop their favourite parts of a track. That said, there is no significant reason tobelieve this form of navigation would be of popular use in a desktop environment; aside
from the relatively weak discussion of the implementation the authors provide no further
argument. Evidence that people would indeed utilise its functionality (e.g. a questionnaire
or user study), would significantly contribute to the value of the proposal.
Aigrain (2001) was one of the first to publish a commentary explicitly

referring tosystems for navigation of musical audio utilising static content-based visual representationsof the audio. He argues that the optimum representation-based approach would be a visual
annotation that could present the content in various levels of abstraction from the acoustic
to the notational:"But [a representation-based browser] faces the extreme difficulty of automatic transcription and also suffers from the fact that the representation
hides some important features of the content (for instance, voices for speech or
performance for music). Static representations of audio content take all theirvalue when they can be presented at different scales and levels of abstraction(2001)and directly associated with sound production. "-Aigrain
The present work does not attempt such an all-encompassing navigation aid for two
main reasons. Firstly, and as Aigrain points out, content-based transcription is a tremendously difficult task and only applicable for relevant music. Secondly, it is an open question
as to how to provide a compact interface to such information. The present work aims toprovide an interface readily comprehensible and adaptable to popular playback software.

2.2.3

General AudioBrowsing

General audio browsing interfaces do not restrict themselves to any particular audio content. Typically this is an easier problem to solve than information retrieval in any one(e.
featuresidentify
differentiate
thetog. musicare coarser
andparticular context sincefrom speech rather than rock from jazz). Since computation power (and general understanding of the problems involved) is forever increasing, there is a significant amount ofdedicated
literature
to this subject.
earlyrelatively
Kimber and Wilcox (1996) present an early example of attempting to augment a basic
(film
for
navigation
systemgeneralsoundtracks is the example given)audio
random-access
The latter technology iswith automatic index generation from content-segmentation.
described at length in section 3.2.3. The browser interface presents several timelines ofeach of the `voices' (speakers, songs, applause etc. ), which the user may utilise to direct

Y
IBg
8
t3
lti
1ý

ýi.

16

CHAPTER

2. MUSIC PLAYBACKNAVIGATIONThis would have several drawbacks for musical audio; inmany types of musical audio there is not a well-defined (or usefully small, at any rate)
number of voices. Furthermore, having multiple timelines takes a significant amount of thethe random-access navigation.

user-interface; only a far more compact representation of the content would be acceptable
for a popular playback application.
and Cook (1999) presented a prototype implementation of an audio
browser which includes basic segment boundaries together with a CD-player like navigation interface of fast-forward/rewindalong with next/previous segment. The display
Tzanetakis

giving the user an image of the audio portrayed as a basic waveuser-study suggests that automatic content-based segmentation can

is representation-basedform.

An informal

help with navigation

through the audio document.

A comparison of different persons'manual segmentation of musical audio reveals common parts labelled as segment boundaries, suggestive of an underlying `objective' ground truth segmentation.
Comparatively,

the present work explores a different approach giving the user more
freedom in their control method (random-access rather than sequential jumping), thoughboth approaches appear quite valid for ultimately

easing navigation for the typical usersof popular playback software.
Tzanetakis and Cook (2000b) present some follow-up work to the above. A small user
study suggests once again that there is reason to expect some degree of objectivity insegmentation of musical audio, with around 70-80% of segment boundaries being agreedof participants. The content segmented includes both musical audio
and music/speech audio; no figures are given for musical audio alone.

on by the majority

SATIE is a piece of interactive software for expert use intended to guide the viewingof, and navigation within, musical recordings presented by Lepain (1997). In order to aid
understanding and navigation within the musical track, it presents multiple
content-based and manually populated representations of the track. The navigation is a
described.
bar
to
the
previously
concept
navigation
simple random access metaphor similarinterpretation,

At the time of publication, only the two basic representations, waveform and spectrogram,had been implemented though plans for more ambitious representations were proposed.
Although founded in a somewhat different context (SATIE being developed primarily foran expert audience), this work does share the notion of providing aid for random-accessnavigation, though presently I will provide a more thorough investigation.
Another interesting outlook on the audio navigation problem is ESPACE2, as presented
by Sawhney and Murphy (1996). Aiming to provide an accessible system for visually
impaired users, it is an audio-only user interface which provides a hierarchical and spatial
interface through the concept of moving in and out of virtual `rooms' (in a similar way to
the work presented by Kobayashi and Schmandt, 1997, mentioned later).
These approaches, while ambitious and possibly useful to a minority, are hardly appro-

2.2. RELATED WORK

17

priate for a mainstream application due to, amongst other things, extra costly equipment
required (multiple speakers), the deterioration of listening experience and the training
required for a relatively unfamiliar interface.

2.2.4

Speech AudioBrowsing

Speech audio browsing is another form of audio navigation.

The content of the audio isthat of spoken-word and applications tend to focus on recordings of either single speakers(e.g. voicemail) or multiple speakers (e.g. diarization). The challenges faced when making
a good speech audio browsing interface share a considerable amount in common with those
focusing entirely on musical audio. The solution, argues Kimber et al. (1995), is similar
to that proposed in the present work:"It is difficult to find specifics in audio recordings because it is necessary
to listen sequentially... Although it is possible to fast forward or skip around[using a random access interface], it is difficult to know exactly where to stopand listen. For this reason, effective audio browsing requires the use of indicesproviding some structure to the recording... These [indices] can be displayed
et al. (1995)
graphically as a navigational aid in browsing. "-Kimber
The interface proposed and prototyped by Kimber et al. is similar to the other audio(1996),
focusing
Wilcoxboth
byKimber
being
the
time
on multiple
andproposedatsystemtimelines depicting the onset of various voices.
Kobayashi and Schmandt (1997) suggest utilising people's natural spatial-awarenessto aid the navigation of audio by mapping the time of the audio document to a spatial
metaphor:"The motion of the sound sources maps temporal position within the audio
into spatial location, so that listeners can use their memory of the spatial(1997)
Schmandt
location to find a specific topic. "-Kobayashi
andThe NewsComm system described by Schmandt and Roy (1996); Roy and Schmandt(1996), proposes and prototypes a device capable of aiding the browsing of audio recordingsof the news. It relies upon the segmentation of the audio into one of a number of speakers
or silence in order to give a discrete representation of the audio and to aid users with a(1999).
described
Tzanetakis
Cookininterface
that
to
and
speakersimilarnext/previous
The Dynamic Soundscape project, a follow-up work to News Comm, spatialises several(spokenthroughoutword) so that they haverecordingaatvariouspointsstreams of audio
the effect of apparently coming from various points around the user. They will naturally
use the `cocktail party effect' to listen to all streams and, when appropriate, `home-in' on

18

CHAPTER 2. MUSIC PLAYBACK NAVIGATIONa stream of interest by stopping all but a single stream of audio which corresponds to asection of the document in question.
The work of Hirschberg et al. (1999), Hirschberg and Choi (1998) and Nakatani et al.
(1998) is particularly interesting to us at present since it details a system proposal together
with empirical data about usage characteristics concerning random-access navigation ofvoicemail documents with visual annotation. The studies suggest, in the field of voicemail
retrieval, that random-access navigation on its own is far from optimal: "users
often lose track of the current audio context, and being unable to determine the sequence

informationand structure of different elements of the audio record". Furthermore:"obvious signposts such as topic/message boundaries may be less helpful
than users expect them to be and perhaps even counter-productive to users
acquiring a basic understanding of the data. Given this result, we are exploring alternatives to simple topic markers including... acoustic segmentation,
particularly as a means of enhancing users' ability to extract the information
they seek from the audio data that has been presented to them. "-Hirschberg
and Choi (1998)
We may take this as a (weak, since the context is somewhat different) indication thatannotation from simple high-level boundaries may not be the best method to approach
the problem of aiding navigation of musical audio. An acoustic-based representation could
yet prove more useful in terms of usability, as indeed the present work postulates.

2.3

Analysisof the NavigationBar

Due to the relatively sparse collection of literature explicitly addressing the real-world usage
characteristics of the navigation bar for musical audio navigation, I conducted two small
investigations; in the first I solicited information from users of musical audio playbackIn(random-access)facilities.
the
how
the
secondtotheynavigation
utiliseassoftwareI devised a set of reference tasks for eight users to carry out; the tasks were largely
five
bars.
The
of
typicaltheofanalysis
results andusage of navigation
representative of
these tasks are given in the second section.

2.3.1

User Study

I conducted a small user study, soliciting answers from 16 potential users of the navigation
system from a questionnaire. The aim was to find out:
" What sort of tasks the navigation interface would be used for.

2.3. ANALYSISOF TILE NAVIGATION

19

13AIZ

" If the tasks were for information-retrieval

purposes, what sort of information would

he retrieved.
views from users is in line with typical I [Cl methodology of determining
an objective method for evaluating the benefit of one interface over
The reference task agenda proposed by Whittaker
et al. (2000) was used as aanother.
This initialquestionnaire

when forming

broad guideline
naire, I included

boththis pilot

study.

my own expectations

To determine

the optionsin the question-of usage as well as the thoughts

A copy of the questionnaire

informally.

solicited

soliciting

given is available

in appendix

of others

I

C.

Content of vocals
Instrument onsets
Repetitions & variations
Limits of verse/bridge/chorus
Onset of vocals
Overall structure
Breaks & pauses
End of song02468

Number of votes
2.2: A bar plot of the results frone the questionnairefor.
interfacetheyutiliseanavigation
users asking what
FigureThe results are plotted

is used typicallythat navigation
Attempting

in figure 2.2. It is reasonably
to find the content

to find overall structure

for navigation,

given to a numberof potential

clear to see frone the small studyof vocals in presumably

popularmusic.

in the music was apparently

not it significant priority
already have some idea of thesome users imagine they would
broad content of the track before attempting
to navigate. Finding the end of the song (i. e.
the outro) and any breaks and pauses inside the track appeared similarly unneeded. Eachof the other2.3.2

suggesting

possibilities

Task

Trace

suggested appear roughly

similarin popularity.

Analysis

In this section I will discuss and analyse the actions taken by candidates asked to performtasks broadly representativeof those classes resulting frone the previous section. This will
and as such I will not begin by building it model;
data and it discussion noting
this chapter aims to provide the reader with well-presented
be
for
I
from
the
to
It
taken
is
that
trends.
thisactions
considerwill
clear
overviewany

not be considereda pure HCl experiment

improvement and thereby build a hypothesis proper.

20

CHAPTER

Although

23 tasks were given in total,
failed

2. MUSIC PLAYBACK

discarding

through

NAVIGATION

those tasks whose entire set

Five of these will be presented
here and discussed, but the interested reader may refer to Appendix B for traces relating
the other 17 tasks. Four of the tasks were finding the time of a particularonset (of eitherof participantsa section,

instrumentof the entire

to answer correctly,

or vocals),and one was collecting

informationabout

the structurerecording.

In all cases, the participant
shouldonly 22 are left.

completeout the task. They were told they
but that they should take every effort towas alone when carrying

the task as quickly

as possible,

ensure they gave the correct answer. They were given six minutes to practise and become
familiarbar should be used towith the user interface and were told that the navigation
reduce the time taken. After completinga task and entering an answer, they were not told
if they answered

VocalsSearchFigure2.3 shows

candidates

correctly.

the rock-electronica

attempted

to find the initial

track

What

I Missthe Most

by The Aloof.

The

onset of vocals in the track.

When is start of vocals?

0
co

y

l0

CL
4)
E
I
0
N

O

u: uu

1 : VN

Figure 2.3: Candidates were asked to find the start of the vocals in What I Miss the Mostby The Aloof. (a) marks this point.
only one (hollow circle) appeared to follow an effective systematic
approach using random-access navigation. She made three large jumps of around 15-20
Of all participants,

2.3. ANALYSIS OF THE NAVIGATION BAR

21

inlisteningforfew
betweenarriveda vocal
each
until
she
only
a
secondseach,seconds
herdeliveredistrinary
at the required point.
essentiallyachopsearchsection; what
A second participant (filled circle) seemed to be comfortable utilising the random-accesspotential of the UI, but her search is less than efficient; she overshoots, undershoots andreviews several portions of the track in quick succession. Interestingly she never actuallylistens to the onset itself, apparently guessing correctly by picking a point somewhere
between the vocals and instrumental sections.
Three participants (the upright triangles and diamond) did not utilise the random
access nature of the navigation bar at all, deciding to listen through almost a minutewithout interruption. It seems unlikely that all three actually liked the music so much todue
bar,likely
it
this
they
took
the
courseofactionso
seems
navigation
usageof
abandonto the absence of clues as to where the vocals might begin (except that they should be
into(upturned
jumpedthe track andAnother
beginning).
triangle)
the
way
a
small
near
essentially listened through, possibly under a similar reasoning.
(the square) found the correct onset point within 15 seconds, fasterthan any other. However, rather than stopping their search they instead appeared to verify
it was indeed the initial onset by listening to a larger portion before it (0: 40 to 0:50), andOne participant

then skipping even further back to 0:22 and listening for another 10 seconds. This themefortask
traces.
thethroughout
continues
verification
multiplerunsrequiringof
In conclusion, it seems most participants either thought they were unable to use the navigation bar without more immediate information to the contents (and so listened throughlarge portions of the song) or, despite trying, found it difficult to make particularly effective
use of it.
Structure

Assessment

As a follow up task to be carried out directly after the previous task, participants werefacts;
firstlykey
they needed
had.
This
two
how
the
requiredmanychorusessong
asked
to assessthe song and find the portion representative of the verse. Secondly they had todetermine on how many separate occasions this part of the music was played.
Figure 2.4 shows the trace of the six participants who correctly answered the task 13'.
Each of the three choruses are shown, notably there is a bridge following the third verse(i. e. where one might expect a third chorus to be) between (b) and (c) at around 3:30-4: 00.
Three participants (upturned triangle, square and diamond) utilised the random access toreview this portion of the track presumably to check that it was not another chorus.
In general, three of the six participants (triangles and square) tended to use the randombuilding
fast-forward-liketoa series of reprequicklyofmechanism,asorteffectaccessby(around
ten
five
the
blocksskippingaroundseconds
songofseconds
each)sentative
between them. This is a reasonable usage for random access given that the question covers

22

CHAPTER

2. MUSIC PLAYBACKNAVIGATION

How many times is chorus played?
a) 1:22-1: 39(b) 2 28-2 45(c) 4 22-4: 39

INX

0
0
c)
0
U,
N
O
VO

N

y
C
m

CL
E
0
Or

0
Ljr)

0
OOOOOOOOOO'OOOOOýOOOOOýOOOOOýOO
N
ONN
t'1
t'1
VNýNMOýýNM

0: 00

Figure

1: 00

Oý

2: 00

3: 00

ýN

týf

O

ý(J

rN

5: 00

4: 00

2.4: Continuingin the same track from figure 2.3, candidates were asked how manytimes the chorus was played in total. (a), (b) and (c) each mark the chorus sections.

the whole song and no further informationTwo(diamond

participants

and hollow

is given.
circle)listenedthroughthe entire

shown by three in the previous task. One of thenbriefly review two sections (the aforementioned
bridge and the final verse).
usage of the random access for verification.
the approach

plicatingtrack,

du-does howeverThis suggests(filled circle) skipped through the track by about 30 seconds
One of the candidates
each time, listening for around five. After getting to the end of the track, she reversed
direction

until

findingthen the second. It seems she aimed to get a broad
listening to only very short sections and making large

the first chorus,

overview
jumps
requiredas quickly as possible,
between them.
Unlike the otherverificationof the middle(upturned
triangle),
participant
she apparently
before making the final answer.
structure

Overall trends in this task seem to be very suggestive of utilising the navigation bar
to skip around large sections quickly for gaining a long-term summary and verification of
suppositions for the structure. This `whole track canvassing' appears to play a consistentrole throughout

the task traces.

2.3. ANALYSIS

Instrument

OF THE NAVIGATION

BAR

23

Introduction

We move on to anotherpiece of music now, this time the classicalposed by Mike Batt and performed
Vanessa Mae taking

by Royal Philharmonic

track

Orchestra

Warm Airwith"childcom-

prodigy"a lead role on the violin.

When is bass instrument onset?

0
0

o
Co
N
NQ
y
ß

o

0
N

0

V: W

Figure1: VV

were asked to find the onset of the bass instrument

2.5: Candidates(Vanessa

in Warn, Air

Mae).

The participant

to finish(diamond),

the task first

aside fromseveral jumps

at thecheck she hadn't passed it around 10 seconds in, finds it very
simply through skipping forward and apparently making a lucky guess at how much before(circle) had a similar
the destination of her penultimate jump it starts. A second participant
beginning

to apparently

approach,thoughhowever, overshot
the point

maintainsby considerably

before, so while almostof the jump

her several jumps

to locateand systematic

search.

The third

participantof onset after aroundthroughfor this;eitherthe onset.

to complete

a perfectbinarysearch it still took

her trace demonstrates

the task correctly

would he similarsuspecting

an effective(square) skips to exactly

the point

this, she reviews the previous

15

to the other two were she not then to continueof the track. I would advance two possible reasonsthe directives for the task and simply wanted to enjoy thealmost anothershe neglectedmakingNonetheless,25 seconds. Presumably

seconds. Her performancelisteningequal skip-listen ratio (about 6-7 each time). She,
more (around eight seconds past) and was very close ata roughly

minute24

CHAPTER

music at the cost of a quickcompletion,

2. MUSIC PLAYBACKor she was unsure

about

whether

NAVIGATION

the onset sheheard was the appropriate

one. In any case she then reviewed the onset twice more before
herself by around ten seconds with the initial jump
undershooting

her answer,

recording
back.
This

task demonstrates

lack of information

an efficient

overshooting

appears

to be an issue.

VocalsSearch 2

use of random

and undershooting

Candidates

were again asked to determine

The music

used in this occasion

arranged

access in searching. but through aknown points for reviewing content

the time of initialis the reggae track

introduction

Money,of the vocalist.

by Pink

written

Floyd

and

by The Easy Star All-Stars.

and performed

When is start of vocals?

0
U)

y

o
N
N

CL
a)
Eo

04

O

0
V; VV

FigureI; w

were asked to find the startpoint is marked (a).

2.6: Candidates

Star All-Stars.

ThisThe two participants
hollow

triangle)(square),participant
to replay
actionlisten

who took
throughthoughthe longest

the track

clearly

without

hearing

the same portion

I can offer is thatof the tracktimeof the vocals in Money

to completeskipping

until

the task (diamondthe vocals start.

to an answer.

andA thirda long run up to the vocals (0: 32 to 0: 46), optsof the track once again; the only explanation
they felt the need to become comfortable
withbefore committingby The Easy

for this course of
the early contents

2.3. ANALYSIS OF THE NAVIGATION BAR

25

The two fastest-completing participants (filled circle and inverted triangle) skip aroundfairly radically listening to about 2 seconds before skipping about 10 seconds. When they
overshoot they reverse direction eventually arriving at the correct point. Notably, one of the two (inverted triangle) reviews the onset point before answering correctly.
Despite having already found where vocals start, one participant (hollow circle) caninevitably

vasses a significant portion of the track (up to around 1:15) before navigating backwards.
They again overshoot into the instrumental part, but again appear to want a `broader
view' of the track since they jump further back (to 0:27), reviewing the early portion again
before skipping forwards and correctly arriving at the onset.
The final participant (filled triangle) is played the onset of vocals three whole times
after canvassing the beginning and finding a perfect excerpt. Despite listening to thesame period before the onset three times she navigates back to 0: 18, almost 30 seconds
backwards, presumably to verify that it is the first onset point.
The overriding theme here is one of verification; three participants found the correct
point before skipping backwards through the track, suggesting they were not confident
that they had not missed an earlier onset. Canvassing also seems to be a recurring theme,
whereby participants make long sweeps, skipping significant portions and listening for only
a few seconds at a time. I would expect this is in order to get a broader view of the track
possibly to back up their mental image of its structure.
The Beginningof the End

The final task I will review here is a search for the outro of the rock track Can't Get
Enough by Suede. The task was phrased to give participants the knowledge that the outro
had novocals and comprised the singer singing a recognisable `aaah'. This,,ýnlike othersearches, gave them a solid search strategy (i. e. find the transition between vocals and`aaah'), once vocals had been found, the transition may be located efficiently through abinary chop search with the end of the track.
One participant

apparently realises this premise and acts accordingly; she skips using a reasonably precise binary chop with the end, listening for around 3-4 seconds per
jump. On hearing `aaah' first time (4: 02), she retraces her steps. Interestingly, another(triangle)
answered correctly first; she employed a similar strategy, though wasparticipant
more aggressive, skipping far more of the song initially. Having listened to a portion ofthe `aaah', she notably decided to jump forwards again, listening for another five seconds
before retracing and arriving at the transition. A further participant exhibited similar
behaviour, apparently requiring a broader context of the `aaah' section before being happy
to retrace.
The participant to finish penultimately, actually managed an almost perfect first jump,
overshooting the transition by only five seconds. Having listened to the `aaah' of the outro

OFYORýt

26

CHAPTER

2. MUSIC PLAYBACKNAVIGATION

When is beginning of outro?

0
Co

0
Co
0
V
N

o

tu
av
a>

E
I-

0
N

0

Figure 2.7: Candidates were asked to find the beginning of the outro of rock track Can't
Get Enough by Suede (a), being informed it started with and consisted of a vocal `aaah'.
for around three seconds, she then skipped back a small amount and actually heard thestart of the transition. Despite this she skipped backwards, further into the vocals several
times over, before eventually allowing the end of the track to play through to the transition
point. This is again suggestive of the lack of confidence the participants have in a 3-5 second
context.
The participantfinishinglast (inverted

before jumping

to within

vocals,retracesher steps twice.

jumps

furtherin, before retracing

again suggestivewithinformationof an unwillingness

Havingarrivedand finding
to trust

to the location

two seconds into

the transition
the logicalpoint.

the outro,

she again

This behaviour

is oncecombinationof a single sampling
Because of this, itin question.

of the point
and seconds of listening before the search can continue

pertaining

costs several more jumps
and the point found.

2.4

aroundcanvassed slowly for around 30 seconds
ten seconds of the transition
point and, finding only
triangle)

properly

Conclusions

I reviewed the current state of the art of popular music track navigation aids, including
the HCI technology, their current usage and the reasons for usage. I conducted a study2.4. CONCLUSIONS

27

resulting in a histogram of task types that are commonly found among users of audio
playback software. Given this I created a set of reference tasks of which five I analysed the
behaviour of various users when carrying out. From this I can make two main conclusions
concerning the behaviour of users while utilising the random-access navigation bar.
Firstly, of those that gave the correct answer, those who used the random access toactively search out an answer generally did better than those who did not, letting thecontent play without interruption or jump. In absence of any other information, I would
suggest the reluctance to use the random access properties of the bar is due to a lack ofconfidence that they would not skip something important. Thus I favour the idea thatgiving an indication, where possible, that they would not skip anything importantlead to users being more confident with using the bar for random access.

would

Secondly, when users managed to use random-access properly and skip to the rightplace, there is a tendency that if the skip destination is within about three seconds ofthe very point they are attempting to locate, they will skip further backwards by several
seconds. This suggests they are unhappy having found an apparent "local optimum" andrequire a review of the point in a wider scope to make sure it is in fact correct. Where
prudent, reinforcing the idea (perhaps by visual indication) that the content is relatively
constant around the apparent "local optimum"
skipping and listening for a wide-area review.

may help them avoid any unnecessary

This concludes the review of musical audio navigation technology. In the next chapter
I will focus on techniques for visualisation of musical audio, in order to determine how
best the navigation bar user-interface metaphor can be augmented in order to provide`visual indications' that I would argue can improve the utility of the navigation bar for thereference tasks.

28

CHAPTER 2. MUSIC PLAYBACK NAVIGATION

Chapter3

Musical

Audio

Visualisation"We have to remember that what we observe is not nature herself, but nature
exposed to our method of questioning. "-Werner

3.1

Heisenberg (1901-1976)

Introduction

Thus far in my argument, I have shown that people find navigation within a musical track
useful. I will demonstrate that the mechanism used for popular audio navigation canbe augmented with a visual generated directly from the audio signal. At the end of thischapter I hope to have shown that these images are, by and large, reasonable visualisationsof the musical content of the audio signal, in so much as they are representative of thekinds of aspects we would want to identify.
Chapter

Summary

Following the present introduction, the chapter will begin with a review of the literaturemost directly concerning musical audio visualisation. Relatively basic methods like theamplitude graph and the spectrogram are covered. More involved techniques such as selfsimilarity matrices and timbregrams follow. Techniques meant for performance analysis(e.g. the performance worm) and for browsing and selection (e.g. music icons) are thenAlso covered are techniques meant for professional analysis used in various
software packages as well as techniques developed primarily for their aesthetic value andreviewed.

not for representative content.
General analysis techniques are then reviewed which, though not explicitly proposed
as visualisations, appear to be potential candidates for the refinement process nonetheless(e.g. segmentation, novelty). A brief review is made of the literature not directly connected

29

30

CHAPTER 3. MUSICAL AUDIO VISUALISATION

with musical audio visualisation,

but concerning the visualisation of music generally; theproblem of representing musical content.
By the end of the literature review I will have demonstrated that a technique accomhas
I
for
the
the
sort of visualisationplishing exactly
would wantnavigation metaphor
not yet been proposed' though several techniques might, with modification, be possible
candidates. I also hope to have demonstrated the challenges of reducing such a complex
inbothimage,
in
to
terms
and
a representativeof meaning
entity as a musical recording
more practical terms of generation from content.
I then propose a general methodology of creating content-based visuals for popular
the1-dimensional
the
to
of
content
series of colours
represent
navigation, which relies on amusic as the recording plays through. I propose several methods to generate those colours,
beforedemonstrate
I
I
the
term
makingmethodaudio.
eachwhichchromatic projection ofan example-by-example comparison between methods over various genres together with afew contrived `test tones'.
By the end of the chapter the reader will understand the strengths and weaknesses ofeach of the methods over several types of music.
Contributions"Aof a general technique for generating visualisations together withthree novel concrete techniques.
formalisationtotheDiscussionthesetechniques,
their
relationsofadvantages and problems and"existing techniques.

Related

3.2

Work

I will break down the various visualisations into five groups:
trathebeAfield
without
completereview would notsignal visualisation
ditional and widely adopted methods for visualising not just musical audio but audio

Traditionaland signals in general.
Self-similarity

The self-similarity

matrix is perhaps the purest piece of work related to
It was introduced as a method specifically for visualisingmusical signal visualisation.
musical audio and has spawned several techniques based upon it.
Structure

extraction

and segmentation.

This

field

Havingconcerns

worksuch a high-levelin music

thumbnailing,refinement

fingerprintingof the data is clearly'or, perhaps more accurately, had not been proposed when the initial survey was madead-

3.2. RELATED

WORK

31

vantageousfor visualisation from the sample graphics in the publications from thisfield.
Visualisation

I conduct a brief study on
and representationof music composition
the thoughts and currents running through the field of music notation. This is notparticularly practical in terms of musical audio visualisation, but does give insight
into the sorts of representations (aside from traditional music notation) that havebeen proposed.
ad hoc visuals

in software

The most practical pieces of work in this field survey; I
review a selection of common and niche audio and music software packages to discuss

the extent of their visualisations.
For an overview of musical audio signal processing generally, I would refer the reader
to the thesis of Hainsworth (2004), who spends most of the document going into far more
depth then is required presently.
A Note on Information

Visualisation

Spence (2001) tells us that visualisation can be broken down into several discrete processes,
however it is useful to define visualisation in terms of two of these in particular, since itwill be the two that I concentrate most on in this work:
Visualisation = Data refinement + Data presentation
Refinement is the stripping and transformation of the initial data into a usually but
not necessarily smaller data set. It is in this process that aspects of what the viewer wishes
to see from the data is extracted. The efficacy of a refinement might be illustrated in loose
terms of precision and recall; a good refinement will contain as much of the data relating
to the information to visualise as possible (recall) and as little data as such unrelated
As such this stage is entirely context dependent, since no decisions can bemade about what data to be cast aside or amalgamated unless one knows what the data(precision).

means and thus how it relates to the information contained within.
Data presentation relates largely to putting this extracted data into a view most befitting the situation, which comprises at least the phenomenon to be visualised and the
expected viewer. As such, data presentation is less dependent on the context of the data,and is more dependent on human factors. The use of positions, colours, sizes, dimensions,layout, topology and symbols play a large part in this stage. It is the responsibility of thisstage to present the refined data as clearly as possible to the viewer.
In terms of musical audio signals this becomes:

32

CHAPTER 3. MUSICAL AUDIO VISUALISATIONMusical audio visual = (Signal preprocessing + Audio analysis) + Visual construction

Signal processing goes largely understated in music visualisation techniques. It will
typically be of the general form of conversion to some frequency-domain representation
with some psychoacoustic processing typically either MFCC or critical banding and someperceptual loudness scale.
A Noteon Formal

Descriptionof Visualisations

For each explicit method of visualisation, I will consider the type of visual construction it
generates. I describe this by formally describing its space-time dimensionality; I denote
this by writing

a single S for each space dimension and aT if the visualisation is timebased (i. e. animated). I also denote the degrees of freedom for each component making(full3
the visual; generally this will be either 1 (monochromatic/linear
colour),orscaling)
though (as will be demonstrated in the next chapter) could conceivably be 2. I will define
0 to mean a purely binary value. A spectrogram, for instance is denoted SS-1 since it has
two spacial dimensions but is viewed statically(i. e. unchanging through time) and each
be
by
in
is
linearA
wouldcontrast
point
space
represented as avalue.
static waveform,
SS-0, meaning it is still represented as a 2-dimension image, but that each point in theimage may either be part of the waveform graphic or not.

3.2.1

Traditional

Audio

Visualisation

Here I review the two main visualisation techniques which can be used on musical audio
signals (and, indeed, any signal) which are widespread but relatively simple; the waveform
and the spectrogram.
Waveform
The waveform is the single most canonical method of viewing audio. It is essentially atime-amplitude graph of the displacement wave which gives rise to the sound. In terms(hence
data
from
but
is
the
it
source audio
a visual constructionof a method,nothing
canonical). Unlike any of the other approaches here it needs no signal preprocessing or
analysis. The visualisation's formal form is SS-0, though it may be seen in an animatedform where short excerpts from the signal are visualised at a time (this is often found inreal-time applications such as the screen of an oscilloscope). In this case the visualisation
becomes extruded through time, formally making it SST-0.2
21 would note with a small amount of humour, the spiral groove of a vinyl record could be considered
an SSS-O'visualisation'-orperhaps actualisation-of

the waveform.

3.2. RELATED

Variations

33

on this visualisation
A commonof amplitude.
wave.

WORK

Anothervariant

involvea change of scale or meaning

is to use the RMS (root mean-squared)

amplitudeof theto construct the visualisation;
scale of amplitudein the Audacity sound editor (Mazzoni and Dannenberg,2005).

An example

3.1:

generally

is to use a logarithmic

both of these are utilised

Figuremethodscale (bottom).

of a amplitude

The audiofor Prague

waveform
Radio

(top)

by Plaid

and the same audio

on a dB

While

the blue

was visualised.

area is taken up by the wave itself, the light blue areas represent
Constructedusing the Audacity sound editor.

the RMS of the wave.

When used to visualise musical audio, a rather uninspiring image is found. Certain
aspects of the audio such as large changes in dynamics or spectral content may possibly
be visible, but the visualisation is difficult for anyone but an expert to understand. 3 Dueto its ubiquity and simplicity this representation is used as a baseline against which othertechniques may be compared.
has some bearing

Basic amplitude

are much more accurate,on perceptual

loudness but psychoacoustic

methods

throughtime helps to give the viewer

a clearerand smoothingidea of mid and long-termchanges by reducing

the apparencyof short-term

dynamics.

and harmonic content
byBrinkmanManually
diagrams
largely
invisible.
those
and
proposedsuchascreatedarethe dynamics
Mesiti (1991) use the basic concept of a time-loudness
plot to demonstrate
The visualisationis still limited

in a musical sense; timbral,

melodic

Asfor
instruments.
thesuch, short and
each
of
piece
progression`noise' is never introduced;
the result being a diagram showing only the most
medium-term
diagram) subjectively-important
long-term and (according to the author of any particularthroughout

an orchestralchanges to the dynamics.

Spectrograms
The spectrogram
equivalent,

to a Fourierof a particular
spectrum'An

It is related, but notimage of an audio signal.
of a signal. Spectrograms show the power or magnitude

is a time-frequency
transform

frequencyof a signalof sinusoid at a particularshows the magnitude

time in the signal, whereas the Fourierof a given frequency

example of such an expert might be Arthur G. Lintgen who, according
1981), is able to `read' vinyl records from the patterns of grooves.

of sinusoid

throughout

to Holland(19 November

34

CHAPTER

3. MUSICAL

AUDIO VISUALISATION

the signal; since the advent of the Fast Fourier Transform (FFT), finding the Fourier
series (as well as other related transforms such as the Discrete Cosine Transform) has been
computationally cheap.
The spectrogramis therefore

a 2-dimensional

spatialof a piece of musicalrepresentationof the figure may take only a linear value (i. e. the magnitude) and so
declare it SS-1. Figure 3.2 shows a spectrogram.
The `dancing bars' animatedaudio; each component
O formallyvisualsoften

seen withis directly

spectrum

analyser,projectionof otherwise

FigureGeddei

The audio
Nite

audio

systems,whichconnected(top)of a spectrogramfor Prague

Radio

In terms of the visualisation

methodI would

call ain that is an SST-O

and the same audio

on a Phon

Constructed

was visualised.

stages, spectrograms
will typicallyby some time to frequency-domain

the signal will be preprocesseddata which will be used to construct
the FFTby Plaid

formally

scaleusing thetool.

audio analysis

In practice,

playback

to this visualisation
the same basic data.

An example

3.2:(bottom).

electronic

have no analysis stage;transform

and it is thisthe image directly.

is used to calculate

using the process of windowing, whereby the signal is split into multiple portions and each portion's Fourier series is
The process of constructing
calculated separately.
a time-frequency
structure in this way
in known

the spectrogram

Special windowing functions areapplied to each portion in turn in order to reduce edge effects or distortionsof the Fourier
series caused by having a finite `audio block' as a signal. Common window functions used
text on this subject by Hammingare Hamming and von Hann4. There is an introductoryas the Short-Time

Fourier

Transform,or STFT.

(1998) himself.
Thetivelyprocessrecently

the Continuous
window-sizeof STFT

is very

typicalin the audio

analysis

literature,

thoughrela-

a newer form
Waveletknown asof spectral analysis has gained some popularity,(CWT). The STFT has one fundamental
Transform
problem; theis fixed for all frequencies,

4often confused and inaccurately

despite higher frequencies

called `Hanning'

being able to be analysed3.2. RELATED WORK

35

with certainty within far smaller windows. The CWT circumvents this problem by varying(dilating) the window size for different frequencies. The outcome (a time-frequency graph)remains largely unchanged, but higher frequencies will generally have better time resolution and the overall frequency resolution is higher. Due to the dilation characteristics, thefrequency axis for the CWT is naturally logarithmic. An concise description together withmathematical definition and some discussion as to how it relates to music can be found by
Alm and Walker (2002).
There are several variations on basic spectrogram which still remain true to the timefrequency graph; the frequency resolution may be scaled differently in order to make certain
varieties of sound or aspects of music more accessible. Scalograms have their frequency(aslogarithmicallymentioned above), and displayed using an octave-based
rescaled
axis
scale, similar to the well-tempered piano scale but of higher frequency resolution.
The frequency resolution may also be broken down through summation into the critical bands of human hearing in the Bark Scale as described by Scharf (1970) and as foundthroughout the literature on psychoacoustics. These are a set of empirically derived frequency ranges to which the human auditory system has particular sensitivity. Anotherexample of a psychoacoustic scale would be the mel scale, used in the popular MFCC
transform, discussed later.
Other variations include changing the magnitude scale; a logarithmic scale (dB) gives
a simplistic loudness-like scale allowing much more of the content to be seen immediately.
More perceptually accurate scales include the Phon scale, which gives a constant logarithmic loudness scale over all pure-tone frequencies, and the Sone scale, a metric that scales
linearly with perceptual magnitude. A short description of these techniques can be foundby Guessford et al. (2004).
Time-frequency plots are not necessarily limited to determining frequencies of sinusoids
at particular times; there are techniques to find frequencies of beats or rhythms over time;such plots might be called beat-spectrographs. This concept will be discussed more fully insection 3.2.2. Other variations on the concept of the spectrograph include the simplified
score rendition described by Brinkman and Mesiti (1991), whereby a time-frequency graphis used once more but the data used to populate it is taken from the score itself; this work
is closely related to that of the music animation machine detailed by Malinowski (2001)
and discussed later in this section.
Using spectrograms directly for music analysis is not uncommon and there is a con(2006)
demonstrate
WalkerDon
literature
devoted
it.
that
to
and
siderable amount ofmusic can be analysed directly with scalograms in a spirit reminiscent of that proposed by
Lerdahl and Jackendoff (1983). Cogan (1984) dedicates an entire volume to the analysis of
the spectrograms of the performances of a wide range of music. In her PhD thesis on thesubject of short-time Fourier transforms of musical audio, Dorfier (2002) states "diagrams

36

CHAPTER

from time-frequency

resulting

3. MUSICAL

AUDIO

be interpreted
can
even
...

analysis

VISUALISATION

musicalas a generalised

notation".
While

the comprehensivenessc>1such graphics
ingly

to experts

to visualise music and the utilityof using spectrograms
is beyond doubt, they are not without faults; they are exceed-

to analyse in even the best of circumstances,and they suffer from otherby being too general; like wave form graphics they can describe speech,complexrepresentations

noise and non-musicalsoundssuffer from information

overload

as generally
which

and precisely

as music.

makes them difficult

In this respect

theyand interpret

to understanddirectly.
As such, the main issue withigationis clear;they

for a visualisation

using spectrograms

are too cryptic

for a casual or novice

aid to music nav-user to become proficientineasily.

Self-Similarity

3.2.2
There

Visualisation

are two basic methods

the self-similarity

matrixfor visualisation

and recurrence

are an older form of numericaland chaotic

systems.

thoughgoes imuoted

thisvisualisations

Similarity

the self-similarity
presently

h'igiure

matrix

discussed

3.3:

used

analysisplots,

techniquecould be said to be a specialisationin the literature

generallypresent

the visualisation

pertaining
is formally

discussed by Eckmann

used in analysis of fractalsof a self-similarity

to self-similarity

the viewer within the literature;matrix,matrices.

Self-

a square image; in the case of

SS-1, whereas the recurrence

plot,

gives an SS-0 image.

A recurrence

by N. i\larwau,

Recurrenceplots.

et, al. (1987),

Theyof data by self-similarity

with

plot

of an auto-regressive

process.

Original

image

constructed

1wrinission.

It1'curreuc e plots rail be expressed

formally

as the Boolean

matrixR such that:3.2. RELATED

WORK

37

ýýý(i)
ýý)+
6(e
ý(9)
x(i)
Rm,
E
=
-

i,

= 1, ... , N,

(3.1)

where N is the number of signal states (i. e. samples) x(i), e is a relatively small
threshold distance, 11" li the Euclidean norm and 9(. ) the Heaviside step function.
This Boolean matrix which may be expressed visually as a bitmap will have a mark at
any two times where the samples are roughly (at most e apart) similar. Figure 3.3 shows
an example of this construction.
The Self-Similarity

Matrix

Self-similarity analysis is a transformation on a signal that generates a self-similarity matrix(SSM), a two-dimensional representation of the signal over time. Foote (1999b) proposes
this transformation as a useful visualisation when the signal is that of musical audio. He
argues this representation makes directly visible aspects of the audio signal such as verseand chorus repetition, thematic repetitions and variations, note transitions, on-going beatsand points of novelty.
The approach prescribed by Foote (1999b) (the seminal work in this field) involves
extraction of some feature-vectors on a frame-by-frame analysis of the audio. Foote used
the STFT with 50% windowing and a Hamming function to reduce edge effects. On the
resultant spectra he uses the mel-frequency cepstrum coefficients (MFCC) transformation,
though different transformations

may be applied in order to view the similarity
aspects of the musical audio, for example chroma, loudness and so forth.

of other

The basic SSM is defined by Foote (1999b) as being the matrix S:
Si,.i = s.. (Fi, Fj),i, j = 1,..., N(3.2)

where N is the number of signal states (i. e. audio feature vectors), F is the series ofaudio feature vectors and s(i, j) is the similarity function.
The key difference between the construction of the SSM and the recurrence plot is thatthe similarity function of the SSM is left undefined. In order to view phase differences, therecurrence plot defines a Boolean similarity function as the value equality (within limits).
The example of the self-similarity

matrix in figure 3.4 may be used to see the form thatthe two share.
The basic similarity function is defined as being the Cosine distance between the featurevectors of the two audio blocks. Foote suggests using either the basic frequency spectrum

from an STFT of the signal or the MFCC features of the signal.
ti

sw(Sxrv) =Sx 0 Sy

IlsxllIlsvll

(3.3)

'138

CHAPTER

FiguretheA self-similarity

3.4:

Cosine-distance

gnat rix at an excerpt
function

similarity

on the3. JIUSICAL

from
Bark

AUDIO VISUALISATION

Plaid's

Praguecriticalband

Radio.

Formedusing

as featuresummations

vectors.

As such the matrix is calculated through the cross combinationof the array of featurefunction.
He also advocates an improvement
vectors by some `similarity'
to the similarity
function

uses several sequential vectors to give the (correct) ordering of the "sound"
There is no mention of any other similarity
vectors a (positive) influence over the similarity.
which

measures testeds or otherwiseof the dot productefficacy

There is potential

consulted.
similarity

for further

measure, or even whether

can make the matrix

useful for extracting

structuralThis

work here to test theother similarity

measuresany other signal phenomena.
(how aFoote postulates that retrieval could not only be made by acoustic similarity(work bypiece sounds) and what appears to be the staple of the associated literaturebybutTzanetakis
Logan
2001,
Salomon,2002),
Pampalk
2001
et at.,
andandet al..
similarity.

played witha different

The self-similarity
techniques

analysismeans thator representing

potentiallyit could matchthe same piece of music

instrument.

matrix therefore has several advantages over other (musical) signal
put forward. It has no need for user-specified cut-offs or other parame-

ters. It does not rely upon absolute musical events (e. g. note onsets) to generate feedback
but rather relative events (e. g. periods of lag-correlation),and thus is highly generalised.
Unfortunately,
involved

due to the increased

complexforth

ratherdiscrete

identifying

amountto get accustomed

than linear representation

data though"`tested'

as it stands,

being a rather`events' in the matrix tends to be somewhat more
it has aof data to look at. Like the spectrogram

to using and, perhapsof time.

The matrix(in(, to its complexity,

invalidmore importantly

can give an enlightening

it is probably

term here, since he makes no quantitative

it is a planarview oil thebest left to specificexpertmeasure of his experiments

39

3.2. RELATED WORK

tasks rather than general use.
as with many signal-based visualisations, it is clear that it works far
better on monophonic music than on more complex polyphonic music. Music in general,
Furthermore,

however, tends to be realised with multiple instruments, presenting the grave difficulty ofthe matrix becoming so "cluttered" with features that it would make it hard to read easily.
In a purely practical and aesthetic concern, integrating a naturally square visualisation intoa metaphor that favours linearity and therefore a rectangular image may also be reason
for concern.
Analyseson the Matrix

Though not presented as visualisations, there are two graphs from the SSM one can makewhich are of interest to us as potential visualisation tools; a graph measuring audio novelty
and the beat spectrum. The progression of novelty over a signal is given as the sum of a
Gaussian-tapered checkerboard kernel when multiplied, element-wise, by subsequent submatrices falling on the SSM's main diagonal and summed. The kernel matrix is given by
the function K:

K(x, y) =G(x, y),(x > 0) = (y > 0)-G(x, y),(x > 0) # (y > 0)

(3.4)

where
G(x, y) _ Gaussian(II

(23

sy)
,

II)

(3.5)

[-2,2]
iskernel
both
fall
in
the
the
of width s.
matrix
range
and
x
and
y
where
The novelty score becomes high when the submatrix is centred around a point before(2000a)Footedissimilar.
is
but
is
the
around which
and after which
signalself-similar,
showed this can be used to segment audio and it has later been used for segmentation ofmusic tracks which I discuss in section 3.2.3.
The beat spectrum is formed by summing the contents of the super-diagonals acrossthe matrix; these sums form a series of lag-correlation scores for a number of lag times
from the distance between successive feature vectors to the Nyquist, which here is given
by half the total length of the matrix. Formally it can be described as the series S:
a-l

M(k, k+ 1)

S(1) -E

(3.6)

k=0

best
be
found.
is
to
the
M
is
the
spectrumoverwhichmatrixself-similarity
where
There are many techniques to extract a beat or rhythm spectrum throughout theliterature, especially in the time over which the present work was taking place. For a

40

CHAPTERreviewof each of themconventional

methods

comprehensive
argues thattrigger,

an audio characteristic

far more generalcharacteristic

be periodic.

the signal

is a prerequisite

rules regarding

rhythmicwell-formedness

Generative

Theoryof Tonal

theirThoughin orderoverlapped

Using self-similarity
allows aThis is because the only audio

patterns.

must be "similar to itself", and the similarity
for a beat to be present, similar to some of thedeclared by Lerdahl and Jackendoff (1983) inMusic.

in the literature,

not mentioned

Foote(2004).

must rely upon some sort ofextraction

repetitive

This

must

VISUALISATION

the onset of a beat.

signallingis thatnecessary

AUDIO

may refer to Hainsworth

the readerof tempo/beatfor detecting

method3. MUSICAL

subsequent

to form a beat spectrogram,matricesan example

may be windowed

andof which can be seen in figure

3.5.

Figure3.5: A rhythm

spectrogram

3.2.3

Visualisation

from

Segmentation

I discuss briefly

the techniquesof an excerpt from Plaid's Prague Radio. Formed using
function on the Bark critical band summations
the Cosine-distancesimilarity
as feature
vectors. Rhythm strength varies from red (highest) through blue to green.

Iii this section
tractionfrom(possibly

hierarchical,

musicalaudio.

Structural

possiblylabeled)music.

Determining

is and how one mightsystematicallythe underlying

for automatic
content-based
structure exfrom music attempts to determine a
extraction

description

exactlywhat

the perceived

regardingthe structure

structureofof any given piece music

it is a musicological quandary, which aside
from referring the reader to Lerdahl and Jackendoff (1983) I will defer until given concrete
examples later in this chapter. Most researchers in the field are happy to forgo the theory
and appeal
similaropinionnot explicitly

proposedmissing the latterthat authors,definition
65-0.5by comparing

to a popular

theirresults

to those of humans

given atask.

Thoughprovideapproachin the literature

stage of the visualisationin describing

pipeline),

as visualisation

the results of their segmentation

algorithms,

I will use this accidentalof the audio tracks.
of the latter stage here and thus formally label it S-16.
more precise labelling,(and thus

they are so well suited to visualisation

visualisations

may be a slightly

methods

since there are typicallytend to accidentally
visualisation

as myvery few segment types, and3.2. RELATEDVisualisations

WORK

from

no other content-based
song easily with41

segmentations
visualisation;

way shared by
are advantageous in one particularthey give a high-level overview to the structure of thethe clear and precise visualfor the segments contrast

Figurewell).

(assumingcues provided

3.6 is an example

3.6: A simple segmentation
visualisation
created fromdetailed by Abdallah
The Aloof. Formed using the algorithm
the x axis; segmentson the musicalvariations
There

are howeverof the same colour

The Most by

What I Misset al. (2006).

represent

multiple

Time

runs

repetitions

ortheme.
The colours have nothis naive way of visualising.
only that a difference in colours represents a differencewithproblems

for what they represent;

meaningshouldchosenof such a visualisation.

Figurealong

the colours

whose content is more similar than others'(1983)
beJackendoffAs
Lerdahl
state and as noted by Paulus
notrepresented.
and(2006), musical structure is typically
best represented as a hierarchy giving
Klapuriin segment

type.

Furthermore

levels of segmentation;

to multiple

level of segmentation

only.

This

will
and
rise

typicalaudio segmentation

approaches

returna single

is likely

due to the evaluation

methodswhich

typically

only a single set of segments for any track.

requiregiven segmentation

technology,

not follow this particular

Currents

route, though

in the field of thumbnailingagain by Bartschand Wakefielda lag-correlation

to generate
repetitiveat a useful visualisation.

I discuss it as a future

directionin the conclusions.

in Segmentation

work

reported

but as ideas for arriving,
The present work, however, does

may be seen not so much as criticisms

The above paragraph

Early

segments

parts of the audio;verse-chorus

structure)theymatrix.

by Bartschand Wakefield(2001)and largely(2005) used a chroma-based self-similarity
matrixThis could he used to determine and retrieve thegiven some structuralused it to determine

assumptions

on the audio

a representative

portionon the song.

did not recover structure as such, but was a first step to solving
that would later by attempted though more thorough analysis and extraction.

This technique(such as aa problem

Logan and Chu (2000) report of basic clustering clearly outperforming HMMs. Notably,
the HMMs could not be shown to be better than random in their early attempts at structure
extraction.
General audio segmentation techniques and musical audio structural analysis are heavily linked fields. Early work such as that by Tzanetakis and Cook (1999) was used forthus any given time, in belonging
values.

to only one segment type, will take one of a very limited

finite range of42

CHAPTER 3. MUSICAL AUDIO VISUALISATION

audio segmentation, such as annotating boundaries between music and speech in radio
broadcasts. This is useful as it can be done
automatically by the application, ahead oflistening. In their augmented sound editor, users
may supplement automatic segmentation
with their own notes.
Other work such as that by Foote (2000a), Raphael (1999)(2000)
Chu
Logan
and
and
show the roots from which modern music structural analysis has grown. It is a large
field with many applications; Aucouturier and Sandler (2001) have used the approach of
segmentation to advance early work by Foote (2000b) in search and retrieval and numerous.
Burges et al. (2005) use it for duplicate detection. Other uses include thumbnail generation(determining a representative excerpt),
video synchronisation (compiling transition points
and fitting them to a video stream) and section alignment.
I will briefly discuss three approaches to segmentation aimed towards the applicationof thumbnailing (either explicitly or apparently); the work of Foote (2000a) & Foote andCooper (2003) and the related work of Cooper and Foote (2003) together with the work of
Abdallah et at. (2005) and of Chai and Vercoe (2003b), Logan
Paulus&(2000)Chu
andand Klapuri(2006).

Maximising

Cross-Dissimilarity

The approach proposed by Foote (2000a) uses the novelty measure described in section
3.2.2 to determine the segmentation boundaries. In later work, Foote and Cooper (2003)
propose using a spectral clustering method to group the similar segments, creating a systemof similar purpose and scope to others described. No quantitative results comparative withother techniques were presented, making it difficult to ascertain exactly how effective thesetechniques were.
Cooper and Foote (2003) attempt to determine segment boundaries in a similar manner
through the novelty graph, though determines the labels for the segments through a further
self-similarity matrix of each of the segments themselves. The similarity measure for thisnew matrix is calculated from the Kullback-Leibler distance. Singular value decomposition
is used to determine the final labels while ignoring fine or unrepeated structures. The
has
brief
but
techniquefurtherthethoughresults published arepromising,analysis ofnosince been published.
Repetition

Detection

Chaff and Vercoe (2003b) use dynamic programming in order to deduce small segments(around 4.5 seconds) which repeat throughout a track
The
precisely.
most often and most
repeating segments that the dynamic programming are typically smaller then the musicalphrase they belong to (which is what is `really' repeating), and thus a further step is taken
to merge them into fully-fledged sections.

3.2. RELATED WORK43

The dynamic programming algorithm can be viewed as doing a similar job to a lagcorrelation matrix determined from the self-similarity matrix as used by Bartsch andWakefield (2001). In both cases a distance measure is used and in both cases the output
is aa likelihood of repetition of data given a specific period. Two distance measures werecompared in this work, concluding that a slightly modified spectral cosine-distance measure(almost identical to that used by Foote, 1999a) was generally better than a slightly moremusically-orientated pitch-distance measure.
Further results of this technique are reported by Chai and Vercoe (2003a) using achroma-based similarity measure much like that proposed by Bartsch and Wakefield (2001)
on a different evaluation corpus. The results show chroma being consistently worse thanthe original two similarity measures. The spectral similarity measure again generally
performed best.
Texture-Cluster

Grouping

The most widespread approach for thumbnailing is to preprocess the audio into somerelatively low-dimensionality series of `texture' tuples. A limited set of texture `prototypes'
is generated such that each texture tuple may be classified as a prototype. Segments aresequences of texture tuples that share the same prototype classification. The key problems
are finding the limited number of texture prototypes to describe the audio best (roughly
known as the dissimilarity)and to reduce the number of segments in the classification to
only those that are significant (known as minimising complexity and unlabeled segments).
The approaches vary in their exact implementation; typically the audio is transformed
to a series of frames of some perceptually significant acoustic quality. Aucouturier andSandler (2001) used the mel-scaled frequency cepstrum coefficients, whereas Abdallah et al.
(2005) used the first 20 principal components of the frequency spectrum.
One approach used by Logan and Chu (2000) clusters the features directly.

Other

approaches use a Hidden Markov Model trained with the series of frames with Viterbi
decoding to determine the prototype textures. While Aucouturier and Sandler (2001) andLogan and Chu (2000) classified segments directly with these textures, Abdallah et al.
(2005) used them as the input to capture features over a longer time-scale by taking histograms of texture types over successive frames with a moving window. The histograms arethen clustered with a version of the soft k-means algorithm modified to favour neighbouringhistograms to share classification. Importantly, they have observed that it is the effecting
data
for clustering rather
longer(theybeats)
time
good
producesof ause seven
which
scale
than their particular approach with histograms of the frames' texture classifications.
Work summarised by Rhodes et al. (2006) and reported fully by Abdallah et al. (2006)
advances this approach with the use of segment duration priors in a modified version of
the Wolff algorithm, which affect the fitness of a given segmentation by incorporating an

44

CHAPTER 3. MUSICAL AUDIO VISUALISATION

expectation to the length of a segment. The authors concluded that by introducing sucha prior, the problem of segment fragmentation, whereby rapidly changing portions of theaudio are themselves segmented, is solved. Of course the prior is entirely `artificial' andmust be experimentally determined. However the authors found that a "suitably broad
prior" was able to generate a realistically wide range of segment lengths.
Paulus and Klapuri (2006) presented a system with an interactive cost function allowing
the fitness of segmentations to be dictated by varying relative values of the segmentation's
complexity (total number of segments), unlabeled segments (number of textures that are
not satisfactorily classed as any texture prototype) and dissimilarity (variance of texturesclassified to the same texture prototype). By varying this fitness function, the "efficient"segmentation algorithm favours differing segmentations. The interactivity allows an individual to play with the cost function to determine an optimum for any given recording.
Exactly why this level of interactivity is of any use is left unexplained, and the accuracy
and precision of segmentations are not benchmarked against other systems.

3.2.4

Visualisationof Tonality

Tonality

visualisation as described by Gomez and Bonada (2005) is a method of musicvisualisation for analysing the various aspects of tonality. The musical key of an audio
block is estimated from the audio signal by combining several low-level spectral features(presumablyamusic-theory-based) tonal model. Gomez provides several involved
withvisualisations for analysing a given piece of music.
This work is clearly meant for musicians since the content of the visualisations isanalytic and fairly complex; though the authors do note that a foreseen use would bein studying the musical content of multiple tracks at once. One particularly interesting
key
he
is
KeyScape,
SS-1
the
is
the
as
presents
visualisation
which
anvisualisation plotting
a hue in a time-locality space. This is similar to the earlier work of Sapp (2001). The longterm key (i. e. over the whole track) determines the colour of the top of the image; the nextrow down is split into two portions whose keys are self-similar and coloured according to
the keys found in either. This splitting and colouring continues on down the y-axis, withthe time-scale getting gradually smaller and thus locality getting greater. The bottom of
the image denotes the keys of the individual chords of the track. The use of the y-axis asa time-scale dimension is reminiscent of the tree visualisation for structural decomposition
in Lerdahl and Jackendoff (1983).

3.2.5

Visualisation

for Content-Indication

The current of work pertaining to the visualisation of audio for indications of that content
of a track started with the Timbregrams of Tzanetakis and Cook (2000c). Timbregrams
are a small part of a larger body of work known collectively as Marsyas; the software3.2. RELATED

WORK

45

and systems which Tzanetakis
bodyof the initial

part

Tzanetakis

give the viewer

a cue of whether

includedaparttelling

and whichof work of the field now known
Timbregrams

introducedon for his PhD,

reported

that

notinga considerable

as music information

being a content-basedin question

the audio

formedretrieval.

visualthey could

was speech or music;

this latercertaingenres of music such as classical and rock.
With the Timbregram,the audio is depicted through a reduction of the data from a
These features are several simple statistics based uponsmall number of features extracted.
Thewith several other values describing aspects of the rhythm.
latter values are derived from a beat-histogram,which is formed by counting successivein each of the spectral bands. A
values of lag found to have the greatest autocorrelation
the spectrum,

togetherbasic dimensionality

is used to reduce the dimensionality
space thatknown

techniquereduction

best encompasses

as Principle

Component

to the three dimensions,the variancetrack).

These three features are normalised

either

red/green/blueof the dataset

Analysisin a rotation(the datasetof the feature

being the music

and used as the three components

or hue/saturation/value.

Thisin a colour,

process is done for each second of

The resultant

series colours are compiled into a vertically striped
is formally S-3.
where time runs from left to right. As such the visualisation
the audio signal.

A small and informal
that

the descriptive

responding

track"blooms"the suggestion

icons could be used to give the user an idea of what

audio the cor-

No further

contains.

work

was openly

generating

published

images representative

et al. (2006) present a system capable of generatingvarying

has an example
augment

done on the timbregrams

image

supporteduser study

however the concept of automatically
been continued:
Kolhoff(PCA)

in form

accordingof such blooms.

the native browsing

to the content

This is integrated

interface.

of musicalwithon thistechnology,of the content

visually

attractive

audiotracks.

the operatinghasgraphic

Figure3.7

system in order to

The blooms vary in form, size and colour (an SS-3

multi-layer
neuralvisualisation)according to the output of back-propagatedperceptron
network. The supervised network is trained by humans selecting tracks and image combinations; the data from all tracks is then fed back into the trained network for calculating
the icon parameters

for each track.

Figure 3.7: Examples of Kolhoff's blooms generated according to the content of audio andtraining parameters given by the user.
Unlike the Timbregram, the colours are determined with the training of the user; this46

CHAPTER 3. MUSICAL AUDIO VISUALISATION

of course means more effort on the part of the user, but should result in icons that areeasier to interpret. The use of the bloom shape is not only more visually enticing than
the stripey rectangular bar but it also fits within the square space that an icon should
inhabit far more naturally. A small user study found that the system had very respectablereal-world results, with visual similarity
agreeing around 70% of the time.

of icons and audible similarity of music content

Other work in this area would include that of Hiraga and Matsuda (2004a), who
presented a system capable of extracting information from audio to produce a rectangleof colours (thereby an SS-3 visualisation) designed to capture and visualise the mood of
the track. Their work stems from that in performance visualisation which I discuss next.
3.2.6

Animated

Performance

Visualisations

Visualisation

Performance visualisation attempts to visualise the particulars of expressive music performance over any other aspects of the musical content of the audio. Hiraga (2006) writes
performance visualisation
9 "Share understandingis used to:of a performance between co-players. "" "Compare expressions of performances. "" "Understandmusical intent of performer. "" "Data-mining

through the mood of performance. "the analysis of performance revolves around qualities that tend to be varied
from a strict interpretation of the score in classical music. Other genres of music, such as
jazz, may be somewhat trickier to visualise effectively, due to the wide array of deviance
Typically

from the score that performances may make.
Due to the concentration on classical music, these aspects visualised are typicallylimited to the relative dynamics and relative tempo of a piece. Characteristics such as
timbre, absolute tempo & loudness, key and melody are essentially ignored. Assumptions
such as the relations of onset intervals (such as in Dixon et al., 2003), or that a precise
MIDI-encoded version of the performance is available, (such as in Hiraga's work) may bemade in order to ease analysis.
Several methods for visualising performance exist; there are two main currents-thework done by Hiraga et al. (described by Hiraga et al., 2002a, Hiraga et al., 2002band Hiraga and Matsuda, 2004b) and that of Dixon et al. A particular visualisation
recently proposed by Hiraga and Matsuda (2004b) is one of form SS-3, comprising a series3.2. RELATED WORK

47

of rectangles organised as a horizontal series. The axes of the visualisation are timeloudness. The rectangles' horizontal spacing denotes relative articulation in the music andtheir relative size denotes relative tempo. Absolute tempo is not shown. A user study
gave mixed results for this visualisation, suggesting that except where the differences are
extremely pronounced (in terms of deviation from regularity) it may not adequately reflect
the viewer impression. The implementation

also relied upon an accurate MIDI renditionof the performance, implementing it to use an audio signal may be non-trivial.
The visualisation proposed by Dixon et al. (2002b), called the performance worm, isbased upon a tempo-loudness graph with the curve on it extending through the time of thepiece in question. The visualisation was designed to be viewed in an animated fashion inreal time (making it SST-0), though the animation can be folded down to a single image,plotting the entire track's trajectory on a single graph, making it an SS-0 visualisation.
The implementation uses a smoothed series of inter-onset-intervals (IOIs) to determine
the trajectory of the curve (which in the real-time visualisation appears as a `blob with atail'). To do this they make an assumption as to the underlying regularity of the musicthat is being played which, for the body of classical pieces they have tested, is reported towork well. The authors imagine it to extend well into non-classical and non-tonal music,
though no empirical tests have been reported to strengthen this claim.
A general and recent overview of performance analysis techniques and literature review
is made by Widmer and Goebl (2004), and for further information the reader may findthis useful.

Non-analytic

Visualisations

There are several interesting though not directly relevant methods of visualising music
information. Many assume MIDI data is available and generate event-based 3D words
to view the music in a discrete manner; an early proposal by Smith and Williams (1997)
would be an example of this, which generates an SSST-3 type visualisation. The CAVE
Automatic Virtual Environment was a somewhat grander scheme, to generate a similar typeof visualisation again from score data in an immersive environment (the Cave), presented
by Kaper (1998).
Malinowski (2001) describes his Music Animation Machine (MAM).

This is an SST-1

visualisation synchronised to the music performance, which uses the MIDI data of theperformance to visualise the music on a time-pitch graph which itself rolls through time.
It shares some similarities to a usual piano roll which finds itself in time-pitch space.
Different instruments are plotted on the same stave, with individual colours to separate
them. One interesting point is that unlike other time-pitch-based visualisations, it rescales
the pitch axis for each instrument; this makes only relative melodic movements importantand, through sacrificing the ability to compare pitch between instruments, simplifies the

48

CHAPTER

3. MUSICAL

AUDIO

VISUALISATIONview.

In a similarvein to the MAM

Chew and Francois(2003) called

and the tonal

visualisations

MuSA. RT. It displaysbyis the work presented

3D spiral projection
being equal to a Major Third (i. e. aof chroma (SST-3),
with each quarter revolution
frequency ratio of about 1.1892). This again uses MIDI data to animate the visual. The
an animated

3D spiral view has the advantage of clearly displaying triads as triangles, though it appears
be be restricteddisplay and does not visualise aspects of the music suchto monophonicas timbreor rhythm.

Projects

such as ImproViz

by Snydal(2005) show the utilityand Hearstof visual aids

in music software, though it is a niche tool for jazz players and not a general purpose tool;
it works only on monophonic
MIDI data.
A quick

search on the World

pieces of software
quite

which

attemptusing 3D to give visualisations;and text

notingcertain

Lillie'sMusic

Visualisation(2007)

MIT

on visualising

Figure3.8: Beethoven's

Anita

Lillie's

audio visuals,

give large analyticalof varying

some of whichareviews of the audio

dimensionality,

waveforms,

values.

The visualisation

spanning

is formed

into the 12 semitones

a time-toneof the equal-tempered

Moonlight

any given

Sonata (left)
Reproduced

columnare visible

as repetitiveand Daft Punk's

graph,

withthescale. Two images are

Superheroes(right)withwith permission.
by some timbrel
loudness of each semi-tone is given by theof the graphof the signal, whereas the relative
Chordsfromin figure 3.8.

music visualisation.

The colour

brightness.

bothsuch as spectrograms

statistical

to SS-2.

as an example

characteristics

musicalbut nonetheless interesting
work atsome as yet unpublishedThe form of the visualisation
is SST-2, though it can also
music.

being ciiscretised

providedcontent-based(2007) reveals several popular

Googlehas proposedbe amalgamatedy-axisWeb with

sndpeek and Armadillo

sophisticated.

LillieWidecombinations

is determinedof loudness in particularrows.

3.2. RELATED

3.2.7

WORK

Scientific49

and Professional

Applications

There are several scientific, professional, and otherwise niche audio playback and editing
tools that include musical audio visualisation components. Some follow a multi-pane approach where the time is once again presented horizontally and whereby various features
are drawn accordingly, each on their own subsection of the y-axis. The CLAM Music
Annotator presented by Amatriain et al. (2005), WaveSurfer presented by Sjölander andBeskow (2000) and the newer Sonic Visualiser presented by Cannam et al. (2006) are eachexamples of this. Each incorporate various visualisations of the same signal; Sonic Visualiser is a particularly interesting project with its extensible plugin architecture, allowing
future music signal analysis techniques to be extended into its interface. The relevant
techniques used to visualise the audio have, however, been discussed elsewhere in thissection.
Variations2
Other visualisation tools, such as Variations2, as reported by Isaacson (2003), addresses
the problem of visualising music content through synchronisation of various analyticalviews with performance playback. These views might take the form of music notation,
having been analysed in a manner proposed by Schenker; a number of arbitrary text labelsat specific points through the music, or an attractive hierarchical construction of segments,
populated perhaps, after the rules proposed by Lerdahl and Jackendoff (1983). Notably
all such views would have to be made manually; none of the visualisation is content-based.
Augmented

Sound Editor

As part of the Marsyas project, Tzanetakis and Cook (2000b) discuss the Marsyas Augmented Sound Editor; it functions as a standard sound editor, where an audio file isdepicted in the usual waveform graphic with time mapped from left to right. However,
the wave is actively coloured dependant on the audio at that particular point, making it
an SS-1 visualisation rather than the normal SS-0. Aside from this niche application, thetechnology never progressed any further, implying either the lack of utility or an opportunity missed.
3.2.8

Representation

Issuesthe representation of music does not end at the common
music notation. In many ways for many tasks, common music notation is simply not theright tool. It is cumbersome, limiting or impossible to note aspects such as hierarchy
As I noted in the introduction,(Lerdahl and Jackendoff in their work on music theory opted for a tree representation
to better analyse motific portions score), timbre, rhythm, pitch nuance and gradation.

50

CHAPTER

3. MUSICAL

AUDIO

VISUALISATION

Because of this we see various attempts

at creating other methods of representing musice.g. the map-based notation of Weyde (2005) and Weyde and Wissmann (2004) which
visualises music structure where linear and even hierarchical methods will not suffice.
Couprie (2004) gives an interesting discussion on possible intuitivegraphical representations of music. He contends that electroacoustic music, not being score-based, has more
problems with analysis than, for example, classical music. He argues that spectrograms
are too complex to ease analysis and that some other form is necessary. The display comprises multiple discrete but iconic elements, with each element having a continuous form.
This display has attributes of a symbolic notation (such as common music notation withmultiple discrete elements), but also of a continuous visualisation (such as a spectrogram).
He argues that navigational aids are helpful in numerous
situations, with the final soundbite "the creation of the visual provokes anenrichment of the listening".
Isaacson (2005) reviews techniques to visualise music from a music theoretic point ofview, pointing out that good visualisation techniques should be backed up with acceptedmusic theory. Middleton (1990) argues quite persuasively that music theory is a "less thanuseful resource" for popular music. In agreement, many scholars in the area of contentbased visualisation and musical audio analysis, when dealing with popular music, typicallytake a more relaxed and empirical approach, opting to collect evidence not through an
explicit appeal to music theory but rather through empirical experiments. The present
work attempts to take an approach respecting both views; empirical evidence will besought, but I will discuss the visualisations features with regard to the relevant aspects oftexts such as Lerdahl and Jackendoff (1983).
Isaacson writes that there are "many facets of music to be visualised". Unfortunately,
he does not go into the specifics of how each of these facets might be useful; in particular
there is no mention of the music navigation metaphor and how the visualisations might fit
into this, despite navigation being an immediate and obvious use for visualisation (enoughfor Spence, 2001 to devote an entire chapter of his book to it).
Dannenberg (1992) tackles the de facto standard in digital music composition representation, Musical Instrument Digital Interface or MIDI. He notes that while being thestandard and doing its initial job adequately (it was designed as an interface between
digital music equipment in order to transfer such information as note onsets), it hardly
fulfils a need for the representation of higher-level features such as structural depictions ofmusic.

3.2.9

Colour

The primary defintion of colour is:"The quality or attribute in virtue of which objects present different appearances to the eye, when considered with regard only to the kind of light3.2. RELATED

WORK

51

reflected from their surfaces."-Simpson et al. (1989)
We can therefore see that colour, in a similar manner to visualisation, is a term concerning our perception. Colour arises, like sound, from our interpretation of a wave. Likesound, a spectrogram can be drawn of this wave; giving the intensities of the various frequencies which make it up. Similarly, these frequencies are typically limited to the range
to which humans are sensitive; this is called the visible spectrum. Whereas sound runs fromroughly 20 Hz to 20 KHz (for the average human child), colour runs from approximately
1.4 MHz (700 nm, `red') to 2.5 MHz (400 nm, `violet').
The visible spectrum is not an accurate representation of the degrees of freedom ourperception has however. In fact, with regard to colour, humans have only three degrees of
freedom; colour-sensitive (cone) cells in the retina are sensitive to (roughly) the blue (.:: 445nm), green (; z:s 535 mn) and orange (:r 575 nm) areas of the spectrum. The perceptionof the `colour' of the visible spectrum comes from the ratios of these three quantities. 7
Since the visible spectrum may be made up of any combination of strengths of frequencies,
humans perceive many different combinations of spectral light as being equivalent; forexample a single frequency of orange light could easily appear indistinguishable from two
frequencies of light which would, on their own, be perceived as yellow and red. This
contrasts to sound, where two pure tones would rarely be indistinguishable from a single
tone of their average frequencies.
The collection of colours which humans can perceive is called the human gamut. Figure
4.6 gives an illustrationof the human gamut; it is bounded by the visible spectrum. On
the gamut the intensity of the colour (i. e. how dark or light it is) is ignored, providing a 2dimensional (i. e. planar) representation of colour. Much of the gamut can be emcompassedby carefully selecting a number of colour points (primaries) on the visible spectrum andcombining them to form a composite colour. A systematic combination to synthesise acolour is called a colour model.
Colour models in themselves do not properly define colours, since they describe only
the basic methodology of creating a colour rather than the specifics. Examples ofcolour models include red/green/blue (RGB, a colour is created by combining differing(HSV a colour is defined byamount of red, green and blue light), hue/saturation/value
where it falls in the visible spectrum, its brightness, and how `faded' it appears) and(CMYK, commonly used in printing; a colour is defined bycyan/magenta/yellow/black
how much of four colours of paint should be combined). Modern computer systems often
use the RGB model, making it convenient for transmission to visual display units which
synthesise colour by mixing these three primaries.
7There is actually a fourth cell in the retina for perceiving light (a rod cell), though it is sensitive only
to the intensity of light rather than individual frequencies, and is used by the eye for accurate judgementof brightness as well as periphery vision, where accurate colour perception is less important.

52

CHAPTER 3. MUSICAL AUDIO VISUALISATION

For accurate specification of a colour, a colour space must be utilised. A colour space(e.
bydefining
itg.
encompasses a coluor model and extends
the parameters properly
the wavelengths of the primaries). Examples of colour spaces include CIE 1931 XYZ (aperceptually motivated space, discussed later) and sRGB, an industry-dominant, properly
defined version of the RGB colour space. Since our system will be evaluated on basic
consumer hardware only, this colour space was chosen to best represent our RGB colours.
Exactly

what colours are presented on the display device depends on a wide array offactors (e.g. the individual device's characteristics
on thesettings
contrast/brightness,
device itself, hardware drivers of the graphics display software (e.g. X-Windows), gamma
be
As
in
it
is
the
the
will
that
correction
system software).
sRGB colour
suchunlikely
reproduced accurately, but it does at least give a theoretical stationary target.

3.3

Proposed

3.3.1

VisualVisualisation

Construction

Methods

MethodIn at least one piece of work (that by Tzanetakis and Cook, 2000a), a visualisation of amusic recording has been created by colouring the points of a plane according to the pointin the recording analogous to the horizontal position of the given point. Indeed, this isdone almost by accident when depicting a segmentation of musical audio, as was shown in
the last section.
When combined with linear progress-bar style GUI navigation widget, however, thecolours afforded by this visualisation method become an obvious metaphor for the music.
Colours on the y axis are constant and therefore it seems clear that the graphic is linear,
be
This
it
from
is
1-1
might
the
that
time
to
x axis.
and reasonably obviousamapping
contrasted to the spectrogram where both axes change; novice and casual users may not
see this directly.
This technique may be formalised by naming a function P, such that it converts fromthe domain of audio blocks to that of colours. For convenience, we will define the audio
block c in terms of our common signal preprocessing. In all instances, the signal's spectra
The
by
STFTs
first
the
window sizerecording.
calculatedusingaseries
of
over
audio
were
CD
being
the
With
input
1024
50%
standard
thesignalsamples, with aoverlap.
used was44100 11z, this puts the lowest frequency to be detected at around 43 Hz, with windows
around 11 ms apart. The stereo signals were first downmixed into mono, to prevent any
problematic stereo separation effects.

c=_P(b)

(3.7)

the
STFT
bE
frequency
by
the
the
cE
and
set of all
where
spectra as output
above
set of all colours.

3.3. PROPOSEDVISUALISATIONMETHODS

53

Some techniques naturally rely upon multiple audio blocks to create a colour (perhaps
in order to build a context); projection function is therefore extended to P':

b)
P'(c,
v =_

(3.8)

Where b is a series of spectra.
For convenience and relying upon the fact colours can be determined through threevalues, one for each of three primaries, the chroma-projection is defined instead in termsof a two-parameter function P":

v-P"(c,b)(3.9)

such that
b))
b),
b),
C6RGB(Pl'(0,
P"(1,
P"(2,

- P'(b)(3.10)For c, 0 represents the red
where bE the set of all audio blocks and cE0,1,2.
channel, 1 green and 2 blue of the sRGB colour space. The colour space function CsRGB
simply converts from the three primary colour intensities (red, green and blue) to thecorresponding element in the set of all colours. This conversion, though likely rendered
inaccurate by the hardware, is a precisely defined colour space, and is a convenient form,in most computer video systems. Thus P converts from audio blocks andcolour channels to an intensity of the given channel.
used implicitlyWith the projection function defined, the visualisation technique may be defined as.
the planar shader S, which maps the points of any given plane (in x, y coordinates where
both are bounded between 0 and 1):
S(x, Y) - P, (Bt... t+w),

t-

x(1-

w)

(3.11)

B is theaudio track's series of spectra, l is the length of the audio track inspectra and I1..,t is the vector containing the ith to jth elements of z7.See figure 3.9 for an
whereillustration.
I call this general method an audio-colour projection (ACP), with P' being the ACP
function. As such, different individual techniques need only to define their particular
projection function to be completely defined as a visualisation method.
Formally, the ACP method is of type S-3, though if a projection function neglects the
be
S-2
S-0,
formallyitit
AlthoughS-1.
becomes
it
type
ever
or
cannotc parameterof
may still be useful to consider projection functions that restrict themselves to only twodimensions of colour.

54

CHAPTER

Figure3.3.2

Signal3.9: An illustration

3. MUSICAL AUDIOof the core visualisation

VISUALISATION

method.

Postprocessing

In order to utilise the full gamut of colour,
a quantile stretch of the data is done into thefull colour space. A 90% quantile stretch wasused, given by v':(v- qo.95)
V, _qo.95 - go.os
where q, ti gives the nth quantile of the dataset of values v. This particular
normalisation technique was used above others to reduce any extreme outliers affecting the distribution of brightness adversely, which can be a problem with other methods (e. g. max-min,when the distribution

mean/normalisation)

3.3.3

Psychoacoustics

In orderto help generate

certainpsychoacoustic

is heavily

accurate image, the audio is preprocessed withdesigned to account for the process that sounds musta perceptuallytransforms,

go throughto reach the brain.

These transforms

on humans.

In particulartransforms

scaling,

critical-handskewed or not normal.

threesummation,

are derived

from empirical

data collected

contours

are used; equal-loudness
and specific loudness sensation for sone scaling.

for phonBark
The Bark scale is a non-linear

scale of `critical'

frequency

hands, between which we have asimilar perception of frequency difference. They are based upon the inner ear, which canbe considered as a complex set of band-pass filters;thefrequenciesofthecentreeach ofBark bands are related to said filters. Figure 3.10 illustrates the edges of each critical band.
As can be seen, the bands increase monotonicallybetween 100 and 500 Hz, showing the
human
then

The width of the bands
to changes in this part of the spectrum.
after this, denoting the ear's indifference
to small changes at higher

ear's sensitivity

rises sharply
frequencies.

3.3. PROPOSED VISUALISATIONMETHODS

55

0
oýýO

0
N

o---o

oý

10
°

Üo

0

0

0

°

o.

f0

m

Lf)

0.1

1.08

2

2.7

3.7

5.3

6.4

7.7

9.5

12

15.5

Frequency (KHz)

Figure3.10: The Bark criticalbands and where they fall on the audio spectrum.

Summinga basic power spectrum into a Bark critical banded spectrum has several
benefits; aside from being a perceptually
motivated frequency scaling, it drastically reduces
the amount of data it is necessary to process; typicallyup to two orders of magnitude.
Formally,

I define the function

as the bark spectrum

Bark(s)s of 20 bands si, 0<i<21where:>-

Si where bi is the frequencyof the ith criticalband, f, is the frequency

x and b0 = 0. The band of any given frequency
Zbark, proposed by

the spectrum
function(3.12)

xiof the ith band of

may be estimated

1(0.76f) + 3.5tant 1(f /7.5)2
)
ZI,,,
13fan
rk(fk! I, --

with

the

(:3.1:3)

DecibelThe level of sound may be objectively
measured as a ratio between the pressure of thesignal in question compared to some agreed base line pressure; the unit of pressure is
the Pascal (Pa). Since ratios of levels tend to vary both very little and very greatly, alogarithmic
the ten tines

the 10th logarithm

decibels (dB),which is definedof the ratio of value a to base line b thus:

scaling is used to represent

the ratio,a1010910(
adB-b _b)=
Typically
confusionin audio processing,

between parties.

to be the lowest possible

it is useful to agree upon a standard(3.14)
baseline to minimise

is used as such, which is consideredof sound pressure that the human ear can sense. If a

The absoluteamountas being

level 20 jtPa,

56

CHAPTER 3. MUSICAL AUDIO VISUALISATION

logarithmic

ratio is given with this level as the baseline, then the meta-unit dBSPL is used
to mean decibels (Sound Pressure Level). Unfortunately, this presents a problem for musicalaudio signal processing, in that recordings typically do not denote where on the amplitude
scale 20pPa falls. Without this information, there is no way to transform accurately the
raw audio spectral data into dBSPL and psychoacoustic scales based thereon.
For the present work, I arbitrarily set the levels
such that the maximum level within apiece of music is 90 dBsPL; any levels falling below zero were raised to zero. This should
mimic a listener playing back the music at the highest healthy level. Thus I define dB,
which operates on a power spectrum of values in positive unity range, as:dB(x) __ d, where

di, 0<i<n:d2 = 10loglo(x; ) + 90

(3.15)

where x; is the ith band of x and n is the total number of bands in x.
Phon
Humans perceive tones of differing frequencies at differing levels of loudness and are mostsensitive to frequencies around 3 KHz. The phon scale is a scale of frequency-independent
loudness. Any pure tone of loudness p phon is defined as being perceptually as loud asa tone of 1 KHz at p dBSPL. To transform (approximately), one linearly interpolates
between the equal-loudness curves given by Pampalk (2001). Figure 3.11 illustrates thesecurves.
Sone
The sone is a scale of specific loudness sensation. It is designed to give a linear rise withrespect to perceived loudness, unlike decibels which give a well spaced a scale for manydifferent uses. Thus a doubling in the scale of sone should represent a doubling of perceived
loudness. If it operates upon the phon scale, it is a frequency-independent measure. As
such this represents the final stage in the present work's psychoacoustic processing chain.
Formally, I define the equation Sone(p), taken from Bladon and Lindblom (1981):
I240vb0,
Sone(p) _(40)Z"642,p> 40(3.16)

otherwise

Figure 3.12 illustrates the scale; notice that until around 30 phon it grows slowly; after
this it increases in sone at a much faster rate.
We may therefore define the Loudness function L, which operates on a spectrum x to
give the first 20 Bark critical bands on a specific loudness sensation scale:

L(x) = Sone(Phon(Bark(dB(x))))(3.17)

3.3. PROPOSED VISUALISATIONMETHODS

57

0
Co -

6ý
Co o
a Co 420 phonz
3 phon

CY

0

100

200

500

1000

2000

5000

Frequency (Hz)Figure3.11: The equal-loudness
loudness for all frequencies.

curves.

3.3.4

Magnitude

Bandwise

Loudness

Each curve denotes a constantlevel of perceived

Definition
Initially,

the spectrum

is processed witha psychoacoustic

pipeline:

The spectrum

is con-verted to use a perceptual loudness scale Phon, which gives a frequency-independentscaleof loudness. After this it is summed into the critical hands on the Bark scale. This significantly
cuts down on the computation
cost in many areas, since the 512 bands of the
FFTbands, and it also gives a good frequency scale
which can be easily split into three portions later. The final bands of the Bark scale arelinear scale, Sone.
then converted to a perceptually
outputis reduced

A windowing

to only 24 criticaltechniqueis then used; as supposed by Abdallah

et al. (2005), we founda moving window mean over the signal was most helpful in refining the output tobecome musically relevant.
The exact window width we used was an experimentallydetermined
three seconds, with it being moved one second between successive windows.
that

The bandwisemean is taken over the windowof spectra.

Each perceptual spectrum of 24 critical bands is then split into three separate channels for red, green and blue respectively. Each sub-spectrum is then used as an eightdimensional vector to which the magnitude is calculated (as the Euclidean distance from

58

CHAPTER

3. MUSICAL AUDIOVISUALISATION

0
0
CD
Co

c
0
to
a

Co

yN

CD
U
O
J0
N

0
u

10

20

30

Specific Loudness

Figurezero).

Formally,

Sensation

60

50

40
(sone)

3.12: The sone scale in terms of phons.

I define PBLSM:(3.18)PaLshM(c, b) -- JIL(6)8c...
8c+811
where cc {0,1,2}
and represents the colour channel
tively) and b is a vector of three seconds of spectra.

figure;3.13: An activity

flow chart of the bandwis('(either

red, green or blue respec-

loudness smoothedmagnitude(BLSM)

te'chnique.
Figure 3.1'3 gives a data-flow
to a network

graph

Expectations

andFiguretrack

Son(-.

in the audio analysisThe

correspondingis shown followedbottomgraph

preprocessing thewith the basic waveform of the audio, the basic spectral
by the critical-hand
spectrum in units of dB, Phon andof the various

shows the BLSM

to the lower Bark

'I'Iie final visualisation

which (hircctly corresponds

Discussion:3.1 1 gives an illustration
Clubbed to Death. Starting

representationof the technique,framework I used.

representation

processing

intensities

stages whilein each of the threebands, blue to the higher

for this track is shown in 3.16.

bands

(red

and green in the mid-range).

3.3. PROPOSED VISUALISATIONMETHODS

59

to

R0RSS,000_EF4524R03RRFYR
2.00
300

1.00

4 00

5.00

600

.'vyg0R4

1 00

20
40
60

-100
-120
°RRSRRRSRRRSRRR4RRRSRRRSRRR4RR

1:00

4:00

3:00

2:00

5:00

6:00

7:00

-10
-15
-20
25

4

ö

6,30
35
40

1 00

4 OG

3 00

2 00

5 00

7: 00

6 00

R

55
50

}

. 010
6

45

'

40

1ý

35
30
ý1"
r1ll

11ý

1 00

4 00

300

200

100

6 00

Soo

I1o
e

s

ý.

z
7

ti

4R-rsr

1:00

2:00

..

5:00

4:00

3:00

0

ztaHkRSÄ

6:00

7:00

ö

ö
ö
ýo
RR4
0: 009

R1:

RRY
009

R R$

22:
009

RR4R4:

R3:
00=

°RR4
00

516:
00°

RR7

RRY

06:
004

517:
00'

RRY

Figure 3.14: A depiction of the track Clubbed to Death at the various stages of the BLSM
transform.

60

CHAPTERThis

is primarily

timbrel
e. g.

Music

terms.

electronica

melodyweighrepresented

refinement.

where the timbre

is very importantbut otherand rock,

types

AUDIO

VISUALISATIONinbe very descriptive

shouldbenefits from this representation

aspects such as rhythm,

and jazz music,

harmony

have their

will

and

`meaning'

poorly.

more mid-range

content

overallpower,

relativescale dynamics

be an indication

denotewill

by clear banding

be more uniformthe relative

Becauseit shouldof this,instruments,

and dynamics

such as tempochanges,

Similarshouldcolours

As such, repetitions

depictin appearance.

Finally

the saturation

surfacechanges (e. g. crescendos)and more generally

and small variations

rhythm,that

colour,such as the onset ofchanges,

clearly.

Other

we would

aspects of the music
less
be
to
obvious.
expectof instruments

combinations
shouldofinto one of the three subspectra.

the spectral

arise from certain

A spectrum

place should give rise to a very saturated

the power will be engulfed

thatof dark and light areas, whereas tracks

balance of power in the spectrum.

much of its power in a particularsince it is likely

portion.

The lightness denotes
and a bluer hue would denote high-range.
Clear largeas in the standard spectral magnitude measurement.

will be represented

wouldof the sound.
A greener hue denotes"brightness"of the relative

more power in the low frequency

whose power changes little

contains

where

hip-hopmore such as classical,

hue will denotethe colour

Colours

a timbre-based

The hue of the colour should
A redder3. MUSICAL

be visible

as portions

in particularof similar

keys.

colour.

Demonstration

Wave

BLSM

OOmÖNN

PositiontOJ:3.15: Altitnnde

and with

2: 00

1:00

0: 00
FigureONbandwise(Red Square Reprise)

loudness smoothedby Hybridmagnitudevisualised(BLSM).

(Wave)
basic
wave
as a

hyClubbed To Death are both electronic/classicalbrid pieces incorporating
aspects of both musical styles. Clubbed contains a short excerpt
from the Theme (Andante) of Elgar's Variations on an Original Theme for orchestra, Op.
Altitude

hybrid's& Rob Dougan's

to two main sections of strings and drums with various DSP
1 andVariations
largely
there
two
of
are
more
piano
parts,
reminiscentsamples,
effects and
12 of Enigma, but composed entirely by Dougan himself. Altitude is a simple string theme
36 ("Enigma").

mixed

with

and holding

In addition

breakbeat,briefly

In Altitude

with

both

before dying

the dynamics

and the rhythmbuilding

to a crescendoaway.

(figure 3.15), it is clear to see the crescendo at 1:28 repeating its figureý:3.3. PROPOSED

four times

VISUALISATION

1: 53, before

until

depicts

the general brightness

around0: 24) with

METHODS

reducing

61

and fading.

More

content

sampledThis is represented

introduces

a high-passstrings

gets progressively

fades into murkycontent)

withbrown,

filteron, whichcream and eventuallyitrestricting

by the blue hue. The track

This is visible

more relaxed.

filter,a high-passwithto the treble region of the spectrum.
and whose filter

the visualisation
the track. It starts (untilof the sound on its way throughsome non-tonalsome quiet

interestingly

thenlouderget progressively
as the red (denoting bass

white.

Wave
BLSM

ýt
t i's,
00000000
Nth

Position

0: 00

l7

0p
00000000000
ONNMVN-

3: 00

Clubbed to Death(Kurayamino

3.16:(Wave)

and with

bandwise

CO CO ONN

4: 00
Mix)

loudness smoothed

3.16 shows the BLSM

0000

CO CO ON-

2: 00

1:00

FigureFigureNN

00000000000000
t7

O-

5: 00

-N

0) VN-

6: 00

by Rob D visualised

Nth

O

7: 00
as a basic wave

magnitude(BLSM).

image for Clubbed to Death. The piano parts are clear toof green. The blue at the very beginning marks the Elgar's "Enigma"
Theme used as the introduction.
The red parts which fade into purple show the portionof percussion (red) fading to pale blue as other voices are added (taking over the mid and
The large shift in
high-ranges and eventually reducing the apparency of the percussion).
see as two portions

loudness from a multitudeof instrumentation

to a single piano is clear at 2: 55 and again

at 5: 52.
Banding

betweensents the addition
throughpink and gray is visibleof a parametric

the frequency

spectrumat 3.58 which

EQ filter(most likely

and changing

the timbre

continues

a bandpass

to 4: 17. This represweeping

or notch),of the various sounds.

ColourWithout

Here I will compare the main method to the basic non-bandwise
version. The only difference is that the magnitude of the entire psychoacoustic
spectrum is taken, rather thansplittingit into three parts.

We may define this formallyas PI Stil:b) -- IIL(6)11
PLSAf(C,and represents the colour channel
where cE {0,1,2}b
is a vector of three seconds of spectra.
tively), andThe well known jazz track
funk guitar
through

between

Green Onions

1: 10 and 1: 50 is visible(at 1: 30), the funk guitar(either(3.19)
red, green or blue respec-

is shown in figure 3.17. The imprint
on bothimages as a brightspot.

of theHalf-way

SBL
key;
is
image
the
this
visibleon
clearly
as
up
a
steps

62

CHAPTER

3. MUSICAL

AUDIO

VISUALISATION

Wave

II
BLSM
I
LSM

I1 I
III

00000o000000000
ý"
MN

PositionC)

0: 00
Figure

.-NMa

1: 00

3.17:

loudness

VN

The

Green

Onionsmagnitude(BLSM)

track

smoothed

III
r
3: 00

2: 00
by Booker

T. andthe MG"sto loudness

comparedwithbandwisemagnitude(LSM).

displayedsmoothed

Wave
..,

BLSM
I

MONMI
BLM

I'MMilli UII,1SIHIIJIM

00000

Position

0: 00

Figure(BLSM)

3.18:

Takl

compared

by

Plaid

displayedwithto bandwise

loudnessmagnitudea change of hue fromsounds);WithouttoNa

howeverpink/yellow

it appears(denotebandwise

mid-pitch

as a constantwhite

loudness

1: 00

magnitude

smoothed(BLM).

sounds)high-pitch

to blue (denoting

tone on the SL image.

Smoothing

We now compare the main method to the unsmoothed version. More formally, the bandwise
loudness magnitude, PBLAMis defined as:(3.20)

PBLAf(c, b) = IIL(b)s,...ac+a
where cctively){0,1,2}and represents

the colour(either

red, green or blue respec-and b is a spectrum.

Tak I is a short
3.113. It contains

trackby electronic

several short

repetitive

music artists

gets higher-pitched

is visualisedwhich

in figurethat
played on an organ-like instrument
background.
heat
drum
in
the
with a chaotic
voice is introduced

witha higher key thatin the final two figures.

It is a good example
makes medium-term`shape' of the individual
does not have this,Plaid,

figures,

progress into a lower overall key throughout,
In the fourth figure, another, louder, organ-like

onsets)channelof where

reduction

progressionof short-termmuch clearer.

features(such as individual

In the unsmoothedversion,thefigures can be seen in terms of note onsets. The smoothed version
but better shows the changes between the figures as they progress3.3. PROPOSEDVISUALISATIONMETHODS
:

63

Wave
BLSM
BBSM

r_
OO

Position

Ö

CO,

Oý

0: 00

ON

O') OON

Ö!

lh

1: 00

NN

2: 00

ÖNON

fý

3: 00

4: 00

Figure 3.19: Zala by Plaid displayedwith bandwise loudness smoothed
Bark smoothed magnitude (BBSM).

to bandwise

comparedthroughthe track,colour slowly changing
of instruments.

introduction

Without(BLSM)

magnitude

to denote the progress and thethroughout

with

Ö

((0

Psychoacoustics

If we remove the psychoacoustic
magnitude

fromthe spectrum,

loudness scalingfrom

we see, cruciallyfor visualisation,

the method,ranges of the three channels decrease. This gives a smaller
formally define this variant as PBBSM:

where ce

{0,1,2}using justthe basic

the relative

dynamic

range of hues than before.

We

PRSSm(c, b) -- IIBark(b)8c... 8C+8I1(3.21)(either

red, green or blue respec-

the colour channel

and represents

and b is a vector of three seconds of spectra.

tively),From the additionby the blue channel

is takenout psychoacoustic
be covariant withloudness scaling,of the psychoacoustic

scaling,takes in defining

the artefacts

blue tends to stay at a roughly

distribute

The frequency-dependent
the red channel.
the channels values far more evenly. To illustrateelectronic

music by the abstractelectronic

artist,Bothimages look remarkably

similar,and botha significantly

greaterin the visualisations.

role

With-

level throughout,constantor

loudnessthis,scale appears to
we look at a piece ofroughly

the same aspects in

Plaid.
identify

a similar way. However in the BBSM image, the red and blue components of the image
are largely covariant, resulting in an image dominated by purple. Whereas in the BLSM
image there is a slightly better use of hue with portions of light green, blue, pink and dark
however, is the lack of differentiationindigo.

A far more important

portionof 3: 30-3: 52 and much of the firstmarkedcontrast

to the BLSM

(pink versus varying
portioncriticism,

image, whichthethe hue.

This

is in

entirely

differently

important.

The latter

half, due to the sharing
colours

the two sections

shaded of blue).

This turns out to be musicallyhas a clear bass component,

and there is no instrumentation

the first half.

betweenin the higher

key ofL[t64

Comparison

CHAPTER 3. MUSICAL AUDIO VISUALISATION
to Mel-Frequency

Cepstrum

VersionAnother common method of psychoacoustic
the
is
the
of
extraction
audio preprocessing
cepstrum co-efficients from the mel-scaled spectrum. Mel-scaling, as proposed by Stevens
and Volkmann (1940), is similar in concept to the Bark scale, whereby the log-amplitudesof the spectrum are mapped according to empirical experiments on the human perceptionof tone change. The cepstrum co-efficients (of which only approximately the first twenty
are taken), are simply the first amplitudes of the discrete cosine transform, when mel-scaled
spectrum is treated as a discrete signal.
This was visualised as before, except with the substitution of the critical banding
and psychoacoustic loudness scaling for the first 24 mel-frequency cepstrum coefficients(MFCC). As with the 24 critical bands, these were split into three subsets, the magnitude
was taken as the Euclidean distance from zero, and the three colour channels were valued
accordingly. Formally, the colour projection function is PBMsM where:
PBMSM(Crb) = JIMFCCo... 23(b)ac...
8C+811(3.22)

where cE {0,1,2} and represents the colour channel (either red, green or blue respectively), and b is a vector of three seconds of spectra.
Generally, the use of three subsets of the MFCCs seems to give a reasonably usefulvisualisation, with images noticeably matching those from the spectral version of the algorithm. However, there are two key points that seem to burden the MFCC variant, making
it on the whole less useful for visualisation of music: Firstly, while the brightness of the
in
the
is
it
hue
loudness,
from
it
is
detachedthetheassignal represents
not quite as
Secondly, the distribution

impliedbrightnesses
hues
their
and
of

spectral visualisation.
from the MFCC data results in a less informative visualisation, due to fewer artefacts.
To illustrate the first point, we use the rock track Moving by British indie rock band
Supergrass. The track features a fairly typical AAB AAB A verse/chorus structure. However the sets of verses ('A's) get increasingly louder, with increasing instrumentation andtheImportantly,
generalloudeningbarelyisthisovermore emphatic vocals.
noticeable
theme.
Figure 3.20 shows the smoothed bandwise loudness and smoothed bandwise cepstral
magnitude, both with amplitude outlines. In both, the choruses are instantly recognisable
MFCC
in
the
(thoughbrightless
blocks
both
in
In
the
two
the
obviouswhite
middle.
asbandwise
In
is
the
theme
smoothed
noticeably a single
version), each verse
repeated.
loudness visualisation it is, however, quite clear to see that the second pair of verses areof the first pair, but only louder (brighter, but same orange-green-black-red
pattern), the third verse (the outro? ) being of the same quality, but perhaps louder still.
In the MFCC variant, with a far more complex set of colours to denote the verse, andrepetitions

with the colours apparently far more noisy from set to set, it is very difficult to extractýiýE3.3. PROPOSED

$low

100

Wave

VISUALISATIONOW65METHODS

so

BLSM
BMSM
0000000o0000pp000p
ýNMONýNMQNN0

Position

0: 00

1:00

Figure3.20: Moving(BLSM)

compareda00Nr00

th

2: 00

by Supergrass displayed
to bandwise

N

t+I

3: 00

with bandwise
cepstralmel-frequency

4: 00

loudness smoothedsmoothed

magnitude

magnitude(BMSM).

Wave
BLSM
®I-

BMSM

I
';

N

Position

0: 00

Figuremagnitude

N

C7

?N

2: 00

1: 00

Gabriel's

3.21:

smoothed

B

N

Vý

t+01

magnitude(BMSM).

Oboe by Ennio(BLSM)

compared,-Morricone
to bandwise

withbandwise

loudnessmel-frequency

cepstralsmoothed

displayed

the same information.
We can see the visualisationsof Cabriel's

Oboe by Ennio

Morriconein figure 3.21. The

structure of the song is essentially AB-AB-A'B'I
with the A' having an extra somewhat
bassy instrument
present compared to the As, and the B' being slightly augmented fromthe Bs to form an outro. The MFCC visualisation
presents this well, though the different
A' from A goes entirely ignored. Very little textural information
within the B portions of
the song is apparent,
Conversely,
two distinct

correspondparts,

instrument.

to the playing

visualisation

by red.

white.

being comprised ofshows the B portions
themes, the second including
to two distinct
corresponding
The stripes between red and yellow in the first part of the Bs
versionclearlyand silence of the oboe. However,

becomes somewhat

Due to the extra spectral
green followed

them being mostly

the loudness

smallera mandolin-like

withcontent

questionable

where the loudness variant's
is in the third A, running from 3: 27-3: 58.

from the extra instrument,

it becomes brown rather

than

66

3.3.5

CHAPTER 3. MUSICAL AUDIO VISUALISATION

Bandwise Loudness Rhythm Magnitude

Definition
The rhythm

magnitude is a novel technique to deliver the `rhythmicity' of audio at a
beat
known
(also
It
is
by
as
the
particular point.
calculatedusingrhythm spectrum(1999a)
Foote
by
its
We
takingspectrum) as a vector, andmagnitude.
use the algorithm
for calculating the rhythm spectra, which involves populating a self-similarity matrix andsumming across the super-diagonals. Section 3.2.2 gives a full definition of the technique.
As before, this is an extension to the standard rhythm magnitude technique done toprovide colour. The output of the critical banding is split into three subspectra, a rhythm
magnitude for each one is found. Each are normalised individually, and used as their
be
more
The
in
final
techniquemaythecomponentcolour.

corresponding red/green/blueformally defined as the projection function PBLRM:(3.23)PBLRM(C, b) = IIRhy(L(b))tc.. tc+týý, t=6

where cE {0,1,2} and represents the colour channel (either red, green or blue respectively) and b is a vector of 128 spectra (approximately 1.5 seconds).
s-l

Rhy(X)t =E

MX(k, k+ 1),

0: 51! 52(3.24)

k=0

from
the
determined
series
is
MM
is
M
the
the
s
size
of
self-similarity matrix
where
and
of spectra x. Figure 3.22 shows the process as a dataflow pipeline.
3ands

to Phon

10Son.

E

SONSimllai$ty
Matrix

DiagonalSum

Mapnft ds

NormaliseRed

U)

Self Similarity
Matrix

Diagonalsum

Magnitude

Normalbe
wine

Groan

Self Similarity
Matrix

Diagonal
Sum

Magnitude

Normalbes,yM ttýiFigure 3.22: An activity flow chart of the bandwise loudness rhythm magnitude (BLRM)
technique.

Expectations

and Discussion

This is a rhythm-timbre

based refinement; it should benefit music whose short-term autoAs
differenttime.
tosuch, morecorrelative properties change with regard
voices overrhythmic music should benefit, such as hip-hop and dance, whose timbre may stay roughly3.3. PROPOSEDVISUALISATION

constant throughout,

METHODS

but whose rhythmic

67

properties (perhaps with regard to different

voices) changes.
The brightness of the colour relates to the strength of the rhythm at that point (i. e.
how self-similar the signal is over a short period of time), whereas the hue describes wherein the spectrum that simplicity lies. If there is little correlation, or if it is compromised
between two successive and unique rhythms, however, then it should have a lower overall
power and thus be darker in shade.
If the rhythmicity is mostly in voices in the upper part of the frequency spectrum,
the hue will be cooler (blue/cyan/green).
If in the lower part, the hue will be warmer(yellow/orange/red).
Hues would therefore change whenever there is a shift in the relative
lag-correlation of the three sub-spectra. As different portions of the total spectrum change
in their self-similarity (perhaps by introduction, removal or interruption of voices), a greater
shift in hue would be expected.
The number of spectra used, and thus the imprecision of the metric, is equivalent to
the number of bands of the rhythm spectrum (or the cardinality of the vector we measure).
Determining the optimum size of the spectrum is rather a black art; a smaller size results
in better time precision and less processing. A larger size gives less noise and allows higherlevel features to be captured; I settled on a window of around 1.5 seconds, which gave agenerally reasonable output.
Demonstration
Rounds, an album by Kieran Hebden (released under the moniker Four Tet), is considered(e.g. by Clarke, 2003) as being a prime and largely seminal example of a genre called 'folktronica'. This is a fusion between the abstract electronic and folk music genres. And TheyAll Look Broken Hearted ('Broken')

is a track from this album which focuses on chaotic

drumming and a repetitive harp melody, underlined with slow bass chord progressions,with other relatively soft `instrumentation' progressing through the track. Unspoken isanother track from the album, again featuring a repetitive melody (this time piano) anda clear progression, though with more esoteric sounds creeping in. Figure 3.23 shows the
tracks under the BLRM visualisation.
In Broken, we can see that despite the instrumentation and loudness being relatively
similar, the visualisation clearly differentiates the repetitive melody of the harp (1: 25-2:30)
from its `chorus' theme (2: 30-2:55). It also disambiguates between the solo harp and theharp with bass and drums which transitions at 1:45. The repetition of the section 1:45-2:55
at 3: 17-4:27 is also clear to see, despite the addition of an extra, rather loud, bass drum.
It progresses throughout by adding
notable in this visualisation is how the addition

Unspoken, like Broken focuses largely on repetition.
and removing instruments8. Particularly

8'instruments'is usedlooselyhere;many of the soundsnot only are not recognisableinstruments,but

68

CHAPTER

3. MUSICAL AUDIOVISUALISATION

Wave

BLRM

00000000000o000000000000000
VI
ýNmV
.-Nma

Position

0: 00

v)

in

-Nmo

1: 00

2: 00

ýn

-Nmo

ýNmaý"N

5: 00

4: 00

3: 00

Wave
111

BLRM

ýýI I

I, ýIý

Iý

1111

I1

iI
ýNMQN

Position

0: 00
FigureýýýNfý'IVN

1:00

3.23: FourýN(7ý9ýN(70N

2: 00

3: 00

Tet's And

ON(ý10ý

4: 00

ýNMÖý

5: 00

6: 00

They All Look Broken

Hearted

9: 00

8: 00

7: 00

(top) and Unspoken (bottom)
loudness rhythm magnitude (BLRM).

as a basic wave (Wave) and with bandwise

visualisedýN(IQýNt70N

ýNMÖý

ýNMÖN

bass, and later (6: 14) the piano, makes very little
difference to the blue stripes of the visualisation.
This is despite changing the dynamics
(5:
00),
What
the
is
the
significantly.
visualisation
addition of chaotic sounds
clearly shows
similar to a feedback loop one would expect to hear from a microphone being moved closeof a loud drum

to a loud(6: 11) to the much quieter

speaker

its signal.

amplifying

The change fromblue to a pink/whiteof the rhythm,

with

3: 40 seems to be caused by the `dirtying'bass sound and subtle

string-like

instrument,

the piano,

at 3: 06 causes no significant

Withthese two tracks,

ble; despiteof rhythm.

Without

ColourInterestingly

the effects of visualisingchanges in timbre,

similarity

theme.

the visuals

A basic version of the the rhythm magnitude
method, and ºnerely notes the lag-correlation

remainthe additionof an extra

the removal of the primary

melodic

change on the colour.
throughrhythmlargely

unaffected

are relatively

notica-due to the extreme

aspect of the original
The formal
in the entire spectrum.

ignores the per-channel
evident

is given by PLRM:

definition(3.25)e) = IIRhy(L(b))II, t=6ýý,ýý1(ý,Figurechestralhue at3.24 depicts

track

the classicalwhose rhythm

track

is maintained

fast-paced orrhythmic
Woodwindand brassthroughout.

Sabre Dance, a highly
by strings

give the main melodic content, whose playing affects the rhythm
it and by reducing the brightness of some primary colours.
The basic LRM
instrumentation

visualisation

does not clearlyexits the foregroundare not even immediately

recognisable

denoteat 0: 50; the BLRM

as being tonal at all

magnitude

the pointin contrast

by disrupting

brass
loud
the
where
from
hue
pink
changes3.3. PROPOSEDVISUALISATIONMETHODS

69

Wave

II1t
BLRM
I1
II.
LRMI

111111M
IABIIIIiN

IIMN

LI!
pi II

NEI

00o000oo00
NOýNONM

Position

0: 00
Figure

1:00

3.24:

The track

Sabre Danceloudness rhythm

magnitude

to cyan/yellow.

The point

clearlymarked withoverallrhythmdifficultto identify

and loudness rhythm

visualised

withbandwise(LRMI).

magnitude

at 1:07 is also
to mid-range'sof the entire spectrum,is subtle andclearly.

Removing

the psychoacoustic
dynamic

definition

KhachaturianThis change, in the context

strength.

Psychoacoustics

formalby Arammelody is introduced
at which the woodwinda change of hue, again to blue/red, denoting disruption

Without

the relative(BLRM)

2: 00

loudness scaling from the method

ranges of the three channels,

givinghas the effect of reducing

a smaller

range of lilies.

Theis given by PBBRM:PBBRM(c, b) = IIRhy(13ark(b))t,... t, +tll,(3.26)t-s

Wave

MI I I"

IIII
I
11111
11111
BLR
M_

BERM

Position

0: 00

Figure :3.25: The track
magnitude(BLRM)

3: 00

2: 00

1: 00

Without

Me by Erninem

and bandwise

4: 00

with bandwise
magnitude (BBRN1).

visualised

Bark rhythm

5: 00

loudness rhythm

this, we use a rap track by Marshall Mathers, also known by the. nameEminem. Figure 3.25 shows the track Without Me. In the BLRM image, we can see it isfar clearer to find the intro, verse and chorus changes; the outro is also clearly visible. BlueTo illustrateappears due to Mathers'
track;

other rhythmic

to be constantvocals being the rhythmic

voices contributing

throughout,therebyappears due to the self-similarity
instruments

being introduced,

element which changes throughout

theloudness, such as the drums,

tend

to the spectral

havingof the whole sound; the spectrum
all withbalance.

The white

is far `fuller'from more

no effect on the colour

a simple repetitive

theme.

70

CHAPTER

Wave

3. MUSICAL

AUDIO

VISUALISATION

ý+ýºý

BLRM

II
I
I

Im

LRM
BBRM

II

rö
Position

ý0

O^

uý

0: 00
3.26: The trackrhythmmagnitude

As a poignant
the excellenttýJa multi-talentedaffordedis the Enemy

with bandwise
magnitude (BBRINI).

visualised

rhythmby Will

loudness

Quantic),
themonikerunder
on guitars, bass, double bass, piano,(released

Holland

who often plays lead duty

musicianV

here, we can look atto the usage of psychoacoustics
by the Bark-based
rhythm spectrum in figure 3.26.

counter-example

is Timeth

3: 00

by Quantic
Bark

OOO

ONN

2: 00

and bandwise

visualisation

1111111 II'OýNTime is the Enemy(BLRNI)

YI
O^

1:00

Figure

The track

OOOOOOONth

I

In the figure, the waveform makes visible only the twoorgan saxophone and percussion.
large sections to the track, with thegap at 1: 56 to 2: 01. Aside from the start and finish,little else is visible.
In the BBRM

image, we can see the very clear sectioning of the track. The introduction
is given by the blue/red stripes at the beginning, the first repeating theme with red-yellowand the second repeating

red striping,withloud noisy guitar

on the piano,

note arpeggio
lower-key

notes played

theme withare white.

green-yellow-green

The first

the second incorporates

theme is justa second theme,

striping.

The gaps

a repeating highas well as having a

The red-yellow striping of the first theme
arpeggio.
is produced by the difference between the piano with the drums (red since the piano isislag-correlative),(since
drums
the
not
the
not immediately
piano
and yellow with onlyand very quiet

introducing

repeatingthe decorrelation).

by the relatively
The BLRM(non-bandwise)

low-key

The green-yellow

tlienie,again spoilingof the second theme is caused
drums.
the otherwise correlating
banding

image, by contrast, is fairly nondescript and barely better than the includedIn
LRM image, showing that each section is in fact two subsections.

the stressed purple marks at 1: 53 and 3: 17, each of which represents a loud andunexpected guitar riff, is far clearer in the BBRM image than in either of the others.

particular,

Novelty

3.3.6

Score

Definition
The novelty

score was introduced

the cross dissimilarity
Like

the rhythm

by Foote

It provides

a value determinedbyof signal both before and after the moment in time.
it relies upon a prior abstractionof the signal known as aof the portions

spectrum,(1999a).

3.3. PROPOSEDVISUALISATIONMETHODS71

self-similarity matrix, which is calculated simply by evaluating the similarity of the signal
to itself at varying intervals (given by x- y). Section 3.2.2 gives a full defintion of thetechnique. Formally, we define the colour-projection

function PNovetty

PNovelty(C,b) = IN(L(b))I(3.27)

where cE {O, 1,2} and represents the colour channel (either red, green or blue respectively), and b is a vector of spectra preprocessed from the signal. Futhermore, we define
the function N:
1<8 j<8

Mj(i, j)K(i, j)
N(x) _E
s=oj=o(3.28)

Mg is the self-similarity matrix determined from the series of spectra z (both of sizes), and K is the Gaussian kernel function given in section 3.2.2. Figure 3.27 shows theprocess as a pipeline in a canonical fashion.
f FVWV-q

Window
SIFT

Sum Crlucal
Bands

Convertto PhonConwR
fo SoneSelf SimiIarf
MatrixCheckerboard Sum

Normallee

GroyFigure 3.27: An activity flow chart of the bandwise loudness rhythm magnitude (Novelty)
technique.
matrix and accompanying checkerboard kernel were experimented with and qualitatively evaluated. I found that a value of around 128 spectra(1.49 seconds) provided a good balance between time precision and larger scale feature
The size of the self-similarity

presentation.
Demonstration
Figure 3.28 shows a visualisation of the well-known James Bond theme, which will suffice
to illustrate the form of visualisation that a novelty score produces. The novelty output isvisibly different to the spectral magnitude, since it is one level of indirection away. Rather
than showing the track directly, and allowing the user to determine when the metric changes
enough to denote a feature, it instead shows the changes directly, essentially providing a
differential view.
It is clear to see the main orchestral figure plays four times between 0:40 and 1:00.
This is despite slightly different sets of instruments and slight variations on the figure.
The repetition of the guitar theme, twice at the beginning and twice at the end, is also
visible as several gray lines followed by a block of black. The main orchestral climax after

72

CHAPTER

3. MUSICAL AUDIOVISUALISATION°y0000ýoaNM

Position

0: 00

1:00

Figure

3.28:

with

the novelty

the figures,James Bond

Theme by John Barry

after it.

This illustrates

is so high that

too little

information

are unable

3.4

visualised

which runs from 1: 06 to 1: 16 is not discernable

will nevertheless(Wave)
basic
wave
as a

and

score (Novelty).

themesguitar

2: 00

the main drawback

different
being
any
as
of this method;

can be made from visual inspection

it, since it may yet show musical properties

consideroffrom thethe refinementWethe output.

that othermethods

to show.

Discussionof Methods

We
discuss
the strengths of weaknesses of each of the t(, (hni(1uPPs proposed.
nowwill do so by means of a side-by-side comparison of these techniques, over several pieces of
We
from
broad
where
will,
including
jazz,
rap.
music
a
and
range of genres,rockclassical,
possible, base the critique on musical aspects of the audio such as long-term structure,
We willvariationand rhythm.

The techniques

for comparison

will be the three previously

proposed:" Baudwise loudness smoothed magnitude (BLSM).
" [3ruidwise

mel-frequency" Bandwise

loudness" Novelty

score (Novelty).

rhythmThese will be compared
9I

oucitness smoothed" Loudness

rliythin

cepstralmagnitude

alongside(13NISMI).

(BLRMVI).

two variations

without

colour:(LSM).

magnitude
magnitude

magnitude(Riff).

We will also provide the amplitude
projected as height, with a one second movingaandcanonical
average (Wave). I consider this type of visualisationasof musical audio
reasonable baseline.
'[lie

even
the
critique,ofand conclusions
for
track
a
The
used
reasonably
withinwell-bounded
genre categories.
selection criteria
it should
Secondlythewere two. Firstly, the track should be representative
genre.
ofspecific

tracksused do affectthe content

3.4. DISCUSSION OF METHODS

properly

represent

The selection

73

how the genre's facets manifest

visualisation

the visualisations.
the main differences

method.

Test Tone

We beginby comparing

synthetic`plucks' with a simple ADSR

a pure tone.
blocksthree

withingiven in this section is small, but is enough to illustrate

between genres in terms of a particular

3.4.1

themselves

visualisations

The plucks(70 bpm,

158bpm).

envelope modulatingof 880 Hz but change in tempo

A further

amplitude

loudness constant.

the audio in order to keep perceived

a series offrequency

are at a constant

105bpm,test tone;

with a manually-generated(attack-decay-sustain-release)

Figureenvelope

is applied

in

over

3.29 shows the generated

images of this audio.
0: 00

Position

oN

Wave
BLSM
BMSM
BLRM
Novelty
LSM
LRM

Position

0: 00

3.29: Several visualisations

Figureof the test tone `plucks'.

The first thing to notice is the lack of a visualisation for the loudness spectral magnitude;that the normalisation
the loudness stayed so constant throughout,stage reduced theimage to complete darkness!
This contrasts to the handwise variant (BLSM),
which,
throughpurely incidental

Of course the colours
apparently

benefited

it produces

guish the first far better

matrix

are quite

arbitrary.

throughbased visualisations

than the latterthe tempos being largely

the tempos by frequency

effects, differentiates

in the same manner

The self-similarity

withinterference

MFCC

based visualisation

the interference.
(Novelty,

two. The novelty

indistinguishable.

Thebands.

LRM& BLRM),shows the boundaries

also distinfairly

well

74

CHAPTER

3.4.2
The

3. MUSICAL AUDIOVISUALISATION

Trip-Hopdiscussion

Stem/Long

and a broad

amount

0: 00
00000
. -NM

Position

beginsof actual musical audio
Stem.
This is tremendously

and timbre

rhythmLevis

by Joshbased, with

both

6: 00
0 Nma
000
i(I

7: 00
0
0 NCý
0
ý0000

a greatrange of samples.

1: 00
00000
ýN(7O

VN

with

track

the trip-hop

2: 00
00000'
N
ýN(")O

3: 00
00000
N.
. -N(7

4: 00
00000
N(7P

VN

5: 00
00000
MO
N

8: 00
VN

Wave

III

BLSM

WIN
BMSM

INIM111Ill! I' iI

IIIIIIIIIIýIýI'llllý'Novelty

;fI

I131

.,

BLRM

NOW

ýýIII

II (IýIIIIL-_IIIIýIIIIIIýIýII
Lý

I

LSM
IýIIý)LRM

IIIIIIII'ýýIIIýýBI

NfhÖNNMÖNN

Position-lÖNNMÖNNMNNMÖNNÖNN
týýl

0: 00

1:00

Figure

2: 00

3: 00

3.30: Several visualisations

In the firsthalf of the track,of the spectral

4: 00

of the track

dominated

Stem/Long

8: 00

Stem by D. 1 Shadow

a foreground

with

7: 00

6: 00

5: 00

melodyof a guitar,eachbased versions denote the thematic variations well, either with
(LSM), patterns (wave) or hue and intensity (BLSM). The latter ismagnitudecontrasting

intensities

particularly

effective

see transitions

changes
to extra instruments,
at 1: 20,1: 40,2: 40 and 3: 00 corresponding
and speed. A string crescendo is visible around 3: 19 in both the bandwise

in intensity

as a brightversionswiththe fairly

blue stripe,

subtle thematic

variationsof this track; we can clearly

and a pipe organ sample at 3: 40-4: 05 is recognisable

as apatch of increasing brightness.
A large brass-dominatedsection (5: 14 onwards)

green-blueyellowin BLSMis not noticeably

same portion
though

melodyalthoughonly the brass.

the textures

found

elsewhere

in the track,

be seen to change.

image proves aroundthe first half, alternatingplaying

from

the
end
5:
58
The
represents
at
can clearlychangeand the texture introduced
at 6: 26 represents the return of the guitar
leaving
At
lower
7:
06
is
tempo.
theat aguitarremoved permanently,

part,

The BMSM

in LSM.

different

the texturesof a guitar

Colours

brighterand is simply

in the second half is coloured greenNotably, however, in the wave image, theare representativeof a stringaroundas useful as the other spectral

green, gray and white for the various differentof the overall

instrument

magnitudein pizzicato.

versions forvariations.

green represents

theThe red block fading in at 4: 08 represents

thetimbreof the music;

e. g.

M

3.4. DISCUSSION OF METHODS

75

aforementioned pipe organ. The second half of the track is less clear overall; the only clear
artifact is the ending of the guitar at 7:06.
The LRM image discriminates well at 1:02 with the onset of drums, and again at 1:20
with a more regular and frequent percussion. The break and subsequent theme repeat are
visualised appropriately as a black bar and with a texture similar to the initial portionof the track. The loud and regular percussion portion between 2:40 and 3:20 is visible asa large bright patch. The break following the crescendo at 3: 23 has only a single blackline to disambiguate it from the silence which follows. The lag-correlation of the silence
is apparently similar to that of the percussion. In the second half of the track, the mainfeature that is noted is the introduction of the occasional brass note in the background at
5: 12. The following texture finishes with the end of the infrequent (but regular) percussion
at 5:57. The further melodic progression and instrumentation goes largely unrepresented.
The BLRM image denotes significantly more information than RM; the silence following
the crescendo of 3:23 is now covered cyan. This a clear if non-obvious clue to the content.
The following organ fade-in is represented in a similar way to the spectral magnitude
methods. Two portions of the track at the beginning are separated and coloured distinctly
differently (orange until 0:28, purple until 1:01); the musical difference being the lack ofbass in the second portion. This goes largely unnoticed on all other methods. The purple
hue is used once more in the LRM version, where similar instruments and no bass is once
again apparent at 2:02. The second half of the song is still relatively featureless, being
mostly bright white.
The novelty method is once again somewhat difficult to decipher. The build up to
the percussion at 1:03 is made visible, though the proper introduction of the percussion
itself is not represented. The removal of percussion at 1:44 is represented as a change fromblack to white, it flips again when the percussion restarts at 2:41, before a large whiteband denotes the climax. The black block after this (3: 23 to 4:05), encompasses silence, anorgan fading in, and the onset of a high-pitched xylophone-like sample. The texture change
and reduction in intensity corresponds to the removal of a (non-self-similar) sample of aman complaining. Subsequent changes in the track, both in terms of melody, instruments
and speed, go without representation.
In conclusion, either the BLSM or the simple wave are the best overall visualisations;
the wave providing more textual information, especially with regard to the end of the track
and the BLSM providing a clearer overall picture with its use of hue.

3.4.3

Rap

A piece of rap music, by artist Andrew Turner, named The Force is now for consideration.
Figure 3.31 depicts the visualisations of it.

The constant intensity and rhythm of this rap track is immediately noticeable in the

76

CHAPTER

0: 00

Position

1: 00

AUDIO VISUALISATION

3. MUSICAL

2: 00

oo0ooooo0000000000

fýtNMQýiN

4: 00

00

Tiýh

0003:
^'l
dýNdý

('III

ýý

Wave

!

BLSM

ý'

ý'

IN
m
all
JIIIIlllllliiiii IIIJill II

ýI
II
BMSM
le
IIIýýýI
BLRMM1111m
ý"1Novelty

II'IIIýIIIIII Iý

-I

III
LSM ,
jljjýýýlilllllilllii
III
LRM
00000000000000000oo
N
vNN
c7

Positionbasic wave display,
With

e

t7

e

mI 1111111
HI II11 11
4: 00

3: 00

of the rap track

away about

NQ

Ln

2: 00

gives little

which

N

Ln

1: 00

3.31: Several visualisations

FigureIIIIIIIIUIIýII'll1

III mill
III Ill

t'1

0: 00

-111

The Force by Aim

the musical(featuring

Q'n'C).

content.

of LSM, the three choruses are
immediately
more recognisable at 1: 15,2: 12 and 3: 28. The inclusion of colour with BLSM
gives a slightly
more informative
view; choruses are blocks of four light and dark-blue
the more advanced

psychoacoustic

preprocessing

blocks represent

the voices of the first and second

instrument

Each of the other visible combinations
or voice. The BMSM method presents similarof colour similarly represent an
information
to BLSM, thoughsomewhatless clear,bands.

Verses of pink and orange striped

rapperrespectively.

brightwith

distribution

seemingto he heavily

biased to theend.

The three self-similarity
track,

thoughthe novelty

"more

white

lines"

all methods,
the trackdifferently

based methods
is still

is immediately

each make an impressivea last place, within the case of choruses.

and certainly

four repetitions

cues for structure

The LRM

of a basic figure.

takingimage is probably

the clearest of the monochrome
visibleon thisthe form ofvisualisation

methods.

the clearest of
The general form ofequal length blocks, each containing
Each of the blocks containing different voices are visiblyas a set of roughly

shaded.

The additionwiththe intensity

the yellow

onset of rapping

30
2:
BLRIN1
in
image
the
the
at
the
piano
recognition ofof colour
allowsinitialHowever
the
blue
the
rappers' voices asstripes, and ofand green.
at 0: 38 is not represented

well,

with

confusingpink

stripes

The red in the break at 3: 09 does not denote thesame as the red at the beginning of the track; both have a large bass content, though the
four
does
break
have
(rathertheactually
red ofsomeminimal) green content representing
to the content, it seems in terms ofwails. Though the colour is slightly more instructive
introduced

and a highly

especiallyvariant

colour.

77

3.4. DISCUSSION OF METHODSa clear expression

that

RM, performs

better.

here by the Dave Brubeck

Quartetthe non-bandwise

variant,

Jazz

3.4.4

The jazz track

Take Five, performed

will he considerednow.
Position

0: 00

2: 00
1: 00
0000000000000000000000000000
ýn
NmaN'Nmo
INmvNINmeNINmvNNm

3: 00

5: 00

4: 00

Wave

1 1BMSM

BLSM

BLRM

lIt
It

li

®IIII

IIý

ýý''

111I111I

I

! (I

II .)

II

II
i'

zu
s111,1al.
11 IN 111I III 1
0000000000000000
MO

00000oNo0000
NmNMTNNa

Position

0: 00
Figure3.32: Several visualisations

The Novelty
have correspondto the saxophone

has a yellow

portionthe smallcounter-melody

withthe lighterof the jazz track

NONNM

5: 00

4: 00

Take Five by Dave Bruteck.
that it doesis once again quite noisy, the areas of lightness

visualisation

BLRM

and again with

3: 00

2: 00

1:00

()

playing.
for the part of the track

the clear and dark

stripes

without

saxophone,

red and blue stripes

at 4: 37 to 4: 50. The drumand notes

at 0: 36 to 0: 49,

and piano duet denotedbyamount of texture change on it; whiter parts corresponddarkerfrequent
drumming
toand stripey partsstyle;
andvisible a more regularrarelycorrespond to infrequent drumming and, specifically, in the quick runs of molto crescendo.
the yellow

hue has a significant

The primary
clear in the LRM

melody(made up of a repetition

image as a double whiteof anothersmaller

melody)is reasonably

bar at 0: 23,0: 50,4: 25 and 4: 54.

shows the end of the first saxophone portion, with the change todenoteslighter
The
to
1:
52.
a somewhat small change of the ridegreen
achangeatgreenof the end of thecymbal being used less frequently; the change to black is representative
The BMSM

ride cymbal

clearly

to dictate

rhythm.

The subtle light

portionaround3: 19 represents

the pocodoes not make clear any of the changes of melodies,difficultdiscern.
is
to
for
the
and
the
noisy
representation
most partandcrescendo of the piano.

The BMSM

The wave gives a basic overview to the song's structure but little else is discernible;
but
beginningthanthat(larger)thelouder
and
end,
other
at
are
visibleportions
slightly

78

CHAPTERVISUALISATION

3. MUSICAL AUDIO

little is clear.
The LSM, withloudness measure, clearly marks the aforementioned
the psychoacoustic(2: 10), with clear changes
introduction
piano poco crescendo (3: 16 to 3: 24) and drummingin brightness.
beginning

The saxophone

Its repetitiveand end of the track.

denotedis clearlyplaying

however,nature,regions

at theis not made clear.

Withthe brightwithbecomes the clearest yet, with a green band
followed by a longer pink/red
hand repeated (at 0: 23,0: 50,4: 25 and 4: 54). This shows thenature of the smaller melody being repeated to make a single main melody, which itself
is repeated four times. The period of minimal drumming
between 1: 49 and 2: 10 is clearlycolour,in the BLSM

markedas a dark

image,

the repetition

the piano crescendoless clear is where the cymbal
is a clear light patch.

On the whole,is the clear winner,

more clearlypatch;

the BLSMof the small counter-melody

withthe BLRM

the musical aspects of the track

depictingnote, however,the LSM method,As withof the variationthat the visualisationin the other images.

is unmatchedClassical

3.4.5
For thePosition

I would

than the others.

stops.

classical
0: 00

track,

we deferto theAllegroNachtmusik

1: 00
o000oNo0000000000000
TtTvIMv

by

4: 00

3: 00

2: 00
Tn

Mozart.

Amadeus

Wolfgang

c?

I

cv

vnI

ýn

°v

Wave

Ali
BLSMII

1E _t

Il

U. I

1
1NIII
In
_II
1
11
BMSM
1111
I'II1111111l
®
I!
JIM
rl
III
1111
!
BLRM

I

II
i`ICII

1ý'!

1111

I.

i

Novelty

l11l1
L,
mm
LSMII
lilt

LRM

mi

Jill

1II

'I

Iqý

I'

Iý

I I'IýI

Figure

3.33:

AmadeusWith

harmony

Severalof thevisualisationsclassical

4: 00

3: 00

2: 00
track

Nacht7nusik

Allegroby

Wolfgang

Mozart.

this track,

the difficultytendency

1: 00

II1

iII

Position

0: 00

IIa0

and indeedof seeing overallmuch classicalstructure.

music,

whatis immediately

I see this as most likely

noticeableiscaused by the greaterof classical ºnusic's structure to stem frone "higher level" musical aspects, such as
from(counter-)melody,"low-level"
thanandratheraspects, readily analysable79

3.4. DISCUSSION OF METHODS

through its

the signal of the performance (e.g. timbre)9. This would happen naturally
tendency to separate composition from performance.

The overall view of structure is improved somewhat by the techniques that use medium(BLSM, LSM & BMSM). The wave depiction suffers from clutter due toterm windowing
the large amount
BLRM
method,dynamics.

of short-term
the slower,Withthe exceptionof the particularly

betweenof strings

cluttered

2: 30 to 2: 56 is readily

quieter

portionpalette,the BMSM

methodof information.

Repetitions

are relatively

largely

the second and the start
colours seem to have little obvious

recognisable.
Thoughhavinga limitedin terms

presentation

the first minutewiththe aforementioned

resemblingof strings,

portionthe figures of the theme,

seems to give the best overall
clear and unambiguous,of the third. Aside frombeing caused by occasional

apparently

withconnectiona criticalnotes reaching

band threshold.
Of all genres, classicalthe most difficultyis the one whichthese directhavevisualisations

signal-basedpresenting.

Downtempo/Electronic

3.4.6

The Dining

Rooms,an Italian

downtempo

electro-acoustic

2: 00

3: 00

band

jazz

the next

provideOcchi Neri.

track,

0: 000

Position

00
000f7

iN

ONiN

01:

0

00000000000oo
ONiN
(ýl

(7

4: 00

ýýiNiN

Wave

BLSM

BMSM

III

I

I.
I1"'I'IIII'ý

0: 00

I:

IIII'ý'1

ýý"

Iý

I

!I

ý'ýq

II''
N

Position

0

II

-I i

1111 II'I'I"'ý -,

LSM
LRM

III

,MI

111 11 illI!

BLRM'®'11"'1!

NoveltyME'IIlýýl ONN

lýýl

1: 00

äNN

lh

2: 00

týl

VNN

3: 00

ONN

tý

O

off

4: 00

Figure 3.34: Several visualisations of the downtempo track Occhi Nerz by The Dining
Rooms.
With downtempo music the visualisations are usually fairly easy to spot; tracks are
typically clearly segmented and use a range of colour. The regular texture of the segments
9Middleton(1990) makes a similar

observation

80

CHAPTER

3. MUSICAL

AUDIO

VISUALISATION

also tends to be visible.
The structureof the music is easily visible with each of the basic magnitude
louderis clear break from 2: 45 to 3: 20, and the track gets progressively
and content

methods.

There

that

break.

With

the bandwise

methods, we can see the change in colour resulting
the loss of a mid-range voice (a background guitar);this manifests itself as paleto purple in the BLSM image and yellow to pale-purple in the BMSM image.
green-yellow
The rhythm
magnitude
measures appear somewhat less obvious; the LRM method's
until
from

textureregularrepresents

littleobvious musical relevance
information
to that of the BLSM

3.4.7

beat in the music, though the brightness has very
here. The colour of the BLRM[ image provides similar

the regularthoughis paler and more faded.

Pop/Rock/Metal

The Foo Fighters'

rock ballad

0: 00

PositionIN

o0000000000000000
l7
dNIN

Generator

will be commented

1: 00

on now.
3: 00

2: 00
fýl

dNNTdINd

I

Wave

"

I.

BLSM

6..
BMSM

f.

BLRM'ý

p.,
'(II1
La'

Novelty
ii.

LSM

"
ii

LRM

Il'

ONÖNNdNNdNNd

Position

0: 00

Figure1: 00

3.35: Several visualisations

In it manner

similar

3: 00

2: 00

of the rock balladto that of the rap track by Aim,

Generator

Generator

by Foo Fighters.
has a largely consistentgiving a wave image largely devoid of artefacts. Contrast this to the
LSM image which, from its better loudness scaling, presents us with information
readily
blocksdark
dynamics.
two
The
to
the
two
relatingmusic's
repeating choruses are visible asamplitude

throughout,separated with a bright band. Left unclear are the relations between the rest of the track;in fact it follows a roughly A-B-A'-B'-C-B'-C-D-C-C
being
A'
similarthewithstructure,
to A but without vocals and B' being similar to B but with a clear guitar riff in the middle
as well as at the end.

The BLSM image has bright and contrasting cues which depict the structure clearly;
the two B' choruses are clear, this time as red blocks with blue in the middle. A (start to-a:3.4. DISCUSSION OF METHODS81

0: 19) could perhaps be identified as a faded and slightly dark version of A' (0: 44 to 0: 56),
and we can see that vocal parts are denoted with red, guitar as light blue/cyan, and both
The initial guitar solo is deep blue; the vocal dominated verses are a clear
red, with a cream band for the distinctive guitar riff. The small guitar solo (D) sandwiched
between choruses (C), is also visible on close inspection between 2:38 and 2:55. Repetitionas pink/gray.

of the choruses (C) is also visible as gradients from white to cyan.
The BMSM image is no better; vocals are denoted by darker faded green; the guitar riff
is a faded purple. The rest of the track is largely featureless, the few faint artefacts haveno musical meaning. Unlike with BLSM, the combination of vocals and guitar appears no
different to guitars alone.
The novelty visualisation gives a reasonable monochromatic display of the track; verses
with the guitar riff are visible as four white lines, with the centre two being closer together(the start and end of the riff). A faint pattern of several closely spaced lines of increasing
intensity at 2:58 and again at 3:26, are neither especially musically significant nor repetitions. The LRM image does a reasonable job of showing the central B'-C-B'-C structure
between 1:00 and 2:38. Quiet vocal parts are typically darker, though there appears to be
little more clear information. This is improved again by the BLRM image, which seems toshow the same structural information as BLSM except with more noise, less precision, andvocals not clearly corresponding to any particular hue. Pale yellow-green denotes vocals
with no major guitar proportion around 0:25,1: 04 and 2:00, but cyan denotes the vocals
when mixed with guitars throughout the choruses at 1:24,2: 27 and 3:05. The entire guitar
solo and the starts and end of the various features are not clearly marked either.

3.4.8

Dance/Classical

FusionFinally, the high-dynamic range track Clubbed to Death by Rob Dougan will be reviewed.
The main structure of the track changes between strings (to 0: 23), percus(to 2:53), piano (to 4: 00), percussion/bass/strings
sion/bass/samples/strings
with asweep filter (to 4:38), percussion/bass/samples/piano (to 5:53), bass/piano (to 6:20) andfinally piano/strings. This is noticeable on the wave image through the small (percussion)and large (classical) patches, with the transitions of 4: 38,5: 53 and 6:20 being somewhat
more subtle. Aside from this only one other artifact is clear; a darker patch at 1:59 signals
the end of the foreground strings.
The LSM image gives a far clearer image of the track with the superior relative intensities throughout, noticable when compared to the waveform image. Despite being
monochrome, not only are all aforementioned sections of the track visible, but so, too, isthe repetition of the strings in the background of the classical regions (3: 28 to 3:58,6: 20
to end) as four lighter bands.
The colour of the BLSM reinforces the above structure to the eye. the strings at the82

CHAPTER

0: 00
1: 00
Positionooooých OIýNMO 00000'ýN

3. MUSICAL

2: 00
3: 00
4: 00
00N
000000000
ý(I Iý100NM
SO- ýNMO
.-NMO

AUDIO

VISUALISATION

7: 00
5: 00
6: 00
0000('7 Q
0000OT.
00000
N'(7 ONIrNrNMVIrN

Wave

BLSM
BMSM
BLRM
NoveltyNI®IIýI IýI'I

I1'I'

II'
11111

II

I IIýýIý

III'

IIý1

LSM
1'I

LRM

000000000000000000000000000000000000oO
N t'> V-NMdN

Position

Figureý'llfýý

III

.-N

0: 00

1:00

2: 00

3.36:

Severalvisualisations

t+) V

V)

N

C) dNrN

3: 00(ýl VN

4: 00

"-

N

track

7: 00

6: 00

5: 00

of the dance/classical0) d

c7 dN-NmdNN

Clubbed to Death by Rob

D.

beginningare noticeably

blue, though

the track a blue tint

throughout

and strings

do notseeni to correspondrobustly.

the bass (as always withred, most evident

in between

the green of the piano figures at 5: 53 to 6: 20. Overall,

BLSM)

is marked

as a deep and darkthechanges in colours

in the percussion portions largely represent voices being added
and removed well. As the hue changes from a red tint to a cyan tint, more mid-range voices
such as the sauºples, strings and piano take precedent in the sound over the bass. The
subtleuºarcatoof the piano during

the classicalas the flecks of green gettingbrighter.

tone of the sound is reasonably

visiblebetween 2: 53 and 4: 00, is distinguishable

portion,Furthermore,

the use of sweep filters

before the strings

to change theget added, between 4: 00 and 4: 19,

between red and gray. This effect becomes subtler, representingºIs a transition
to Sohle degree, in the 20 seconds following when the strings are added.
Using MFCCs

for the preprocessing

cal sections are noisy and overall
in the percussion

texturegives a somewhat
is difficult

less informative

do give rise to musical changes;
the introductionof a foreground sample;

pink at 1:02 denotes

image; the classi-and hue changes
the change from cyan into

to make out.

portions

the sound

Texturethe important

introduction

Movement of strings from foreground toof strings at 1:20 goes withoutvisual artifact.
background and removal of the sampled voice is denoted by a change to white. In all, thevisual changes are subtle,
The self-similarity

withinIn particular,

and not immediately

based measures perform,

useful changes in the timbre
line.

non-exhaustive,

Novelty

notes many

located,brightwhite

at 0: 24,2: 55,3: 28,4: 00 and 5: 53. However,pointson the whole, poorly.

space of the track withthe transition

pointssections are either entirely

without

representative.

featurea single, precisely(generally

in the percussive

portions),or3.5. CONCLUSIONS

83

laden with details that are of no immediate musical importance (in the classical portions).
The LRM image suffers in much the same way, though the transition between bass/piano
at 6:20 is perhaps a little clearer. Use of colour in BLRM has less clarity
in the classical portions with a generally noisy use of all colours. On closer inspection,
to piano/strings

the repetition of the theme at 3:27 again at 6:20 is denoted through the use of the same
(complicated) pattern of colours; as a nice touch the same theme played once more at 6:58
in marcato is denoted as the same pattern once more but in a lighter shade.

3.4.9

Summary

Each of the proposed methods of visualising music audio signals perform sensibly withcertain types of music, and typically outperform the trivial wave representation, despite
being comparable in terms of screen space required. Notable points include:" Generally, bandwise loudness magnitude performs either the best or not much worse
than the best.
. For rap music and in some manners jazz too, the bandwise rhythm-spectrum
typically performs best.

method" Classical music tends to be for more difficult to visualise than other genres for thesemethods.
9 With few exceptions, the addition of colour through the bandwise step improves thevisualisation.

3.5

Conclusions

I have provided a thorough review of literature concerning the visualisation of musical audio
signals. I have proposed two novel and efficient methods for visualising musical audio. Eachof the proposed methods are single-stage audio-analysis methods not employing complex
machine learning algorithms or multi-stage pipelines. They produce a simple visualisationof form S-3 making them good for usage on a popular navigation system.
I have discussed and provided examples for each non-obvious step in creation of thevisualisation methods. I then discussed the aspects of music that each visualisation method
depicted over a number of tracks from a board range of genres. I finished by summarising

the findings from that informal inspection.
In the next chapter I will consider one particular path for improving the simplicity andlegibility of the image; the use of more advanced dimensionality reduction techniques inorder to project an audio block into a colour. In particular, principal component analysis

84

CHAPTER 3. MUSICAL AUDIO VISUALISATION(PCA) and the self-organising map (SOM) will be utilised as methods to project an audio
block onto a chromaticity plane.

Chapter

4

Chromaticity

Plane Trajectories"The painting has a life of its own. I try to let it come through. "-Jackson

4.1

Pollock (1912-1956)

Introduction

In this chapter I propose and discuss a novel music audio visualisation technique, designed to utilise the topology-preserving dimensionality-reduction capabilities of the SelfOrganising Map.

4.1.1

Chapter

Summary

I begin by reviewing work related to that presented here; dimensionality reduction methods
and examples of their usage in music and musical audio signal processing. I then detailthe proposed visualisation technique able to utilise linear and non-linear dimensionality
reduction methods; first giving an overview and then detailing and defining each stage.
This is followed by demonstrations of the technique in action. Finally I review each of
the proposed variations on the technique, using the same pieces of musical audio in theprevious chapter.
4.1.2

Contributions"A

formalisation of a dimensionality-reduction-based core technique, drawing upon
the earlier visual creation technique, together with two novel concrete visualisation

methods.
" Discussion of these techniques, their advantages and problems, and their relations
to existing techniques.
85

86

4.2

CHAPTER 4. CHROMATICITY PLANE TRAJECTORIES

RelatedWork

The self-organising map together with principal
component analysis are two common methods for visualising high-dimensional data. This is essentially the problem which the chromaprojection function must solve.

4.2.1

The Self-Organising

Topographic

Map

Introduced

by Kohonen (1982), and later described by Kohonen (1990), there is a large
amount of literature describing the Self-Organising Map (SOM) and its various applications. Oja et al. (2003) compiled a bibliography of all its various uses, fittingly illustrated
by a SOM visualisation of the types of subjects covered. Notably, music is one of the
few areas without a large amount of attention. It is regarded
as a general-purpose unsupervised neural network, and as such is covered in overview and reference texts including
those by Skapura (1996), Fausett (1994), and Callan (1999). `Neural networks' are a set of
biologically-inspired

methods and techniques for recognising patterns, typically through aconnectionist paradigm.
There are several variations on the SOM algorithm. Kohonen (2007a) gives a goodstarting point. The underlying concept in all of them is to have a set of model vectors,
each of which represents a point in input data space. These model vectors are thenorganised in some form of topology, typically a quadrangle, though various alternatives toplanar and indeed Euclidean geometry have their uses (Ritter, 1999 goes into more detailabout this). The topology is important to the algorithm in one way only; it helps define
the neighbourhood function. This determines how the reinforcement of one node will affecteach of the other nodes. Of course this topology plays an important part when the SOM isinspected, but that is not part of the algorithm per se. Once properly trained, nodes thatlie close to each other on the grid have relatively similar model vectors, and thus represent
similar data.
Perhaps the most archetypal use for the SOM is the classification of Finnish phonemes.
A map is trained to decode the low-level audio features of various Finnish phonemes into
points on its planar surface. Figure 4.1 illustrates the trained map together with thephoneme represented. Spoken audio may then be visualised as trajectories in the SOM
space.
Definition
Formally, we may define the SOM as a topologically defined set of model vectors m, as
ICohonen (1990) does. Each model vector defines a point in high-dimensional (input)
space and has an implicit
a point-to-point

position in the lower-dimension (output) space. It thus acts as
mapping between spaces. An input point may therefore be mapped to a

4.2. RELATED

[WORK

87

a8ý

ýý
'uuoaaaaeeitai
oýý

oýa

,ý aýý

mA

14 il a

II a

aýt

ý_
ýýý"ýi

aýý

SýýS

(1ý

1

Qlj-ý!A!
Figure4.1:

low-level

data, right are the phonemes represented by the data.
on Scholaipedia by Kohonen (2007a), according to the licence. (Jo njri. ght remains

articleAn illustration

trained

SON! on Finnish

correspondingLeft

are the/(eprodticed fromm

phonemes.

spectralwith the original

Formally,of a fully

author.

point by determining

outputthe pointin output

the model vector that most closely snatches it.

space of input

vector x is equivalent

to the implicit

positionof node i where:(4.1)

III in{jjx - mill)i

The SOM may be trained by an iterative learning process, where each step of time
may be denoted by:(4.2)

in7(t+ 1)= mi(t) +tz,i(t)[X(t)- Ini]It,

is it neighbourhood

functionto determine

creases to zero throughout
distribution:

training

and is typically%lr1where ( and a are decreasing-r,a(t)e-llr,functionsthe size of structure
implemented

4.2.2

Ij2/°(t)2(44.3)

over time.

to the definitionof the model vectors is of lesser importance(sectionand is discussed fully in the description of our particularalgorithm

The SOM with Musicalthis de-as it Gaussian-centered

The initialisation
SOM,

enforced;of Ilhe1.3.1).
.

Audio

The SOM has had several applicationsin the field of musical audio infortnatiott
retrieval.
The three typical uses are as a means of a similarity`proxy' of musical audio, as a method
in itself for visualisationof data concerning

music and as a classifier.

88

CHAPTER 4. CHROMATICITY PLANE TRAJECTORIESFor Visualisation
The SOM has been instrumental
limitedto music.

in a wide range of visualisation tools, and is by no means
However in the various fields concerning music it has seen use as apreprocessor to a self-similarity matrix of tonal content by Toiviainen (2005); a methodof visualizing melodic data from a symbolic format for comparative analysis of folk musicby Toiviainen and Eerola (2001). It is also heavily used
as a visualisation aid in music collections; Lubbers (2005) use it for combining visualisation with "auralization" forcollection browser. Morchen et al. (2005) uses it to visualise collections
according to perceptual distance, in a similar manner to the original Islands of Music
by Pampalk (2001); Rauber et al. (2002); Rauber and Frühwirth (2001); Pampalk et al.
(2002):a multi-media

Islands of Music is a project spanning several research avenues. It is centred aroundthe utilisation of a SOM to navigate around a collection. It works by appealing to avisualisation representing clusters of music on a plane as islands and mountains. Several
concepts are trialed, including feature vectors for SOMs (which we use in the present work),
and utilising SOMs as a dimensionality reduction method for a large audio feature vector
to generate 'meta-features' (an idea adapted for use in the present chapter).
For a Similarity

Proxy

The SOM has also seen considerable use in the field of music analysis algorithms as what
we will term a `similarity proxy'. We use this term to mean `a spacial mapping using the
topology preservation characteristics in order to give a more efficient, and perhaps effective,
measure of similarity between two feature vectors'. Vembu and Baumann (2004) use tags
generated from online descriptions, (in this case from the retailer arnazon. com), to train
the SOM. Artists can then be recommended according to where they fall on the resultant
map, rather than comparing the given tags directly. Aside from being more efficient, thiscan make queries more effective, since `neighbourhoods' of artists can form where several
This allows artists to be grouped together that do notnecessarily share the same tags, but have close associates that do.
tags are shared between artists.

Dittenbach et al. (2003) presents PlaySOM which, stemming in part from the Islandsof Music project, allows an interface for hierarchical navigation (`drilling down' in datamining slang), and area selection for playlist generation. This again relies on the topology
preserving characteristics of the feature space, and on the feature space being reasonably
representative of the perceived qualities of the music. Toiviainen and Eerola (2002) use
the SOM to determine melodic similarity; trained on many high-level vectors modelling
melodies, the SOM functions as an effective alternative to string-matching.

Finally Wood and O'Keefe (2003) describe some basic work attempting to utilise the
SOM as a measureof similarity for musical pieces. They attempt to predict whether two

4.2. RELATED WORK89

pieces share the same album by their distribution
For Classificationof Musicalof transformed points on the map.

Audio

SOMs have had uses in musical audio feature classification. With genre classification being
a popular challenge in the field of music information retrieval, SOMs have been used asa pattern classification tool. Knees et al. (2006) utilise the SOM to project music onto a2-dimension `map' of genres. Reminiscent of the Islands of Music work, the various genresof music may be visualised as portions of the map segregated by a learned viewer, or as inthis instance, a web-based artist-tag retrieval system. Some 66,000 different features wereextracted for this task, being reduced to the 20 that most accurately described genre-splits
with Pareto-Density Estimation. Ponce de Le6n and Inesta (2002,2003) also attempt
to identify musical style but in a more limited sense; to discriminate between jazz andclassical. The SOM is utilised as a feature space partitioning tool, the features here being
derived from symbolic data.
Rather than classifying music from symbolic data, Cemgil and Gärgen (1997) attempt
to use the SOM to classify instruments from audio data. They discuss several types ofneural network with respect to the problem of instrument sound classification. They findthat SOM did not perform as well as the other networks. However they found that theSOM proved more useful for being able to see what the network was doing and how it wasordering the data internally.

4.2.3

Principle

Components

AnalysisPrinciple Components Analysis (PCA) is a technique used in statistics typically for simplifying a dataset, reducing it to a lower dimensionality (perhaps for visualisation). Thismay be done by ignoring high-order components in favour of lower-order, thus retaining as
much of the variance of the original dataset as possible, with as few dimensions as possible.
Indeed, PCA is the optimal linear transformation to determine the subspace of the largest
variance.
Introduced by Pearson (1901), and described in depth by Jolliffe (2002), the principle
component of a set of n-dimensional vectors is the n-dimensional vector which describes
the projection giving the greatest variance over the dataset. Formally, we can describe theprinciple component on a zero-mean dataset x as wl where:
wl = arg max var{wTx}
IIWII=1(4.4)

If the dataset is projected by this vector (i. e. the variance is removed), the principle
component of the new dataset is the secondprincipal component of the original dataset.
Each principle component is naturally orthogonal to all others. Thus the full set of n

90

CHAPTER 4. CHROMATICITY PLANE TRAJECTORIES

principal components can said to be an n-dimensional rotation, which, when applied tothe dataset, orders the dimensions in terms of maximum covariance. We may call this theresult of Principal Components Analysis. Formally, we can describe the k-th component
as wk where:
wk = arg

Ilml

x var{wTkk_1}

(4.5)

where
k-1

Rk-1 =X

wiwTX

(4.6)

i=1

PCA is typically implemented by:
1. Subtracting the mean from the dataset.
2. Determining the covariance matrix of the dataset.
3. Determining the eigenvectors and eigenvalues of this matrix.
4. Sorting the eigenvectors in terms of corresponding eigenvalues.
The eigenvectors represent the principle components of the dataset.
In Musical

Audio

IR

Being a conceptually simple, effective, well-understood and accessible technique, PCA isused in a wide variety of situations throughout numerical processing, to reduce the amount
its
An
data
by
discarding
is
of
use
that
exampleofwhich
statistically most redundant.
in the processing of musical audio, which serves us well since it was directly adapted for
PCA(2004,2003)
inbePampalk
utilised
thewhothat ofuse
present work, would
et al.
to reduce two high-dimensionality

feature vectors (around 1000-dimension) to just tens ofdimensions, with very little loss of quality in the resulting visualisation.

4.3

Proposed

Visualisation

Methods

In this section, I discuss a visualisation method which we term chromaticity-plane trajectories. This method builds on the core visual construction method in the last chapter. In'positwo-dimensional
block
this
to
particular,amethod reduces aof the musical audio
tion' representative of its content. A particular two-dimensional colour-space is used to
decode this position into a colour. This colour is considered representative of the musicalaudio. Therefore, as the positions determined by the audio blocks change, a trajectory
forms though the colour space.

4.3. PROPOSEDVISUALISATIONMETHODSThis technique

can be consideredin three distinct

extraction,(2) vector projection

informationof the colour

partly)contingent

about

the inter-relationshippreviouswhich

is availableof colours.
mappedfairlyThis

Since no contextual

colour encoding.

(2), the performance

to partis unlikeabsolute

featureof thiscolour-feature(istechniquebut only

mapping,in theof the visualisationsof the audio (e. g. loudness andmanyelements

to colours.

The two proposed
differenceand (3) perceptual(1) perceptual-audio

parts;

on users not caring about an absolute

chapter,

spectral-ratio)

91

methods

share in common

most of the technique

as a whole;thebeing the `projection'

Figure 4.2 shows a diagram of the threemethod.
A large feature vector of the song is created. Ittransforms
that the data goes through.
is then projected down with a dimensionality
This portion of thetechnique.
reduction
only

is based largely on the Islands of Music work. After a low-dimensionality
positionhas been attained,by way of a colour-space into a single colour. This
it is transformed

methodtechniquetherefore

fits well into the audio-colour

framework

projection

devised in section

3.3.1.

FeatureExtraction

Audio Block

Featureset

Colourspace
Projection

Position

Colour

L-S

06.

00*0

FigureDimensionality
Reduction

Lobw

-

av

4.2: The stages and intermediate

projection

CPT

colour-method.

In particular,
refinement,

for the generaldata types required

perceptualand perceptual

differences
differencesin audio should
in colour should

be well represented
be approximately

in the firstlinearfrom thelatterstage. Thus by assuming a linear mapping in the middle stage, the results is a mnapping between audio and colour that to some degree preserves perceptually-proportionality.
These attributes

are discussed in more detail

We can not directlyin the relevant

map a high-dimensionality

sections.
featurevector to aplanar surface trivially,so we propose two methods. Firstly, we preserve linearity
at thelow-varianceby discarding
dimensions in acost of discarding some of the information,
Secondly,
PCA.
input
the
we propose sacrificing
with
space
variance-orderingofrotation
properlinearity

witha non-linear

perceptual

but topologically-correctaudio

mapping,in order to represent

92

CHAPTER 4. CHROMATICITY PLANE TRAJECTORIESThe latter method also has the advantage of providing a natural
all the information.
method for content-based (i. e. not necessarily linear) quantisation of the output.
Initially

I describe the feature vector preprocessing method. Secondly I discuss colour

spaces. I finish with two forms of dimensionality reduction; a linear method using PCA
and a non-linear method using the SOM.

4.3.1

FeatureVector

Processing

Following the work of Pampalk et al. (2004), I used the spectrum histogram featureset(SP) to model the perceptually motivated audio feature space. SP, although it does notmodel aspects of timbre such as instrument onsets, has been found by Pampalk et al.
(2003) to significantly

other feature extraction techniques in grouping tasks.
When compared to several other more complicated and computationally intense methods,
it outperformed each of them at distancing musical audio of differing style, artist, genre
disamforThoughdeliver
is
this
it
that
and album.
a useful space
no guarantee
willbiguating musical features in an individual piece, it is encouraging. Using Bark `critical
band' summation

outperform

as well as specific loudness scaling, SP is a psychoacoustically robust

preprocessing method.
Furthermore SP, unlike e.g. rhythm-based techniques, takes only a small amount ofsignal to collate a feature vector. The amount of signal to collate a robust rhythm-based
feature vector is typically significantly more (e.g. the periodicity histogram rhythm-based
feature takes 12-seconds to collate). Furthermore, it is not clear how a rhythm-based
feature vector would describe the sorts of aspects of music that users typically wish tohave visualised.
The two other techniques evaluated as performing worse at disambiguating objective
Aucouturierby(2003).
Pampalk
These,
their
proposers,
music attributes
et al.
named afterand Pachet (AP) and Logan and Saloman (LS) notably used mel-frequency cepstrum-based
been
has
(i.
features).
Although
MFCC
clearly shown as
this
never
e.
signal preprocessing
being a worse method of preprocessing music signals than the Bark/Sone preprocessing,histogram
I
the
For
that
theory.
these
spectralthis work would support
reasons, adopted
allas the algorithm, with no major changes from that proposed originally.

The Algorithm
Figure 4.4 shows the process as a dataflow pipeline. More formally, the basic form of thefeature extraction method is:

1. Take a window of STFT spectra; I used three secondsworth of spectra which appeered to deliver good results.

4.3. PROPOSED VISUALISATIONMETHODS

0
0
0

0
0

0
N

6L
ý-

6L

ýL

ýýL
ýý6itbk.
6KiL
bik.
is,ih,
w,WNhilý

WýýýýýLLLý

ý6

ýL

ýt

ý

ýýýwe

w-

0
v

ý6 6m6k
0
N
N

O
N

01.

lop.
-

O
O

O

O
iý

O
O

0
<V

ý- ý

Figure4.3: The full set of histograms

6kkk1trRý

for the track

Lebanese Blondeby Thievery

Cotpo-

ration.
2. Modify

each spectrum to a critical-banded
described in section 3.3.3:(a) Rescale axes logarithmically(b) Sutra into first 20 critical(c) Scale each frequency

Sons; scaled psychoacoustic

spectrum,asto c113-SPL.
bands according

to the Bark scale.

band to pliou units by interpolating(Phon).
(d) Scale each band into specific loudness solle units(sore).

equal lo lidness curves

9.1

CHAPTER 4. CIIROMATICITY PLANE TRAJECTORIES

3. Rescaleso maximum loudness over the piece equals 1 sone.
4. Sum a spectrum histogram matrix H with each spectrum s:

1,, s>> YO-

Utl(
p =_
s

to,

(4.7)

otherwise

5. Scale the histogram so that the maximum value contained is 1.
H' __H= max(Hf, i)(4.8)

This gives a 1000-dimension perceptually-based audio `timbre' feature-vector, which I
use as a model for human perception of the musical audio. Figure 4.3 illustrates the track
Lebanese Blonde as 280 distinct histograms.
The eigenvalues may be inspected to reveal the internal dimensionality

in figure 4.5.

In almost all cases, 90% of the total variation of the datasets is captured in just the firsttwo principle components. This serves as an initial validation of the argument to project
the data onto a 2-dimensional plane. In the next section, I will consider the perceptuallyopposite notion of turning such a 2-dimensional coordinate into a colour. After this, I will
discuss two methods of reducing the high-dimension feature space in order to `project' theaudio features Into a colour.

4.3.2

ColourSpace

In this section, I will discuss colour spaces, In particular the mapping from a position toa colour.
In most colour spaces, chrominance (chromal)subspace, (which I will define here as
being the aspect of colour which does not change with the amount of light received by
the viewer, i. e. non-luminosity dependent), are naturally representable with 2 dimensions.
Brightness Is Ignored for two main reasons; firstly over concern that it would be of greater
apparency over chroma. As such, it may mislead the viewer to identify it with some particular aspect of the sound (e.g. spectral brightness, loudness). Secondly, at the extremities,
luminosity

naturally folds all colours down to one (i. e. white or black), making the otherdimensions Indistinguishable.
ichro, na actually has at least two meanings; together with being a shortened form of the word chrominanr, it has a more traditional meaning, which I will not use in the present work, that makes it synonymouswith saturation or purity of a tone.
Wind"
BTFT

Sum Criocd
Bonds

Convertb PhonConvertýto SonsNormýlla

Figure 4.4: An activity flow chart of the CPT preprocessingtechnique.

Histogram4.3. PROPOSED VISUALISATION

ap

`NNO

J10

METHODS

95

I,

0

J^
Ca

ca
°-°

1.961

c>
ä°a
241

c0
15 WO1^Q000"00""'
5
10
15
Principle Components

000000QQQQ9951.02
5
15
20
10
Principle Componentsp20

Principe Components

4.5: Each of 8 tracks'

tempo,

trip-hop,

jazz, rap).

5_6-

WýO___--____-_--0-00292__-_o_-

00-

O

5
10
15
Principle C rnponents

251

15
Principle Components

Principle Components

Principle Components

Figure°p2888

0000000000

000000pýRPrlnclple Components

eigenvalues (Lebanese Blonde, classical, rock, fusion, downThe dashed red line denotes the 10% cutoff for the sum of theeigenvalues.

An advantageous

qualityof a colour

space would

be to use as much colour(i. e. asfeatures. A
wide a gamut) as possible, in order to better aid an onlooker in distinguishingfurther related quality would be to capitalise, where possible, on the primary colours ofbe the best defined on the display system
the colour model RGB, since they will naturally
used. 2Three planar

subspaces are depicted

chromaPerceptual

Similarity

In addition

to the consideration

perceptual

similarity.

This was addressed withsubjective

similarity

of colours.

In a mannerlarge number

in figure 4.7.

of Colour

analogousof gamut,to the Bark criticalof human experiments

an experimentbandingof frequencies(see section 3.2.1), a

were carried out to form a general view of how people
by asking people to match a
MacAdam experimented

perceive colour and changes therein.
pure spectral colour, (i. e. one that can be denoted
light),

byof distribution
by MacAdam (1942) on thethere is a second question

perfectly

by only a single frequency

ofcolour of which they had control over two chroma parameters; x andforfixed).
found
He
the
that
chosenx and y varied between
valueswasto a composite

y (the luminosity
subjects,

formed an ellipsoid distributiononover the course of the experiment
plane. These ellipses (one for each colour tested) became known as thebut that

the x-y colour
in human perception of colour.
MacAdam Ellipses, each denoting a `region of indifference'
It is useful to define a (theoretical)
space where differences in the position are proportional

to differencesin the perceived

colour space as function
2Unfortunately

colour.

Formally,

S which maps a colour to a point.

for this document,

I define a perceptually
The apparent

this is at ends with the ink-based printing

euclideanchange in colour

system.

96(71APTE'R.!.

CHROMATICITY

PLANE TR.AJEC Ii)RIES

0.9
D. N

uO0

0.4

o'

0.3

o0B

02
0.0

9e

u1un40.4

0.5

0.6

0.7

0.8

A
Fit; um I(i:
ill

Olf.

'I'Iu

pure

('IE

SpectralThe

ellipsesOle nituni size.

193 1 xy chromaticity
cctlOIIIS(the

diagraus,

wit lt the human-visible`tongue'-shaped(7urve)the

MarA(lam

Ellipsesis I)rupurt i(, nal tu the 491(1i(1(901distance

betweencudpoints

I>((*I, Q) A IIS(ei)S(r2)11-1)11Fpli's.
shillscorrespond

to

the

and
but

gtunutliuc

areenclosedof extratipcctral

reproduced

at

ten-

of the change:(4.9)

whore !' is it (unI't iun defining list eivecI linear (Iiffcrcucc betwccu two colours.
SimpleSpaces( )ne silIv colcmr space is I»wcct aroundthe hue and saturation

portionsof the IISV colour

I use it polar coordinate
nutclel. Since thc cnttput, is the satire for all Imes at zero saturation,systcnt for the splice rat 111I tIinn it CartesituI system. We cleli11 V the spaces thus:(fant '("t', 4).

('list'(

---

5)
27r

7r 10.5,(aý(4.10)

0.5)z -f- (y -- 0.5)2,1)

On inspection,
Mist rilnitiuei;

it, is clear tct tics tIirve aspects of non-linearity
in the perceptual colour
firstly, the e"e"eete"rprovides far too small a jump in quality whe"u compared to

Ow e"elges. tie"e'e,eully, flit- changes in Ieue"are far greater
Yellow,IIinge"nta and e"ynee) thanaromielthe primaries

the secondary

colours(i. e.

(reel, grce"te amt blue).

Finally,asaroiuniis ufte"n flit, o'ose", Olt- sI>iu"e given to green Imes lams far fewer changes in colourof he"r hues.

to that of

4.3. PROPOSEDVISUALISATIONMETHODS

HSv Colour Plane97

YUV Colour Space where Y=0.5

YIO Colour

Space

V-0.5

where

00

oÖ

>

°O

c\!

0

0.0

02

04

Figure4.7:

almostcertainly

06

Three

08

1.0

-1.0

0.0
U

-0.5

0.5

1.0

05

-1.0

00

0510

is
colour planes. Note: The colour reproduction
and should be taken as a demonstrationand not a reference.

simplechrominanceinaccurate

YUVand YIQ(top middle and right of figure 4.7) are both used for the transmission

analogue

television

signals; they are designed to map colours in such a way as to minimise

obviousdegradation

space except
information

withwhen noise is introduced.

the same colour

exactly

Thoughinterestingas 2-dimensions,

neitherof these proved a particularly

space due to excessive concentration

CIE

representof 30°.

a rotation

explicitly

They

of

since they encode the chromagood mappingof the same colour over a wide area.

SpacesAs describedby Kasson and Plouffe (1992), CIE (Commissiond'Eclaiiage)
Internationale
XYZ colour space (CIEXYZ,
Y) wasbottom left of figure 4.8 with a fixed luminosity
defined colour spaces. It is particularly
one of the first mathematicallyspecial since it
is based on perceptual experiments done on the human eye by Wrightdepicts the famous chromaticity
diagram derived from his experiments.
is designedin the human gamut, each as
chrotnaticities
all distinguishable
As such, it forms the basis for many colour spaces which encompass all

can be translated

into the sRGB colour space by:

RiinearGlinear
Blinearhistogram

As such, CIEXYZ

colours.

CIEXYZ'doingfigure 4.6

to encompass

a single point3.
visible(1929);

so without-1.5374-0.4986X-0.9692
0.0556

1.87600.0416Y-0.2040

1.0570Z

each colour mapped to only one pointof the EM spectrum

guaranteeingof the visible portion--

3.2410is trivial;

one simply

uses a

98

CIEXYZ

Colour

Space

where

Cflitl'EU..4. CHROMATICITY

Y=0.5

CIELAB Colour Space where L' = 110

PLANE TRAJECTORIESCIELUV Colour Space where L' = 110

8

ý>
,>
ý>
N

>o

ýa

0

LJ
00

0J

04

06

08

1.0

-100

x

certainlyinaccuratecliroiuiuancethet. ril)Iu1,

0.055
('.

-100

1(l

C(lfl

the

values100

Note:

The colour reproduction
and not a reference.

<=0.0031308
sir01.12)

Clinear > 0.0031308

f ýtýý IiýLCf1r a,
takes

and

50

0

-50

u.

and should be taken as a demonstration,near,

ýý.
vHCH
a

100

colour planes.

fl2.92(-'!

where

50

a.

1 inure 4. ti: Three CIE-specified
is almost

0-50

Riinear,of

Giineari

Biinear

in

turn

to

make

I

Ir(: It"For the i«: I3 space,uºnitlstreatill
computersystº'iiis furºu colours

I assume sR.C: 13,
it widelysR( 13 is it practicalsystems.
fromadoptedthe threeprimaries

standard,
space;

colourred, green

and blue,and common
most

light

on

imaging

and as such it is

Fur software to store colours in such terms. 'T'hese however do not accurately
advnntligeuuti
model Immun vision, which for it perceptual system is desirable.
The ('II;
1976 I, *; º*I>* (('II; LA13) colour space, depicted middle of figure 4.8, is a
it is well-defined and encompwises
colour space. I3aseci on CIEXYZ,
like most outer colour spaces,
Its three parameters,
all c"oloitrs in the hiuuun gai iiit.
It is
to it single luminosity
correspondvalue
and two chroma values (`a*', `h*').

perceptually

motivatedto describe all the coloursc"umi>Itre I the most, complete colour model used conventionallylinearwas designed largely as a perceptually
Visible to the hiunmui eye, and furthermore
colourspace. It inaiy he converted

{fi;,
(fl,

to ('I EXYZ

space by:Iý

(V + 16)/i16(.1.1:;)f,fy + a'/500__(4.14)

f= = fy - b'/200(. 1.15)fd >6l (i/ 11(i)3d2Y,, otherwise(4.16)

4.3. PROPOSED VISUALISATION METHODS

99

fXnfx3e

fx >8

X
(fx

16/116)3b2Xn-(4.17)

otherwise

JZnfz'

fz >8

Z=(fz -16/116)3ö2Zn(4.18)

otherwise

The systematic perceptual linearity of CIELAB is determined from the MacAdam
ellipses (see section 3.2.9). The CIELAB colour space uses these to warp the CIEXYZ spacein order to better preserve human perceptual linearity. The space is designed so that the
euclidean distances between points in it are proportional to those given in the Munsell bookof colour, a seminal text on colour notation by Munsell (1912). CIE 1976 L*u*v* (CIELUV)
is a similarly perceptually motivated system, but whose colour differences present different
values, smaller than those in the Munsell book of colour. Of the two, I found visually
better results from CIELAB.
CIELAB

has, in some manners, been superseded by the CIECAM97c, and later
CIECAM02.
These spaces however have several drawbacks for this situation. Firstly,
they do not provide a clear
unlike the a*b* and u*v* subspaces of CIELAB/CIELUV,
planar subspace. Furthermore, they are high-dimensional colour spaces; CIECAM02 uses
seven parameters (lightness, brightness, colourfulness, saturation, hue quadrature, chroma
and hue), not including whitepoint adjustment and surround correction.
Proposed

ColourPlane

On the video equipment available, I found the CIE space simply far too non-linear. Becauseof this, a colour plane was devised manually which can be seen as a modified red-green
colour plane; a blue component is added which takes the value of the maximum of the red
and green value subtracted from unity. Formally, I define CRGb, a function which maps
positions on an x/y plane to colours:
C!RGb(X, Y)
CSRGB(X0.3+ry,
=

y0.

Tt7,

(1

-

max(x,

1/))0.4+7)(4.19)

wheremax(a, b)

a,

a>b

b,

otherwise(4.20)

I found y had to be varied between 0 and 0.2 depending on the characteristics of thevisual display unit.

From inspection of figure 4.9, one may seethat the plane is reasonablyuniform in terms(e.
Althoughg. the outer perimeter generally
perfectof perceived changesof colours.
not

100

(11Af''1'E'R. 1. CHROMATICITY

PLANE TRAJECTORIESRGb Colour Plane
0

0
ö

ýo
ö
0

a

00

02

0.4

0.6

0.8

1.0

R

Figureis almost
X1.9: 'I'lle proposed I M. 1) colour plane. Note: The colour reproductionj L(('iu"ate and skimild be taken as it demonstration
certainlyand not a reference.

htº.v is higher degree of change than thy internals),
typicalgood use of all primaries,

PCA

4.3.3

uniformon

it makes

Furthermore,a wide range of hues.

its well as supporting

Projection

In t.Iiis section,

I will describe

fcaturv

to the 2-dimensionalwill

as any other spaces discussed.

video equipment

commodity

it is at least as perceptuallyvectors

be illustratedof how t.l1v filial

withand illustrateexamplespositionbasic linear approachitvectorfor mappingof the final visualisation,

for reducing

to it colour.

togetherwiththe audio

Tue processdemonstrations

vitiwclisactioie is determined.

Definition
The approachcaul he Sill 1111

I by takingAlleither informationis (Iiscar(lecl;(Iittii., et.

the two greatest
dills

principleof thecomponents

this is it linear approach toInitially,
the two eigenvecentirely.

although

is ignored
it small nntotutt of information
tors are fotuul which have the highest eigenvalues of all the data in the dataset X. To
t.rnnsfurtn it given feature vector x, into it position vector pi it should be projected into theprojection,

After
snl>space defined I>y those two eigenvectors.
be ntn"nucliseci by tne>ut-sul>tra ctiou told (T-division,
vectors

positiont.hnt. /' (xc)tear the track.

l"ortnally,at position

projection

function

(xý

J
--

p' is defined

stichp, where:

Pt
Where j> and v(p)of the track,

calculated

the position vector should
frone the distributionof allprojection,

are respectively

e is is c"liunping

functionz)V(4. 'll)a(P)the Iueau and standard

deviation

and V is the 1000x2 matrixof position

vectors

containing

the two4.3. PROPOSED VISUALISATIONMETHODS101

eigenvectors (e) of highest corresponding eigenvalues (e):
e1J,

V-

Vn, 0<n<1000:(4.22)

en>e+1Lea

As has been shown in section 4.2.3, the two principle components account
90% of the total variance of the dataset, thus less than 10% of the statisticallyinformation
discardis discarded.

content

It should

be noted,

however,thatfor aroundsignificant

the clamping

does

more information.

The visualisation

is finalisedby correcting

the position

and mapping

it into the chro-plane to get a colour:

maticityx<aa,wcA(Xi)--CRGb(0.5 + 0.51_i, 1(P (Xi))),la,b(X) =b,x>b(4.23)

otherwise
Illustration

histogramsfor Lebanese

the twoprinciplecomponents
most significant
and only
represented by these two componentsof Lebanese

for by the visualisation.

these which
are vectors in the data

Figure4.10: The first

Blondeby Thievery

FigureBlonde;two principle
Corporation.

4.10 illustrates
it is the features

are accounted

componentsof the spectral

Since principle

components

cyan is to be considered zero and thus the component being
dimension.
The first component can be seen to cover whatagnostic to that particular
is essentially the DC component,in that it has only a positive effect on the loudness.
space they can be negative;

Notably,

the low and mid-levelsof loudness of the bass are weighted

more than the rest of

the track.

The second component represents the weighting of the higher-frequencies over thebass; while the bass (especially the mid-levels of loudness) is excessively negative, much of

102

the the higher

frequencies

rcnt; e frectucucieti,

CHAPTER

4. CHROMATICITY

are positive,

especially

PLANE TRAJECTORIES

of loudness aroundmid-levels.cud ctuieter levels in the treble.

This component

the mid-would

therefore

Trajectory(PCA)

likelyrepresent, the presence of speech in the music, or lack thereof.

TopologicalTrajectory(PCA)

Topological

0

m
ö

0

ö

a
ö

O
N
0

0
0

0.0

020.4

(it) Reel throughFiguretu I he I'('Aiu, ºI/1X'd t

Iiirperandof tingea Small(it) Shows
II
'l.

line

is uuuletheplayingtinge

of t.h(' track.

held

directly

nlwve

the

track

thinnerwit Ii it t.irnlrrec"hnuges

thinof Irreim..

of theis at thebottomnlrrul)i

track

the

playingin t he uutnic
theof Gaiissiat1of thewhen

tuet.ht"urigh the Irucx of' the visible spectrum,

locus;

is mapped

farther

unlikethat

notdrizzled
period,it. proves

startingare (Irawir at, every 15 seconds throughaccordingit largearoundby cycling
withmoved,

endingand thediamondsareis denoted

by a

Small

by a fintr-sample-widetrack

Lebanese

its a trajectory

are spaced

an image

the

with

minuteeuchaccordingis added.

noise

point'sfor an exteu(k'cI
iftcrn,thetrack.

1.0

plane in theBlondedistancesare smoothed

track

wit Ii the paintly andof theright,

tor the dowutcurpo

ruapl)cci

gives

time

trajectories

t.iurc

This

along

Lebanesefor smaller

is increased'I'irnce i, (1e11(,t('ci in two ways here; firstly
diamonds

0.8

0.6

0.4

by

width

ath ouut

it flat. calivas,

if f he t itnlrn'
streaks02

followedt he trajectory

'I'le

stays

0.0

trajectories

throughI, t .y CUT7)O1'(L16UN. 'I'Ire'l'l(uIiIrrn(-.

line'sFor vase of viewing,

average,

FigureThechanges

15 seconds

diamond.

movingIw(op right.

theof thebeginningthemapping;

plot tecl ('very

1.0

yu Ilow, t; mrn, blue and back to rcul. (b) Black to white, with the chromaticity
background.

monotonicallycolour0.8

A demonstration

11:

"L

0.6

apart

the

aroundrespectwithof it can of paintplaying.

of paint

is dispersed;

the canvas

Ifforming

accordingly,

the colour of the proverbial
red and ending similarly.

to

heim;

to the timbre

amountthe track, whereby

Blonde'paint'Secondly,larger one representsit4.3. PROPOSED VISUALISATIONMETHODS103

The rainbow colours are changed to levels of gray (starting
the passing of a minute.
atblack) and RGb chromaticity
plane added as a background, one may visualise how colours
are picked for the creation
One may now imagine

a streakof colours

4.11(b)on a linear

has fallen on the chromaticity

to where the paintexactlyaid. Figureof the navigation

illustrates

time

this.

scale whichcorrespondplane.

Demonstration(known henceforth as
Figure 4.12 shows the principle-components-based
chroma-projection
PCP) of Lebanese Blonde. The colours may (with some effort) be matched to those under
Clearly, the PCP visualisation
is far more informationthe trajectoryof figure 4.11(b).
rich than that of the wave-based
track

is a typicalmethod.

of PCP. A fairly

trait

The fluctuatingclear `top-level'

Fluctuation

timbrefoundthroughoutof the track

segmentation

then blue/orange/purple/violet

seen as it starts red/purple/green/orange/green,green/blue/gray/yellow.

texturepatterns

stemmingfromrhythmicthecan befinishingvarianceof theare also visible.

Wave
PCP

PO$IilOfl'-

0000o0000o000ooäoN
NMONN

0: 00
Figure4.12:

Thievery

t7

O

ý1f

th

ýN

cý

N

4: 00

1:00

2: 00

3: 00

Corporation's

Lebanese Blonde

visualised

as a basic wave (Wave)

PCA chrorna-projection.

and withWith

heavy use of colours, no underlying

vibrant,

The variance-preserving

projection

structure seems especially apparent.
misses the mean values, being expanded to the edges

before clamping.

This is not a triviallye. g. normalising

to withinfixable trait;

informal

experimentation

showed thatto interpret;

faded and difficult
In the next section,

we willthe self-organising

2c made the majorityof the visualappearoutliers had vibrant colours.
only relatively unimportant
method, utilising
propose and discuss a non-linear projection

map in an attempt

4.3.4

SOM

to overcome this issue.

Projection

In this section,

I will introduce

vector mapping.

The SOM algorithma discussion

togetherwithgenerated

output.

the method

I constructed

for a non-linear
methodwill be described,of the map size, the training

process and theused and the initialisationand illustration

feature-to-2D

1011

Description

CHAPTER 4. CHROMATICITY PLANE TRAJECTORIESof Algorithm

The set of model vectors m are defined to have their positions on a square grid of size mxn.
For direct visualisation, a hexagonal grid is typically preferable, however as the SOM isbeing utilised in order to map directly onto a square plane, the slightly simpler squaregrid Is used. The initialisation phase (discussed and described in the following section) is
completed before training In earnest.
The Datch Map variant of the SOM algorithm, described by Kohonen (1999), was usedwhich was found to have similar results to the traditional incremental-learning method,
but taking under half the time (with the same number of 'iterations'4) for the equivalent
The Batch Map learning algorithm used may be defined in terms of the set ofmodel vectors mp (p Is a pair), the dataset x, and parameter or by the following equation:
training.

h(c(k), p)Xk

dp : m. _Ekp
Ek h(c(k), p)(4.24)

where c(k) denotes the position (as a pair) of the model vector which most closely
represents the input vector k in the mapping:c(k) = arg min 11
xk - mp112(4.25)and h(a, b) defines the neighbourhood function on two map coordinates, which can be
described as a Gaussian-modulated proximity measure:h(a, b) __ae

o-e 2

.=

(aj-bt)3+(ai-bi)2

ae

Vom

(4.26)

I Implemented this by first calculating the sum of all vectors in a Voronoi5 set V1, thenusing them to calculate the weighted average,thus:
Vi = {xiIc(i) = j}

(4.27)

x{(4.28)

Ek h(c(k), p)sa

(4.29)

af-E
xievi

dp : m.p__ Ekh(c(k),

p)lVkI

A further parameter, k denotes how the former parameter o should be changed foreach iteration:
4Technically, the Batch SOM's iterations are in groups spanning the entire training data, so iterations
are only comparable if one considers an entire round of training data in the incremental method a single
Iteration.
'for an overview see the Voronol reference by Okabe et al. (2000)

4.3. PROPOSED VISUALISATIONMETHODSI (r)v" -- ve-k('1.30)As such there are three parameters to determine; the number of codes s2, the learning.
the latter two parameters is
rate k and the initial neighbourhood
radius a. Determining
difficult;

with

v=

The training

s/4.

significant

extensive

I had reliably

experimentation

halts on the first iteration

changes applied,which was determinedarg maxp

II inl*,good results

fromwhere the model vectors

k0.01 andhave no moreby:mn11 < 0.001(1.31)

ýt-ý--ýt ý- k Imý
ý- im-1M
6RLL ihLW
wob.
w- - m-

04

01

00
Swe
FigureFigure4.13: An example
4.13 illustrates

of an 8x8 SOMV1trained

a trained

SOM. Noticeon the track

how similar

Lebaiirsefeatureljlonde.

vectors are grouped

near to each other, but also the non-linearityof the map. The model vectors representing
high degree of low bass and mid-range are found along the top of the neap.
a consistently
Vectors representing quiet portions are found towards the lower right, with silence found atrighthand corner.

effect throughout

the mid-highthe bottom

the startof the fourth

minuteOn the righthand side, vectors sporting

range bands;of the track.

this representsa heavily

the high-pitched

Vectors on the bottom

and middle

stippled
strings

at

left hand

106

CHAPTER 4. CHROMATICITY PLANE TRAJECTORIES

sidesrepresenttemporally inconsistent, but generally loud, parts of the track through mostcritical bands; musically, these correspond to periodic notes from the brass instuments.
Initialisation
It has been suggested by Attik

et al. (2005); Kohonen (2007b) that the training of the
SOM may be optimised by initialising the nodes' vectors as a plane through the subspaceof the first two principle components. More formally, the initialisation may be defined:
ini j_(2im-2j1)eoeo + (n1)e1el+x-(4.32)

where the map in is of size mxn and x Is the mean of all input vectorsover the dataset.
On and en are respectively the nth unit eigenvector and value ordered with decreasing
cigenvalucs,i. e. satisfying:en > en+t(4.33)

Vn : IIe,,Il =1(4.34)

Despite extensive attempts, I could not find a set of training parameters which provided
a significantly faster training mechanism with such an initialisation than without. Thismethod was therefore used for initialisation only when generating different sizes of SOM
to compare against each other (since the final mapping is more likely to be similar). An
lnitlaliser where all dimensions of all vectors were set to a pseudo-random was used for thegeneral case:

Vd : mi,j (d) = r, o <= r <= 1

(4.35)

where v(d) Is the dth clement of the vector v, r Is a value chosen at random.

Input Vector PCA
PCA (discussed In section 4.3.3) is a method of calculating a linear projection such that a
number of dimensions may be omitted from the data with minimal loss In total varianceof the dataset. This can therefore be used as a data compression mechanism, reducing thehigh dimensionalityof the Input vector space.
Although conducting PCA on such a high dimensionalityof data Is computationally
Intensive and omits some details from the dataset, its advantages are significant. It has
a dramatic effect on speeding the training for maps with a large number of nodes, andalgorithms that rely on random values for clustering are more likely to have consistentoutcomes over multiple runs, since the number of possible solutions and variables thatneed to be determined are reduced significantly.

4.3. PROPOSED VISUALISATION

Inspectionof the output

1000 dimension

original

METHODS

revealed

vectors

and onlyaround 99% of the dataset's

containing

no apparent

107

differencesthe 20 mostbetweenprinciplecomponents

As such, input

total variation).

training

with

the

(typicallyvectors put through

PCA are prefered on two conditions:
1. PCA

has alreadycomparedtionalthe results of PCA

In the case of many of my experimentsI
and the SOM, thus using PCA came at no computaanyway.

cost.

2. The training
PCA
Figurebeen conducted

time for the SOM at 1000 dimension
This is generally

time.

computation
4.14 illustrates

the first 20 principle

input

vectors

is greaterthan thethe case at a map size of n2 = 25.
componentsof the track

Lebanese Blonde.

As in the previous section, cyan is to be considered agnostic to that particulardimension.
One may see that the first component represents a varying bassline together with a level ofgeneral loudness in the music.
amountof loudness centred
2

The fourtharoundconversely

represents

4.14:

4

ýý r ý3

The firstof a signifcant

the 600 Hz bands.

:

Figurethe varying

20 principleof the spectral

components

ý. M"
histogramsfor Lebanese

Blonde.

Size of SOM
The topologyof the SOM

is the final

parameter

to determine

prior to training.
in the lower-dimensionality

The

number of nodes used varies the degree of quantisation
space.
A lower level of quantisation
toallows subtler differences in the high-level dimensionality
be denoted by changes in colour, however this comes at a significant cost of computation.
isthe number of nodes,
n
where
with
algorithm
scales approximately
thus the difference in training time when the size of a map is doubled from a 4x4 map to
The SOM

an 8x8 map is around a factor of 16.

O(n2)

108

CHAPTER

4. CHROMATICITY

PLANE

TRAJECTORIESAs we have a relatively

small quantity of data, one concern is that the SOM might
overfit data, causing a uniform distribution of points over colour plane. Instead all thelarge-scale features would be captured (with their distanceon the map corresponding totheir perceptual distance), but intra-cluster distances would be left smaller. This would
have the effect of keeping a reasonable mapping between perceptual audio difference andthe perceptual chromatic difference. Clearly, this cannot happen when the number ofnodes Is significantly smaller than the quantity of data, so from this point of view the 4x4
SOM Is favoured.
A higher-level of quantisation gives a simpler visual with a more consistent mapping of
low-dimensionality variation to high-dimensionality variation, since reducing the number ofpossible states 'flattens' the degree of possible separation between those states. Generally,
there should be fewer nodes than items in the dataset. Since there is one datum per secondof audio In a track, and many popular music tracks are between three and five minutes
long, this would suggest an upper limit of the map size of 12x12; in this section I discuss
the results of 2x2,4x4,8x8
and 16x16 maps. The visualisations are called SOM2, SOM4,
SOM8, SOM16 respectively.

Demonstration
Figure 4.15 shows the trajectoryof Lebanese Blonde by Thievery Corporation through the
RGb colour plane, as projected by four sizes of SOM. Since the line width is increased
when the trajectory moves a relatively small amount over the running time of the track, it
may be Identified as paint dribbling from a brush held aloft. Notably, the image of the 2x2
SOM demonstrates that all transitions are between neighbouring nodes. This suggests a
track without relatively abrupt changes in timbre; something not wholly unexpected froma downtempo (`chillout') track.
From the SOM4 through to the SOM16 mapping, it is clear to see that the main formof the initialisation from the two principle components has been preserved through thetraining process. This suggests that the main advantages to using the SOM should be thatfinding(inhavinglower
technique
that
the
of
a
naturalquantization
andof
map sizes),and better expressing substructure (in the higher map sizes).
Figure 4.16 shows the final SOM-based visualisations for the same track. The varying
sizes of SOM demonstrate the varying simplicity of the visualisations; the 4-node SOM2
representation essentially segments the track into four portions. On inspection, thesesegments correspond to:Blue Tabla playing only.
Red Tabla with drums/bass.
Yellow Drums/bass with vocals/keyboards.

4.3. PROPOSED VISUALISATION

TopologicalTrajectory

METHODS(SOM2)

Topological109(SOM3)

Trajectory

TopologicalTrajectory(SOM4)

0

N
O
O
0

0.0

0.2

0.4

Topological

0.6

0.8

Trajectory

1.0

0.0

(SOME)

0.2

0.4

0.6

0.8

Trajectory

Topological

1.0

0.0

0.2

0.4

Topological(SOM8)

0.6

Trajectory

0.8

1.0

(SOM16)

O

N
Ö

0
Ö

0.0

0.2

Figure

0.4

0.6

0.6

4.15: A demonstration

1.0

0.2

0.0

of the trajectories

to six sizes of SOM. The line's width
tonicallychanges fromdiamonds

are plotted

is denoted

is increased

to white

0.6

0.8

followedDrums

movingdistancesthe playing

average, and a small amount

only or with

0.6

0,8

according

moved, and mono-time of the track.

of Gaussian

noise is added.

in the yellow

sections,

The 3x3 and 4x4 SOM representations
have an overall structure
2x2 SOM, although they increasingly
add detail to the monotonic
from scatting

to singing

introductionof a brass accompaniment

2: 24.

of these changes are also visible

visualisation,

0.4

Small

quiet keyboards.

The onset of brass instruments
go unrepresented
break in drums in the green section.

Both

0.2

by Lebanese Blonde

for smaller

throughout

0.0

1.0

locus;
the
in
15
time
the
each minute
music
along
of
every
seconds
by a larger diamond.
For ease of viewing, the trajectories
are smoothed by afour-sample-wide

Green

black

OA

at 1: 12 is hinted

visible as thin `dribbles'

bar at 4: 09 corresponds

as does theto that of theblocks.
The change

similarthe change in the shades of green. The
is denoted by the change from green to yellow at
at with

in the higher

on the paint drizzle,

sizes of SOMs.

Flecksin thebegin to show; the green/yellow

to a quiet tabla note in the background.

1.0

110

('11: 1I', I,! U -1. ('llliOýtili%Tl(

I'I'Y PLANE 1'It.AJF; (,"L'OHII:: S

Wuvo

jdý

SOME

i1

SOMH
SOM 16rT(TT71__TTTTrTý..

T-T_ý.
O(8.

N`: 1111111

N

0: 00
Figur

.

1: 00

Itu I WithaL 2x2

tiOM,

lx"l

SOM,

8x8

be svvu,of t lu(huntsTexturing, truc"ttil

added

4: 00

/3londc

visualised

16x16

SOM.

SON1 andshown

ahcnttthe track;

Ihce bass at, 0: 21 becomesreadilyvisibleits it change

fromof theSOM2visualisation

makes

ftu"thertoe clear,

while

and

dissimilarity

with

but

I lie twc> rhyming

first,

the'still

details

yellowmaintaining

surroundings.
Iinvs

of vocals

block
the

overallorange'l'Itechangefromare also clear

Finally,/ all flowing

throughiinagenoisy

the introduction

to denote
to vocalsof (lark/light

blue

its

to purple.
internal

thecol>esivcuessis againorangeclear,

blocks:/ I've seeti this place before,siuniner ill allotlivi. Hille.
(I: 23 dark uruugeJ) Now I (,;ill hear the sound,
drifting(1: 26 (light orango'j) orals
through the blinds.
t'hought's,

andin particular

scatting

as it pair

(I: 13 Ids rk orimp, j) 'Rx) low to find my wary,(l: Ili IIigIit ornugej) too high to wonclar why.

texturedhue(wave)

ALSa I) LSic Wavean increasingly

and

ON

(h

8x8,

but, with
over

N

ýV

M

ýA

3: 00

'ti Lc, hafl('S('AS Ihce SO? I's sirre iucrc'a>svs to (ix(i
may

ýN
O

fh

2: 00

Corporatimi..

7711r urry

1.16:

ON

, (1

r'

/I

have a million

lily tnincl.

in ºtII but thc" two lower sires of SUM, it is clear to see four blocks of colour
from block to block;
III(I 3: 115 and 3: 55. ('liis c"ulcmr changes a small :womit

Icc"tmeel (Ic".g. in tiONI'1, it, starts

nti gray, c"hall es to hoIVcl blue-); rVVu, thou to blue, and finallytoIºurl>Iv. '1114-seit'erat'ive changes are reflected ill the ºuusic iLs the timbre niakeup changes
is of theby IL Iºc-rc"c"1ºt.
unlly sºnnll Imt, significant, ºuºuººnºt.. In this case, the first transitionºnuiu drains

leaving strings and bongos, the second is the introduction
stopping,
t.ºthlit and I he third, the re-int. rucluc"ticm of the- clrcuus and removal of the strings.

of the4.4. DISCUSSION OF METHODS

111

Effects of Training
The effects of training

are quite pronounced; the SOMs are initialised with the two principleof the dataset (which itself has been reduced through PCA to 20 dimensions).

components

The visualisation

on Lebanese Blonde by the 4x4 SOM with this set of model
SOM training
vectors is given by i=0 in figure 4.17. With each iteration of the Batch-Map
performed

one may see that the map gets increasingly
as indicated by the colours becoming more vibrant.

algorithm,

0: 00

Position

1: 00
00000000000000000000000
my
vý

ý1cý

compressed,

2: 00
ýn

iNmv

better

3: 00
ci

i

mv

uý

fitting

the data,

4: 00
ýn

iNmv

cý

i

v

Wave

i_o
i=l
>lý

e

i=2

OR

i=3

i=4
,_5

Vii'

/#ýt

Ifiý

'9IFFI

III

i=10

i=20
N

Positionfýl

V'NM<N'N

0: 00

Figure

mo'

I

t7

2: 00

1: 00

4.17:

ThieveryCorporation's

8x8 SOM. i denotes the number

aNN

t'J

aN

4: 00

3: 00

Lebanese Blondeof iterations

[ý

äNN

in the various stages of trainingof the learninganphase.

break in the drums and keyboards becomesthe musically important
increasingly
evident at 2: 05. A one-off, quiet, but nonetheless perceptible,
collection of
bongo beats in the background at 2: 50 also becomes increasingly
noticeable through the
In particular,

training

algorithm.

at around

4.4

Furthermore,

the previously

mentioned

break from scatting

to singing

1: 10 becomes clear.

Discussionof Methods

I will now conduct a brief qualitative evaluation of four of the variants of visualisations
fordiscussing
the same
by
in
do
their
I
this
visuals
generated
proposedchapter.
sowill
test tone and seven pieces of music from the discussion of the previous chapter.

112

CHAPTER

4.4.1

Test

4. CHROMATICITY

PLANE TRAJECTORIESTone

The test tone consistingin figure 4.18. There
fairlyinterpretation

good visual

to he expected,
represented
describeof several plucks in three blocks of differing tempos is visualised
is little of significance to discuss here; each of the methods presents aof the audio.

SOM2 is clearly the best, which is somewhat

since the states of the audio

by the uwdel

the threevectors

in this simplisticof the map. Notably, withare the secondary

statescolours,

but in the centre of the edges. In general,
as the Other two SOMs.

colour-spacefoundexamplePCP,

may be directly

the colours

used tonot at the extremitiesof thehowever, it performs

about

as well

0: 00
PosiUOn

o000

M

WavO

SOM2
SOM4
some
PCA

Position

0: 00
Figtire'I'llehigher

4.18: Several visualisations

sires of SOM

to the interference

structureeclge-cnuV of t he preprocessing

become warped

slightlyeffects of the audio
methodof the test tone `plucks'.
as they attempt

windowing.

This

to give some subhighlightsanto accountforperhapsmade worse by the SUMS' attempts

it.

4.4.2

'I)"il)-llop

Stem. This is a fairly
track Stun/Long
evaluation
with the trip-hop
complex, irregular (i. e. wit Ii no repetitive structure),
subtle (changes are often graduni) anddiverse piece of ºuusic. The t rack has two main crescendos preceded by similar progressions,
I begintue nwvic

smaller one at. 1:4t), the second at 3: 23. After this it changes style somewhat,
reºnoving Hit, originallypercussion altogether and changing the ensemble.
quite important
The SOh12's depiction of the overall structure of the track is niediocre; crescendos arein thetelºre, vntecI clearly in blue, and red denotes dominant samples of brass instruments
the first,

foregruiuul.

However,

at, 3: 24, suggesting
simpler,clnieterthe SON12 misrepresents

it repeatof the startuºutiic of the saue

the changeof the track.

basic style to thatin style with

Iºº general,of green.

it yellow

block

the yellow

represents

Unfortunately,

applying

4.4. DISCUSSION OF METHODS

0: 00

Positiono_.oooo
N 7ON

1: 00

2: 00

0ý

00 NINMVNINMO
00m

I90-OT

IN

113

3: 00

070N

4: 00

00000

0000on

5: 00

00000
.-NMONINNINVi

6: 00

7: 00

0000

8: 00

00ý0ý

imm

Wave

Milwemmr

IMIM

SOM2

a

§IM

SOM4

I

II
!

@IM

Willi
?
SOM8

W9M"

PCP

Pos

oo.

ooooo
N

i t ion

ooooo
0)

ONN

0: 00

Figure

0)

1: 00

N

00000
M<
'N

N

2: 00

does not adequately

The SOM4 utilises a significantly

00000

ONN

3: 00

4.19: Several visualisations

such a boundary

00000
l')
.-N

.-

5: 00

of the track

represent0C)C)výý.

0) VNi.

4: 00

ýoý

0

C')

6: 00

Stem/Long

t')

7: 00

.>N

8: 00

Stern by DJ Shadow

its diverse content.
to construct

greater palette

its image, with

the SOM8

block

an image with more detail, noise, and only a few significant changes. The rose(around 0: 30) represents a note from a string instrument,at the beginningmost

likelya double bass, in the background.

erratic

drums,

presenting

the following

The change to dark green represents

the onset ofWith

the SOM2,

and light green to sustained

percussion

change to gray does not clearlyat the beginning.

and the crescendo.

show that

The SOM4 and SOM8 represent

the content

the variation

is a variation

reasonablyof thatwell, depicting

it as faded yellow/yellow/dark
faded green, and finally a longer crescendo of bright(rather than yellow/rose/yellow/darkgreen).
green/light
The brightness
Howeverof green denoting

like the SOM2,

SOM4

the veracity

and SOM8of percussionis also a reasonablesuffer from reusing

yellowgreen

analogue.

in the second half

(at 3: 23), which,withaside from being on the whole quieter, does not share much in common
The use of orange (at 3: 43) for the organ is reasonable, however it is
the beginning.

as a clear block in the SOM4, and a gradient from yellow in the SOMS; the latter
is a better representationof the gradual fade-in of the organ.

depicted

Betweenblocks),

4: 08 and 5: 12, the track

althoughspeech apparently(yellow
becomes relatively similar to the beginning
sampled from a film as well as some other instruments

are mixed over the top; as such, the mild orange and pink hue changes may be considered
Following this, the two higher SOMs improve on the large red block of thewarranted.
SOM2 by introducing

loud brass notes are consistently
represented assome substructure;
red and purple blocks; red being for the higher notes (there are only three main notes).
On initial inspection, the PCP image is visually pleasing. In particular,
the gradient
However,
like
the
the
organ.
approachingofreflective
other methods, it suffers from being the same general hue as other portions of the track(not least that which follows it directly) which are musically different. The end portion of
between

3: 24 and 4: 05 is largely

11a('11AV'1'I ºU .1. CHROMATICITYt he I nick with
tiO, \IH.

1)1,1vand reel Imes are somewhat

Wit le tIu

P CI), withthe blue-purple

In general,

the 1'('l'images.

track

its minimalist

properly

arguably

tx

4.1.3

Hap

Figure'i/o"1.20

work excellently
Position

Forrr,

rite track.

0: 00

or SO1MI8, both

the diversityof thelost by the SOIVIH over the SOIVH isof extratrack of roughly

1: 00

Li_ ý. ý__.

to represent

information.

All four of the visualisationon this tincl)ne-dominatedandf'roin the(listinguishiible

the SONN

that

The SON12 seems unable'tan'sof SOJI4

so.

palette; the clarity
great ei price to pay for the small .einoiint
withare barely

no more information

providesof, Which are clearer

less clear than tue banding

striping

Inne; the Iirnss nute,, ill the iuusic are readily

PLANE ýrºtýý.rl; (ýrORIES

constant

2: 00

al pear tomethods
intensity.

4: 00

3: 00
Rgä9°Rä°

III1

Wnvo

ýIIýIIIýIII
SOMI

SOMA

Mill

_'

! ''-

JII

ý-

'IItII!
SOMtf
mi
PcII
1111MIIIIMMIM1III
fir:
R S3 S

PositioFigtirc

'

Si

0: 00

N

Si

Iic>1I>tluv I'('1'and I Ii

of tli

RS

IMMIý
R

Si

5ý

°

2: 00

3: 00

4: 00

rap track

The Force by Aim, (featuring

Q'n'C).

1: 00

Several1.20:
visualisations
.

IIIIý

SON 12 do a very rvnsuuahlc' top-levelof theis visible (bluetrack; the other two ii>tre>ciuee it little more tiubstreu"teu"e. An introduction(yellow/green
ill the tiOMs and yellow in Hie 1'('1'),
I>me repetitious
thet.
chorusateofits
in t.lu. SONIs nt)el I>Iwe/reel iu I he I'(: 1'), and two repet. il. ic>ns of fluý verse (red iu the SON Is
visual representationand green in the I'('1'). SON H and SONIM clearly show that the initial chorus is a variationin SON H
oil the following versions, wit Ii it slightly different coloured section (green/cream
All four visualisations
its
Einei gray/c r ºm in SOMS).
represent the chorus substructure
fuur neI)Vtit ions of nutuiPs(rif)l,vocals.

anti loving identical to tlºe first section;
nlt. licwgl1 ho t.h ion. cluic't the first is purely prrcvtisionand the latter purely nondescript
Vocals. Iluwe ver, nsicle Il-oll that, SON12's tic heule is generally robust. Grenu represents
S( )iNI2 misrvllr

vents t he sect ion sta i tilig

3: IIpercussion,

with Yellow and reef repº'ese"nting the vocals of the ti'ºn8ln and male respectively.

'I'lle striped

portions

The otherSUM

clouc/tc' broken
visualisations

vocals over it bac"kgrcnnul of percussion.
largelybuild

on theof SOM2,

simplicity

with

SONIM

rý

4.4. DISCUSSION OF METHODS

115

removing

some advantageousquantisationof colour.

is to split

the vocals between

each vocalist

into

the othera perceptuallysimilar-but

noticeablyof the apparent

misrepresentation

the cost of a significant amountbetween vocals and non-vocals.

less clear.

Selecting

simplicity

should4.4.4

betweenSOM2

be prioritisedof SOM4

contribution

one purple-red

anddifferent-orange.

SOM8

the

sectionremoves

at 3: 11, but does so at

and a less clear demarcationof noise added throughout,close, SOMV12and SOM4 arguably
information
to SOM2 as does SOM8

main

two clear sections,of the first

repetition

AlthoughsimilarThe

perform best overall. PCP tends to provide
to SOM4, but in both cases it is simply

and SOM4

woulddependupon whether

clarityandinformation.

over substructural

Jazz

Figure4.21 depicts

is immediately

Position

Dave Brubeck's

jazz classic Take Five.

The complexityof each image

noticeable.

0: 00ON0lÖpf

1:00ONmä2:

00ONMÖ

p(

3: 00

NpN4:

00OOO5:00

OOON

IIII1I
Wave

SOM2

rr
umoll
SOM4
Ellm1
EII
PCP

iII40
IiIII
11ý

Moll

0

II Iý

U. II

ONhdNON

Position

NOW

I.

,

ÖONhÖNONlaANhONON
PO

0: 00

5: 00

4: 00

3: 00

2:00

1:00

Figure 4.21: Several visualisations of the jazz track Take Five by Dave Brubeck.
In terms of a top-level

overview,

the trackhas three main

portions;

an initial

theme

percussion-dominatedsection (until
4: 23), and the reprise of the theme in the last section. This structure is broadly noticeable
SOM2 segregates the track into red/blue and green/yellow,
the otherin each visualisation;
witheach of the instruments(until

1: 52), a central

SOMs into cooler colours green/blue and the warnerseparates similarly but with the opposite meaning.
Eventsin the track

are also depictedcommonly;

the PCP

red/orange/yellow,

while

each of the SOMs

makes

visibleand piano between 1: 52 and 2: 10 when the
PCP,
but
is
discernible
inthebecomesdominant.
This
is
clear
so
not quite
percussion
loud
SOM2
PCP
both
Notably,faintasectionoftinting.
misrepresent
the
and
with
green
is
this
theme;
the
block
to
3:
15
presumably
mainanpercussion aroundpertaining
as a
the consistentand relatively

quiet percussion

Ui

CHAPTERlinac"cunntecl-for

1.
CHROMATICITY.

PLANE, TRA. 1b,'CIOUIº: '.5

entt.lier of the distribution.

In terntti of substructure,by it similarly

the two SOMIs reveal the gradualcrescendo achievedby thein lute:From orange to pale orange through grey,
yellow with bale green, yellow with grey specks (representing loud and irregular drumming)and finally tuiltntken yellow. The PCP and SOiMI2 are less instructive here, however in terms
cirutusgraudual changeof repetition
and Variation of the nto-tin figure of the saxophone, they perform significantly
Itet. Irr than the codier ntet. hcxis. The theme is clearly visible ill the PCP as two bold violet.
Itnrs on a yellow backing repeated; the reprise of the theme at 4: 24 is clearly recognisable,
despitethe bars being reel denoting a slightly louder accompanimentTheof instruments.
two larger SOhIs (it) show this information(but gis double green/greybars), and a't stichit, is entirely
the culuiu

t hat, the clearer

possible

vitictals is siutply

clue to an unfortunate

rotationofspace.

4.4.5

Classical

Figtiru

1.22
slows
.

-art's Nachtmusik Allegro, a classical stringof 1110track; ininircliatcd, y uot. iepnl)Iv is t Ile difficulty
structure from such music withof extractingf)urcly t iinl)rc'l information.
0: 00

P-111llll

various representation',

:ýV
1I

I1I11

"A
Il
l.

1:00

11
N

2
'ý

J

(ý
ýA

"!

2: 00

OOpOOOO0O0O
N

4: 00

3: 00
V

[h

AI

NTV

ýA

I1I11.1

W11w1

Mm
SOM211
IL

SOMA

11 ME

-Wf- M

PCPIMM ýii

PosiuunFigure

n

ý3

0: 00

tiewral"1.22::1 nlnrlrus

3

i"
iý

1: 00

I

It

is im

of the classical

1: u
-T

4: 00

3: 00

2: 00

visualisations

m
°°4ä

5ý

:ýv

w

WM

M 1M

Ellin

-T

IN I

.I

ýMNM.

NEIRWOM

Mug
F1
SOM8
f

mo miI

track

Nachtimisik

Allegrnby Wolfquteyh10'(01.

series of primary colour ntripus of the SOM2 visual does not appear to beetil4ev"ially intit. ruc"t.iVP its to t.Iiv content of' the track; the large blue portion at around 2: 31)
background,Oletwill',
intheperformed
ctuietwithout.
secondset
strings
itofele"neºte,'I'Iii

givingcomplext114 t r, u"k its fntit-paced

rlryt. hm.

It tu IIu. IºIue port ion, comprises

On close inspection,

we can see thatthe track,

it single t10911v plºryv(I twice at they beginningand again

the first, minute'. It ºwiy lxs seen that, ºº variation
lust, aft.
.
more, directly nfte'r they blue' portion, at 2: 56.

of this tii

iio' is plikyed Once

4.4. DISCUSSION OF METHODS

With

bettera slightly

above facts slightly

point,

the SOM4

packed gamut,

a similar

However,colour.

motif

and SOM8

Despiteof the music, which

considerable

detail,apparent

as the first

The PCPvisualisation

makes a similarof information

amounttwo, although

the quiet break between 2: 30 and 2: 55 is not as apparent,,
reds used, which are flecks elsewhere in the track.
Althoughnone performed

suited to provide

especiallyas clear a picture

well,

the SOM4

withoranges andseemed the bestvisualisationof the track as possible under the circumstances.

Downtempo/Electronic

Figure 4.23 shows the visuals generated
The Dining Rooms.

for Occhi Neri, an electronic

0: 00

Position

00000
oo0 t")iN

Wave

oi'00odowntempo

03:

ßl1

iN

track by

OOo

000

No02:

VNiMONiNMO

o004:
(")

ViN

aim
-foe-peeoeea

1I_

SOM2
SOM4

1U
PCP
00
ýNýN

Position

0: 00
4.23:

INN!

I

INN
l1I 111
IIII III1 I_l

IIII

jý
I III

SOM8

Figurenois apparent.

substructure

4.4.6

before the blue

theme

this is not reflective

played in a lower key.

make thevisualisations

as the non-repeatedeasier to spot, especiallysection at 2: 06 is of a different
is, at that

117

UI
oý

o000ooooooogl7VýNýNýN

1:00
Several visualisations

2: 00
of the downtempo

4: 00

3: 00
track

Occhi Neriby The Dining

Rooms.
Completely

is that the music does
unclear on both the PCP and Wave visualisations
not start until 0: 15. All three SOM visualisations
are sensitive to this fact, with clear andaccurately abrupt colour changes. A further aspect largely missed in both the SO\12 and
PCP visuals

is the very noticeable

startof percussion

at 0: 48. Although

there is at slight
subtle and barely

it is undeservingly
introduction
noticeable. The quite importantof bass at around 1: 05 is well represented
each SOM image, but once again unclear with the PCP.
change of tintin the PCP image aroundthis time,in

For the rest of the track, each of the visualisations performs roughly equivalently, eachpresenting some sub-structure between 3:50 and 4:20. The PCP image delivers the mostsubtle substructure, giving very blurred stripes between blue-purple and purple. Arguably,

PLANE TRAJECTORIES

4. CHROMATICITY

CHAPTER1

this could be considered this the most accurate the music at this point is a stable runishowever,track
theConversely,
very
the
top.
of percussion with samples merged overbutfor'norrnalising'
is
small
there
actually
throughout;to
this
an argument
as suchsimilar
Fuitherniore
featUtes.
lange
X'
:
lart
t1.
to
visualeqýiwaXrý eovcliE£ece\Wce.,
relativelg
ax"loxigh the stripes may not denote particularly significant timbrel features, they dorepresent well the repetition of a figure, and so may be considered as musically important.
The end of the track isa run of percussion ending at around 4: 40 (clear on the SOM4
and SOM8),This appears to disrupt thewith fairly quiet samples of chatter mixed.
SON12 and PCP algorithms,
which result in an unclear combinationof colours for the last
25 seconds.
SOM4 and SOM8,
providesit subtle

extra

periodicslightlywith

presenting

4.4.7

cue to the equally

SOM8

subtle change in the loudness and the additionof an

are especiallyat 3: 35. For this track,

sampleits harsh segmentation

it slightly

in this track;

except for the smoothing,confusednotingsimilarthe SOM2 representation

pointsimage in the latter

underperformedin the music, andof lesser significancehalf.

Pop/Rock/Metal

Our penultiinate
track

simplerock

overallstructural

with a verse/bridge/chorus
structure,and is well-represented
level by each image, shown in figure 4.24.

0: 00

Positionchi

toN

This is a relatively

Generator.

piece of music to consider is the rock ballad

vo

1: 00
$Io000p0000000N

3: 00

2: 00
c+l

V

on an

NIN

ch

8

RIN

ao

11

Wavo

I

-

SOM4

_il

SOM2

ý

SOM8

PcP_II5

II

=ME

Iii

OaNN

Position

th

0: 00

Figure1.24: Several visualisations

e leºu" segmentation
structure of the track. Afterthe abrupt,

VNNMäNN

1: 00

The

with

.

guitarafforded

3: 00

2: 00

of the rock ballad

riff of the latterby the Foo Fighters.

an instant idea of thethe three verses are clearly depicted in red,by SOM2

the introduction,

Generator

gives the viewer

two depictedin thewithan abrupt

green stripe

as a waypoint

betweenred and green here,

middle.

131ue appears

to be used essentiallyoccurringat houiularies

between the two with no apparent

equivalent

in the music.

Unlike4.4. DISCUSSION OF METHODS

SOM4, which clearly

119

defines the three keys of the chorus (dark green/grey,
to the SOM2 algorithm

the lack of colours available

yellow),from becomingThis is particularly

apparent.

noticeable

prevents(especiallyin the red-dominatedis introduced

extra

substructure

guitarriff at 0: 44 and 0: 50, visible

withany such substructure

after the bridge between

2: 57, after which the SOM2 remains green until the outro.
The SOM4 and SOM8 once again build upon the SOM2
slightly

light green and
2: 38 andbut blurrepresentation,it

Small amounts ofverses), reducing the clarity.
the SOM8, such as the repetitionof a six-second
On the whole,orange stripes.

as a pair of yellow/palethe extra structure added with the SOM8, while perhaps following the music perceptually(which changes timbre fairly frequently),
in terms of recognisable
does reduce its utility
features.
The PCP

provides

Althoughview overall.

a more subtle

the chorus sections

visible,of yellow are not quite as clear as in the other images, especially the musically quite clear
separation at 0: 56. In the blue-dominatedchoruses, the PCP provides a subtle cue to thechange in key, as it changes from blue-green
2: 38 fading
different

into a murky

however,green-blue,

The bridge at

to blue and then to blue-purple.
is not clearly

separatedfromthe obviously

vocal section after it at 2: 56.

In general, the simple structure of this track is captured well with the SOM4 image. The
PCP image does not provide a clear guide to the contents; emphasising a large difference on
the overall structure, it gives too slight a difference on reasonably important substructure.
Dance/Classical

4.4.8

FusionFigure 4.25 shows the visuals generated for the dance/classical fusion track Clubbed to
Death.
0: 00oNMON

Positionp

1:000 0000

3:
00
N
ýNýYO

2: 00 Nm

4: 00ýN

pp

ppp

(1

5: 00ýN

dN

pp

ýd

6: 00ýN

7: 00ýN

o

+Ö/

f

op

d

Wave

SOM2

,

-_

_II

SOM4
SOME
PCP

-

"s

II

Ti,
Position

-Nt9O

0: 00

-N(ýl
iA

1: 00

O

uni

ýNtýf

2: 00

Ö

uff

ýNtOY

3: 00

N

ýNtOO

4: 00

N

ýN

5: 00

t+i

dN

ýNt+f
ý!

6: 00

N

1'1 Y

7: 00

Figure 4.25: Several visualisations of the dance/classical track Clubbed to Death by Rob
D.
The PCP visual shows a dubious separation of the classical portions of the track from120

CHAPTER 4. CHROMATICITY PLANE TRAJECTORIES

the percussion; yellow Is used to denote the quiet bass and piano section around 5:56, withsimilar yellow-green used to denote the onset of heavy drums and bassaround 0: 24 and 4: 00. The bold red and blue blocks do well at separating the extreme
timbres present In the track, but the simplistic dimension mapping does not seem to reflect
the multi-timbre nature of the first two and a half minutes, most of it being a (difficult tothe perceptually

mix of yellow/green/blue colours with no clear boundaries (which are present
In the music as changes In key and voice ensemble). This track illustrates the problemsof relying upon a linear map which provides too few degrees of freedom in a long andInterpret)

timbrely-complex

track.

The SOM2 method once again conducts a reasonably clean segmentation. Blue represents the purely classical portions of the track that Rob D himself composed and played
Including both piano and strings. Red represents quieter strings and piano. The red/blue
striping at the beginning represents the excerpt of strings taken from the Main Themeof Elgar's 'Enigma Variations'. Thus the change from red to blue at approximately 6: 25
represents the Introduction of strings and a loudening of the piano. Yellow represents
louderIf
represents
green
any,
stringpercussion with a small,accompaniment, whereas
percussion with more voices (sampling, strings, piano, bass) in the background. The
SOM2 functions here as a simple method that draws attention to the main changes andthat acts essentially as a top-level segmenter.
The SOM4 and SOM8 methods build upon the SOM2 representation considerably;classical portions of the track are again represented by the red-blue portion of colourBlue
light
dark
helps
but
the
represents aandpurplesubstructure.
use ofrepresent
space,
raised key in the piano solo. The pinks at the beginning represent the violins; notably, after
they are introduced around 6:55, the intense blue and purple become less so, becoming'dragged' towards the pinks of violins. The Intense red portion denotes the low-tempo bass

and quiet piano portion of the track.
The percussion portions of the track are represented well in the orange-yellow-green
SOM8
in
the
instrumentsThethechanges;
medley ofcolours change subtly as
colours.
high-pitchedstrings withandofasampledvoice
abruptsubstitution
anvisualisation,
far
is
This
key,
brightlower
denotedbyIs
In
clearertoayellow.
achangeaslower strings
faded-yellow
faded-yellow
before
It
to
the
two
to
and
of orange-yellow
change comparedto pale green, denoting the Introductionof a single extra sample and background strings

respectively.

4.5

Conclusions

I have detailed and discusseda novel audio visualisation technique, and evaluated it interms of being useful as musical audio navigation aid. I considerthe results of the discussion

4.5. CONCLUSIONS

121

of the images to be encouraging. Usage of the self-organising map as a dimensionality
reduction method appears to be largely vindicated over simpler techniques such as PCA,
due to the amount and clarity of information perceived by the viewer. The linearity of
the PCA seems well suited to depict overall structure clearly and attractively, however
subtler and transient extreme changes are often ignored or misrepresented. The natural
usage of the SOM as a quantisation and clustering technique also seems reasonable, withthe 4x4 map often presenting the majority of information of that given by the 8x8, butwith increased clarity.
I have now examined a range of simple musical audio visualisation

techniques thatare appropriate for augmenting a navigation aid. In the next chapter I detail a quantitative task-oriented evaluation of the better methods of visualisation that have so far been
identified.

122

CIIAPTER 4. CHROMATICITY PLANE TRAJECTORIESChapter

5

Evaluation

Navigation
of

Aids

"The most exciting phrase to hear in science, the one that heralds new
discoveries, is not `Eureka! ' (I found it! ) but `That's funny' "...

(1920-1992)
Azimov-Isaac5.1

Introduction

My aim is to determine whether automatic content-based visual aids actually do help
people navigate. A common method for visualising sound is the loudness waveform, which
is likely to be easy for people to understand and use. I use this as a baseline to determine
whether other visualisations are more effective. To test the theories presented over usage
improvement in chapter 2I determine what learning effects there are, if any, over time,and how this changes between those' who use the visualisation nandthose who do not. To
determine whether the theory concerning irrelevance of absolute information Is founded
together with the theory that the SOM presents information better than the PCA methods,
I test whether users utilising the SOM or PCA methods can perform as well as with theoriginal methods.
In general, I actively look for evidence to support or refute these theories. I test thesehypotheses by constructing a null hypothesis which I may attempt to refute, statistically,
to some degree of certainty. In all the statements I make in this chapter the statistically
accepted probability of their truth is at least 95%, and considerably more In many Instances.
5.1.1

Chapter

Summary

I begin by detailing work related to the evaluation of visual navigation aids, of which there
is very little. I continue with the first two studies, which, though essentially simple pilot
studies, resulted in enough information to make some statistically significant statements.
After these, I move on to the major study, where, with realistic use cases,I determine
123

124

CHAPTER 5. EVALUATION OF NAVIGATION AIDS

usage patterns and comparative performances over a variety of what I consider the best
proposals of visual aid. I then detail the final study carried out to investigate the learning
effects over several weeks of usage (prior to this all studies had assumed a minimum of
training).

Our Contributions

5.1.2

With the studies made here, I present, with statistical significance, evidence supporting
the theory that content-based visualisations can indeed guide users around the track,
contributing to their performance over a variety of tasks. I show that both the basic
bandwIso-loudness and SOM-based visualisations aid users in determining answers to avariety of questions posed on the content of pieces of music, over simpler methods suchas the widely used waveform. Furthermore, I present evidence suggesting the reasonsfor this are a pronounced reduction on the number of seek operations around the music
and an increased accuracy of seek operations. I also make a number of other significant
observations about individual

5.2

usage and methods.

Notes

Statistical

Testing

For determining

general relations and making categorical statements concerning theseresults, I defer to the Tukey Honest Significant Difference (Tukey IISD) method of inferenceof p-values (hypothesis certainty) as described by Yandell (1997). This Is more conservative(e.g. than a basic pairwiso two-way t-test), since it takes into account experiment-wide
variation of mean, and Increases p-values accordingly.

Problems with Testing
The nature of tests means that finding statistically significant results Is difficult, due towide ranges of people's skill and experience. This could be mitigated though conducting
significantly more trials. However the resources to do this were not available. In total, 106
trials were conducted. 82 Individuals took part In the studies, and, with several
tasks in each trial, I collected 2686 individual trial results.

Individual

Softwaro
In augmenting the software with which to do the trials, the principle of least surprise wasfollowed and the playback interface was changed as little as possible. The media playerthat I set about to augment, amaroK, already provided a highly intuitive Interface with
the now-standard navigation bar. In amaroIC's case, a triangular pointer scrolls across5.3. BOUNDARY-FINDING

Figure5.1: The amaroK

the top of the bar, denoting

TESTS

to the visualisation

the BLSM

music player withthe current

only change I made to the interface
according125of the player

position

was to have the internal

algorithm.

Boundary-Finding

throughoperational.

the track.

Theof the bar coloured
the user was free to click anywhere

As before,on the bar, to warp the player to the corresponding

5.3

visualisation

position

portionin the song.

Tests

The first two performance examinationsin this chapter are based around the concept ofboundaries in music. By this I mean distinguishingpoints in the music which someone
would want to remember or know in advance. These might include onsets or removals of
instruments
and voices, chorus/verse/bridgechanges and crescendos. I leave exactly what
constitutes a boundary as a decision for the user, but conduct a preliminary
study to verify
that there is some agreement

5.3.1

Objectivity

I initiallyconduct

between the opinions

of multiple

users over such boundaries.

of Boundaries

considerably

a small study to test the hypothesis that multiple people do not disagree
about where they consider major boundaries to fall in a given piece of music.

The validityof this hypothesis

participants

may utilise

the following experiments.
will be assumed in interpreting
Furthermore,
I expect that there will be more agreement on boundaries in popular music,
owing to the presumption
thatof typical and familiar structure (i. e. verse/bridge/chorus)
to determine

prototypeboundaries.

There is prior work by Tzanetakis and Cook (2000b), who concluded in favour of thishypothesis.
The results showed that around three quarters (0.73,0.76
or 0.79 dependingon the configuration
reported wouldof participants)of the total number of boundaries
typicallybe agreed upon by at least 50% of participants.

(twice ours), where they were given a special segmenting

suggested to them potential boundaries.

This was a study of 20 participants
audio editor

which automatically

126

CIIAPTER 5. EVALUATION OF NAVIGATION AIDS

Althoughwas randomised in eachcase, the measured learning curve for the audio editor was steep, suggesting that there may
be a significant amount of (albeit fair) noise in the results. Participants were being timed,
the order of audio tracks given to the participants

potentially biasing them Into selecting at speed rather than accuracy and precision. Audiowas not discriminated; music together with spoken word was used. This experiment Is afurther test of the hypothesis under slightly different conditions (entirely music, specificgenres), and thus largely complementary. Since the music used in this test also happens
to be that used in two of the later experiments, It also helps give some direct validation
for the interpretingof their results.

Motliod
Ten participants were given nine pieces of music each, to determine the boundaries in. For
the music, three tracks were used from each of the following genres; classical, rock/pop
and alternative clectronica. The tracks used are given in table 5.3.1.

Naino

Genre

Air on a C-String (J. S. Bach)
Siren Song (Devics)
Elios Therepia (Blue States)
Country House (Blur)
Minuet in A (Luigi Boccherini)
LebaneseBlonde (Thievery Corporation)
Paranoid Android (Radiohead)Classical

They're Hanging Me Tonight (Red Snapper)
Barber's Adagio for Strings (T. C. Albinoni)

Pop/Rock
Alternative

Electronica

Pop/Rock
Classical

Alternative Electronica
Pop/Rock
Alternative

Electronica

Classical

Table 5.1: Tracks used for determining agreement upon boundary positions.
listened to the tracks, free from a time limit, on their own
Inboundary
the
They
down
Important
8±2music.
pointsnoted
mostequipment.
music
Exactly what constituted a boundary was left for them to decide, although as a guideline
they were given the same directions for selecting them as those taking part in the futureforWesternlistened
Notably,
to
at
the
music
popularparticipants
each ofexperiments.
least one hour each day, suggesting that they may have a relatively shared appraisal ofwhat would constitute a boundary.
Each of the participantsOnce collected, the points were collated and cross-referencedto test for agreement.

5.3. BOUNDARY-FINDING

Alternative

Classical

o

TESTS

Rock

o

127

Alternativeo1.

Classical

o

Rock

o

o

I

0.

0.

c
IL
c
0

O.

0 0.
9
rn

ä

0,

ä

0

d

0.

a
0

ä
0.;

0..

0.(

0.4

People Sharing Agreement(a) 1 second disparity

Figurefor agreement

5.2: The approximate

between a number

People Sharing Agreement(b) 2 second disparity

probabilityof any given boundary

for agreement

being agreed uponpointof people.

Results
Havingtabulated

agreement

the results,

of a given boundary

can be graphed.

Figureprobabilityof findingof othera disparity

opinions sought
between points of at

Figure5.2(a) shows the samethe change in approximate
point

with

the number

5.2(b) shows the graph, requiringmost two seconds for them to be considereddata with a stricter disparity of one second.
An analysis of variancein agreement.

may be conducted,

a degree of

persons'

to check how each factor apparently

affects

the result.

Table 5.3.1 shows the results of the analysis, confirming the significance of thecurve. Notably, for the stricter appraisal of agreement, the small differences in agreement
between genres appears statistically
A post-hoc Tukey HSD test reveals thatsignificant.
we can be 95% certain that classicalonly by a mean difference of 0.068).

Degrees

PeoplelGenremusic has a greater

amountof agreement(albeitP-valueof Freedom

1-second10.::
22-second

00.014650.1721

From the figures, it is clear to see the difference of overall curve of the graphs, withthe two second disparity yielding a shallower curve overall. This shows how, among 2-3

128

CHAPTER 5. EVALUATION OF NAVIGATION AIDS

people, points tended to have a very high level of agreement, if being slightly imprecise.
The probability of a point being agreed upon byat least half of the participants iseither 0.68 and 0.80, depending upon the degree of forgiveness applied for the duration
between points. This compares as slightly lower than Tzanetakis's
measured 0.76 at a
disparity of only 0.5 seconds, but nonetheless supports theprimary hypothesis.

Conclusion
I have demonstrated

that people typically have a reasonable level of agreement about
what constitutes a boundary in Western music. Furthermore, I have found evidence tosuggest the Invalidity of the second hypothesis, showing a small but significant increase inagreement at a point-disparity

5.3.2

level of ±1 second.

Basic Methods

The first study made was essentially a pilot study to test the hypothesis that casual musicnavigation can be aided by provision of an image. Five visualisations from those proposedin chapter 3 were integrated Into the navigation barof the popular Free-software music
player, amaroK (see The Amarok Team, 2005 for more information). I aim to either refuteor strengthen the hypothesis by finding that at least one navigation aid does `help' people.
In this experiment, `help' Is defined with the task of finding boundaries in music, similar
to that of the previous section; here, however, a strict time limit is imposed for each track.
Their performance Is determined by comparing their given boundaries to a ground truth
formulated by the combination of several persons' boundary delimiting.

MethodInitially,

five tracks were selected from a range of music. The tracks were selected to give agood range of different types of music and of different difficulties of problem. Tracks werechosen for their Interesting features that would best test the systems. Mood changes (bothsubtle and blunt), instrument changes, vocal changes and rhythm diversity are among thefeatures I attempted to utilise in order to best examine the systems. Table 5.2 describes
the tracks that were chosen.
For this study, I use the ground truth provided by two people who delimited the musicin question manually, with no aids. In this early experiment, it was supposed that thiswould be enough to get a good Idea of the outcome. Further experiments use a significantly
greater corpus of ground truth. 18 subjects were then each given five trials-one for eachof the five tracks. For each track, each of the five analysis techniques given in table
5.3 together with a 'blind' control with no annotation were rotated through. This study
therefore assumes that any cross-learning effects between the visualisation algorithms are5.3. BOUNDARY-FINDING

TESTS

129

Track

Genre

Walk in the Night (D. McMurray)

Jazz

Reasonable, consistent beat structure and loudness. Minimallysitions.
Plug In Baby (Muse)Rock

defined tran-

Simple rock ballad with clear verse/chorus structure.
Goodnight Moon (Shivaree)
Pop/Folk
Fluid pop song with little beat structure and hazy verse/chorus structure.
Prague Radio (Plaid)
Electronic/Abstract
Timbrally complex, highly dynamic with multiple moods and well definedbeats.
Keep Hope Alive (Crystal Method)
Timbrally

Electronic/Dance

simple, well defined beats/transitions,consistently

loud, few

moods.
Table 5.2: The five tracks chosen for the user study.
minimal, something of which we cannot necessarily be sure; this is circumvented in later
studies.
Giving multiple tracks and algorithms to the same user means that there is no chanceof an individual learning the tracks, nor how to use the algorithms during the experiment.
It also means that a relatively large amount of data can be collected (five sets of times
for each of 18 people). However the issue of learning effects between different algorithms(countered slightly with change of order) goes unanswered. Furthermore, using different
people means that there are no matched pairs, thus the results are far more difficult tomodel statistically. As different people are likely to perform very differently at this task,
the variance is likely to be large. This is exacerbated by having different people with adifferent permutation of tracks/algorithms.
Name

Abbrev.

Spectral MagnitudeSM

Bark Bandwise Magnitude

BBM

Rhythm Magnitude

RM

Bark Bandwise Rhythm Magnitude

BBRM

Novelty

Novelty

Table 5.3: The five visualisation algorithms chosen for the user study.

Each subject was given an initial period of training (somerequired more than others),
until they felt familiar with the controls of the player. Aside from getting to grips with

130

CHAPTER 5. EVALUATION OF NAVIGATION AIDS

the "look and feel" of the application, they were given no specific information on thevisualisation algorithms. For each trial, the subject was given 60 seconds with the player,
Incorporating the given analysis technique with the track loaded. They were allowed toutilise the random access of the navigation bar. During this time, they had to determine
as many boundary points in the music as they could.
boundary points would not be penalised.

They were told that additional

Participants

had to manually write down the times of the boundaries, thus limiting
their time even more. As with other studies, participants came from the undergraduate/postgraduate student body, as well as postdoctoral staff. Following the trials, a short
Interview was conducted, calling for comments on each of the algorithms.

User Commentary
The overall feeling of those interviewed was that they preferred SM/BBM visualisations
over any of the others. Having utilised all five of the methods, many also indicated thatthey intuitively

related the intensity of a point with the loudness at that point. Somewent on to suggest that they would then intuitively relate the colour of a point with any
Instruments playing at that point. Only one candidate suggested that brighter parts inthe visualisation might mean increased dynamics and otherwise less constancy.
The standard RM measure as well as the novelty measure were generally disliked.
Specific comments were "daunting" and "less predictable". The general feeling was thatmeasurement (I. e. novelty) was unsuitable for intuitive learning; people expected to see "chunks" of similar sounding portions of time, rather than specific points at
differential

which the music changed. Those who commented felt that the rhythm magnitude measuresimply looked overly populated and excessively contrasting, and thus determined it to be
too "daunting" for general use.
Cosmetically, almost everyone preferred colour over monochromatic visualisations (oneThe majority of those suggested that theyeven went so far as to say It was "pretty").
also found colour to be the better visualisation In respect to usability also. They found it"easier to distinguish",and "more Informative". The opinion of colour In the BR.M wassomewhat more divided. While nobody made It out to be worse that the mono-chromevariant, most favoured the look of the BBM, finding the RM less well defined.
As for usability and comfort, the participants were quite polarised in their opinions as
to whether adding colour was more helpful in the experiment. While some decided thatit gave them more Information and thus was more useful, others felt that the addition ofcolour increased the learning curve too much. Most went on to suggest that perhaps, given
enough time to learn, the colour might eventually be better anyway.

One participant suggestedthat the three colour componentsused to create the colour
from the low, mid and high portions of the spectrum should be switched. Apparently they

5.3. BOUNDARY-FINDING

TESTS

131

expected bluer hues to relate to "warmer" (i. e. bassier) sounds with redder hues related
to harsher, sharper (presumably higher) sounds.
Results
For each of the five trials per person, the times are collated given against the ground truth,
creating a single `score'. A two second linear kernel is used on the distance from the nearest
ground truth point to determine the score of any single given point; the final trial score is
the sum of all such points. Formally, to score a point t;, given the closest point to thattime in all of the collated ground truth for that track r:
3-It;s(t, r)_E
t

-rjI,

0,

It; -rjI<2,otherwise

j=argmin(lri-til)
3

(5.1)

The (perhaps slightly lenient) two seconds of allowable deviation was chosen due to thenature of the experiment. Not only were the participants heavily rushed, but the player'stime reading is accurate only to within ±0.5 seconds, and they had to write the time aswell as inspect it and listen to the music. Once collated, the distributions over the genres
may be plotted. Figure 5.3(a) shows the distributions as a box-and-whisker plot. The
box represents the lower and upper quartiles, the inner line is the median and the outerlines are the minimum and maximums each genre. It is clear to see that BSM and SM
visualisation algorithms appear to have helped more than the others. I make a standard
analysis of variance test on the findings to check if they can be analysed further. This ispresented in table 5.4.
P-value
Algorithm

0.0006228

Person0.0019703

Track0.1041948

Table 5.4: Analysis of variance of the factors of the experiment.
This shows that the algorithm is highly significant at a level of > 99%, and, to a lesser
degree, the individual in question (less fortunately). Performing a Tukey IISD pairwise
comparison gives the comparison matrix of figure 5.3(b). Two statements may be madeimmediately from the results concerning the tracks:" SM and BSM are better than RM, Blind.
" SM is better than Novelty, BRM also.
'or the upper quartile plus one-and-a half the interquartile range

132

CHAPTER. 5. EVALUATION0OF NAVIGATION

AIDS

AAAAv

BBM
20
Blind

°°

BRM

15

Novelty

°°V

10

5

RM

vv

SM

AAAA

0
III

m
-C
m
i2l

'3

arc

d¢

mm

c)

CO

o

Zz

(a)

Box-and-whisker

of distributions

plot

of(b)

genre scores,
5 3: The.:

FigtireIISU pairwise mean score difference
Filled triangles are significant to 95%.

'llukeymatrix.
box

plot

Emd scoredifferencevii

for allmatrix

music

over

the

experiment.

If the results data is limited
statisticalsignificancecase for pop which
and the comparison

to only that of the electronic music, there is still reasonablefrom the ANOVA test (p 0.003898 for algorithms).
This is not thehas p -0.1924. The distributions
may again be graphednnitrix; this is shown in figure 5.4.

0

A

BBM

A

A

A

0

0

v

.

D

0

20
0

Q
15

Blind

rIý-,

BRM

A

Novelty

o

p

RM

A

Q

I

-*-'
10

5

is box plots

.

o 7
v

7

SM
0

(it)

Ilox-nwl-whisker

genre

Figureplot. of distributionsof(b)cores.

inatrix.

5.4: The box plot and score difference

pairwise mean score difference
Filled triangles are significant to 95%.
IISt)'I)ikeymatrixfor electronic

music over the experi-

cnent..
'I'Iu

nvsults of the electronic

music suggests similarfindingof I313M being significantly

betterthanbetterfindings

than Novelty,

to the others, with the extra

and SM being not so significantly

I31ZNI.

I5.3. BOUNDARY-FINDING TESTS

133

Conclusion
I have shown that basic music visualisations can help people navigate around a track forbasic boundary-finding tasks when given a tight time limit. In particular, the two spectralThereisbased
evidence toallothers.
magnitudevisualisations significantly outperformedsuggest that the BBM may be more helpful for the timbrally-clearer
for e.g. rock and pop.

electronic music thanI have further presented evidence suggesting that people may themselves find such aids
both aesthetically pleasing (especially in the case of BBM) and helpful. These are crucial
points for a system that is to succeed as a popular navigation aid.

5.3.3

CPT Projection

This experiment is to determine if, under a similar test framework to the previous experiment, we can show any differences in the performance of the methods proposed in chapter
4. In particular, I would like to test the hypothesis that the SOM-based navigation aids
do actually provide help over navigation bars without such visuals; I construct a null hypothesis that they do not and attempt to refute it. Furthermore, I wish to find evidence torefute the hypothesis that timbrely-well-defined electronic music is more likely to be susceptible for help through visual aid than other types of music. One other null-hypothesisthat I wish to test is that the direct loudness visualisation is at least as helpful as SOM
better).
SOM(i.
I
believe
the
that
are
visualisations
visualisationse.
The music used for this experiment is different to that of the previous; firstly to test
the latter hypothesis, I wish to use as much electronic music as possible to minimise thepossibility of the tested tracks being grossly unrepresentative. Secondly, I wish to have aslightly narrower distribution, refocusing on three key genres; classical, rock and electronic.
Thirdly, since these tracks are a subset of those used for the previous study on boundary
agreement, there is a good ground truth to work with.

MethodThe experimental method is generally similar to that of the previous experiment. As
mentioned before, six tracks are used, detailed in table 5.5.24 participants were usedin this experiment; each given all six of the tracks In a randomised order with the six
visualisation agorithms. The order of the algorithms was changed for each participant,to reduce the bias from learning. The visualisation algorithms used were the four from(SM)
frontIs4,
tothetogetherthemeasure,which
akinmagnitude
spectralchapterwith(i.
'blind'fromthealgorithm
e. no visualisation ataandprevious experiment,
runnersall).

The advantagesand disadvantageswith using this experiment outline are highlighted

134

CHAPTER

5. EVALUATION

Track

Genre

Country House (Blur)

Pop/Rock

OF NAVIGATION

AIDS

Simple rock ballad with clear verse/chorus
structure.
Paranoid Android (Radiohead)
Pop/Rock
Log, progressive rock with multiple instrumental
sections. Well defined transitions.
Minuet in A (Luigi Boccherini)Classical
Timbrely narrow strings-based track. Repeating andvarying figures arise inunclear transitions.
Barber's Adagio for Strings (T. G. Albinoni)
Classical
Long strings/organ track with pronounced dynamics. Several timbrellycontrasting regions give clear transitions.
Lebanese Blonde (Thievery Corporation)
Fluid rhythm-dominatedfine boundaries well.

Alternative

Electronica

downtempo track with some timbre-contrast

They're Hanging Me Tonight (Red Snapper)

Alternative

to de-

Electronica

Log, complex and dynamic track with many interwoven timbres.
Table 5.5: The six tracks chosen for this user study.
in the previous chapter so, I will not repeat them here. The only significant change tothe method was to implement a feature in the software allowing participants to press thespace bar on the keyboard to denote a boundary, rather than having to break focus andwrite it manually. They were allowed to delete the boundaries after pressing the space bar
by simply right-clicking

on the boundary represented by a small rectangle in the bar.

Results
The results are collated in a similar way to those of the previous experiment; scores arecalculated with the same two-second linear kernel on the closest matching ground truth
for each set of ground truths (10 in total for this experiment). The box-and-whisker plots
depict the distributions

over all genres in figure 5.5(a). There is an approximately normal
distribution of the points, with the trials of the SOM-based aids having higher but broader
scores on average. A standard ANOVA test is used to attempt to refute the null hypothesison the data that scores across all algorithms are on average the same. The test reports asignificantly low probability of any of the factors leading to a consistent mean, leading us
to refute it:

The post-hoc Tukey HSD test is conducted to check for any significant comparative
differences in the data; the results of this are depicted in figure 5.5(b). For this, we5.3. BOUNDARY-FINDING

TESTS

135

P-value
Algorithm

0.002400

Person0.001736'luck

1.363 x 10-11

can see that scores for each of the trials of SOM-based navigation aids are significantly
higher than those with no such help. There was no significant difference between any
other algorithms, although the means (i. e. size of triangles) are suggestive that the SOM
methods do typically result in a higher score then either of the PCA and SM algorithms.
This is continued by splitting the data into three genre-sets. Figures 5.5(c), 5.5(d) and5.5(e) show the distributions of data for the classical, rock and electronic music respectively.
Using a similar method to that of the above, the probability of the differences in scoremeans being significant may be found with an analysis of variance test:
P-value

Degrees of Freedom

Classical

RockElectronica

Algorithm

50.1256 0.776173

0.0007852

Person200.40070.002981

0.0052872

Track

1

0.2406 0.473618

0.7357833

The only genre with a significant difference in means is that of the electronica, which has
a particularly high probability of significance (> 99.9%2). Post-hoc tests on the electronica
corpus of trials give us figure 5.5(f).
One may see from the' post-hoc tests that restricting the corpus to only electronica
results in exactly the same significant conclusions as those of all genres.
User Feedback
Short informal interviews were conducted following the trials, to solicit any major opinions
the participant had after their small period of usage. Few found the visuals detrimental
to the experience of audio software; those that did found no utility or aesthetic value intheir combination of tracks and visuals. Of the visual aids, the basic spectral magnitude
aid was invariably placed last; opinions over the chromatic plane trajectory methods werequite diverse, but on the whole the SOM4 was the most popular, with a common criticism
of the SOM2 being too little information, and PCA of being too difficult to read though
its complexity. As the participants had so little training time, these opinions are indicativeof only the initial reaction.
2This P value is only accurate if this subset is considered as independent from the other two. I continue
anyway since it is small and the post-hoc analysis gives only supportive results.

136

CHAPTER

5. EVALUATION

OF NAVIGATION

AIDS

Conclusions
I have shown statisticallysignificant evidence to support the hypothesis that SOM-based
Although I
aids help in the task of boundary selection under a time-limit.

navigation
have no statisticallysignificant evidence to refute the claim that more basic methods suchas spectral magnitude are at least as good as the SOM-based methods I have, at least,very suggestive circumstantial evidence. Furthermore, the experiment suggested that the
PCA-based aid may not perform as well as those of the SOM.
In this task, there is no significant nor apparent difference in the performance of eachof the SOM variants. In terms of feedback however, users typically preferred the SOM4
over all others.

5.3. BOUNDARY-FINDING

TESTS

137

120

Blind

100

PCA

0

80

SM

D

SOM2

60
40
20

F9H
vÜ
85

(a)

SOM4
SOME

0

v

7

0

v

v

v

v

v

v

v
o

ý

o
D

o

-

0

ý

A

o

H5

a00

ifeÜtioD

tÄ

a000

0

°

D

v

v

o
V)

Box-and-whisker

plot

of all scores.

(b)

Thkey

HSD

pairwise

mean

score

diff.

ma-

trix.

70
120
60
100 -

50
40

80 -

30

60 -

20

40 -

10
20 7VQmNO

OD

65

(c)

-r-

2N

Box-and-whisker

plot

of classical

7<2NY

C8

5)
scores.

100

(d)

60

A

SM

A

SOM2

A

SOM4

20

v

PCA

40

a22S

Box-and-whisker

Blind

so

00

-

ih

a

(e) Box-and-whisker

C"

oooa
U)

plot of electronics

0vof rock

D

A

D

scores.

v

y

v

c

p

Q

p

V

V

o

p

o

ýýýýýBýI
SOM8

plot

n

D

o

n

y22

scores. (f) Pairwise mean electronic

score diff. matrix.

Figure 5.5: The experiment and genre-wise box plots and score difference matrix
electronica and in general.

for

138

FigureCHAPTER

5.6: The amaroKmusic player

5. EVALUATION

the BLSM

with

OF NAVIGATION& Loudness

HeightAIDSvisualisation

operational.

5.3.4

Learning

Rate

The final boundary-times
cerning

the learning

based experiment

numberwork were tested in three groups;
varied the heightI expectlittlesince participantsof navigation

throughout

aid, basic aids of the varietyof loudness.

comfortablenavigation

with

con-

evidencetask, over the course of aaids proposed

curve for the `blind'be typicallyshouldempirical

no visualas a function

or no learning

I collect

boundary-finding

the time-limited

A selectionof weeks' usage.

aids that

aids. In particular,

curve of navigationalon how well people perform

was one designed to test the hypotheses

aid withthe present

tested so far andno visualisation,

the concept of random-accessnav-

igationand are thus unlikely to improve in their performance of the task over the courseof three weeks. As such, I expect this curve to denote a control that shows the degree oftask-learningthat happens with increased familiarity
with the media. With no reason tosuspect otherwise,
sising thattraining

I imagine

the learning

the other two methods

curve for the algorithms

given is enough,or so shallow

is either

the proposed
loudnessof the classicalvisualisation

waveform.

I expectFigure`waveform'

methods.

so fast thatcurves, thus hypothethe initialperiodofas to be insignificant.

The aids that vary the height of the graphic
an integration

to yield similar

regard to the loudness are essentially(discussed in section 3.2.1) withvisualisation
with

They form a silhouette

5.6 gives an exampleof the visualisation

with

the

of this in use.

users to be on the whole more comfortable

withutilising

the height loudness

visualisations.
aspects of the visualisation
more readily than the proposed colour-projection
is a particularly
traThis is because, as mentioned previously, the waveform visualisation
software3. As
one, being found in all manner of professional and semi-professional(shaldetectableifislearning
I
be
less
there
the
to
a
curve,
pronounced
such,expectcurve
ditional

lower, since part of the learning
3an example being theopen-source

should already
Audacity

be complete).

audio editor

I also expect performanceto5.3. BOUNDARY-FINDING

TESTS

be generally better with the loudness silhouette, since more information
to the user.

139

is made available

Four participants were used in this study; although this may seem a low number
compared to the other studies, it is important to note that for testing the hypotheses thisproved adequate. Each participant had a total of nine sessions over the course of fourweeks. Each session comprised nine trials on varying genres of music, and thus a total of
324 individual trial scores were collected. Participants did not use the software between
sessions.
MethodPrior to starting the first session, participants were given around 10 minutes training on
the software with each visualisation used in the trials (but not the music). A total of('Blind').
bar
blank
in
to
the
navigation
additioneight visualisation algorithms were used
Four included the loudness silhouette, four did not. The four chosen were SOM2, SOM4,
BMFM and BLMS, and were assumed a diverse but reasonably performing selection fromthe proposals. The different methods of visualisation were chosen to help prevent learning
effects from multiple occurrences of the same combination of track/trial, and to maximise
the number of trials per session in order to maximise throughput of data.
In a similar manner to the previous two experiments, this does not take into accountlearning effects through trials. To minimise this effect, the order of music was randomised
for every session conducted. As before, participants were given only 60 seconds per trial,and had to identify boundary points. Unlike in previous experiments, participants werelearning
This
the
to
to
the
the
effectsofmitigate
was an attempt
study.
given
music prior
the music through the course of the study. For this experiment, the time read-out on
the software was disabled to prevent users from learning the individual boundary pointtimes in the track. The space-bar functionality described in the previous section made it
redundant.
Scores were made in exactly the same way as those of the previous two studies. The
detailed
boundaryin section 5.3.1;
fromtrackstheagreement,
of
study
music
used were
three each of classical, rock and alternative electronica. The ground truth found from thatstudy is used here also.
Results
Once the scores are collated, an analysis of variance on the results is conducted to checkthe probabilities of significance of difference of means, which, as table 5.3.4 shows, isreasonably certain.
Following this, the learning curve proper is plotted. Figure 5.7 shows this as eachtrial's point together with the approximate regression lines to show the trend. One may

140

CHAPTERAIDSOF NAVIGATION

5. EVALUATION

P-value

see that over the period
has largelyof initial

learning

parallelRun

1.019 x 10-05

Algorithm

1.622 x 10-05

Person9.731 x 10-07

Track

0

learning(untilcurves,day 13) each of the three methods
this is the time required for participantsaroundsuggesting
to learn how to do the tasks generally (in terms of becoming
well as learningat random

efficientaccess as

the music).

In the latter half of the experiment, however, the curves split, with the `blind' and`both' (proposed visualisations with waveform) level ling off and the `colour' (i. e. proposed
visualisations) alone continuing to rise to meet the level of `both'.

Blind

0

Both

Colour

0

o

x

x
150
x

x
x
x

x

x
M

x
ý
x
z

x
x

x

x

x

xx
xx

xx

x

x

#
x

100
N

M

8
N

x

Ä

x

x
x

x

x

x

1t

50

#
x

x
ý
11

x

x

x

x

Ix
xýM

8

XY

x
if
ft

ý

x
x

ý

x

x

x

x

X
J[

x

x

0
5

25

20

15

10
Day

Figure5.7: Scores of every one of the 324 trials

throughout
denotedthe study

by different

thatcolours.

the trial

The x-axis

conducted.
The different
was conducted.

denotes the point

navigation

aids are5.3. BOUNDARY-FINDING

TESTS

141

User Feedback

Interviews were conducted following the trials; participants were asked how useful they
considered each of the visualisations, what uses they could see for it, and whether they
thought these would be an overall improvement or detrimental to the interface. Candidates
generally considered the visualisation to be less useful for classical than for the other genres.
In terms of learning, they generally considered that the curve ended after half the numberof sessions, also suggesting that their increasing knowledge of the song played a big role.
The loudness `silhouette' was generally accepted as being useful initially, though again
less so for the classical music. The high degree of changeability was also commented on,
suggesting that this `noise' was too dominant. BLMS was generally considered one of thebest, although like the loudness silhouette, the high frequency variation was pointed out
as a defect.
The 2x2 SOM-based visualisation received both considerable praise and criticism. It
was considered one of the better representations of the classical tracks, and was typicallyconsidered to "do the right thing" in terms of representing "tonal variations". However
precision, aesthetics and accuracy were issues. Considered with the loudness silhouette,
it overcame many of these issues and was typically considered very useful. In particularhaving the duality of colour for tone and height for loudness was considered advantageous.
Notably, there was disagreement as to the efficacy of the SOM4-based aid. Some considered it a "more cluttered version" of the SOM2-aid, despite considerable improvementfrom the silhouette.

Others considered the SOM4 a significant improvement, suggestingthat the previously problematic greater levels of colour were a specific advantage. Therewas similar disparity in the opinions as to which was more aesthetically pleasing of thetwo, though both were generally considered the two `best looking', and there was little
doubt over their improvement by the silhouette.
The MFCC was typically considered the weakest visualisation, being a particularly"unintuitive representation". One participant suggested he was less likely'to use it due tothe excess "clutter and general ugliness".
In terms of the usefulness of the aids, most were generally impressed; as one said it
is "easier to jump around to a certain bit". One thought it would work well as a good
iconic representation of a track on a newly bought album; perhaps as a visual thumbnail.
Exactly when it would be useful was particularly polarised. Some felt that it would be
most useful before a track was heard, and thus in terms of 'jisting' over `bookmarking'.
Others were of the opposite conviction, suggesting that it would be largely useless initially,
but be good for getting to know a piece having heard it once already, and further increasein usefulness with knowledge of the piece.

142

CHAPTER 5. EVALUATION OF NAVIGATION AIDS

Conclusion
In general, the original hypothesis is supported. Each of the curves roughly resemble that ofthe `blind' curve. The curve `both', denoting the combined waveform and colour-projection
visualisation, gives a similar but shallower curve. Interestingly, the `colour' actually meets`both' during the time frame chosen for this experiment. I take this as suggestive that any
extra information provided by 'both' becomes redundant after a user is familiar enoughwith the colour-projection, and that this level of familiarity
weeks of intermittent usage.

5.4

Generalcan be attained within three

Task Tests

We arrive at the final and more significant experiment conducted as part of the present
work. In all of the previous studies, the utility of a navigation aid was judged by how
much it improved a user's ability to determine boundary points that best agree with someground truth. While this is an interesting use that no doubt requires navigation to succeed,
it is a slightly indirect method of measuring, whose results, as we have seen, tend to bequite noisy.
This study was conducted under a different design; an apparent `score' will not bemeasured within a set time limit, but rather the time taken for a user to complete atask is measured. These tasks not only require use of navigation, but are
representative of the various use cases for navigation, and are the same tasks used for theparticularreference in chapter 2.
I have shown how the proposed visualisations, and in particular the SOM4 and BLMS,
represent the music visually. I have further provided evidence that users are able to benefit
in terms of their navigation from having a visualisation.
I therefore submit the following proposals:" To be a useful navigation aid, the visualisation need not provide absolute information
regarding the content but rather relative information.
9A single-level visual segmentation of the track will not aid so much as one focusing
instead on relative-distance information.
"Asuperior navigation aid will reduce the amount of time to complete a task byincreasing the accuracy of individual navigation seek operations, and reducing theamount required.

As such, I predict that computer literate people will be able to complete a set of highlevel music information retrieval tasks faster when given an audio-derived visualisation as5.4. GENERALTASK TESTS

143

a navigation aid than those not. This assumes the users are given a short self-training
period, and that similar accuracy requirements are placed upon the answers given.

5.4.1

Method50 participants

are taken and split them into six groups of eight people and one pair.
Each participant is given a set of instructions and left alone. This document covers basic
disclaimers (withdrawal, anonymity) and notes that the participants are being timed, butthat it is the system being tested, not them. It relates the usage of the audio player
application, in particular that of the timeline bar.
When ready, each person is allowed five minutes of training time to familiarise themselves with the navigation system on a track unrelated to the tasks. This is timed automatically by the system; they are left on their own throughout the session. They are given
a ten second warning before the first question is given. After completing the tasks they
fill out a questionnaire concerning themselves and their thoughts on the software before
leaving.
Each group of eight people is associated with one of six conditions; the pair is associatedwith a seventh condition. Each condition generally relates to a specific visualisation given
as a navigation aid. Aside from this unique condition, each participant is subjected toexactly the same script, detailed below. Throughout the trial, neither the participant
nor the supervisor (the question script) know of which condition they are under and theexperimenter is not present, thereby giving, in this context, a triple-blindvisualisations associated with the conditions are given in table 5.6:
Name

Abbrev.

Sequential; no random access

(Seq)

Blank

Blind

Bandwise Loudness Magnitude Smoothed

BLMS

4x4 SOM Chromatic Plane Ilajectory

SOM4

Loudness Waveform

LMS'

Sandler/Levey Segmentation

Seg

As BLMS but for a different track

Bad

Condition

experiment. The

Table 5.6: The seven conditions in the user study.
A few conditions need a little more explanation; `(Seq)' refers to a baseline made whereusers were not only given no visualisation, but did not use the random-access navigation
facilities of the software. They listened to the music without seeking, until they decided
upon the answer for the task and then moved on to the next. This, therefore, represents asensible lower-bound for the average duration of tasks. Only two people were subjected to144

CHAPTER

5. EVALUATION

OF NAVIGATION

AIDS

this condition due to the lack of variance in the possible approaches that could be taken.
Blank or `blind' is simply where no visualisation is given, as with the other studies.
Loudness Waveform `LMS`' is where an untextured
silhouette of the loudness is depicted.
This was used as a good benchmark to measure the performance of our methods. I expect
users to perform well with this method generally, since it is the traditional and widelyused form of representing audio, and in a fairly acute test such as this study, I expect themost obvious representation to fair the best.
Sandler/Levy Segmentation or `Seg', is a visualisation derived from the segmentation
method recently reported by Levy et al. (2006). It can be considered a state-of-theart method of performing a high-level musical feature-based segmentation. The basic
segmentation information comes in the form of contiguous chunks, with start/end times
and an arbitrary index according to its `type'. With this data, a visualisation is constructedin the same manner as is exemplified in their publication, distributing

hues to `types' evenly

preference to their content.
Finally, the `Bad' visualisation is one which misinforms and misleads the user deliberately. An unrelated track's BLMS visualisation is provided. For the initial minutes ofand without

training, users received a valid BLMS visualisation.
The tasks take place at a normal computer terminal.

Questions appear at a console,and users must navigate through the track (played automatically) to determine the answer.
The entire navigation trace, including the duration between and length of seeks is recorded
silently. Once an answer has been entered, the duration taken is recorded silently and thenext question is given. Answers are not checked for accuracy at this stage.
5.4.2

Results

The results are inspected in three sections; firstly, the mean time taken which is theprimary objective, quantitative metric for evaluating the performance of the visualisations.
Secondly, the seeking behaviour is analysed and finally the questionnaire is discussed.
Importantly, before any results are analysed we discarded all incorrect answers. Although the analysis could be conducted on the general behaviour, I consider using only thedata for correct answers far more telling of `proper' usage. As an aside, I informally analysed the incorrect answers and found no significant differences between that corpus andthis; even the differences in proportion of correct to incorrect were statistically insignificant
between conditions.
Time

Taken

Having collated the durations taken for each of the 26 tasks by each of our 50 participants,
the distributions of time taken for each of the conditions is depicted (see figure 5.8(a)).

5.4. GENERAL145

TASK TESTS

a three-way analysis of variance is conductedto check for significant differences in the means; this is given in table 5.7.

Seeing the relatively well-formed distributions,P-value
Algorithm

4.309 x 10-12

Task

2.2 x 10-16

Person1.065 x 10-15

Table 5.7: ANOVA of task based study's collated durations.
Not only the algorithm is highly significant, but both of the other dimensions of the data
and task). This suggests that it may be difficult to make all inferences fromour experiment. Nevertheless, a set of visualisations for the data may now be constructed,(participant

including a conditionwise matrix of comparison with the Tukey HSD. Figure 5.8(b) showsthese visualisations.
The results show a statistically significant improvement (i. e. reduction) in time taken toof the loudness waveform and the SOM4
Indeed the SOM4 clearly outperforms all visualisations. The only one thatcomplete a task, between the direct-visualisation
visualisation.
is not statisticallysignificant is BLMS, although the suggestion is there nonetheless. The
results are further suggestive of the BLMS visualisation being better than the loudness
level.
The
95%
it
is
figurethebut,
5.8(c)
confidence
not
quite
at
shows,
aswaveform,
loudness waveform outperforms the blind at 95% confidence levels, though the margin is
17%.
in
time
of
aroundreduction
relatively small with a mean
One may see from figure 5.8(d) that is takes on average around 40% longer to get
SOM4,
LMS*
than
the
the
and an extra 65% with noconditiona correct answer withvisualisation at all.
Seek Behaviour
With the seek behaviour, two metrics are used; the number of individual seeks (i. e. clicks
distance
bar)
determinethetravelled
to
the
and
mean
a
correct
answer,navigation
usedon
in a seek (i. e. the difference between the playback positions before and after the seek).
The theory is that a more expressive visualisation (i. e. one that conveys the information
it contains better), should effect an increase in the average length of seeks as users learn
to utilise the graphic to direct larger-scale navigation on faith. A generally more accurate
visualisation (i. e. one that corresponds well to the music) should effect a decrease in total
desiredinthe music without blindly
the
to
they
point
as
navigate
number of seeks required,skipping through the track as we saw in chapter 2.
As with the durations after plotting the graph, a three-way ANOVA is used to checkfor relative amounts of variance in the results, and find the means are very significant

146

CHAPTER

600

5. EVALUATION

0

0

Bad

500
400

0

e

A

A

A,

Blind

v

BLMS

y

v

LMS'

v

v

n

Sag

v

v

0

v

O

8
200

p

D

Q

v

-

o

.

-

A

-

A

A A A A A

SOM4

V

n
aamv
(n

3J-

Co

(a) Box-and-whisker

a

(SeQ)
100

V

va

0

A

A

0

300

AIDS

OF NAVIGATION

v

v

V V

U)

b,

mge

A

-y

plot of all durations.

(b) 'I`ukey

HSD pairwisemean duration

diff.

matrix.

S

LMS"-SOIA4

SOM4
LMs",
S0wsor44
OlrdGOM4lS,-SOM4
LMS"_BLMS((SeqySOM4LMS", BLMS

SOWOLMS

B:,dgMS
B. S(Seq,s(Soqyuis"
BS-9(S-Q)-S-9
eW-B

iswvs.wd

B.dSold(Seq)-BGM(S. Q)/afim(SgWBd(soq)-Bad

01234
01234

0
20
40
60
80
Differencesin meanlevelsof condition(c) Mean difference
Figureindeed.

The results

95% confidence
5.8:

The

are plotted(d) Mean factor 95% confidence intervals.

intervals.

collated

Factorsin meanlevelsof condition

durationresultsof thetask-based

study.

in figures 5.9 and 5.10.

P-value
Seeks

Seek Lengths

2.612 x 10-15

2.779 x 10-16

Task

2.2 x 10-16

2.2 x 10-16

Person5.077 x 10-06

6.210 x 10-09

Algorithm

we can be 95% certain thatLMS* requires, on average, more than twice as many seek operations as the SOM4 forfewer seeks fordetermining
correct answers. The SOM4 generally requires significantly
Using the Tukey

determining

correct

HSD post-hoc

test, results

answers thanany otherindicatethatof the tested

navigation

aids, includingthe5.4. GENERAL147

TASK TESTS

0

100

Bad

A".
AI.

Blind

80

O

BLMS

60

00

OO

40

20

.AAA

(Seg)

vvvvv

SOM4

vvvvv0

y. -4

y
a

co

(a) Box-and-whisker

vv

Qa

rn

mm

J

b,

Cl)

M3

CCi)

xge

02

vii

(b) Pairwise mean seek count diff. matrix.

plot of all seek counts.

3LMSISOU4

SOM4-(Seq)&LMS-(Sea)BI d-(Seq)Bad-(Sea)
S. 9-(Saa)LMS'-(So

A"AA

Seg

8o

0

vvvvAA

LMS'

0Oc

AA
AA

84,
Ba 4
S6LMS".
ýLMS)&LMS-SOM4

Blw-Saa
8 d-saw
Seg-SOMALMS'-SOM4
Blind-&MS

LMS'BLMS

Bad-BLMS
Se9-9LMS
LMS'-BIMS
Seg-8Ws.
w
Qlm:
LMS*M., d

LMS'-Bfrid

seoSadBad-ei.

a

Seg-a

d

LMS'-8dLMS"/Sad

LMS'-SegLMS (S g

o5

2.0
2.5
0.5
1.0
1.5
Factors in mean levels of condition

0.0

15

10

Differences in mean levels of condition(c) Mean difference 95% confidence intervals

3.0(d) Mean factor 95% confidence intervals.

Figure 5.9: The collated seek count results of the task-based study.
BLMS, which itself requires fewer seeks than the segmentation-based visualisation.
In terms of the lengths of seek operation, a distribution
be seen. nirther
investigationshows that the apparently

very few (< 5%) of the total

outsidethe plot in fact constituteconsiderthe distributions

ignoredhere, since there are no seek operations

increasing

to be approximately

aid has a pronounced

the SOM4 navigation

with an apparently long tail canlarge number of samples fallingnormal.

number,The `(Seq)'

and I therefore

navigation

One may see quite clearly that
effect on the mean length of seeks; significantly
to analyse.

them over all others.

From these results, one may say with 95% confidence that for determining
answer, the SOM4 navigation aid leads users to seek around 85 f 40% further
no visualsystem isnavigation

visualisation.

aid, and around

Perhapsmore interesting

75 f 40% than with

only the traditional

is how both the `bad' BLMS

navigationa correct
than withwaveform
aid and

148

CHAPTER

5. EVALUATION

OF NAVIGATIONAIDS

the legitimate BLMS aid resulted in significantly larger jumps than having no aid at all.
Indeed, according to the confidence intervals they have near identical means.
600

o°g
°

500

A

Bad
Blind

400

v

V

A

BLMS

300

100

n

A

-

o

7

v

A

111111
LMS'

200

A

v

Seg

v

o

v

SOM4

A

A

A

7

A

A

gýýýý4em(a) Box-and-whisker

(0

ö

m

plot of all seek lengths.

(b) Pairwise

LMS'-Blind

LMS'/BIind

Sog-Bind

Se8'BIYM

Sod-S dWd/Blold

BLUS-Blind

BLMSHIhd

mean seek count diff. matrix.

SOMVBIiq
Seg21AS'
Bad-LMS"BaNLMS'

BLMs-LMS"

BLMS&LMS'

SOMW-LMS"

SOM4/U4S'

Bed-S.

9

BLMS-S. g
SOM4-Seq
BUMS-Bad

BIMSBad

SOM1-Bea

SOMMBed

SOM4-BlMSSOMVBLMS

05
10
15
20
Differences in mean levels of condition-5(c) Mean difference

95% confidence

25intervals.

0.0

2.0
1.0
1.5
0.5
Factors in mean levels of condition(d) Mean factor 95% confidence intervals.

Figure 5.10: The collated seek length results of the task-based study.

Questionnaire
The questionnaire
found in appendix

the session was filled out by the participant
alone. It can be
D; all questions were multiple choice, most had five ordered responses.

following

The answers given from the questionnaire
were collated in order to determine any statistically significant findings. This was done for two reasons; for the objective questions about
the participants'
i. i. &4,

background,a precondition

I want to attempt

to the previous

4Independent & Identically Distributed

findings

to refute the groups being approximately
being

significant.

Secondly,for opinion

5.4. GENERALTASK TESTSQuestion

149

Mean

MedianP-valuelisten3.6

40.4405musician3.0

30.9812computersoftware

4.7

0.10161.1

51

usefultraining

3.6

40.0018291.5

10.3101

N/A

3

0.73331.2

10.4245representativeknow

0.5053

Table 5.8: The averages and ANOVA probabilities of non-equal means over each questionin the questionnaire.
questions, this is done to check for any statistically significant trends in perceptions.
An analysis of variance test was conducted on each of the sets of the questions as a
response to the condition as a predictor; table 5.4.2 shows the P-values. The question
pertaining to usefulness of navigation aid has an exceptionally low chance of a consistentmean over each condition. Aside from this, none qualify as having a particularly significant
chance of differing means, and certainly not enough to refute the assumption of approximate
i. i. d..
The averages are largely in agreement over each question. People typically had heard
at least one of the tracks before, found the five minutes of training time either `too much'or `about right', and found the tasks generally representative of how they use navigation
facilities; we plot the latter in figure 5.11, `Never navigate' was for people who said theyeither did not listen to audio, or at least never needed to navigate in audio. `Not these' wasfor people who did not consider the tasks given as being representative of their navigationneeds. This accounted for one in three participants.
tasks representative of their navigation needs.
to
(V

-

The rest, around 50%, considered the

These

Q
LL.

O

IN

O

Figure 5.11: The histogram of opinions as to the degree of representation of the tasks.

150

CHAPTER

Finally,the `useful'for difference

capabilities

whatsoever

to a less significant

is investigated;

question

OF NAVIGATION

how people perceived

namelyAIDS

the helpful-aid. The Tukey HSD significant difference matrix and 95% confidence
are given in figure 5.12. On the whole people that had no navigation

ness of the navigation
ratios5. EVALUATION

the `aid' less than useful.

considered

degree the BLMS

People using the SOM4 andand Seg, typicallyconsidered them more useful than
rated
people using the SOM4 visualisation

people using other visual aids. In particular,
it on average higher (at 95% levels of confidence)

those using Blindthanand the LMS*

visuals.

LMS'-(Seq)
Blirq-(Seq)Bad-(Seq)
Sep-(Seg)
BLMS-(Seg)

A

Bad
Blind

v

A

v

p

v

p

SOM4-(Seq)Lp

Blind-LMS'
Bad-LMS'D

Seg-LMS'
BLMS-LMS'

BLMS

SOM4-LMS'

LMS'

v

v

Seg

D

v

(Seq)

o

v p

SOM4

0 A

V

L
Q

0

v p

®

m

Bad-Blind

Y
v

V

A

Sep-Blind

BLMS-Blind
SOM4-81ind
Sep-BadBLMS-Bad
SOM4-Bad
BLMS-Seg
SOM4-Seg
SOM4-BLMS-

2to(a) Pairwisemean useful diff.

012345-1
Differences in mean levels of condition(b) Mean difference

matrix.

95% confidence intervals.

Figure 5.12: The collated `useful' question on the questionnaire for the task-based study.

5.5

Conclusions

The initialassumption

the music information
retrieval tasks are, on the whole, repObjective boundarytasks carried out by people is supported.

that

resentative of navigation
points do exist in music that can be agreed upon by many people.
The(null)

hypothesis

the traditionalthatwaveform

methodof visualising

audio

is

trajectory
method of visualisation
as at least as good as a SOM-based chromatic-plane
for conducting
tasks is refuted, and I instead advance the theory that the SOM4-based
visualisation

will substantially

aid the casual navigationof musicalaudio.

The current

worse than evenat all is significantly
standard of supplying no visualisation
for certain tasks. There is evidence leading us to refute the theory thatbasic visualisations
methodof dimensionality

the SOM methods

for projection.

the PCA

reduction

for chromatic

plane projection

is as good as5.5. CONCLUSIONS

151

A simple visualisation is a reasonable augmentation for a popular music player inthat the learning curve is minimal with my proposed methods, that it is both objectively
measurable to be helpful and generally perceived as being so too. A visualisation is helpful
for navigation because it provides enough of a cue to direct individual seeks accurately tothe desired point in the music, thus decreasing the number of seeks required and increasing
the mean seek distance.

152

CHAPTER 5. EVALUATION OF NAVIGATION AIDS

Chapter6

Conclusions"The whole problem can be stated quite simply by asking, `Is there a meaning
to music? ' My answer would be, `Yes. ' And `Can you state in so many words
what the meaning is? ' My answer would be, `No. ' "-Aaron6.1

Copland (1900-1990)

Discussion

In concluding a work such as this, it is best to review the original aim. At the beginning,
I wished to learn more about navigation aids, and in particular how useful they might bein the context of popular music playback software. There was reason to believe that adepiction of the waveform might be useful through its ubiquity among audio applications.
However, I sought to explore further, to determine if other methods might be better suitedin the context.
To determine if, scientifically, there was any truth in these thoughts, I first conducted
initial studies concerning navigation in popular music playback software. Largely becauseof this, I hypothesised that content-based visual navigation aids based around a projection
from audio to colour would prove helpful for common music playback tasks due to providing
a visual analogy, allowing better directed random-access seek operations. I went further,hypothesising that even more useful would be a non-linear psychoacoustic transformation,
whose mapping of colour to sound was arbitrary, but which rather attempted to preserveI
betweenthethought that this would
used.
shadesmaximisingcolours,while
relationships
better express the variety of sound, the inter-relations and the constancy of individual
portions.
After comparing my proposed visualisations over a range of music, and conducting
I
have
body
100
involvingaparticipants,of evidence tooverseveral user evaluations
conclude the following:
153

154

CHAPTER6. CONCLUSIONS" For my chosen context, the core method of augmenting the navigation system with acontent-based visualisation is a success. It does have a significant effect on how thenavigation facilities are used over a multitude of users and on a multitude of tasks.
Unsurprisingly, this changes dependent upon what algorithm is used to generate theimage.
" With the notable exception of classical music, I found basic spectral-surface-based
visualisations to be adequate for the visualisation of most important aspects. I did
not test many other more `musically-inspired' types of feature extraction, but it may
be reasonable to expect that a combined approach may yield superior results. The
two non-spectral-surface-based feature extractions I proposed proved themselves lesseffective than more basic methods.
" Despite mapping an entirely arbitrary colour to any particular moment of music
diindividually,I
foundthroughthewhen consideredchromaticity-plane projection
mensionality reduction techniques to be, on the whole, at least as good as any othertechnique. In particular, this turned out to be such an unimportant property of visualisation that it allowed the 4x4 SOM-based aid to perform statistically significantly
better than almost all others. This means that the same technique could be usedwith a different chroma-plane to provide a visual that better fits the colour scheme ofa particular player with no loss in performance (assuming the plane had a similarly
good evenness of perceptual colour distribution).
hypothesis
SOM,
best
4x4
The
the
to
originalmymethod,agree
withappeared"The
improvement
throughevidence showsmeans.
ofnavigation
visualconcerningthat a better performance in terms of time taken to complete a task correctly arose
better directed-random-accessfrom fewer-but
seek operations.
To summarise, the main contributionsof this work are:

The proposal and implementation of a general signal-processing metamodel which
can be used to model a number of structures difficult to express in other graphicalmetamodels of computation.
" Empirical data concerning how people navigate around tracks in order to complete
appropriate tasks along with an analysis of the data.
"A

comprehensive review of literature

concerning the content-based visualisation ofmusical audio.

" Several distinct novel methods for generating visualisations of music.

6.2. FUTURE DIRECTIONS

155" The implementation of these methods, together with a comparison and discussion of
their output over several styles of music.
data from user studies conducted to determine the performance of thevisuals as navigation aids, together with a thorough statistical analysis of the data.

" Empirical

6.2

FutureDirections

The present work, although making some important contributions, has only scratched thesurface of this interesting topic. I will now discuss the future directions in which this work
could be taken.
Alternative

Visualisation

Methods

The feature set used for the SOM-based visualisations, while proving its mettle in thisstudy, can almost certainly be improved upon. The low-level features used model little
other than the timbre, and therefore do not model time-varying changes in music. A rangeof musically significant features, e.g. rhythm, melody, key and harmony, could prove to bea useful addition to the feature set.
I have made few efforts to analyse the effect of combining the waveform visualisation
with the proposed colour-projection visualisations. Further work is necessary to betterverify my tentative hypothesis that the addition of the loudness silhouette is not usefulafter the brief period of learning has finished.
A second visualisation route, based around segmentation technology, could attain avisualisation similar to the 4x4 SOM by hierarchical segmentation. Tracks could be segmented as per the literature, but then each segment could be further `subsegmented'.
Colour could be based upon the model feature represented by the segment (unlike the
SOM method where colours are chosen arbitrarily).

But the colours used in subsegments

would be a blend of mostly their parent segment and the other segments to which thesubsegment model leans (if it is not exactly equal to the parent segment, that is).
In so far as the bandwise mechanism (for introducing colour) was implemented, ourstudy conducted only a simple 3-way fair division of the critical bands. Informal experimentation suggests that uneven division could significantly improve the fidelity of theresultant visualisation. Two points through the critical band spectrum could, for example,
be chosen such that the three subspectra minimised their cross variance.
Alternative

Navigation

Aids

In the present work, a static image is given on the navigation bar. Another approach
by
bar
This
the
be
determine
image
the
the
current
point
toplaying.
navigation
of
would

156

CHAPTER 6. CONCLUSIONS

might be a single corresponding row of the self-similarity matrix, for instance.
One further avenue of investigation would be in the translation of abstract pseudocontinuous linear audio data into a discretely annotated format similar, perhaps, to that
presented by Couprie (2004). The translation represents a far easier problem than thatof generalised music transcription, since the representations are similarly low-level andimprecise, dealing with such concepts as speed, brightness and dynamics.
Miscellaneous
One of the most exciting aspects (indirectly) opened up in this project (as far as I'm concerned) is the prospect of a `unified audio visualisation architecture'. Such an architecture
would help develop visualisation components that are able to actually deliver what they
(ourimThroughinmetamodel
context
a real-timepromise.
either a pre-analysis stage or
plementation supports both), the musical audio could have specific features extracted andcorrelated to the rest of the track, in order to deliver a useful and consistent visualisation.
Because the visualisation can then be programmed separately to the analysis framework,
the visualisation author could concentrate on the task at hand, and the feature extraction author would be able to improve the visualisation, precision, and accuracy, without
needing their co-operation.
The information could be presented to the visualisation module by way of an opaqueinterface, and the format could be in either a continuous form (as I have already demonstrated), or a higher-level, discrete form as discussed in the preceding section.
The annotation information could be in the digital music file itself, as technologies
devices
both
in
This
track.
toportableallow
metadataawould
encapsulatealready exist(such as Apple's !Pod) to utilise the information accordingly and, with proper standardis"fingerSuch
browsing"jist"
inbetotracksa
musicstores.
music
usedwhilstation, could
determinetodescribe
topurchasermusic adequately enoughallow a potential
print" maytheir interest in a record.

Appendix

A

Tools Used
Document

Preparation" LATEX

Typesetting software.

" Bibtex

A bibliography administration" KBibtex

and translation tool.

A bibliography collection manager by Thomas Fischer.

" Kile A Latex document editing environment.

" Winefish A Latex document editing environment.
" OpenOffice.
" GNUplotorg An office suite featuring figure-authoring

software.

A data plotting tool.

" GNU RA statistical analysis and visualisation language.
SoftwareCreation" KDevelop

and Analysis

A multi-language integrated development environment.

" Qt A cross-platform GUI toolkit by Trolltech of Norway.
A cross-platform cross-language compiler.

" GNU

Compiler

Collection" GNU

DebuggerA run-time debugger.

" Perl The Practical Extraction" SLOCcountand Report Language interpreter.

A source code analysis tool by David A. Wheeler.

157

APPENDIX A. TOOLS USED

158

Experimentation"K

Desktop" amaroK

Environment

The Unix desktop environment.

The Unix music player.

" PostgreSQL

A database management system.

All software used for this project is Free software, as defined by the Free Software
Foundation.

Appendix

B

Navigation

Tasks
When is theme repeated?
(b) 2:26-2: 36(a) 1:24-1: 34

O
O
M

O
cm
N

O
O
N
0
4)
N0
cC

a
aý
E

0
U)

0
ýNMäýýN

0: 00

(ý

dNýNM
ýt

2: 00

1: 00

FigureýN

B. 1: Task 7: (a) is the original

159

theme,

ý(1

1"f

3: 00

(h) is the repeat.

16(1APPENDIX

B. NAVIGATIONTASKSStart of repetition of melody?

a

C
`13
CL
E
H
c

c

Figure13.2: '['ask 8: (a) is the repetition
When is repetition,of the melody

not quietera) 1: 04-1: 14

slowerat the beginning.

variation?
(c) 2: 29-2: 39(b) 1: 18-1: 32

8
N
ZU
ZZ
O
N

0

0
2N2Js0N9d2222O2

0: 00
Figure1:00

13.;3: Task 9: (a) is the original

repetition.

theme,

2: 00(b) is a quieter/slower

variation,(c) is the161

When is second onset of instrument?
(a) 0: 00-0: 20(b) 1:33

O

O
N

O
O
N
N

O

CL
Q)
EO

0

N
N
CO

0
a

O
N

O
0000000ýQ

N

ONN

t7

0:00

1:00

Figure B. 4: Task 10: (a) is original period of playing, (b) is the second onset.
When is 6-second

break in instrumentation?
(a) 1: 10-1: 16

0
0

0
Co

o

:o
a>
N

(0

y
tu

a>
E

o

0
N

0
ýN
f+l

dNýN

f+1

1:00

0: 00
Figure

ONýN

t+ý

2: 00

B. 5: Task 11: (a) is the 6-second break.

i

162

APPENDIX%A/h-..

;.. i 4-1-4.. --^*

h......

:....

ý...........

B. NAVIGATIONe h. s......

4.4

nV)

(0

U)
a`.
13

E

FigureB. 6: Task 12: (a) is the onset of the instrument,

Wh.

-

ie

firef

rennetni(b) marks

1: 10.

nrn-7

Co
N

IL
d
E
F=

0: 00

1:00
Figure

2: 00

3: 00

4: 00

B. 7: Task 13: (a) is the onset of the organ.

5: 00

TASKS

163

When is first onset of drums?

C
4

C
U

C
(1)
Co
N

cu
dc

0

C

C

Figure B. 8: Task 14: (a) is the onset of the drums.

What is first line of second vocalist?
(a) 2: 59-3: 02
0
co

0
N

N

NO
07

CL

w
E
F

0
CV

0
t9

ýA

1:00
Figure

N

7

N

Ö

f+l

ýA

QN

2: 00

B. 9: Task 17: (a) is the first line of the second vocalist.

3: 00

164

APPENDIX

B. NAVIGATIONTASKSWhen is onset of vocalist after other vocals?
0
N

O
O

O

00

N

V
Nt/ý

O
CO

l0
a
E
H0

O
N

0

2: 00
Figure

3: 00

B. 10: Task 18: (a) is the period

4: 00

of the second vocalist,

5:00(b) is the onset of the first

vocalist.

What are first words sang?

°

äi
N
N
a-

a

0)

°

EN
F=

°

°

0:00
Figure B. 11: Task 20: (a) is the period over which the first words are sang.

165

How many times is verse-chorus(a) 0:35-1: 06
MXX

O

played?
(c) 2:46-3: 17(b) 1:23-1: 54

O
N

O
O
Vl
O
N
Cl)
N
as
dO
co

(1)
E
I-

O

0
N

XN

0
tV

MdN

.-N0ON-"tJdN

.-

2: 00

1:00

05

QN

3: 00

B. 12: Task 21: (a), (b) and (c) all mark the verses. There

Figure(V

4: 00

is a bridge

between(b)and (c).

When is start of guitar solo?

C
Lf

U)
Ur
CL
CL
E
F=
0
U,

0

1:00

0: 00
Figure

2: 00

3: 00

B. 13: Task 22: (a) marks the start of the guitar

4: 00
solo.

166

APPENDIX

B. NAVIGATIONTASKSWhen is end of guitar solo?
u
c

c
c

L
N

N
tu

IL
aý

{

H

0

Figure13.14: Task 23: (a) marks the end of the guitar

solo.

When is onset of voice after quiet portion?
a) 2: 35-2: 45(b) 3: 10

O
N

8
N

Op

NO
NO

t

E_

o

ý
15E

0
N

0

ý/

N

0: 00
Figure

RZ

l+f

dNN

:ý114 /ý

:
t0

1:00

N

O,

2: 00

13.15: Task 24: (it) marks the quiet

fý

N

O,

V
th

3: 00
portion,

N'

4: 00

N

VO280Nch5: 00(b) marks the onset of the voice.

167

When is onset of instrument after break?

0
Co

0
Co

m
a
E
I0
N

0

Figurement.

B. 16: Task 25: (a) marks the end of the break,(b) marks the onset of the instru-

168

APPENDIX B. NAVIGATION TASKSAppendix

ReferenceC

Questionnaire

Please circle the tasks you are typically utilise a music playback application's navigation
facilities for.
1. Finding the onset of vocals.
2. Finding the onset of an instrument.
3. Finding the starts and ends of verses, choruses and bridges in songs.
4. Finding the ending of the song.
5. Finding repetitions and variations of the music.
6. Finding breaks and pauses in the music.
7. Determining the content of vocals (e.g. wanting to listen for a certain lyric).
8. Determining the overall structure of the music (e.g. whether it has a verse/chorus
structure).
Please indicate in the space below any other uses you would typically have for a navigation facility:

169

170

APPENDIX C. REFERENCE QUESTIONNAIRE

Appendix

Main

D

Task Trial

Questionnaire

1. How often do you listen to music?
(a) I rarely listen to music.
(b) I usually listen to music for upto 3 hours a week.
(c) I usually listen to music for upto an hour a day.
(d) I usually listen to music for upto 3 hours a day.
(e) I usually listen to music for at least 3 hours a day.
2. What is the extent of your experience as a musician?
(a) I have never voluntarily

written music or played an instrument.

(b) I have not played an instrument or written music for over 10 years.
(c) I have not played an instrument or written music for over 3 years.
(d) I have played an instrument or written music in the past 3 years.
(e) I regularly play an instrument.
3. How much time do you spend on a computer?
(a) I rarely use a computer.
(b) I usually use a computer for upto 3 hours a week.
(c) I usually use a computer for upto an hour a day.
(d) I usually use a computer for upto 3 hours a day.
(e) I usually use a computer for at least 3 hours a day.
4. How familiar are you with audio playback software?
(a) I am familiar with WinAmp, iTlmes, Windows Media Player or some otherapplication for playing music on a computer.

171

APPENDIX

172

D. MAINTASK TRIALQUESTIONNAIRE(b) I have rarely used applications for playback of music on a computer, but amfamiliar with the idea.
(c) I am unfamiliar with using a computer to playback music.
5. Of each of the media in the tasks, please indicate which you were already familiar(summed):6. How useful did you find the software for helping to answer the tasks given?
(a) Unhelpful; I think I could have done better with my usual means of music/audio
playback.
(b) Not useful; I don't think this helped make the tasks any easier or quicker thanmy usual means of music/audio playback would have.
(c) Potentially

useful; I don't think it helped me this time but given more time tolearn I think it might be helpful.

(d) Fairly helpful; I think some tasks were made substantiallyeasier to complete.

(e) Very helpful; I think most tasks were made substantially easier to complete.
7. How much training time do you think you need to use the this player applicationand navigation features?
(a) I think the six minutes given was overkill. It's quite clear anyway.
(b) The six minutes was about right. I don't think I could have learnt much more
given more training time.
(c) The six minutes was too little.
I think another 10-20 minutes would haveresulted in a substantially better performance.
(d) The six minutes was far too short a time. I think more than 30 minutes solid
training is needed before the system would be effective.
8. How representative do you think the tasks given are of your need for navigation?
(a) I never need to navigate in music or audio.
(b) When I do navigate in music/audio I would not need to do these tasks or tasks
like them.
(c) When I do navigate in music/audio I would need to do these tasks or tasks likethem.

Bibliography
S. A. Abdallah, K. Noland, M. Sandler, M. Casey, and C. Rhodes. Theory and evaluation ofa Bayesian music structure extractor. Proceedings of the Sixth International Conference
on Music Information Retrieval, JD Reiss and GA Wiggins, Eds, pages 420-425,2005.
S. A. Abdallah, M. Sandler, C. Rhodes, and M. Casey. Using duration models to reducefragmentation in audio segmentation. Machine Learning, 65(2): 485-515,2006.
P. Aigrain. Audio Retrieval and Navigation Interfaces. Readings in Multi-media Computing
and Networking, 2001.
P. Aigrain, P. Joly, P. Lepain, and V. Longueville. Representation-based user interfaces forthe audiovisual library of year 2000. Proceedings of SPIE, 2417, Multimedia Computing
and Networking, pages 35-45,1995.
J. F. Alm and J. S. Walker. Time-Frequency Analysis of Musical Instruments. Society for
Industrial and Applied Mathematics Review, 44(3): 457-476,2002.
J. Massaguer, D. Garcia, and I. Mosquera. The CLAM Annotator: A
Cross-platform Audio Descriptors Editing Tool. Proceedings of the 6 thlnternational

X. Amatriain,

Conference on Music Information

Retrieval, London, UK, 2005.

M. Attik, L. Bougrain, and F. Alexandre. Self-organizing Map Initialization.
NOTES IN COMPUTER SCIENCE, 3696:357,2005.

LECTUREJ. J. Aucouturier

and M. Sandler. Using Long-Term Structure to Retrieve Music: Representation and Matching. Proc. International Symposium on Music Information Retrieval, 2001.

M. A. Bartsch and G. H. Wakefield. To catch a chorus: using chroma-based representations
Applications of Signal Processing to Audio and Acoustics, 2001
IEEE Workshop on the, pages 15-18,2001.

for audiothumbnailing.

M. A. Bartsch and G. H. Wakefield. Audio thumbnailing of popular music using chromabased representations. Multimedia, IEEE Transactions on, 7(1): 96-104,2005.
173

BIBLIOGRAPHY

174

M. Biggs. Visualisationand Wittgenstein's

Tractatus.

2002.

S. Blackburn and D. DeRoure. A Tool for Content Based Navigation of Music. In Proceedings of ACM Multimedia. ACM, Bristol, UK, 1998. ISBN 1-58113-036-8.
R. A. W. Bladon and B. Lindblom. Modeling the judgment of vowel quality differences.
The Journal of the Acoustical Society of America, 69: 1414,1981.
A. Brinkman and M. Mesiti. Graphic modeling of musical structure.
Research, 3: 1-42,1991.

Computers in MusicC. J. C. Burges, D. Plastina, J. C. Platt, E. Renshaw, and H. S. Malvar.

Using audio

for duplicate detection and thumbnail generation. Acoustics, Speech, andSignal Processing, 2005. Proceedings. (ICASSP'05). IEEE International Conference on,

fingerprinting
3,2005.

R. Callan. The Essence of Neural Networks. Prentice-Hall,

1999. ISBN 0-13-908732-X.

C. Cannam, C. Landone, M. Sandler, and J. P. Bello. The Sonic Visualiser: A Visualisation Platform For Semantic Descriptiors From Musical Signals. Proceedings of ISMIR,
Victoria,

Canada, 2006.

A. Cemgil and F. Gärgen. Classification of Musical Instrument Sounds using NeuURL
S1U97,1997.
Proceedings
In
Networks.
theciteseer. nj. nec. com/
of
ral
html.
cemgil97classification.
via Structural Analysis. Proceedings of theconference on Multimedia, November, pages 02-08,2003a.

W. Chaff and B. Vercoe. Music Thumbnailingeleventh ACM internationalforindexing and thumbsignalsofmusicalanalysis
Digital Libraries, 2003. Proceedings. 2003 Joint Conference on, pages 27-34,

W. Chaff and B. Vercoe. Structuralnailing.
2003b.

E. Chew and A. Francois. Real-time Music Information Processing. Proceedings of the
31st International Conference for Computers and Industrial Engineering, ICCIE2003,
San Francisco, CA, February, pages 2-4,2003.
P. Clarke.

Folktronica.

BBC

Collective,

Editor's

Review, 2003.

URL

http:

//wwtir. bbc.

co. uk/dna/collective/A1120555.

R. Cogan. New Images of Musical Sound Harvard University Press, 1984.
In
Analysis.
Similarity
Structural
Music
PopularSummarizing
J.
Foote.
Cooper
M.
via
and
ACM.
US,
2003.
California,
Berkeley,364-373,
ACM
Multimedia,
Proceedingsof
pagesBIBLIOGRAPHY

175

P. Couprie. Graphical Representation: An Analyticaland Publication Tool for Electroac-coustic Music. Organised Sound, 9: 109-113,2004.
R. B. Daunenberg. A Brief Survey of Music Representation Issues, Techniques, and Systems. Computer Music Journal, 17(3): 20-30,1992.
M. Dittenbach,R. Neumayer, and A. Rauber. P1aySOM: An Alternative Approach to
Track Selection and Playlist Generation in Large Music Collections. 2003.

A. Dix, J. Finlay, G. D. Abowd, and R. Beale. Human Computer Interaction.
Saddle River, NJ, 2004.

Upper

S. Dixon, W. Goebl, and G. Widmer. The Performance Worm: Real Time Visualisation of Expression based on Langners Tempo-Loudness Animation. Proceedings of theInternational

Computer Music Conference (ICMC 2002), 2002a.

S. Dixon, W. Goebl, and G. Widmer. Real Time Tracking and Visualisation of Musical Expression. Music and Artificial Intelligence: Second International Conference,
ICMAI2002,

pages 58-68,2002b.

S. Dixon, E. Pampalk, and G. Widmer. Classification of Dance Music by Periodicity
Patterns. In Proceedings of the International Conference on Music Information Retrieval(ISMIR '03), Baltimore, Maryland, US, 2003. Johns Hopkins University. URL http://ismir2003.

ismir.

net/papers/Dixon.

PDF.

G. W. Don and J. S. Walker. Music: A Time-Frequency Approach. Journal of Mathematics
and Music, 2006.
M. Dorfier. Gabor Analysis for a Class of Signals called Music. Ph. D. thesis, Universityof Vienna, Institut of Mathematics, August 2002,2002.
J. P. Eckmann,
Europhysics
//dx.

doi.

S. O. Kamphorst,
Letters,

and D. Ruelle. Recurrence plots of dynamical systems.
doi: 10.1209/0295-5075/4/9/004.
5: 973-977,1987.
URL http:

org/10.1209%2F0295-50757,2F4%2F9%2F004.

L. Fausett. Fundamentals of Neural Networks. Prentice-Hall,
J. Foote.

1994. ISBN 0-13-334186-0.

Automatic

Audio Segmentation Using A Measure of Audio Novelty.
In
Proceedings of IEEE International Conference on Multimedia and Expo, volume 1,

pages 452-455. IEEE, 2000a. URL http: //www. fxpal.
footeICME00. pdf.
J. Foote. ARTHUR:

com/people/foote/papers/

Retrieving Orchestral Music by Long-Term Structure.

Symposium on Music Information

Retrieval, 1,2000b.

International

176

BIBLIOGRAPHY

J. Foote. Methods for the Automatic
1999a. URL citeseer.
J. Foote.

Analysis of Music and Audio.

nj. nec. com/foote99methods.

Technical report,

html.

Visualizing

Music and Audio Using Self Similarity.
In Proceedings of ACM
Multimedia, pages 77-80, Orlando, Florida, US, nov 1999b. ACM.

J. Foote and M. Cooper.

Media Segmentation using Self-Similarity Decomposition. In
Proceedings of the SPIE Storage and Retrieval for Multimedia Databases, volume 5021,

pages 167-175, San Jose, California,
people/foote/papers/SPIE02.

US, 2003. SPIE. URL http: //wvw. fxpal.

com/

PDF.

E. G6mez and J. Bonada. Tonality Visualization of Polyphonic Audio. Proceedings of theInternational Computer Music Conference, pages 57-60,2005.
Google. Google web search, 2007. URL http: //www. google. com.
J. Guessford, H. Kaper, and S. Tipei. Loudness Scaling in a Digital Synthesis Library.
Proceedings of the International Computer Music Conference, Miami, Florida, 2004.
S. Hainsworth.

Techniques for the Automated Analysis of Musical Audio. PhD thesis, Ph.
D. thesis, Cambridge University Engineering Department, Cambridge, UK, 2004,2004.

R. Hamming. Digital Filters. Dover Publications,
R. Hiraga. Musical Performance Visualization,-rhiraga/researchhi.

1998.

2006. URL http: //www. bunkyo. ac. jp/

html.

R. Hiraga and N. Matsuda.

Graphical expression of the mood of music. Multimedia andIEEE International Conference on, 3,2004a.
Expo, 2004. ICME'04.2004

R. Hiraga and N. Matsuda. Visualization of music performance as an aid to listener's
comprehension. Proceedings of the working conference on Advanced visual interfaces,
pages 103-106,2004b.
R. Hiraga, R. Mizaki, and I. Fujishiro. Performance visualization: a new challenge tomusic through visualization. Proceedings of the tenth ACM international conference on
Multimedia, pages 239-242,2002a.
R. Hiraga, F. Watanabe, and I. Fujishiro. Music learning through visualization. Web Delivering of Music, 2002. WEDELMUSIC 2002. Proceedings. Second International Conference on, pages 101-108,2002b.

J. Hirschberg and J. Choi. "I just played that a minute ago!:" Designing User Interfacesfor
Audio Navigation. Proceedingsof Content Visualization and Intermedia Representations
CVIR'98,1998.

BIBLIOGRAPHY

177

J. Hirschberg, S. Whittaker,

D. Hindle, F. Pereira, and A. Singhal. Finding information inaudio: A new paradigm for audio browsing/retrieval. Proceedings of the ESCA workshop:
Accessing information in spoken audio, pages 117-122,1999.
B. Holland. A man who sees what other people hear. The New York Times, page C28,19
November 1981.
K. A. S. Immink.

The compact disc story. Journal of the Audio Engineering Society, 46(5): 458-465,1998.

E. Isaacson. Content visualization in a digital music library. Third International
on Information Visualization Interfaces for Retrieval and Analysis (IVIRA)

Workshopat JCDL,

2003.
E. Isaacson. What You See is What You Get: On Visualizing Music. Proceedings of the
Sixth International Conference on Music Information Retrieval, pages 389-395,2005.
1. Jolliffe. Principal Component Analysis. Springer, 2002.
Manifold compositions, music visualization, and scientific sonification in an
immersive virtual-reality environment. 1998 International Computer Music Conference

H. Kaper.

(ICM98), Ann Arbor, MI (US), 1010111998-1010611998,1998.
An analysis of selected computer interchange color spaces.
ACM 7)-ansactions on Graphics (TOG), 11(4): 373-405,1992.

J. Kasson and W. Plouffe.

D. Kimberciteseer.

Acoustic segmentation for audio browsers, 1996.
and L. Wilcox.
html.
ist. psu. edu/article/kimber96acoustic.

URL

D. Kimber, L. Wilcox, F. Chen, and T. Moran. Speaker segmentation for browsing recorded
audio. Conference on Human Factors in Computing Systems, pages 212-213,1995.
P. Knees, T. Pohle, M. Schedl, and G. Widmer.

Automatically

Describing Music on a

Map. Proc. LSAS '06,2006.
M. Kobayashi and C. Schmandt. Dynamic Soundscape: mapping time to space for audio browsing. Proceedings of the SIGCHI conference on Human factors in computing
systems, pages 194-201,1997.
T. Kohonen. The self-organizing map. Proceedings of the IEEE, 78(9): 1464-1480,1990.
T. Kohonen. Fast Evolutionary

Learning with Batch-Type Self-Organizing Maps. Neural

Processing Letters, 9(2): 153-162,1999.
T. Kohonen.
scholarpedia.

Kohonen

Network,

page 7421.

org/article/Kohonen_Network.

Scholarpedia,

2007a.

URI.

http:

//www.

178

BIBLIOGRAPHY

T. Kohonen. The self-organizing map (som). WWW, 2007b. URL http: //www. cis. hut.
fi/projects/somtoolbox/theory/somalgorithm.
shtml.
T. Kohonen. Self-Organised Formation of Topologically Correct Feature Maps. Biological
Cybernetics, 43: 59-69,1982.
P. Kolhoff, J. Preuß, and J. Loviscach. Music Icons: Procedural Glyphs for Audio Files.
Computer Graphics and Image Processing, 2006. SIBGRAPI'06.19th
sium on, pages 289-296,2006.

Brazilian Sympo-

T. Kosonen and A. Eronen. Rhythm metadata enabled intra-track navigation and content
modification in a music player. Proceedings of the 4th international conference on Mobile
and ubiquitous multimedia, 2006.
P. Lepain. SATIE: an interactive software for listening to musical recordings. Proceedingsof the fourth ACM international conference on Multimedia, pages 413-414,1997.
F. Lerdahl and R. Jackendoff. A Generative Theory of Tonal Music. MIT Press, 1983.
of high-level musical structure from audio
data and its application to thumbnail generation. Proc. ICASSP, 2006.

M. Levy, M. Sandler, and M. Casey. Extraction

A. Lillie.

Visualizing

Music.

URL

2007.

http:

//flyingpudding.

com/projects/viz_music/.

B. Logan and S. Chu. Music summarization using key phrases. Acoustics, Speech, andSignal Processing, 2000. ICASSP'00. Proceedings. 2000 IEEE International Conference

on, 2,2000.
B. Loganand A. Salomon.

port, Cambridge

A Content-Based

Research Laboratory,Music

Similarity

2001. URL http:publications/techreports/reports/crl-tr-2001-2.

D. Lubbers.
Exploration
Information

Sonixplorer://cri.

Function.

Technical

research.

compaq. com/

re-

pdf.

and Auralization for Content-based
Proc. of the 6th International Conference on Music

Combining Visualizationof Music Collections.
Retrieval (ISMIR05),

London, UK, 2005.

D. MacAdam. Visual sensitivities to color differences in daylight. J. Opt. Soc. Am, 32(5):247-274,1942.
S. Malinowski. The Music Animation

Machine. 2001. URL http: //www. well. com/user/

smalin/mam. html.
D. Mazzoni and R. Daunenberg. Audacity Sound Editor, 2005. URL audacity.

sf. net.

BIBLIOGRAPHY

R. Middleton.

179

Studying Popular Music. Open University Press, 1990.

F. Morchen, A. Ultsch, M. Nocker, and C. Stamm. Databionic visualization of musiccollections according to perceptual distance. Proc. of the 6th International Conference
on Music Information Retrieval (ISMIR05), London, UK, 2005.
A. Munsell. A Pigment Color System and Notation.

The American Journal of Psychology,

23(2): 236-244,1912.
C. Nakatani, S. Whittaker,

and J. Hirshberg. Now you hear it, now you dont: Empirical
Studies of Audio Browsing Behavior. Proceedings of the Fifth International Conferenceon Spoken Language Processing, (SLP98), 1998.

M. Oja, S. Kaski, and T. Kohonen. Bibliography of self-organizing map (SOM) papers:
1998-2001 addendum. Neural Computing Surveys, 3(1): 1-156,2003.

A. Okabe, B. Boots, and K. Sugihara. Spatial tessellations: conceptsand applications ofVoronoi diagrams. John Wiley, 2000. ISBN 978-0471986355.
fur Softwaretechnik und Interaktive
Systeme der Technischen Universitat Wien, December 2001.

E. Pampaik.

Islands of Music. PhD thesis, Institut

E. Pampalk,

A. Itauber,Music

Archives.

ACM.

URL

and D. Merkt.

In Proceedings

citeseer.

Content-basedof the ACM

Organization

Multimedia,

and VisualizationofJuan les Pins, France, 2002.

nj. nec. com/pampalkO2contentbased.

html.

E. Pampalk, S. Dixon, and G. Widmer. On the evaluation of perceptual similarity measures
for music. Proc Intl Conf on*Digital Audio Effects, 2003.
E. Pampalk, S. Dixon, and G. Widmer. Exploring Music Collections by Browsing Different
Views. Computer Music Journal, 28(2): 49-62,2004.
J. Paulus and A. Klapuri. Music structure analysis by finding repeated parts. Proceedingsof the Ist ACM workshop on Audio and music computing multimedia, pages 59-68,2006.
K. Pearson. On lines and planes of closest fit to systems of points in space. Philosophical
Magazine, 2(6): 559-572,1901.
P. J. Ponce de Leon and J. M. Inesta. Feature-driven recognition of music styles. Ist Iberian
Conference on Pattern Recognition and Image Analysis. Lecture Notes in Computer
Science, 2652:773-781,2003.
P. J. Ponce de Le6n and J. M. Iiiesta. Musical Style Identification
Maps. Proceedings of the Second International
WEDELMUSIC,

2:82-92,2002.

Using Self-Organising

Conference on WEB Delivering of Music

180

BIBLIOGRAPHY

J. Preece, Y. Rogers, H. Sharp, D. Benyon, S. Holland, and T. Carey. Human-Computer
Interaction. Addison Wesley, Wokingham, UK, 1994. ISBN 0-201-62769-8.
C. Raphael. Automatic

Segmentation of Acoustic Musical Signals Using Hidden Markov
Models. In Transactions on Pattern Analysis and Machine Intelligence, volume 31.
IEEE, 1999.

A. Rauber and M. Frühwirth.

AutomaticallyAnalyzing and Organizing Music Archives. In
ECDL '01: Proceedings of the 5th European Conference on Research and Advanced Tech-

nology for Digital Libraries, volume 2163, pages 402-414, London, UK, 2001. SpringerVerlag. ISBN 3-540-42537-3. URL citeseer. nj. nec. com/rauberOlautomatically.
html.
A. Rauber,

E. Pampalk,

Maps to Create

and D. Merkl.

a Hierarchicalings of the Internationaltre Pompideu,

Paris,

rauber02using.

html.

Structuring

ConferenceFrance,Using Psycho-Acousticof Musicon Musicby Sound Similarities.

Information

2002. IRCAM.

URL

Models and Self-Organizing

Retrievalciteseer.

(ISMIRIn Proceed'02), Cen-

nj. nec. com/article/

C. Rhodes, M. Casey, S. A. Abdallah,and M. Sandler. A Markov-chain Monte-Carloapproach to musical audio segmentation. Proc. of IEEE International Conference on
Acoustics, Speech, and Signal Processing, 2006.

H. Ritter.

Self-organizing maps on non-euclidean spaces. Kohonen Maps, pages 97-110,

1999.
D. Roy and C. Schmandt.

NewsComm:a hand-held interface for interactive access to
structured audio. Proceedings of the SIGCHI conference on Human factors in computing
systems: common ground, pages 173-180,1996.

C. Sapp. Harmonic Visualizations of Tonal Music. Proceedings of the International
puter Music Conference, pages 423-430,2001.

Com-

N. Sawhney and A. Murphy. ESPACE 2: an experimental hyperaudio environment.
ference on Human Factors in Computing Systems, pages 105-106,1996.

Con-

B. Scharf. Critical bands. Foundations of Modern Auditory

Theory, 1: 157-202,1970.

C. Schmandt and D. Roy. Using acoustic structure in a hand-held audio playback device.
IBM Systems Journal, 35(3): 453-472,1996.

J. Simpson, E. Weiner, et al. The Oxford English Dictionary. 1989.
K. Sjölander and J. Beskow. WaveSurfer-an open source speech tool.
464-467,2000.

Proc. ICSLP, 4:

BIBLIOGRAPHY

181

D. M. Skapura. Building Neural Networks. ACM Press Books, 1996. ISBN 0-201-53921-7.
S. M. Smith and G. N. Williams.
8262-0.

A Visualizationof Music. IEEE, 1997. ISBN 0-8186-

J. Snydal and M. Hearst. ImproViz: Visual Explorations of Jazz Improvisations.
In
Proceedings of the Conference on Human Factors in Computing Systems (SIGCHI '05),
Portland, Oregon, US, 2005. ACM Press.
R. Spence. Information
S. Sterrett.
Depiction.

Pictures

Addison-Wesley
Harlow, 2001.
visualization.

of Sounds:

2004. URL

h: ttp:SterrettPictures0fSoundsRl.

Wittgensteinon Gramophone//philsci-archive.

pitt.

Records and the Logic of

edu/archive/00002019/01/

pdf.

S. Stevens and J. Volkmann. The Relation of Pitch to Frequency: A Revised Scale. The
American Journal of Psychology, 53(3): 329-353,1940.
E. Terhardt.

Impact of computers on music: an outline.
Neuropsychology of Music, pages 353-369,1982.

Music, Mind, and Brain:

The

The Amarok Team. amaroK Music Player, 2005. URL amarok. kde org.
.
P. Toiviainen. Visualizationof tonal content with self-organizing maps and self-similarity
matrices. Computers in Entertainment (CIE), 3(4): 1-10,2005.

P. Toiviainen and T. Eerola. A method for comparative analysis of folk music based on musical feature. Conference Program, Proceedings & List of Participants, VII International
Symposium on Systematic and Comparative Musicology III International

Conference on

Cognitive Musicology 2001 Jyväskylä, Finland, 2001.
P. Toiviainen and T. Eerola. A computational model of melodic similarity based on multiple representations and self-organizing maps. Proceedings of the 7th International
Conference on Music Perception and Cognition, Sydney, pages 236-239,2002.
G. Tzanetakis and P. R. Cook. Audio Information Retrieval Tools. In Proceedings of theInternational Symposium on Music Information Retrieval (ISMIR). ISMIR, 2000a.
G. Tzanetakis and P. R. Cook. Experiments in Computer-Assisted Annotation of Audio In Proceedings of the International Conference on Auditory Display (ICAD '00).
.
ICAD, 2000b. URL http: //www. icad. org/websiteV2.0/Conferences/ICAD2000/
PDFs/Tzanetakis. pdf/experiments-in-computer-assisted.
pdf.
G. Tzanetakis and P. R. Cook. MARSYAS: A Framework for Audio Analysis. OrganizedSound, 2000c.

182

BIBLIOGRAPHY

G. Tzanetakis

and P. R. Cook.

Annotation.
ing

In

to Audio

http:Proceedingsof IEEE

and Acoustics,

New//soundlab.

G. Tzanetakis,

In Proceedings(ISMIR

citeseer.

cs. princeton.

Audio

Workshop

Paltz,New

Automatic

US,

oct

Conference
Indiana,andof Signal

Process-

1999. IEEE.

URL

waspaa_mfas.

MusicalBloomington,

for Browsing

on Applications
York,of the Internarional'01), pages 205-210,

Segmentation

edu/publications/1999

G. Essl, and P. R. Cook.

dio Signals.
trievalMultifeature

pdf.

Genre Classification
on Musicof Au-

Information

Re-

USA, oct 2001. ISMIR.

URL

html.

nj. nec. com/tzanetakisoiautomatic.

S. Vembu and S. Baumann. A Self-Organizing Map Based Knowledge Discovery for Music
Recommendation Systems. Proc. of the 2nd Internation

Symposium on Computer Music

Modeling and Retrieval (CMMRO4), Esbjerg, Denmark, 2004.
T.

Weyde.

Dynamic

Presented

resentation.
Network,

and

2005.

InteractivetoURL&url=http%3A%2F%2Fwww.

the
http:

Visualisations
Fifth

Open

//wvw.

Workshop

doc\&ei=0

oftheco. uk/url?

google.

interactivemusicnetwork.

OpenWorkshop_2005%2Fpapers%2F12.

Symbolicof MPEGMusicInteractive

RepMusic

sa=t\&ct=res\&cd=4\

org%2Fevents%2FFifth_
xNRtvJL42EOgTi-4CKDg\&usg=

AFrgEzctpTKDy21nw-gYOaNJHk4DUBGHHQ\&sig2=YluV3NUZpfy2IN7iOAfW8w.
T. Weyde and J. Wissmann.

Visualizationof musicalthe First Conference on Interdisciplinary
S. Whittaker,

structurewithmaps. ProceedingsofMusicology. Graz, Austria., 2004.

L. Terveen, and B. Nardi.

Let's stop pushing the envelope and start addressing it: a reference task agenda for HCI. Human-Computer Interaction, 15(2/3):

75-106,2000.
G. Widmer and W. Goebl. Computational Models of Expressive Music Performance: The
State of the Art. Journal of New Music Research, 33(3): 203-216,2004.
L. Wittgenstein.

Tractatus Logico-Philosophicus. Kessinger Publishing, 2004.

G. Wood and S. E. O'Keefe. Quantitative

Comparisons into Content-based Music Recog-

nition with the Self-Organising Map. In Proceedings of the Internarional Conference
on Music Information Retrieval (ISMIR '03), Baltimore, Maryland, US, 2003. Johns
Hopkins University.
G. Wood and S. E. O'Keefe. A Case Study of Distributed

Musical Audio Analysis Using theGeddei Processing Framework. In Proceedings of the International
Information
Institute,ismir.

Retrieval (ISMIR

Conference on Music'04), pages 44-47, Barcelona, Spain, 2004. Audiovisual

Pompeu Fabra University.

ISBN 84-8804244-2. URL http: //ismir2004.

net/proceedings/p009-page-44-paperi58.

pdf.

BIBLIOGRAPHY

183

G. Wood and S. E. O'Keefe. On Techniques for Content-Based Visual Annotation

to AidIntra-Track Music Navigation. In Proceedings of the International Conference on Music
Information Retrieval (ISMIR '05), pages 58-65, London, UK, 2005. Queen Mary, University of London. ISBN 0-9551179-0-9. URL exscalibar.
ismir-2005. pdf.

sourceforge.

net/files/

R. Wright.

Art and Science in Chaos: Contesting Readings of Scientific Visualization.
Proceedings of the 5th International Symposium on Electronic Art, 1994,1994.

W. D. Wright. A re-determination of the trichromatic coefficients of the spectral colours.
Transactions of the Optical Society, 30(4): 141-164,1929.
B. Yandell. Practical Data Analysis for Designed Experiments.
1997.

Chapman & Hall/CRC,