Proceedings of ISon 2016, 5th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 2016

HEART ALERT: ECG SONIFICATION FOR SUPPORTING THE DETECTION
AND DIAGNOSIS OF ST SEGMENT DEVIATIONSAndrea Lorena Aldana Blanco

Steffen Grautoff

Thomas Hermann

Ambient Intelligence Group
CITEC, Bielefeld University
Bielefeld, Germany

Emergency Department
Klinikum Herford
Herford, Germany

Ambient Intelligence Group
CITEC, Bielefeld University
Bielefeld, Germany

aaldanablanco@techfak.unibielefeld.de

steffen.grautoff@klinikumherford.de

thermann@techfak.unibielefeld.de

ABSTRACTThis paper presents two novel sonification designs forElectrocardiography (ECG) data: (a) Water Ambiencesoundscapes aim at turning heart activity into an ambience
which exhibits salient patterns as specific ECG properties
deviate from a normal heartbeat, (b) Timbre Morphingsonification aims at supporting analysts to quickly assess if anabnormality in terms of the frequency, rhythm or amplitude in
the signal occurs. Both methods are embedded into aninteractive setting where the users can upload a dataset andinteractively adjust sonification parameters, for instance insearch of settings that optimize the contrast between a baseline(regular) and abnormal (ST deviated) case, based on prerecorded real ECG data sets. In result, we qualitatively analyze
how a small group of users interacts with the system and what
their overview regarding the proposed methods is. Also, we
conduct a study with eight participants in which they are asked
to classify a set of sonifications according to two categories;
healthy or unhealthy. The study results suggest that the
proposed sonification designs allow users to correctly classify
the datasets without having prior knowledge about ECG signals.

1.

INTRODUCTION

In recent years, the interest for using sonification as a method
for exploring ECG signal features has increased. Researchers
had exposed the advantages of using heart rate sonification to
support medical diagnosis [1], and they have proposed sounddesigns to guide attention to the specific ECG segments [2].
Additionally, part of the research efforts focused on improving
the detection of the ECG components of a signal [3], in order toprovide accurate segmentations that result in better starting
points for the sonification method definition.
One of the challenges when analyzing ECG recordings is todetermine which parts of the signal are meaningful data, andwhich others are the result of noises and artifacts. Moreover, adifficulty for accurate diagnostics is given by the fact that anabnormality in the ECG might be present in only a specificsubset of leads, while other leads remain closer to a healthy
signal. Sonification appears to be a good approach to help
clinicians in the analysis and identification of importantvariations of the signal. First, because the human ear has thecapability to rapidly and robustly detect changes and patterns
even in very noisy signals, and second, because by giving theusers the possibility to interact with the resulting sonifications,they could refine the sonifications themselves in order toenhance any pattern they regard as relevant, thus increasing thesaliency of patterns. For example, if an abnormal and normal
signal would differ in rhythmical features, the adjustment ofparameters that control a nonlinear time warping might increase
the perceptual difference between the sonifications of thesedatasets and thus help to better distinguish patterns.
The paper will first provide some background on ECG in
Section 2. After presenting the used data (Section 3) andmethods to extract relevant features (Section 4) the paper
introduces the two new sonification approaches in Section 5. A
pilot study with discussion and conclusion complete andsummarize the paper.

2.

BACKGROUND ON ECG

The electrocardiogram was first developed and introduced in
the year 1903 by Willem Einthoven. An ECG is a visual
representation of the electrical potentials generated by the cellsof the heart muscle [1] during the depolarization andrepolarization process of each cardiac cycle.
The resulting ECG signal is structured into intervals orsegments that represent the electrical current flow within theheart over a period of time. A standard ECG recording contains
12 leads, which allow to measure the electrical potentials in
distinct heart walls.
From the 12 standard leads, a few subgroups are formed.
Their division depends on the heart’s wall they measure. As aresult, the standard leads are divided in lateral, inferior, septal
and anterior leads. When a set of leads belong to the same
subgroup it is said that they are contiguous.

2.1. ECG reference points and intervals
An ECG signal is typically analyzed by looking at six standard
reference points [2], which are produced in every heartbeat. The
references are shown in Figure 1.
The P wave represents the process of atrial depolarization,
while the T wave occurs during ventricular repolarization. The
U wave is normally not seen because of its low amplitude;
however, it also takes place during ventricular repolarization.
The J-point is located where the QRS complex finishes and the
ST segment begins.

48
ISon2016-1

Proceedings of ISon 2016, 5th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 2016

Figure 1. ECG standard reference points (P, Q, R, S, Jpoint, T and U), ST-segment and ST-interval. This
image is a derivative of ‘Normal ECG 2’ byMadhero881
Additional to the standard reference points, there are
general segments in the signal that are important for its analysis.
First, the PQ segment, followed by the QRS complex that
represents ventricular activation. Right after the QRS complex
there is the ST segment. It appears when the heart muscle
contracts, which is the time between depolarization andrepolarization of the ventricles. In healthy patients, thissegment should be isoelectric, which means that the potential
difference is zero. Finally, between the T wave and the P wave,
the TP segments appears.
The most common ECG intervals and their normal
durations for a heart rate of 60 beats per minute (bpm) are listed
in Table 1 [2].
Interval
PQ/PR Interval
QRS
QT

Duration
160 ms ± 40 ms
100 ms ± 20 ms
400 ms ± 40 ms

Table 1. ECG standard interval durations for a heart
rate of 60 bpm.
2.2. Importance of the ST-Segment
The ST segment is an isoelectric part of the healthy ECG.
During that period of time there is normally no electric activity
in healthy subjects. The ventricular depolarization has reached
its maximum and the repolarization has not started yet.
Therefore, no current is flowing during that time frame. The ST
segment begins with the so-called J-point which is succeeding
the QRS complex, the J-point represents the end of ventriculardepolarization. The end of the ST segment is the beginning of
the T wave corresponding to the ventricular repolarization.
Under certain conditions the ST segment can vary its
appearance. The most important ST segment abnormalities are
ST elevation or ST depression. These can be due to an ischemiaof the coronaries, the vessels that are necessary for direct
oxygen supply of the heart. ST segment elevation occurs in an
event of sudden closure of these vessels. In this situation a
myocardial infarction is present, because the tissue of the heart
is in constant need of oxygen supply. Medical doctors use the
ST segment evaluation as a major tool to decide whether thepatient needs urgent intervention for recanalization of thecoronaries.

1

According to the guidelines of the European Society ofCardiology, patients presenting with ECGs containing an ST
segment elevation at the J-point of greater than 0.1 mV in two
contiguous leads are ruling in as a so-called ST elevation
myocardial infarction (STEMI) [4]. Slightly varying rules exist,
depending on age and sex of the patient. The elevation at the Jpoint should be compared to either the PQ segment or the TP
segment, which are both supposed to be isoelectric. The
repolarization of the atrium occurs during the PQ segment [5],this segment can show variants and might lose its isoelectricity
if there is pathologic involvement of the atrium, e.g.
perimyocarditis [6].
If the above mentioned criteria of ST segment elevation are
met the patient will undergo a heart catheterization, because
described ST segment changes are most likely due to a STEMI.
In case of a STEMI the shape of the ST segment elevation caneven hint to the time of onset of the infarction. By evaluating
the ECG and the localization of the ST segment elevations
medical doctors can distinguish which coronary vessel isacutely affected.
The ST segment is a very important part of the ECG,
because disturbances of this segment are evaluated for different
pathologies, the most important pathology being myocardial
infarction.
2.3. Overview of ECG Sonifications
State of the art methods for analyzing ECG signals rely merely
on visual representations. Nevertheless, the interest forsonifying ECG signals has been growing in recent times. Someefforts are focused on detecting the main components of the
ECG signal for further use in the sonifications [3], and someothers had proposed sonification designs to represent the data,
either implementing audification methods [7, 9] or parameter
mapping sonification techniques [1,9, 7, 8].
So far, the sonifications were mainly focused on heart rate
representation and the pathologies that derive from abnormal
heart rate values or rhythmical patterns. For example, Ballora etal. [1] carried out a study in which they sonified the heart ratevariability in four cardiac states. Additionally, Mihalas et al.
[10] proposed sonification designs to monitor heart rate during
exercise. Also, Terasawa et al. [9] directed their work towards
making ECG components such as the T and P waves moresalient.
3.

To develop our sonification designs described below in Section
5, we utilize two ECG data sources. The first one is the publicly
available PTB (Physikalisch-Technische Bundesanstalt)
diagnostic database [11] from Physionet2, and the second one isown data recorded via a General Electric’s MAC2000 Resting
ECG System3 in the hospital.
3.1. Physionet dataset – PTB diagnostic database
The PTB database is composed of 549 files sampled at 1000
Hz, each recording contains the standard 12 leads plus threeFrank leads. We use only the first 12 leads for our sonifications.
Along with each ECG recording, a clinical summary is
included, which encloses information such as age, gender and2

Licensed under CC BY-SA 3.0.
https://commons.wikimedia.org/wiki/File:Normal_ECG_2.svg

ECG DATA AND PREPROCESSING

3

https://www.physionet.org/physiobank/database/ptbdb/
http://www3.gehealthcare.com/en/products/categories/
diagnostic_ecg/resting/mac_200049
ISon2016-2

Proceedings of ISon 2016, 5th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 2016

diagnosis. From the total number of files, there are 148 patients
whose diagnostic was Myocardial Infarction. Additionally, 52
cases are included as healthy controls.
3.2. MAC2000 Resting ECG System
The files from the MAC2000 Resting system are provided byour clinical partner. Since the system outputs the recordings in
XML format, we first extract the relevant leads into a CSVformatted file.
The database built from the Hospital’s system contains files
from patients diagnosed with myocardial infarction and subjects
that are considered healthy. All recordings include the standard
12 leads sampled at 500 Hz.
4.

the duration of each segment. For the latter calculation, we
normalized the ratio of the ECG parts in relation to the RR
interval duration when the heart rate is 60 bpm. We take as areference the values initially presented in Table 1.
Typical interval duration for ahealthy adult (60 bpm). RR
Interval duration = 1000 ms
QRS Width

100 msQT Interval

400 msPQ/PR Interval

160 msRatio of ECG Intervals
in relation to RR
interval duration when
the heart rate is 60 bpm
44 56"789:;
10.0
44 56"789:;
2.5
44 56"789:;
6.25

ESTIMATION OF ST-SEGMENT DEVIATION

In order to calculate the ST-segment deviation, first we estimate
the location and duration of the sections and standard reference
points of the ECG signal. Initially, we carry out the R peaks
detection and subsequently, based on this information we define
the other segments.

Table 2. Ratio of ECG intervals in relation to the RR
interval when the heart rate is 60 bpm.
Together with the peak’s location and the estimation of thesegments shown in Table 2, we determine the location of the J
point, which happens right after the QRS complex ends (cf.
Figure 4). Additionally, we calculate the ST interval duration as
EF/GH = IF/GH − I4E/GH

4.1. R Peaks detection
The R wave is the standard point with the biggest amplitude in
the ECG signal over a heartbeat cycle (Figure 2). The detectionof the R peaks is commonly used to determine the heart rate.
We perform the R peak detection of every lead
implementing the procedure proposed by Worrall et al. [3] asfollows: First, we remove the DC component from the signal
and apply a high-pass filter to remove the lower frequency P
and T waves. Subsequently the envelope is estimated using theHilbert transform operation and the resulting signal is nonlinearly scaled. Finally, the R peaks are detected and grouped
when the time difference ∆" among them is lower than 300 ms(corresponding to a rate of 180 bpm). The R peak with thehigher amplitude among the grouped peaks, was chosen as thepeak for the analyzed heartbeat. We chose to use a lower ∆",than the one proposed in [3] for grouping the peaks because weobserved that when there is an ST deviation the heart rate could
be higher than 60 bpm, leading to a lower duration of the QTinterval and the other segments. Hence, setting a smaller ∆"diminished the risk of missing peaks and increased theperformance of the R peak detection algorithm.

(1)Finally, we determine the duration and location of the TP
segment as
FK/GH = 44/GH − IF/GH − KI/GH(2)

As was mentioned in Section 2.2, the estimation of the TP
segment is important because it provides the interval fromwhich we estimate the isoelectric reference. Figure 3 shows theestimation of the points and segments that are necessary to
calculate the ST segment deviation. Although the end of the T
wave (that marks the end of the ST interval) is not exactly
determined, the estimation of the J point and the TP are better
achieved.

Figure 3. ECG segments estimation for one lead (black
line: J point, green line: end of ST interval, red line:
end of TP segment).
Figure 2. R peaks detection.

There are other methods that focus on a more accurate
detection of the segments [12, 13, 14], yet we postpone such
improvements as our current approach is sufficient for thedevelopment of the presented sonification designs.

4.2. ECG Segments detection
Once the R peaks are detected, we determine the heart rate of
every lead (as 60/RRinterval,-./ , where RRinterval,-./ =2 3<=2 4456"789:;< , k is the kth heartbeat) and subsequently
3

4.3. ST-Segment deviation estimation
With our previously estimated J point and ST interval duration
per lead, we can proceed to calculate the amplitude difference50
ISon2016-3

Proceedings of ISon 2016, 5th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 2016

with respect to the TP segment (which we assume to define
isoelectric) in every heartbeat of the analyzed lead.

4.4. ST deviation in contiguous leads
Besides estimating the ST deviation, it is important to detect if
an abnormal value of the deviation is present in contiguous
leads (as defined in Section 2) to determine the heart’s zone
where problems occur. For this, we take as a reference the set of
the 12 standard leads and cluster them into three groups. We
don’t include the aVR lead as part of the subgroups5.
•
Anterior leads (joining septal and anterior leads):
V1, V2, V3, V4.
•
Inferior leads: II, III, aVF.
•
Lateral leads: I, aVL, V5, V6.
Then, if in any of these three groups a deviation exceeds athreshold \ = 0.1 mV [4] in two or more leads, we conclude
that the ECG signal exhibits ST deviation in contiguous leads
and thus the corresponding section/s of the heart is/are affected.

Figure 4. ECG segments detection. This image is a
derivative of ‘Normal ECG 2’ by Madhero884

5.

First, we apply a band-pass filter to remove frequencies
below 0.6 Hz and above 70 Hz in order to remove frequencies
out of the accepted range for ECG diagnostic [2]. Then we
calculate the average amplitude within the TP segment, which
ideally should be 0 mV, and serves as DC-offset or
isoelectricity reference. Practically we computePQ
1
N " O""L − "M PR
by summing the sampled signal g(t) between the segment
borders t3 and t4, using

FK =

(3)

2

"M = 4S7:T + I4E/GH + EF/GH and "L = EF/GH + FK/GH .
V
Likewise we calculate the amplitude at the J point as theaverage from the J point until a time "V .
PW
(4)
1
EF =
N " O""V − "2 PX
2V

Finally, the average ST-segment deviation with reference tothe isoelectric reference results in
EF.YZ = EF − FK(5)

For the proposed sonifications we consider only theabsolute value of the ST deviation STabs,amp = | STamp |. Next to
the ST elevation calculated per heartbeat we determine theoverall elevation per lead by1
6

We propose two sonifications, water ambience sonification andtimbre morphing sonification, that can be used in different
medical scenarios. The first one is meant to be used as amonitoring tool, which implies that when a signal ischaracterized as healthy, the sound should not be intrusive nor
distract the doctors from any activities they are performing. The
idea is to turn the ECG signals systematically into a soundscapethat can be constantly played in the monitoring room, leading to
audible sound events as soon as the signal exhibits abnormal
behavior, from weak cues to a number of sounds appearing
simultaneously in the soundscape to make the ECG features
more salient.
The timbre morphing sonification is intended to work as an
emergency signal that produces very clearly distinguishable
sounds between ST-isoelectric (healthy) and ST-deviated(pathological) signals. The motivation is to provide an auditorycue that allows doctors to quickly assess the overall state of themain ECG components.

5.1. Water ambience sonification sound design

where "V = "2 + I4E/GH .

EF.YZ[-./ =SONIFICATION DESIGNS

3

EF< − FK<

(6)

<=2

Where, n is the number of heartbeats per lead, and k is theindex for the heartbeat in the data recording.

The idea of using water sounds for representing the ECG starts
by considering that the heart itself works as a pump that sends
blood (fluid) to the body tissues. Therefore, we can expect thatwater flow provides a metaphoric association to facilitate its
interpretation as blood flow across the heart. However, we
propose two variations in comparison to the real blood flow
model: (i) Discrete representation of blood flow: instead ofusing a continuous sound stream for water flow we quantize thestream into perceptual units, i.e. water drop sounds as basic
elements. (ii) Opposite representation of Healthy vs.
Pathological ECG: rather than representing the blood flow of ahealthy signal triggering several sound events, we propose an
inverted approach where a healthy signal is represented with theleast possible amount of sounds. In this way, the listener only
receives auditory cues when the signal presents abnormal
variations that call for attention. Thus, during each heartbeat anumber of drop sound events are triggered if the calculated STdeviation is greater than 0.05 mV; otherwise no sound is
produced. We chose to trigger the drops from a 0.05 mV
threshold in order to subtly start calling the attention from the4

Licensed under CC BY-SA 3.0.
<https://commons.wikimedia.org/wiki/File:Normal_ECG_2.svg>

5

The aVR lead is not normally included as part of any subgroup.

51
ISon2016-4

Proceedings of ISon 2016, 5th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 2016

listener before an amplitude value catalogued as ST deviated(0.1 mV) is reached.
We propose a parameter-mapping sonification [15] where
we determine the number of drops within each ST deviatedheartbeat by linearly scaling the amplitude of each ST segment(input values) to the number of drops (target values). Based on
the ST amplitude characteristics that we have observed in the
ECG data, we define the range (min, max) of the source values
to (0, 0.5) and the output (min, max) is set to (0, x, where x is
the maximum number of drops defined by the user (cf. Table 4).
Data
FeatureST
deviationofheartbeat

Data
Range
(min,
max)
(0mV, 0.5
mV)

Parameter

Parameter range(min, max)

Number ofdrops perheartbeat(0, x)
x is defined by theuser, according to:
1 ≤ ` ≤ 10

Table 4. Linear mapping of data features used in the water
ambience parameter-mapping sonification.
In order to avoid wrong interpretations of the number ofdrops due to their temporal coincidence, but also to provide thelistener with a cue for the heart rate even if a high number ofdrops is triggered, the sound events can be evenly distributed
over the RR peak duration, or they can be distributed over a of
the RR interval (a ∙ 44/GH , Figure 5). We set the possible
minimum duration for the drops distribution parameter to a =0.4 and the maximum to the full RR interval (a = 1.0).

Figure 6. Water Ambience sound design drops and extra
ambience sounds representing contiguous leads.

5.2. Timbre morphing sonification sound design
For the timbre morphing sonification, we introduce a parametermapping sonification that represents abnormality via aninterpolation between two different waveforms, so that thedegree of deviation becomes salient as a morphed timbre.
The sound is synthesized by superimposing two oscillators
over different wave tables, specifically (i) a sine wave and (ii) asquare wave. For this sonification we wanted to interpolate
between a sinusoidal waveform to one of the other three mostcommon audio wave shapes (Triangle wave, Square wave, andSawtooth wave). We chose the sine wave as the starting point inorder to represent healthy signals with the most spectrally
simple sound, a pure tone, and then interpolate to a more
complex tone. We chose the square wave. We did not use thetriangle wave because it is perceptually too close to the sine
wave and we also ruled out the sawtooth wave because it wasperceived as too sharp as it contains all harmonics.
As a result, the overall timbre can be produced only by asinusoidal wave, a square wave, or a combination of the two.
The cross-fading weighting factor a between the waves depends
on the ST deviation value, thus when the ECG is consideredhealthy and the amplitude in the J point is low, the sinusoidal
wave is predominant. On the contrary, when the deviation ishigh the square wave is more noticeable. The cross-fading
between the waveforms is given by,
f " = a ⋅ f. " + 1 − a ⋅ fh "Figure 5. Water Ambience sound design of healthy ECG
and ST elevated ECG.
Finally, we use the information from the ST deviation incontiguous leads to add selected ambience sounds. When theinferior, lateral or anterior leads present a deviation, a newsound event is triggered. Each of the three groups is assigned toa specific sound: the group of lateral leads is assigned tothunder sound, inferior leads to wind sounds, and anterior leads
to rain sounds. All these sounds are added by playing recorded
sound samples of few seconds duration. They are not meant tobe played as long as the groups’ deviation prevails, but occur
only once every 10 seconds duration if the ST deviation ispresent.

(7)

As mentioned before, we intend that this sonification
provides cues for clinicians to assess the overall state of the
ECG components, thus besides representing the ST elevation
we wanted the sonification to also represent the amplitude andlocation features of the R peak and the T wave in every
heartbeat. For this, we trigger sound events of the synthesizedsound only when the R peak and the T wave occur in eachcardiac cycle (Figure 7). The duration of the two sound eventsis initially set to 600 ms but can be modified by the user withina range from 0 ms to 1000 ms.

52
ISon2016-5

Proceedings of ISon 2016, 5th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 2016

sonification for exploratory data analysis has been discussed byHerman and Hunt in 2004 [16]. Since one of the goals of ourresearch is to provide sonification tools that can be used in
medical scenarios as a supporting tool for diagnosis andmonitoring, and given the fact that clinicians are normally used
to work with medical devices that present data on a visual form,
we decided to develop a Graphical User Interface (GUI) whereusers can interact with the sonifications and at the same time
have a visual feedback of the data that is currently being played.
6.1. Heart Alert GUI
Figure 7. Timbre morphing sound design.
The frequency of the synthesized sound is modulated by asine oscillation resulting in an audible effect of vibrato. We use
a linear mapping function (eq. 10) to map the amplitude of the
R peak to the fundamental frequency of the synthesized sound.
We take the same approach for the sound event triggered when
the T wave appears but in this case the data feature we map isthe maximum amplitude value of the T wave. Furthermore, we
count the number of leads that are catalogued as ST deviatedand map this value to the depth parameter of the vibrato (cf.
Table 5).
i = ` − `. / `h − `. ⋅ ih − i. + i.

The Heart Alert GUI is divided into four modules; the first one
allows the user to select an ECG file and it’s basic properties(sampling rate, etc.). Then the user can select a group of leads
or an individual lead for plotting and sonifying. Module 2
contains a menu for selecting the type of sonification and a‘play’ and ‘stop’ button. The third module plots the selectedleads of the ECG signal and when the sonification is triggered,provides visual feedback about the current time in the data. The
last module includes controls for interacting with thesonifications.

(8)

where, ` is the value to be mapped. The source range is givenby (`. , `h ) and the destination range by (i. , ih ).
DataFeatureParameter

Parameter
range (min,max)

Amplitude
R peak

Data
Range
(min,
max)
(0 mV,
2.0 mV)

Fundamental
frequency

Amplitude
T peak

(0 mV,
0.8 mV)

Fundamental
frequency(100, x)x is defined by
the user,according to:100 ≤ ` ≤ 1000
(200, x)x is defined by
the user,according to:200 ≤ ` ≤ 1000
(0, 1)

Number of
leads
catalogued
as ST
elevated(0, 12)Depth ofvibratoTable 5. Linear mapping of data features used in the timbre
morphing parameter-mapping sonification.

Figure 5. Heart alert GUI (Green rectangle: Options
module, blue: Sonification module, orange: Plot
module, purple: Interaction module).

6.2. Interacting with the sonifications
The user can interact with the sonifications by adjusting thesliders of the GUI that control the parameters of each sounddesign and selecting the leads to plot and sonify.
From the available parameters that determine the water
ambience sonification, the two parameters maximum number ofdrops and drops distribution were chosen for the GUI, using
sliders as shown in Figure 6. Additionally, we include three
more sliders, one to control the level of the ambience sounds
that represent ST elevation in contiguous leads, another one forthe level of the drops sound, and the last one to control the
playback speed.

Lastly, we take the value of the RR segment duration anduse it as the rate parameter of the vibrato.
6.

INTERACTIVE ECG SONIFICATIONInteractive sonification can provide tools for the user to have abetter understanding of the data and to make features of interest
more noticeable. The advantages of using interactive53
ISon2016-6

Proceedings of ISon 2016, 5th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 2016

Figure 6. Interaction Module of the Water ambiencesonification.
For the timbre morphing sonification, we chose threeparameters: (i) max fundamental frequency that the R peak
amplitude can be mapped to, (ii) maximum fundamental
frequency that the T peak can be mapped to, and (iii) the soundevent duration. The interaction module (see Fig. 7) also
includes a slider to control the level of the sonification and one
for the playback speed.

timbre morphing sonification (10 audio files). For the other four
users the sonification designs were presented in the opposite
order. We also asked the users to select on a six-points Likert
scale (1 being the lowest score and 6 the highest score) if thesonification was pleasant to listen to and if they found the soundacceptable to be listened to for a longer period of time. At theend they were asked to select their preferred sonification.
The classification accuracy (i.e. the fraction of examples
correctly classified as healthy or ST elevated, reported as k ±m) is 0.975 ± 0.046 for the water ambience sonification and
0.9 ± 0.093 for the timbre morphing sonification. A two-sided
t-test shows that this difference is significant (t(18) = 3.0, p<
0.0199. Even do the high classification scores for bothsonifications suggest a ceiling effect, the water ambiencesonification allowed a more accurate classification.
Pleasantness of each sonification was rated from 1 to 6,
where 1 refers to ‘strongly disagree’ and 6 refers to ‘strongly
agree’ (results are reported as k ± m). The water ambiencesonification was rated as 5.12 ± 0.64 and the timbre morphingsonification as 3.5 ± 1.69.
When the users were asked about the sonifications being
acceptable to listen to for a long period of time, they rated thewater ambience as 4.88 ± 1.13 and the timbre morphing as 3.5±1.93. Finally, all users selected the water ambience design as
they preferred sonification.

Figure 7. Interaction Module of Timbre morphingsonification.
7.

8.

USER FEEDBACK AND STUDY RESULTS

As a first preliminary qualitative assessment of the new designs,
we asked three subjects to comment on the insight provided by
the sonifications when comparing healthy and non-healthy
datasets. They also provided feedback regarding the graphicaluser interface and the interaction tools. At the time of thisevaluation the current version of Heart Alert did not allow toalter the generated sonifications in real-time, but instead
required to trigger a new sonification by pressing the play
button in order to apply the changes produced by the interaction
module sliders.
During the pre-test the users stated that the morphed soundworked closer to an alarm requiring their immediate attention,
while the water ambience sound felt more calm and didn’t
evoke the same level of urgency compared to the timbre
morphing sonification. We also noticed, that for the water
ambient sonification the wind and rain sounds were often
confused by the users, which made difficult to identify the roleof this sounds when they appeared. Concerning the GUI, they
indicated to feel comfortable with the interaction elements
provided and one of them suggested to include a set of presets
in the interaction module that could be used as a starting point
to interact with the sonifications.
After conducting the preliminary test, we improved theinteractivity of the system so that the changes in thesonifications were produced immediately when the sliders weremodified without having to trigger the sonification again. Also
we selected new rain and wind samples to allow the users tobetter identify when these sounds appeared.
After this iterative improvement, we carried out a second
study in which eight participants were asked to listen to ECG
sonifications (each with 10 seconds duration) and classify themin one of two possible categories; healthy or unhealthy. Fourusers evaluated the water ambience sonification first (theclassification task included 10 audio files), followed by the

DISCUSSION

This work presents a sonification system for ECG signals that ismeant to be used as a supporting tool in the diagnosis of ST
segment deviations. We propose two sonification designs, one
for monitoring the patient and one for providing quick insighton the state of the ECG segments. Furthermore, we provide
tools for visualizing and sonifying a specific lead or a set ofleads, which allow users to focus their attention on the heart’s
zone they are more interested in, or to have a general idea on
the characteristics of the signal. Presently the sonifications arestereo, but the playback capabilities of the actual system can beextended to generate multi-channel sonifications which can beplayed over N loudspeakers for a more spatialized sonification.
The interaction module provides clinicians with tools thatcan be used to adjust the sonifications based on the needs andpreferences of the user and at the same time find the sounds thatare more meaningful to them when analyzing and exploringclinical data. These sonifications should differ from alreadyused auditory displays (e.g pulse oximeter), because it ispossible to modify the sounds and assign them to a different
spectral bandwidth that doesn’t interfere with the sounds
produced by existing auditory devices or soundscapecomponents in a regular medical scene. For future system
implementations we think it would be interesting to implement
the control of the Heart Alert features using mobile devices thatclinicians already use on a daily basis.
Currently the estimation of the ECG segments is
implemented under a basic approach that leaves room for
improving the detection methods. Also the peak detection
algorithm can be further improved to yield better performanceon ECG files with distinct rhythm pathologies. Improving the R
peaks and segments detection would open a door for using thesystem as a supporting tool in the diagnosis of other heart
abnormalities.
The study results showed that the proposed sonifications
provide a functional first approach to ST-segment elevation
sonification since it is already makes it possible for users withno previous ECG experience to differentiate healthy datasets

54
ISon2016-7

Proceedings of ISon 2016, 5th Interactive Sonification Workshop, CITEC, Bielefeld University, Germany, December 16, 2016

from ST-elevated ECG. A significant difference was found in
the classification task, however the classification scores are
considerably high for both sonifications, which suggest a ceiling
effect that would not make possible to select one of the twosonifications as a conclusive winner. The ceiling effect can be
the result of an easy classification task; this means that thecomplexity of the task needs to be increased for future studies.
9.

CONCLUSIONS

We have introduced an ECG sonification system that uses real
medical data to provide information on the overall state of thesignal an its variations. According to the study results obtained,
the sonifications showed to offer interesting insight on the datawhen a healthy dataset is compared to an ST elevated one.
However, further work needs to be done in order to improve thesegment estimation so that the sonifications better convey
information regarding changes in the leads that conform astandard ECG.
We consider ECG sonification research to be promising andable to provide meaningful insight on several features of theheart’s signal. Not only could it be used for diagnostic, but also
as an educational tool for medicine students that are in theprocess of learning how to interpret the variations in the signal,and for whom an auditory-type feedback could be useful.
We aim to improve the methods presented in this work andto use sonification to further represent and explore thecharacteristics of the ECG signal. For example, one of thefollowing steps in ECG sonification, would be to explore
methods for sonifying the polarity of the ECG axis.
10. RESOURCES
Examples of the sonifications are provided in
http://dx.doi.org/10.4119/unibi/2907475

ACKNOWLEDGMENTS
This work has been supported by the German Academic
Research Service (DAAD) and the Cluster of Excellence
Cognitive Interaction Technology ‘CITEC’ (EXC 277) at
Bielefeld University, which is funded by the German Research
Foundation (DFG).

[5] Z. Ihara, A. van Oosterom and R. Hoekema: “Atrial
repolarization as observable during the PQ interval. J
Electrocardiol”. 30: 290-297, 2006.
[6] M. Imazio and F. Gaita: “Diagnosis and treatment of
pericarditis”. Heart, 101: 1159-1168, 2015.
[7] G. Mihalas, S. Paralescu, N. Mirica et al., “Sonic
Representation of Information: Application for Heart Rate
Analysis”. In Proc. of APAMI Conference (2012). Beijing
23-25, Oct 2012.
[8] V. Avbelj: “Auditory display of biomedical signals
through a sonic representation: ECG and EEG
sonification”. In Proc. of the 35th International
Convention of Information Communication Technology,
Electronics and Microelectronics MIPRO (2012). Opatija,
Croatia, pp. 474-475, 2012.
[9] H. Terasawa, Y. Morimoto, M. Matsubara, et al., “Guiding
Auditory Attention toward the Subtle Components inElectrocardiography Sonification”. In Proc. of the 21th
International Conference on Auditory Display (ICAD2015). Graz, Austria, 2015.
[10] G. Mihalas, L. Popescu, A. Naaji et al., “Adding Sound to
Medical Data Representation”. In Proc. of the 21th
International Conference on Auditory Display (ICAD2015). Graz, Austria, 2015.
[11] R. Bousseljot, D. Kreiseler and A. Schnabel: Nutzung der
EKG-Signaldatenbank CARDIODAT der PTB über das
Internet¨. Biomedizinische
Technik,Band
40,
Ergänzungsband 1, S 317, 1995
[12] S. Bulusu, M. Faezipour, V. Ng et al., “Transient stsegment episode detection for ecg beat classification,” inLife Science Systems and Applications Workshop (LiSSA).
IEEE/NIH. IEEE, 2011, pp. 121– 124, 2011.
[13] B.U. Kohler, C. Henning and R. Olgmeister: “Theprinciples of software QRS detection”, IEEE Engineeringin Medicine and Biology Magazine, 42-57, 2002.
[14] N. Maglaveras, T. Stamkopoulos, K. Diamantaras et al.,“ECG pattern recognition and classification using nonlinear transformations and neural networks: A review”.
International Journal of Medical Informatics 52, 191-298,
1998.
[15] T. Hermann, A. Hunt, J. G. Neuhoff, editors. “The
Sonification Handbook”. Logos Publishing House, Berlin,
Germany, 2011.
[16] T. Hermann and A. Hunt: “Guest Editors' Introduction: An
Introduction
toInteractive
Sonification”,
IEEE
MultiMedia 12.2: 20-24, 2005.

REFERENCES[1] M. Ballora, B. Pennycook, P.C. Ivanov, L. Glass, and A.L.
Goldberger: "Heart rate sonification: A new approach to
medical diagnosis." Leonardo 37.1: 41-46, 2004.
[2] G.D. Clifford: “ECG statistics, noise, artefacts and missing
data.” Ch3 in: G.D. Clifford, F. Azuaje and P.E. McSharry(Eds): Advanced Methods and Tools for ECG
Analysis. Artech House Publishing, October 2006.
[3] D. Worrall, T. Balaji and N. Degara “Detecting
Components of an ECG Signal for Sonification.” In Proc.
of the 20th International Conference on Auditory Display(ICAD-2014). New York, USA, 2014.
[4] P.G. Steg, S.K. James, D. Atar et al., “ESC Guidelines forthe management of acute myocardial infarction in patients
presenting with ST-segment elevation”. European Heart
Journal, 33:2569-2619, 2012.

55
ISon2016-8