st srInternational
The
InternationalConference
ConferenceononAuditory
AuditoryDisplay
Display(ICAD(ICAD-2015)
The
2121
2015)

July 8-10,
2015,
AustriaJuly
8–10,Graz,
2015,
Graz, Austria

GUIDING AUDITORY ATTENTION TOWARD THE SUBTLE COMPONENTS
IN ELECTROCARDIOGRAPHY SONIFICATION

Hiroko Terasawa1,2, Yota Morimoto3, Masaki Matsubara1, Akira Sato1,
Makoto Ohara1, and Masatoshi Kawarasaki1
1

University of Tsukuba
1-2 Kasuga, Tsukuba, Ibaraki 305-8550 Japan
2

JST, PRESTO
7, Gobancho, Chiyoda, Tokyo 102-0076 Japan
3

Royal Conservatory of The Hague
Juliana van Stolberglaan 1, 2595 CA The Hague, The Netherlands
terasawa@slis.tsukuba.ac.jp

ABSTRACT

control heart beating. ECG signals reflect this charging andreleasing pattern that occurs at different parts in the heart.

ECG (electrocardiography) consists of a few components andfeatures that indicate the details of electrical conduction ineach part of the heart. The simple pitch-mapping sonificationof ECG cannot indicate these components to listeners,
limiting its potential for diagnostic usages. We present an
improved method to emphasize such key features in ECG
sonification, using the principle that “we are attentive to loud,high-pitched, and rapidly fluctuating sounds.” With thisprinciple, we can guide the listener’s attention to subtle yet
critical elements of the ECG waveform by controlling thedegree of auditory saliency for each component. In this report,
we describe ECG sonification step by step and discuss the
cognitive and attentive issues as well as the emotional
responses of the listeners.

1.

INTRODUCTION

A normal ECG signal shows a pattern within each cycle.
Figure 1 shows one cycle of a sinus rhythm (a normal
heartbeat pattern). A cycle is defined as a combination of fivewaves, specifically, the P, Q, R, S, and T waves. The Q, R,
and S waves occur very rapidly and are often denoted as the
QRS complex. The P and R waves occur with the atrial andventricular discharges, respectively. The atria charge during
the QS interval, and the ventricles charge during the T wave.
The R wave is the most notable wave, and the cardiac
monitor most commonly used in hospitals (the type which
generates periodic beeps) utilizes the occurrence of the R
wave as a vital sign. The interval between consecutive R
waves is called the RR interval and is the most criticalinformation to be considered. Therefore, previous works on
ECG sonification have mainly focused on displaying
information related to the R wave [1, 2].

In the diagnostic use of ECG (electrocardiography), medical
doctors watch for key features that indicate the details ofelectrical conduction in each part of the heart. This paper
describes a sound design methodology to emphasize such key
features in ECG with the objective of guiding the listener’s
attention to the important information.

1.1. ECG components
ECG measures the electrical activity in the heart using
electrodes placed externally on the chest or limbs. The
electrical charging and releasing of the heart muscle cells

This work is licensed under Creative Commons
Attribution – Non Commercial 4.0 International License.
The full terms of the License are available at
http://creativecommons.org/licenses/by-nc/4.0/

Figure 1: One normal ECG cycle, consisting of a P wave,
QRS complex, and T wave. Usually, the U wave is hidden by
the T wave and is thus invisible.

ICAD 2015 - 231

sr
The 21
21st
International
The
InternationalConference
Conferenceon
onAuditory
AuditoryDisplay
Display(ICAD-2015)(ICAD 2015)

However, the waves other than the R wave also carry criticalinformation [3]. For example, the shape of the P wave
reflects the functional balance between the left and right atria.
The T wave is generated when the heart restores in theresting stage, and deformation of the T wave may indicate
conditions such as heart muscle problems or cardiomegaly(heart enlargement). There are many other important featuresin the P and T waves and in the intervals between waves aswell. Cardiologists read such information from ECGs inorder to perform diagnoses.

1.2. Goal and expected usage
Our objective was ECG sonification such that the soundwould provide details of the P and T waves that would yielduseful information for arrhythmia diagnosis.

This project was a collaboration between cardiologists andsound engineers. We decided that descriptive sonification of
the P and T waves would be a research question of interest toboth cardiologists and sound engineers. Cardiologists still
examine these intricate waves visually, seeking critical signs
for arrhythmia without assistance from information
technology. For sound engineers, creating a design beyond
monotonous beeps that can describe subtle changes in thedata poses a significant research challenge.

Our expected users are cardiologists and clinical staff. The
system is currently working offline (not in real time, but on
stored data), but our approach is simple enough to be applied
to real-time systems. We do not expect that the users will
make diagnoses only based on sonifications, but rather thatsonifications will contribute informative supplementary
displays to be used along with conventional ECG graphs. In
other words, we hope this kind of sonification will be usefulas part of multi-modal (i.e., audiovisual) displays, increasing
the robustness and decreasing the cognitive stress in the
diagnostic process.

July 8-10,July
2015,
Graz,
Austria
8–10,
2015,Graz, Austria

2.

GUIDING ATTENTION WITH SOUND DESIGN

2.1. Simple pitch-mapping approach
We first applied the simplest pitch-mapping approach to our
ECG data. The amplitude of the ECG waveform was
associated with the pitch of the sonification: a higher
amplitude corresponded to a higher tone pitch. We employed
granulated sinusoids in a musical scale, like that of the piano
keyboard shown in Fig. 2, instead of a sweeping sine tone.
The advantage of using the musical scale is that changes in
the waveform are more easily recognized as they form a kindof melody. It is very difficult to recognize the extent to which
the frequency of a sweeping sine tone increases. Listeners
can more easily recognize the pitch intervals of a musicalscale. Thus, in order to sonify the form and amplitude of an
ECG signal, the use of the musical scale has significant
advantages over sweeping sinusoids.
The use of the musical scale also has aesthetic merits. By
choosing a diatonic scale (or some other scale, such aspentatonic) the occurrence of beating and inharmonicintervals can be limited. For listeners who are accustomed to
traditional Western music, this property is highly desirable,facilitating the recognition of sonification as a result of theiralready trained music recognition skills.
Another synthesis technique we employed was “thinning,”that is, generating a sinusoidal grain only when the pitchchanged, as illustrated in Fig. 3. This design does not
generate sound constantly but rather only when a significant
change is observed. It yields a kind of arpeggiated rhythm inthe sonification with fluctuating micro-silences between
sinusoidal grains. This technique produces a light texture in
the sonification, rather than the flat and heavy texture of acontinuous sound stream.

1.3. Cognition and attention in auditory scenes
This goal cannot be accomplished by simple pitch mapping,in which the signal amplitude is mapped to the frequency ofthe sonified sound. This simple pitch-mapping approachcauses cognitive issues in interpreting the P and T waves, as
we describe in Section 2. In this paper, we describe theprocess of resolving this problem and how we guided thelistener’s attention to the ECG P and T waves by effective
sound design.

Section 2 describes the cognitive problem with the simple
pitch-mapping approach. Section 3 offers a solution based on
the principle that “fast and high-pitched sounds grab auditory
attention.” Section 4 describes the technical framework of thesystem, and Section 5 is devoted to the additional discussion,conclusion, and future work.

Figure 2: Simple pitch mapping in a piano-keyboard style.
The amplitude of the ECG waveform is associated with thediscrete pitch (note in a musical scale) of the sinusoidal grain.

ICAD 2015 - 232

st srInternational
The
InternationalConference
ConferenceononAuditory
AuditoryDisplay
Display(ICAD(ICAD-2015)
The
2121
2015)

July 8-10,
2015,
AustriaJuly
8–10,Graz,
2015,
Graz, Austria

With our simple pitch-mapping approach, the QRS complex
becomes a high-pitched, suddenly fluctuating sound,satisfying the above criteria that are required for auditory
saliency. That is why the QRS complex is audibly distinctive
and focusing on the P and T waves is extremely difficult.
However, since our goal was to better represent the P and T
waves, the QRS complex being salient was not desirable inthis investigation. In order to accomplish our goal, we had tocause the P and T waves to be salient and to de-emphasize
the QRS complex.

Figure 3: Illustration of sound thinning. The sinusoidal grains
are produced at constant time intervals, but if the pitch of thesubsequent sinusoidal grain is the same as that of theprevious grain, only the first sound is played (dark triangles)
and the later sound is omitted (white triangles). This processemphasizes the changes in the waveform and produces anirregular rhythmic texture in the sound.

2.2. The cognitive issue in P- and T-wave perception: “We
are attentive to loud, high-pitched, and rapidly fluctuating
sounds”
The next problem is the difficulty in differentiating the P and
T waves with the simple pitch-mapping approach, because
the sonified sound changes pitch very rapidly in the QRS
complex, while the changes in the other waves are less rapid.
This problem is actually one of auditory attention andperception. Auditory scene analysis is the process ofrecognizing the meaningful segments of an auditory scene,
which consists of the overlap and connection of many sounds[4]. In this process, some sounds are more distinctive thanothers and attract more attention. This attention-grabbing
characteristic is called saliency, and recent publications havefurther elaborated on its concept and models [5, 6, 7].
The most fundamental principle of auditory saliency is thatloud and high-pitched sounds are very difficult to ignore [7].
Most auditory alarms are designed based on this principle;
for example, fire alarms are extremely high-pitched and loudto attract attention and motivate evacuation. Furthermore,since our hearing is most sensitive to high frequencies, wetend to perceive high-pitched sounds as louder [8].
We propose another influence on saliency, roughness, whichis the rapid fluctuation of sound. Alarms, bells, and sirens
often are accompanied by rapid changes in amplitude andfrequency, resulting in a rough texture. Roughness isgenerally regarded as the cause of annoyance [9, 10]. We do
not yet understand the causal relationship between annoyanceand attention, yet we can reasonably speculate that the
emergent events, which should attract attention, areassociated with annoying, negative emotions, so that people
intuitively react repulsively to the events [11]. Anotherpossibility, based on a Darwinian perspective, is that
individuals who learned to interpret an emergency as
emotional arousal have survived over the generations.
Considering the alarming and annoying nature of roughsounds, we can conclude that rapidly fluctuating sounds draw
attention in an auditory scene.

2.3. Attention-guiding strategy
The saliency principle of “we are attentive to loud, highpitched, and rapidly fluctuating sounds” can be an influential
tool for sound design. By inverting this principle, “quiet,
low-pitched, and slowly fluctuating sounds” are less salient.
Considering these principles enables effective sound design.

3.

SONIFICATION DESIGN

The sonification design described in this section resolves thisproblem by inverting the order of saliency for the QRS
complex and the other waves and by replacing lessinteresting information with a symbolic sound. We describe
the sound design concept in this section and further technicaldetails in the next section.
3.1. Control signals
In order to produce the control signal, we first conducted asimple QRS complex detection. We analyzed the signal withrunning windows and identified the segments showing thelargest variance as the QRS complexes, since the rise-fallpattern of a QRS complex segment shows the greatest
variation from the baseline signal. The segments between
consecutive QRS complexes were simply divided into firstand second halves. Figure 4 shows an example of the threecontrol parameters in a sinus rhythm; we associated control
parameters 1, 2, and 3 with the QRS complex, PR interval,
and ST interval, respectively.

Figure 4: Control parameters for sonification. Parameters 1, 2,
and 3 indicate the QRS complex, P-wave segment, and Twave segment, respectively.

ICAD 2015 - 233

sr
The 21
21st
International
The
InternationalConference
Conferenceon
onAuditory
AuditoryDisplay
Display(ICAD-2015)(ICAD 2015)

The detection of the QRS complex and allocation of PR and
ST intervals can be further sophisticated, but that is beyond
the scope of this paper.

3.2. Achieving a symbolic, less salient QRS-complex
sound
In the pitch-mapping sonification, the QRS complex sounded
too salient. Therefore, we replaced this segment with a belllike, low-pitched, slowly decaying sine tone (which is,according to our principle, a less salient sound) at the time
when control parameter 1 turned positive. This replacement
step significantly facilitated the listener’s ability to
distinguish the P and T waves. Thus, listeners could perceive
the details of the P and T waves yet simultaneously maintainawareness of the basic heartbeat rhythm, so the QRS
complex part no longer interfered with attention to the P and
T waves.

3.3. Achieving a more salient P-wave sound
The P and T waves were then better recognized, yet thesegmentation between the two consecutive waves was notevident. The cardiologist we consulted advised us toemphasize the P waves more because these include morecritical information than do the T waves. Therefore, whencontrol parameter 2 became positive, we raised the pitch of
the P waves by a half octave, so that these could attract more
attention. After doing so, the P waves were more salient, andlisteners could focus on the P waves while recognizing theother parts in parallel.

4.

July 8-10,July
2015,
Graz,
Austria
8–10,
2015,Graz, Austria

4.2. Sound synthesis
We designed the internal sonic aspects of each of the threestreams such that the sonic images of each stream would be
distinct from those of the others: the P wave employed
Hanning-windowed sine-tone grains and was rendered in ahigher pitch range than that of the T wave (in order toemphasize the critical information); the T wave used a sine
wave oscillator with phase-modulation feedback, adding
clarity (harmonics) to the otherwise dull-sounding frequency
range; the R wave was generated in the pitch range between
those of the P and T waves and was distinctively articulatedwith a percussive envelope.
Additionally, the grain density was controlled for both the P
and T waves by dynamically controlling the number ofoverlapping grains according to the amplitude of the ECG
signal: a lower amplitude of the ECG (i.e., lower pitch ofsonification) corresponded to fewer overlapping grains. Thisoverlap was controlled to avoid the dull sonic image of thesaturated lower frequency range and to smooth out theotherwise jagged articulation in the upper frequency range.
We decided to render the R wave weaker in amplitude than
the P and T waves because of its simpler role; it only informs
the regularity or irregularity of the beating intervals.
However, we rendered it as a chord (notes: C, F, A, and B),
and because of its slightly dissonant character (F major #4th)
in the key of C major, it could be heard more clearly than if itwere, for example, a single tone or a C major chord. As
exemplified in this harmonic approach, by limiting thecontinuous pitch space to the discrete steps of a musical scale,we expanded our parameter space by also incorporating themusical parameter space (i.e., tonality, melody, harmony,etc.). The range of notes and chord used for sonification are
depicted on a musical staff in Fig. 5.

TECHNICAL DETAILSP

We implemented the sonification system in the SuperCollider
programming environment combined with Ruby. As
described in Section 2.1, we employed granular synthesis andmapped the amplitude of the ECG data onto the discrete stepsof a musical scale. In order to enable sonic distinction of the
PR interval, QRS complex, and ST interval, our systemconstructed three granular synthesis streams per ECG signal,
each corresponding to one of the three waves. In this way,each sound stream could be adjusted independently (scale
range, transposition, grain articulation, panning, etc.)
according to its intended cognitive role.

4.1. Data analysis
First, we analyzed the original ECG data by using thestatistical analysis class implemented in Ruby, since
SuperCollider only minimally supports mathematicalanalysis. The variance of the windowed signal was calculated,
and the sections with largest variance were identified as QRS
complexes. The calculated control parameters 1, 2, and 3
were then used to control the associated granular synthesis
streams. All three of the granular synthesis streams shared
the original ECG data but were controlled (turned on or off)
independently according to the three control parameters in
Fig. 4.

&œ

œ

T

œ

œ

R

˙˙˙˙

Figure 5: The range of notes used for sonification for the PR
interval (left), ST interval (middle), and QRS complex (right).

5.

DISCUSSION

5.1. Saliency, alarm, and emotion
The phenomenon that “loud, high-pitched, and rapidly
fluctuating sounds are difficult to ignore” occurs quite often
in sonification, even with data from sources other than ECG.
Furthermore, most of the annoying sounds produced bysonification are of this nature. These sounds easily cause very
negative emotional responses, and the Darwinian ideas
described in Section 2.1 may explain why. For example, thesmell of rotten food is perceived as repulsive and arouses anegative emotional response. This reaction is very natural
because those who did not have negative responses to rotten
food did not survive over the generations, so we are thedescendants of those who experienced repulsion to rotten
food. In the same way, considering that “loud, high-pitched,
and rapidly fluctuating sounds” are most likely caused bypotentially dangerous events with high energy and activity, it

ICAD 2015 - 234

st srInternational
The
InternationalConference
ConferenceononAuditory
AuditoryDisplay
Display(ICAD(ICAD-2015)
The
2121
2015)

is also natural that many of us experience negative responses
to these sounds.
In sonification, when the original data are “fast and strong,”
we may or may not wish to express the sound as such. If we
do not wish to express the signals as originally generated, we
can sonify them with various degrees of saliency by carefully
reducing the relevant features in order to avoid unnecessary
negative responses.

5.2. Compressing information based on expected function
The other interesting aspect in this sonification process is the
mixture of symbolic and analogic representations, whichwere described by Kramer [12]. Symbolic representations
display the processed information, while analogic
representations directly display the original data or signal. In
our case, we combined these two types of representations,representing the QRS complex as symbolic and the P and T
waves as analogic.
A symbolic representation is equivalent to the compressionof the signal information. In our case, the details of the QRS
complex were omitted, and only the information about its
timing was utilized, in order to mark the rhythmic intervals.
Of course if we had wanted to utilize all of the original data,
this kind of compression would have been impossible.
However, knowing which aspects of the data to retain or
discard enables the combined use of the symbolic andanalogic representations.
Mixing the symbolic and analogic representations facilitates
the parallel processing of multiple kinds of information.
Symbolic representations tend to generate impressions
similar to those of musical sounds, while analogic
representations generate impressions similar to those ofenvironmental sounds. Humans are able to distinguishmusical and environmental sounds and process them inparallel. Perhaps because of this skill, it was easy torecognize and distinguish the two types of sound in oursonification.
As described in Section 4.2, by limiting the pitches to thoseof a musical scale, our degree of expressive freedom
increased, enabling the use of musical parameters.
Furthermore, the waveforms became more easily
recognizable due to the melodic patterns. Flexible switching
between symbolic and analogic representations, or
superposing of those representations, provides a sonicresource that can be further exploited.

5.3. Collaborative process and evaluation
This project was conducted as a close collaboration between
sound engineers and cardiologists. Although we have notconducted a formal evaluation, the frequent meetings of thetwo parties facilitated the estimation of the validity of thesound design. The cardiologists and engineers sometimes had
different opinions (or complaints) about the sound design,and such conflicts usually developed into creating a better
solution.

July 8-10,
2015,
AustriaJuly
8–10,Graz,
2015,
Graz, Austria

6.

CONCLUSION AND FUTURE WORKIn this paper we described a design method to guide alistener’s attention in sonification, using ECG sonification as
an example, according to the principle that “we are attentive
to loud, high-pitched, and rapidly fluctuating sounds.” The
QRS complex was replaced with a symbolic, less salient
sound, and the critical P wave was emphasized with adistinctive, higher pitch. We also discussed the emotional
response to and auditory saliency of the sonification.
Topics of future work include investigating the potential forarrhythmia diagnosis and sophisticating the data analysis
process. A formal listening test is also desirable, but further
close collaboration between domain scientists (cardiologists)and sound engineers could be equally advantageous.

7.

ACKNOWLEDGMENT

This project is funded by PRESTO program of Japan Scienceand Technology Agency.
8.

REFERENCES[1] D. Worrall, B. Thoshkahna, and N. Degara, “Detecting
Components of an ECG Signal For Sonification,” inProc. of the 20th Int. Conf. on Auditory Display (ICAD2014), New York, USA, 2014.
[2] M. Ballora, et al., “Detection of Obstructive Sleep Apnea
through Auditory Display of Heart Rate Variability,” inComputers in Cardiology, vol 27, pp. 739-740, 2000.
[3] S. Watanabe, and I. Yamaguchi, ECG Perfect Manual A Comprehensive Tutorial with Theory and Waveform
patterns (in Japanese), Tokyom Japan: Yodosha
Publishing, 2006.
[4] A. Bregman, Auditory Scene Analysis, Cambridge: MIT
Press, 1990.
[5] M. Elhilali, et al., “Interaction between Attention and
Bottom-Up Saliency Mediates the Representation of
Foreground and Background in an Auditory Scene,”
PLoS Biology, Volume 7, Issue 6, e1000129, pp. 1-14,
June 2009.
[6] V. D. Delmotte, Computational Auditory Saliency, Ph.D.
Thesis, Georgia Institute of Technology, Atlanta, 2012.
[7] S. H. Chon, Timbre Saliency, The Attention-Capturing
Quality, Ph.D. Thesis, McGill University, Montreal,
2013.
[8] B. Moore, An Introduction to the Psychology ofHearing: Sixth Edition, Leiden, Netherlands: Brill, 2013.
[9] W. A. Sethares, Tuning Timbre Spectrum Scale: Second
Edition, London, UK: Springer-Verlag, 2005.
[10] I. Peretz et al., “Cortical Deafness to Dissonance,” Brain,
vol 124, pp. 928-940, 2001.
[11] D. Evans, Emotion: A Very Short Introduction, Oxford,
UK: Oxford University Press, 2003.
[12] G. Kramer, Auditory Display, Reading: Addison-Wesley,
1994.

ICAD 2015 - 235