The 20th International Conference on Auditory Display (ICAD-2014)

June 22-25, 2014, New York, USA

ELECTROENCEPHALOGRAM STEADY STATE RESPONSE SONIFICATION
FOCUSED ON THE SPATIAL AND TEMPORAL PROPERTIES
Teruaki Kaniwa1, Hiroko Terasawa1,2, Masaki Matsubara1, Shoji Makino1, Tomasz M. Rutkowski1
1

University of Tsukuba
1-1-1, Tenno-dai, Tsukuba, Ibaraki, 305-8577, Japan
2
JST, PRESTO
7, Gobancho, Chiyodaku, Tokyo, 102-0076, Japan
kaniwa@mmlab.cs.tsukuba.ac.jp{terasawa, masaki}@slis.tsukuba.ac.jp{tomek, makino}@tara.tsukuba.ac.jp
ABSTRACTThis paper describes a sonification approach of multichannel
electroencephalogram (EEG) steady-state responses (SSR).
The main purpose of this study is to investigate the possibilityof sonification as an analytic tool for SSR. The proposed
sonification approach aims to observe the spatial property(i.e. location of strong brain activities) and temporal property(i.e. synchrony of wave forms across channels) of brain activity.
We expect to obtain useful information on brain activity
locations and their dynamic transitions by taking advantage ofspatial sound with multichannel loudspeakers that represent
EEG measurement positions, while expressing the temporal
property of multiple EEG channels with timbre by integrating
respective auditory streams. Our final sonification evaluation
experiment suggests the validity of the proposed approach.
1. INTRODUCTION
We report a sonification of auditory-evoked steady-state
responses (ASSR). ASSR is a type of the steady-state brain
responses (SSR) [1]. Most of the previously proposed studies
with electroencephalogram (EEG) data sonification aim for aclinical medicine application, such as diagnosing orcomprehending the abnormal cerebral function, which occurs inepilepsy [2-6]. In this study, we investigate EEG sonification of
more commonly observed brain activity phenomenon, such as
SSR. This study extends the sonification of SSR [7-9] with
better expression of data properties.
Unlike epilepsy, SSR is observed among many healthypeople. SSR is a brain activity phenomenon such that when aperiodic stimulation is presented, EEG signals show a periodic
pattern of the exact same frequency at many parts of the brain.
While the most significant application of ASSR is the screening
test for newborn’s hearing, developing brain-computer interface(BCI) using ASSR is also regarded as promising [10].
The mechanism of SSR is still under investigation.
However, in general, a response to an external stimuli occurs
from a certain location and that response eventually propagatesto all over the brain: For example, when an auditory stimulus isThis work is licensed under Creative Commons Attribution– Non Commercial (unported, v3.0) License. The full terms of theLicense are available at http://creativecommons.org/licenses/by-nc/3.0/.

presented, its initial response starts on auditory pathway andquickly propagates to the other parts of the brain [11]. Taking
arithmetic mean of multiple EEG measurements is a typical way
to detect SSR, but in order to understand the mechanism of SSR,
observing subtle changes from single-trial data is expectedly
more effective. With this goal, we propose a sonification
method for single-trial SSR data that can represent the locationand the temporal transition of brain activity.
In order to analyze the mechanism of SSR, we focus on
the spatial property (i.e. location of strong brain activities) andthe temporal property (i.e. synchrony of wave forms acrosschannels), so that we could understand where and when the SSR
start and propagate in the brain. To represent the spatial property,
we map the spatial arrangement of the brain activity to thesound spatialization, using a multi-channel, three-dimensional
loudspeaker setting.
Moreover, to represent the temporal property of data, we
map the temporal patterns of data onto the temporal pattern of
the synthesized sounds (i.e. patterns of onset, sustain,
modulation, and decay). Because of the auditory streaming
effect, the sounds of a similar temporal pattern fuse together,and the sounds of different temporal patterns segregate eachother [12]. The synchrony across multiple data channel (i.e. data
sharing a same temporal pattern across channels) can beperceived as the fusion of sounds with a composite timbre.
Detection of neural responses (including SSR) usuallyrequires repeated measurements and detailed statistical analysis,taking long time to unveil the interesting features in the neuralactivities. We expect this sonification method provides aflexible framework for single-trial EEG data analysis for bothexperimenter and subject. Such a framework will help thedevelopment of the real-time data analysis and the applications
such as auditory biofeedback and auditory BCI.
2. EEG STEADY STATE RESPONSE SONIFICATIONMETHOD FOCUSED ON THE SPATIAL AND
TEMPORAL PROPERTYThis work focuses on the “synchrony” property of SSR suchthat a response occurs in the same frequency band as a stimulus
frequency. We propose a sonification method using theamplitude change of a frequency bin in a power spectrum
corresponding to the stimulus frequency. Fig. 1 shows theThe 20th International Conference on Auditory Display (ICAD-2014)

proposed framework. First, we conduct short-time Fouriertransform (STFT) for input EEG data, and then we extract the
spectral information corresponding to the stimulus frequency.
Then we normalize spectral amplitude to emphasize aninteresting data property: we propose two normalization
methods, one to focus on the spatial property, and another tofocus on the temporal property. The transitions of the EEG
signals are sonified as mixture of sounds with each of themcorresponding to a single EEG channel: each component’s
sound can exhibit the transition from textured density to thetone-like quality of the timbre as the brain activity becomesstronger. And we employ three-dimensional sound as the soundreproduction environment. By associating timbre and the soundimage position with EEG measurement position, theinformation of strength and location of brain activity can beobtained.
2.1. STFT and extracting stimulus frequency bin
First, we conduct STFT for input EEG data, and calculate thespectrogram by taking its power. Then we extract the spectral
information corresponding to the stimulus frequency from
calculated spectrogram, and we use its spectral amplitude as aparameter for the sound synthesis. By presenting only the
information that is associated with the SSR, the proposed
sonification method decreases the effort on users to recognize
SSR. The power spectrum of the stimulus frequency bin (k0) at
the ith EEG channel is obtained as follows:

EEGnXi,k0 [n] =∑ x [m]ei" 2π %
− j$
'k0 m
# N &

...

STFT

Extraction of stimulus
frequency bin(1),

m=n−N+1

	
  	
  	
  	
 

where N is the window length (0.1 ms) of STFT and xi is the ith
EEG channel signal.
2.2. Normalization according to data properties
The normalization is to deform data based on a certain rule to
make it easier to use. In this study, we propose two
normalization methods according to data properties, one tofocus on the spatial property (normalization across EEG
channels), and another to focus on the temporal property(normalization within EEG channel). We aim to make aninteresting data property clear by normalizing spectral
amplitude. In addition, since the value of spectral amplitude islimited to the range from 0 to 1, the association with soundsynthesis parameter and data feature quantity is facilitated.
2.2.1. Normalization across EEG channels
The normalization across EEG channels is performed by
dividing the power spectrum of each channel in the time
domain using the maximum amplitude value of the stimulus
frequency bin across EEG channels. The spectral value after thenormalization is obtained as follows:

X! ai [n] = Xi,k0 [n] / X max ,EEG Processing (MATLAB)...

Stimuli

June 22-25, 2014, New York, USA(2)where Xmax is the maximum amplitude value of the stimulus
frequency bin across EEG channels. In this normalization, thepower ratio across EEG channels is conserved and thisrelationship is reflected in the sound. Therefore, the user canfocus on the spatial property of data and identify the EEG
channel in which strong brain activity is happening.
2.2.2. Normalization within EEG channel

Normalization1

according to data property

Spatiality

1

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

Temporalit\

00

OSC Communication

Listening

Sound Synthesis (SuperCollider)
Temporality

6SDWLDOLW\

FM

$0ٔ)0

#2

#2

#n

...

#1

...

#1

#n

Figure 1: The framework of proposed SSR sonification method.

The normalization within EEG channel is performed by
dividing the power spectrum of each channel in the time
domain using the maximum amplitude value of the stimulus
frequency bin in each EEG channel. The spectral value after thenormalization is obtained as follows:

X! wi [n] = Xi,k0 [n] / Xi,max ,(3)

where Xi,,max is the maximum amplitude value of the stimulus
frequency bin at ith EEG channel. In this normalization, it ispossible to observe the time change of EEG signals regardlessof the magnitude of spectral amplitude. Therefore, the user canfocus on the temporal property of data and identify thesynchrony pattern across EEG channels.

The 20th International Conference on Auditory Display (ICAD-2014)

2.3. SOUND SYNTHESIS METHODSingle-channel sonified sound is generated from single EEG
channel data according to the sound synthesis method described
below. For multichannel EEG data, the proposed method
generates multichannel sonified sound according to the numberof EEG channels and reproduces them respectively frommultichannel loudspeakers associated with EEG measurement
position. A user listens to the auditory stream that is spatiallyformed by these sounds.
2.3.1. Sonification focusing on the spatial property
In sonification focusing on the spatial property, the power
spectrum value normalized across EEG channels is mapped tothe amplitude of pulse wave and pulse period. Table 1 shows
the relationship of the spectral value and sound synthesis
parameters. A pulse wave has a characteristic that it isperceived as an interval “length” if pulse interval is long (lowfrequency), but on the other hand it is perceived as a "pitch" ifpulse interval is short (high frequency). From such difference ofthe perception, we aim to perceive the occurrence of SSR and
the intensity of brain activity intuitively.
Table 1. The mapping of sonification focusing on the spatial
property.
EEG parameterSound Synthesis Parameter

Power spectral [0, 1]

Pulse frequency [10Hz, 400Hz]
Gain [-20dB, 0dB]

2.3.1. Sonification focusing on the temporal property
In sonification focusing on the temporal property, the power
spectrum normalized within the EEG channel is mapped to only
the period of pulse wave. Table 2 shows the relationship ofspectral value and sound synthesis parameter. It is possible toobserve the time change of EEG signals regardless of themagnitude of spectral amplitude because of the constant soundvolume. We aim to perceive the synchrony pattern and the
propagation process of SSR.
Table 2. The mapping of sonification focusing on the temporal
property.
EEG parameterSound Synthesis Parameter
Power spectral [0, 1]

Pulse frequency [80Hz, 400Hz]

3. EVALUATION TEST OF PROPOSED SONIFICATIONMETHOD3.1. Dataset used for the sonification
In order to generate sonification sounds, we used single-trial
EEG data of four persons that were measured under theconditions shown in Table 3 and 4. This dataset was acquiredpart of the study on ASSR [13]. The online EEG experiments
were conducted in accordance with The World Medical

June 22-25, 2014, New York, USA

Association Declaration of Helsinki - Ethical Principles for
Medical Research Involving Human Subjects. The proceduresfor the EEG recordings were approved by the EthicalCommittee of the Faculty of Engineering, Information and
Systems at the University of Tsukuba, Tsukuba, Japan. The
experimental protocol was explained in detail to the subjects
who agreed voluntarily to participate by signing the consent
forms. During the EEG recording, they were presented
modulated sine waves as auditory stimuli for ASSR using
headphones. The acquired EEG data consist of sixteen EEG
channels according to the international 10-20 system [14].
3.2. Overview of the evaluation test
The evaluation test of the proposed sonification method was
conducted in a studio at the University of Tsukuba, TARA
Center. The procedures were approved by the EthicalCommittee of the Faculty of Engineering, Information and
Systems at the University of Tsukuba, Tsukuba, Japan. We
conducted two types of experiments to evaluate the proposed
sonification method: The first task is to identify the location ofstrong brain activity; the second task is to rate the strength ofsynchronization across EEG channels. Subjects who participatedin the first task were 15 people (4 females and 11 males, averageage being 23.6 years old), and subjects who participated in thesecond task were 15 people (3 females and 12 males of the early20s). They were all healthy people with no clinical history
related to speech and hearing.
Fig. 2 shows the association between EEG measurement
positions and the arrangement of loudspeakers. We present
sonified sound with 15 channel loudspeakers associated with
EEG measurement position to obtain useful information on
brain activity locations. The signal measured by the electrode on‘POz’ is not used for sonification because of the limited numberof playback channels. The loudspeakers were set in two levels:
eight at the middle level (about the height of the listener’s ear)and the ceiling level. The listener sat in the center of the room
and was able to turn the head, but not allowed to stand up or
walk around.
Table 3. EEG recording conditions.
Number of subjects
4g.USBamp with g.SAHARA dry
EEG amplifier
active electrodes
Electrode
F5, F1, F2, F6, FC3, FC4, C5, Cz,arrangement
C6, CP3, CP4, P5, P1, P2, P6, POz
Sampling frequency
512 HzStimuli
Modulated tone of a sine wave
Stimulus presentation
10 sectime
Table 4. The number of trials for one subject
Types of modulated tone
Number of trials
Carrier frequency
Modulated[Hz]frequency [Hz]
500
40
10
500
77
10
1000
40
10
1000
80
10

The 20th International Conference on Auditory Display (ICAD-2014)

June 22-25, 2014, New York, USA

3.3.1. Experimental method



























TaskⅠ: Identification of the presence of localization
We first analyzed the strength of the localization patterns using
RMSE (Root Mean Square Error) of the power at stimulus
frequency bin across EEG channels, and selected 30 data to besonified: 15 data with strongest localization patterns and 15 datawith weakest localization patterns. We then synthesized 30
sounds with the dataset, and presented them to our listeners in arandom order. Listeners evaluated the presence or absence of
localization. We finally determined if the listeners judgments
were correct or not, by comparing them with our analysis ofstrong/weak localization patterns.





(a)
















TaskⅡ: Identification of the localization channel
For this task, we used the same 15 data with strongest
localization patterns from the task I, although the soundplayback locations of the EEG channels were shuffled. The
sonified sounds of the 15 data were presented to the listeners ina random order. Listeners judged the location where thestrongest brain activity happens, and answered speaker locations
from which outstanding sound was heard. Since the original
signal may have contained multiple channels that exhibit
similarly strong power level, the listeners were allowed toanswer up to two outstanding locations. We then determined if
the listeners answered the speaker channel that corresponded to
the EEG channel with the strongest activity.
3.3.2. Experimental results




(b)
Figure 2: The association between loudspeakers and EEG
measurement positions. (a) shows 16 EEG measurement
positions. (b) shows the arrangement of 15 loudspeakers
associated with EEG measurement position (the outer circle
corresponds the middle height, and the inner circle and thecenter speaker is at the ceiling height.)
3.3. Identification experiment of the spatial localization ofbrain activity
In the identification experiment of the spatial localization, we
use the sonification focusing on the spatial property. The
purpose of the experiment is to examine whether subjects canidentify the presence of localization and the localization channel.
Here, “localization” means that the strong brain activity appears
in a particular location. Experimental task is carried out in twostages of "Identification of the presence of localization" and "Identification of the localization channel". We examine theeffectiveness of the sonification focusing on the spatial property
by comparing the chance level and correct answer rate of thetask.

TaskⅠ: Identification of the presence of localization
Fig. 3 shows the accuracy rate in TaskⅠ. Subjects identified thepresence or absence of localization at 77% accuracy on average,
ranging from 50% to 97%. The average correct answer rate ofall subjects including 95% confidence intervals exceeded thechance level (50%). Therefore the sonification focusing on
spatial property is capable of determining the presence orabsence of localization.
TaskⅡ: Identification of the localization channel
Fig. 4 shows the accuracy rate in TaskⅡ. Subjects identified the
localization channel at 88% accuracy on average, ranging from
53% to 100%. The average correct answer rate of all subjects
including 95% confidence intervals exceeded the chance level(13%). Therefore the sonification focusing on spatial property iscapable of determining the location where the strongest brain
activity happens.

The 20th International Conference on Auditory Display (ICAD-2014)

Figure 3: The accuracy rate in TaskⅠ.

June 22-25, 2014, New York, USA

Figure 5: The analysis of the cases where a subject answered
two channels in TaskⅡ.
3.4. Pairwise comparison experiment of the strength ofsynchronization across EEG channels
In the paired comparison experiment of the strength ofsynchronization across EEG channels, we use the sonification
focusing on the temporal property. The purpose of theexperiment is to examine whether subjects can assess thedegree of synchronization across EEG channels. In thisexperiment, subject listens a pair of sonified sound and judges
the dissimilarity of the perceived strength of synchronization
across EEG channels between two sounds. We examine theeffectiveness of the sonification focusing on the temporal
property by comparing the average correlation coefficientacross EEG channels and the estimate of the judgment obtained
from the experiment.

Figure 4: The accuracy rate in Task Ⅱ.
Furthermore, we analyzed the cases where a subject
answered two channels. In these cases, we defined that theanswer is correct if the RMS of the power at the corresponding
EEG channel is ranked within the top three of the powers fromall the channels. The percentage of answers with two channels
in the task II is 33 % for all the trials. Fig. 5 shows the rate of(1) either one of the two answered channels being correct, (2)both of the answered channels being correct, and (3) both of theanswered channels being incorrect. The rate of both correct
answers was 85% on average, meaning that subjects could
determine multiple outstanding channels at once. Therefore thelisteners were capable of determining multiple locations wherestrong brain activities happened, with the sonification focusing
on spatial property.

3.4.1. Experimental method
Table 5 shows the three dataset for this experiment. Dataset A,
B, and C hold low, middle, and high RMS values, respectively.
Each set consists of four data with gradually increasingcorrelation coefficient while RMS values were fixed as much as
possible. We select them based on the average value of thecorrelation coefficients across EEG channels and the averagevalue of the RMS at stimulus frequency bin: the strong
correlation across the EEG channels means that they share asimilar temporal pattern, and the strong RMS value means thestrong ASSR.
This experiment is based on Scheffe's pairwise comparison
method [15]. Each dataset contains six pairs of sonified sound.
Subjects listen to the pairs in a random order and judge thedegree of synchronization across EEG channels with 5-point
scale shown in Table 6.
3.4.1. Experimental results
Fig. 6-8 show the results of regression analyses from thesubjective evaluation value and the correlation coefficient across
EEG channels. The coefficient of determination R2 is 0.88 on
average for all dataset, that is, the result showed high reliability.
Therefore the sonification focusing on temporal property is

The 20th International Conference on Auditory Display (ICAD-2014)

June 22-25, 2014, New York, USAcapable of determining the degree of synchronization across
EEG channels.

A

0.84
0.81
0.85
0.87

0.06
0.28
0.49
0.71

B

1.12
1.11
1.12
1.18

0.22
0.41
0.61
0.79

C

1.51
1.63
1.65
1.64

0.20
0.34
0.41
0.64

1.5

Subjective Estimation

Table 5. The dataset for identification experiment of thestrength of synchronization across EEG channels.
Average value of
Average value ofDatasetRMS
correlation coefficient

2

1

0.5

0

−0.5

−1

−1.5
0

0.2

0.4

0.6

Correlation

0.8

1

Figure 7: The result of regression analysis for dataset B.

2

Table 6. The 5-point scale of pairwise comparison
Category
Judgment
1
Earlier sound has a much stronger
synchronization
2
Earlier sound has a slightly stronger
synchronization
3
Both sounds have a similar level ofsynchronization
4
Later sound has a slightly stronger
synchronization
5
Later sound has a much stronger
synchronization

Subjective Estimation

1.5

1

0.5

0

−0.5

−1

−1.5
0
2

Subjective Estimation

0.4

0.6

Correlation

0.8

1

Figure 8: The result of regression analysis for dataset C.

1.5

4. CONCLUSIONS

1

0.5

0

−0.5

−1

−1.5
0

0.2

0.2

0.4

0.6

Correlation

0.8

1

Figure 6: The result of regression analysis for dataset A.

We proposed a method of a multichannel EEG sonification that
focused on the temporal and spatial characteristics of brain
activity. The proposed method targets at the “synchrony”
property of SSR such that a response occurs in the same
frequency band as a stimulus frequency. From this property, theamplitude change of the stimulus frequency bin in a power
spectrum was used for sonification. In addition, clarifying thedata property to be focused by the normalization, we aimed todecrease the effort on users to understand the data. In this study,
we proposed two normalization methods according to data
properties, one to focus on the spatial property, and another tofocus on the temporal property.
From the evaluation experiment, we found that listeners
successfully identified the location where strong brain activity
happened with the sonification focusing on the spatial property.
With the sonification focusing on the temporal property, theThe 20th International Conference on Auditory Display (ICAD-2014)

subjective judgment of synchrony across EEG channels wellmatched with the correlation coefficient across EEG channels,suggesting that listeners can accurately assess the degree ofsynchronization across EEG channels with sonification. These
results suggest that our attempt on sonifying the spatial andtemporal properties can help observing such properties ofsingle-trial EEG data. The future work of this research includes
simultaneous evaluation of the spatial and temporal synchrony,and comparing analysis performances and user’s effort (i.e.
cognitive load on users) between proposed method andtraditional visualization methods.
5. ACKNOWLEDGMENTS
We thank Yoshihiro Matsumoto for conducting the EEG
data measurement. We also thank our reviewers for theirvaluable suggestions on this paper. This project is supported by
PRESTO program no. xxxxxxx, Japan Science and Technology
Agency.
6. REFERENCES[1] E. G. Williams, Fourier Acoustics: Sound Radiation and
Nearfield Acoustic Holography, London, UK: Academic
Press, 1999.
[2] G. Baier, T. Hermann, S. Sahle, and U. Stephani, “Sonified
epilectic rhythms,” in Proceedings of the International
Conference on Auditory Display (ICAD2006), pp. 148–
151, London, UK, 2006.
[3] T. Hermann, G. Baier, U. Stephani, and H. Ritter, “Vocal
sonification of pathologic EEG features,” in Proceedings ofthe International Conference on Auditory Display (ICAD
2006), pp. 158–163, London, U.K., 2006.
[4] G. Baier, T. Hermann, and U. Stephani, “Multi-channel
sonification of human eeg,” in Proceedings of theInternational Conference on Auditory Display (ICAD2007),
pp. 491–496, Montreal, Canada, 2007.
[5] B. K. Min, A. Bystritsky, K. Jung, K. Fischer, Y. Zhang, L.
Maeng, S. Park, Y. Chung, F. Jolesz, and S. Yoo, “Focusedultrasound-mediated suppression of chemically-induced
acute epileptic eeg activity,” BMC Neurosci, vol. 12, no. 1,
pp. 1–12, 2011.
[6] H. Terasawa, J. Parvizi, and C. Chafe, “Sonifying ECOG
seizure data with overtone mapping: A strategy for creating
auditory gestalt from correlated multichannel data,” inProceedings of the International Conference on Auditory
Display (ICAD 2012), pp. 129–134, Atlanta, GA, USA,2012.
[7] T. M. Rutkowski, F. Vialatte, A. Cichocki, D. Mandic and
A. K. Barros, "Auditory Feedback for Brain Computer
Interface Management – An EEG Data Sonification
Approach," in Knowledge-Based Intelligent Informationand Engineering Systems, vol. 4253, pp.1232-1239,
Springer US, 2006.
[8] T. M. Rutkowski, A. Cichocki, and D. Mandic,"Information fusion for perceptual feedback: A brain
activity sonification approach," in Signal Processing
Techniques for Knowledge Extraction and Information
Fusion, pp. 261–273. Springer US, 2008.

June 22-25, 2014, New York, USA[9] T. Kaniwa, H. Terasawa, M. Matsubara, T. M. Rutkowski,
S. Makino, "EEG steady state synchrony patterns
sonification," in Signal Information Processing Association
Annual Summit and Conference (APSIPA ASC), pp. 1–6,
Paper ID 364, 2012.
[10] H. Higashi, T. M. Rutkowski, Y. Washizawa, A. Cichocki,
T. Tanaka, "EEG Auditory Steady State Responses
Classification for the Novel BCI," in Proceedings of the
33rd Annual International Conference of the IEEE EMBS,
pp. 4576–4579, Boston MA, USA, 2011.
[11] R. Draganova, B. Ross, A. Wollbrink, and C. Pantev,“Cortical Steady-State Responses to Central and Peripheral
Auditory Beats,” Cerebral Cortex, vol. 18, no. 5, pp. 11931200, 2008.
[12] A. S. Bregman, Auditory scene analysis: Hearing incomplex environments, Oxford: Clarendon Press, 1994.
[13] Y. Matsumoto, N. Nishikawa, S. Makino, T. Yamada, TM.
Rutkowski, “Auditory steady-state response stimuli based
BCI application - the optimization of the stimuli types andlengths.” in Signal Information ProcessingAssociation
Annual Summit and Conference (APSIPA ASC), pp. 1-7,
Paper ID 285, 2012.
[14] R. Oostenveld and P. Praamstra, “The five percent
electrode system for high-resolution EEG and ERP
measurements,” Clinical neurophysiology, vol. 112, no. 4,
pp. 713–719, 2001.
[15] R. S. Lehman, Statistics in the behavioral sciences: A
conceptual introduction, Pacific Grove, CA: Brooks/Cole
Publishing Company, 1995.