EEG Steady State Synchrony Patterns Sonification
Teruaki Kaniwa∗ , Hiroko Terasawa∗† Masaki Matsubara∗ , Tomasz M. Rutkowski∗‡ and Shoji Makino∗∗ University of Tsukuba, Ibaraki, Japan

E-mail: kaniwa@mmlab.cs.tsukuba.ac.jp† JST-PRESTO, Tokyo, Japan
E-mail: terasawa@tara.tsukuba.ac.jp‡ RIKEN Brain Science Institute, Japan
E-mail: tomek@tara.tsukuba.ac.jp

Abstract—This paper describes an application of a multichannel EEG sonification approach. We present results obtained witha multichannel-sonification method tested with steady-state EEG
responses. We elucidate brain synchrony patterns in an auditory
domain with utilization of the EEG coherence measure. The
transitions in the synchrony patterns are represented as timbre(i.e., spectro-temporal) deviation and as spatial movement of
the sound cluster. Our final sonification evaluation experimentwith six subjects confirms the validity of the proposed brain
synchrony-elucidation approach.

I. I NTRODUCTION
Data sonification is a method of converting data into nonspeech audio and to observe characteristics of the data throughhearing [1], [2], [3]. Most of the previously proposed studies
with EEG data sonification aim for a clinical medicine application, such as diagnosing or comprehending the abnormal
cerebral function, which occurs in epilepsy [4], [5], [6], [7],[8].
In this study, we apply sonification to EEG steady-state
responses (SSR) [9], which are often used in brain-computerinterface (BCI) technologies [10]. SSR is a brain activity
phenomenon such that when a periodic stimulation is presented, EEG signals show a periodic pattern of the exact same
frequency [9]. The mechanism of SSR is still not clear inneuroscience. The purpose of this study is to elucidate, using
sonification, the process of an occurrence and propagation of
SSR that is captured by scalp-surface EEG. In this report, we
focus on the “synchrony” of SSR and propose the sonification
method reflecting the degree of a spectral correlation between
two EEG channels by using coherence analysis that represents
the similarity between time-series data. Lastly, the sonification
is evaluated by users in order to verify the effectiveness of theauditory representation of the coherence analysis.

overtone mapping [11]. The final outcome is a four-channel
surround audio representation of the EEG coherence values.
A. EEG Measurement Experimental Protocol
The EEG recordings for the proposed sonification experiments were conducted in the Laboratory for Adaptive Brain
Signal Processing, RIKEN BSI, Japan. The experiments were
designed and performed following the institutional ethicalcommittee guidelines for experiments with human subjects.
The experimental protocol was explained in detail to thesubjects who voluntarily agreed to participate by signing
consent forms. We performed the experiments using active
EEG electrodes connecting to BIOSEMI activTWO amplifierswith electrodes connects at the 𝐹 1, 𝐹 2, 𝑃 1, and 𝑃 2 sites
according to the international 10/10 system. The sampling
frequency was 2048 Hz.

Stimuli

Coherence Analysis (MATLAB)

EEG
Coh12

Coh13

OSC communication
Hearing

Sonification (SuperCollider)II. M ETHODSIn this paper we describe the EEG sonification method
toward the understanding of EEG SSR coherence patterns.
Figure 1 shows our sonification framework. In the firststage, EEG coherence analysis is performed in MATLAB,
using the pre-measured EEG signals that show the SSR. Using
the OpenSound Control (OSC) protocol, the results of theEEG coherence analysis are transmitted from MATLAB to
SuperCollider. With the SuperCollider software package, we
performed the sonification of the coherence analysis, using the

Coh n-1n

．
．．

．
．．

#n

#n

#n

．
．
．

．
．
．

．
．
．
#3

#3

．
．．#3

#2

#2

#2

#1

#1

#1

Fig. 1. The framework of EEG SSR sonification.

B. Steady-State Visual Potential Sonification Based on Pairwise Electrode Coherence Analysis

𝐺𝑖 (𝑓 ) = 𝑆𝑖 (𝑓 )𝑆𝑖∗ (𝑓 ),

20

15

10

amplitude [ V]

5

0

−5

−10

−15

−20

−25
0

2.5

3

3.5

4

50
40
30

(1)

20
0

20

40

60
[Hz]

80

100

120

80

100

120

80

100

120

(2)
stimuli(3)

An example of four EEG channels carrying SSVEP response is presented in Fig. 2, while the Fourier power spectraof the pre-stimuli, stimuli, and post-stimuli periods are depicted in Fig. 3. The results of coherence analysis conducted
for 1 s-long EEG periods with 500 ms overlap are presented
in Fig. 4.
C. Overtone Mapping

60
50
[dB]

2
0 ≤ 𝛾𝑥𝑦≤ 1.

2

pre−stimuli

40
30
20
0

20

40

60
[Hz]
post−stimuli

60
50
[dB]

2

∣𝐺𝑖𝑗 (𝑓 )∣𝐺𝑖𝑖 (𝑓 )𝐺𝑗𝑗 (𝑓 )

1.5

60

where ∗ is a complex conjugate. The coherence estimate forthe two channels 𝑖 and 𝑗 is calculated as in:
2𝛾𝑖𝑗(𝑓 ) =

1

Fig. 2. The single EEG channel with SSVEP response. The visual stimuli waspresented in a range of 0.5 − 2.5 seconds.

and the cPSD for a pair of channels 𝑖 and 𝑗 is:
𝐺𝑖𝑗 (𝑓 ) = 𝑆𝑖 (𝑓 )𝑆𝑗∗ (𝑓 ),0.5time [s][dB]

As introduced earlier, the steady-state response is a periodic
brain-activity pattern evoked by the presentation of periodic
stimulation [9]. When visually flashing stimuli are used, theresponse is called a steady-state visually evoked potential(SSVEP), and is often applied for brain-computer interfaces(BCI). However, the mechanism of SSR is still not clear
in neuroscience. Thus we aim at spatial sonification of thephenomenon to help scientists to elucidate it.
Coherence is a quantitative measure of the spectral consistency between two signals. The coherence measured between
two EEG electrode signals provides a measure of mutual
influence or “synchrony.” The magnitudes of the estimatedinfluences in complex, non-linear dynamic systems can bequite different at different frequencies or different spatial
scales [12]. Therefore, sonification of pair wise coherence
dynamics in multichannel EEG recordings with known steadystate responses can bring new value to spatial brain-activity
analysis compared with classical spectral features [13]. The
range of coherence value is from 0 to 1, and its value reflects
the degree of spectral correlation. The paper by Kaminski [14]clearly indicated that using coherence for EEG data processing
is effective in analyzing a correlation within the pairs of EEG
channels.
Coherence is calculated with the auto-power spectral density(aPSD) and the cross-spectral density (cPSD) between twotime series. The single 𝑖𝑡ℎ EEG channel signal 𝑠𝑖 (𝑡) yields thecomplex Fourier transform 𝑆𝑖 (𝑓 ). The aPSD for the channel
𝑖 is obtained as follows:

One channel of raw EEG data with SSVEP response in a range of 0.5−2.5
25

40
30
20

We previously proposed a sound synthesis method called
overtone mapping in which the fluctuation of multichannel
EEG data are assigned to time-varying amplitudes of overtones
from a harmonic series, generating “an auditory gestalt” fromcorrelated time-series data [11]. We used this technique in thisstudy and assigned the coherence between two EEG channels
to amplitudes of a harmonic series. The basic algorithm of the

0

20

40

60
[Hz]

Fig. 3. The power Fourier spectra of the single EEG channel with SSVEP
response as shown in Fig. 2 divided into three panels. The top and bottom
panels represent EEG power spectral without SSVEP for pre- and post-stimuli
periods respectively. The middle panel depicts the EEG power spectrum with
SSVEP response with two distinct peaks at 20 and 40 Hz.

III. SSR DATA AND ITS SONIFICATION
coh(1,2)

coh(1,3)coh(1,4)

1

1

1

0.8

0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.2
0
0

pre
stim
post

0.4
pre
stim
post

0.2
50

We used EEG data of four channels containing SSVEP
obtained by LED flashing for sonification in this study. The
length of the data is approximately 10 seconds and SSVEP occurs between approximately 0.5 and 6 seconds. We calculated
4 pairs of coherence from the data.
The fundamental frequency and Musical Instrument Digital
Interface (MIDI) note number of the harmonic series corresponding to each coherence pair is shown in Table 1. The
synthesized sounds are played in the loudspeaker setting as
shown in Fig. 5. The subscript number for coherence (coh)represents a combination of EEG channels; e.g., speaker channel #1 plays back the synthesized sound from the coherence
between EEG channels 1 and 2, and so on. We designed thefundamental frequencies to sound in harmony when multiple
sounds occur at the same time in order to provide comfort andfamiliarity to the listeners.

0
0

pre
stim
post

0.2
50

0
0

coh(2,3)50
coh(2,4)

1

1

0.8

0.8

0.6

0.6

0.4

0.4
pre
stim
post

0.2
0
0

pre
stim
post

0.2
50

0
0

50
coh(3,4)1
0.8

TABLE IT HE FUNDAMENTAL FREQUENCY AND MIDI NOTE NUMBER

0.6

OF THE COLUMN OF OVERTONE CORRESPONDED TO EACH
COHERENCE

0.4
0.2
0
0

pre
stim
post

Coherence
coh12
coh13
coh24
coh34

50

Fig. 4. The results of pairwise-coherence analysis as shown in Eq. (3) for
all four pairs of the analyzed EEG signals as in Fig. 2. The results of theconference analysis is easier to classify already viable when compared with
spectral representations as shown in Fig. 3. The two distinct “hills” at 20
and 40 Hz are related to the stimuli (solid red lines), while the dotted anddashed lines are from pre- and post-stimuli intervals without coherence inthose frequency ranges, respectively.

Fundamental frequency [Hz]
130
164
195
233

MIDI note No.
48
52
55
58

Coh12
１

r = 1.5m

synthesis is as follows. First, we determine the fundamental
frequency then generate a harmonic series of sine wave that
contains as many harmonics as frequency bins of coherence.
Coherence is assigned to amplitudes of overtones, that is,
the fluctuation of the nth bin of coherence is used as theamplitude of 𝑛𝑡ℎ overtone. When a frequency bin from thecoherence array has a high value, the overtone assigned tothat frequency bin sounds loud. When a frequency bin fromthe coherence array has a low value, the overtone assigned tothat frequency bin sounds quiet. To summarize, the fluctuationof the coherence array delivers timbre change. We applied
the above method to the sound synthesis in SuperCollider by
analyzing the results in MATLAB with open sound control(OSC) protocol. In MATLAB, we calculated the coherence
between specific combinations of EEG channels, which arephysiologically well connected. We converted the coherence
into amplitudes of overtones as the following equation:2

𝐴 = 10𝛼(𝛾𝑥𝑦 (𝑓 )−1)/20 .

(4)

In the above equation, 𝐴 represents amplitude of overtone
2(𝑓 ) represents coherence. Since the coherence valueand 𝛾𝑥𝑦
ranges from 0 to 1, the amplitude value 𝐴 ranges from -𝛼 to0.

θ = 90°
Coh13

Coh24

２

３

４

Coh34
Fig. 5. The environment of hearing the sound.

IV. E VALUATION EXPERIMENTS OVERVIEW
We conducted two evaluation experiments for the proposed
sonification method: The task for Experiment 1 was to detect
the presence of SSVEP; the task for Experiment 2 is to identify
the direction of SSVEP propagation. Our subjects were fourmen and two women in their early twenties. All the subjects
were healthy and did not have any clinical histories in hearing
andspeaking. They participated in the experiment upon theconsent of the condition of the experiment. The experiment
took place in the studio at the Life Science Center of Tsukuba
Advanced Research Alliance.

Results of the sonification evaluation experiment
1

0.9

0.8

V. E VALUATION EXPERIMENT 1 : SSVEP DETECTION

0.6
coherence

The purpose of this experiment was to verify if the subjects
could detect the presence of SSVEP by hearing the sonified
sound. The experiment consisted of the listening tests and aquestionnaire.

0.7

0.5

0.4

A. Procedure
1) Listening test: In the listening test, subjects were asked
to detect and report the onset and offset time of the SSVEP
response period using a Graphical User Interface (GUI) witha stopwatch function while listening to the sound. Before
the experiment, subjects practiced understanding the sonified
representation of SSVEP response using the sounds that did
not appear in the experiment. The sonification sound for theexperiment session was generated from the data described in
Section III. The sound was repeated three times, and the total
length was approximately 35 seconds.
2) Questionnaire survey: After the listening test, the subjects answered a questionnaire survey with the following
questions about comfort and comprehension:
A. How comfortable was listening to the sonification sound?
B. How comprehensible was detecting the presence of
SSVEP?
The evaluation was done with a 5-point scale, where point 1
corresponded to “bad” and 5 corresponded to “good”.

0.3

0.2

0.1

0
0

5

10

15

20

25

30

35

time [s]Fig. 6. The reported onset and offset times for Experiment 1. Solid lines:
individual response; dash-dotted lines: mean onset and offset times across four
subjects. The waveform plot shows the mean of the four coherence arrays.
TABLE IIQ UESTIONNAIRE RESULTS FROM EXPERIMENT 1:
THE MEAN SCORES FOR COMFORT AND
COMPREHENSION ON A 5- POINT SCALE (1: BAD ,5: GOOD )Questionnaire
Mean score

A(Comfort)
4.3

B(Comprehension)
4.5

B. Results
The summary of the reported start and end times of the
SSVEP response period is provided in Fig. 6. Short, dashedlines represent the mean value of the reported onset and offset
times from all the subjects, and the solid lines individually
represent the SSVEP response period reported by each subject.
The wave plot provides the mean value of the four coherences.
All the subjects detected the periods where the mean value offour coherences was the highest, and the variance of the onsetoffset times between subjects was very small. All the subjects
were able to detect a period in which a strong synchrony in thewhole area of the brain occurred, and therefore were able toidentify the SSVEP response by listening to the sonification.
The coherence values identified as the start and end of the
SSVEP were asymmetrical: The majority of subjects reported
the onset of SSVEP at 0.5 − 0.6 mean coherence value,
while the offset of SSVEP was reported at 0.3 − 0.2 mean
coherence value. The possible reasons for this asymmetry are
the adaptation to the sound (usually the threshold of hearing
takes a lower value, when the amplitude of a sound is gradually
decreased, compared to when the amplitude is increased), andthe steepness of the coherence change of the original data

(i.e., more gradual in the rising phase in contrast with a sharp
decline in the decreasing phase.)
The results of the questionnaire survey are shown in Table
2, with the mean scores for questions A (comfort) and B (comprehension). Both mean scores are over 4, which corresponds
to the judgment of “slightly good”, showing that the presenceof SSVEP is well represented by the proposed sonification
method.
VI. E VALUATION EXPERIMENT 2: P ROPAGATION OF
SSVEP
The purpose of this experiment was to verify whether
subjects could distinguish the process of the occurrence andpropagation of SSVEP by hearing the sonified sound. The
experiment consists of the listening tests and the questionnaire.
A. Procedure
1) Listening test: For the listening test, we presented thesonification in stereo and four-loudspeaker settings, and asked
the subjects to describe the direction of the SSR propagation.

In the stereo-setting listening test, the sonification of twocoherence arrays was presented. By using only three EEG
channels out of four, we obtained two coherence arrays (e.g.,coherences between channel #1 - #2 and #1 - #3, when leaving
channel #4.) A session consisted of 32 trials with a threesecond silence between trials, where each trial presented astereo sonification based on two coherence arrays.
In the four-loudspeaker listening test, subjects listened tofour-channel sonification based on four coherence arrays withall the EEG channels. The task of the subjects was similar
to the stereo listening test, in which the subjects answered
the direction of the SSVEP-response propagation. A session
consisted of 24 trials with a 3-second silence between trials.
The direction of the SSVEP response between loudspeakerchannels was varied for every trial by taking different EEG
channel combinations.
In the experiment, subjects listened to the sonification, andanswered the direction of the propagation by selecting theorder of which loudspeaker sounded earlier compared to theothers. Before the experiment, subjects practiced the direction
judgment of the SSVEP-response propagation.
2) Questionnaire survey: After the listening test, the subjects answered a questionnaire with the following questions
about comfort and comprehension:
A. How comfortable was listening to the sonification sound?
B. How comprehensible was the propagation of SSVEP?
The evaluation was done with a 5-point scale, where point 1
corresponds to “bad” and 5 corresponds to “good”.
B. Results
Table 3 provides the chance level and the mean value ofthe accuracy in identifying the direction of the propagation inExperiment 2, for both the stereo and four-channel settings.
In both settings, the mean value of the accuracy is better thanthe chance level. And the chance level of the stereo setting
is half the value of the four-channel setting. By contrast, thedifference in the accuracy of the subjects between both settings
is only about 20%. We are considering the possibility that
this difference in accuracy is due to the design of sound or
the attention problem across multiple channels. These results
confirmed that the subjects can distinguish the process of theoccurrence and propagation of SSVEP by hearing the sonified
sound.
Table 4 shows the mean score for the questionnaire forExperiment 2. The mean score on comfort is at the same level
as Experiment 1, but the mean score on comprehension wasmuch lower, at “slightly good” level. Even the subject who performed at 100% accuracy throughout Experiment 2 provided“slightly unintelligible” rating as well. Figure 7 shows thisdiscrepancy between the actual accuracy (objective judgment)
and the comprehensibility score (subjective judgment), showing that these two measures did not directly correspond. Many
subjects who performed well on the identification reported thetask being difficult.

TABLE IIIT HE CHANCE LEVEL AND MEAN
ACCURACY SCORE FOR
EXPERIMENT 2. T HE SUBJECTSANSWERED THE DIRECTION OF
THE SSVEP PROPAGATION

Setup
Chance level
Mean accuracy

stereo
8%
87%

4ch
4%
69%

TABLE IV
Q UESTIONNAIRE RESULTS FROM EXPERIMENT 2: THE
MEAN SCORES FOR COMFORT AND COMPREHENSION ON
A 5- POINT SCALE (1: BAD , 5: GOOD )Questionnaire
The evaluation points

A(comfort)
4.5

B(comprehension)
2.7

VII. C ONCLUSIONS AND FUTURE WORKIn the presented project we have focused on the synchronypattern analysis of the brain SSR by sonification. We proposed the sonification method as a representation tool thatreflected the degree of the spectral correlation within pairs of
EEG channels. We used the power-coherence analysis as ameasure for the synchrony, which allowed us to capture thespectral similarity properties of the multichannel EEG data.
The transitions in the synchrony patterns are represented as
timbre (i.e., spectro-temporal) deviation and as spatial motionof the sound cluster.
To evaluate the proposed method, we conducted the experiments with human listeners. The listeners were asked to
identify the SSR onsets and offsets from the multichannel sonified EEG. Our results showed that the proposed sonification
approach based on the pairwise EEG coherence allowed forvery accurate identification of the steady-state responses.
In future work we plan to improve the accuracy of our
method in order to further elucidate the propagation patternsof steady-state responses. We also plan to construct a spatial
three-dimensional “sound field” to simultaneously analyze a

Fig. 7. The relationship between the comprehensibility score of the questionnaire and the accuracy of the identification in the case of the four-channel
setting.

larger number of EEG channels carrying steady-state response,with an improved sound design that helps the listeners to better
understand the propagation from both objective and subjective
perspectives.
The presented research results are a step forward in developing multichannel brain synchrony analysis tools.
R EFERENCES[1] S. Barrass and G. Kramer, “Using sonification,” Multimedia Syst., vol. 7,
pp. 23–31, Jan. 1999.
[2] T. Hermann, A. Hunt, and J.G.Neuhoff, The Sonification Handbook.
Germany: Logos Publishing House, 1st ed., Nov. 2011.
[3] “International community for auditory display,” http://www.icad.org/(2012/08/04).
[4] G. Baier and T. Hermann, “Sonification: Listen to brain activity,” inMusic that works - Contributions of biology, neurophysiology, psychology, sociology, medicine and musicology (R. Haas and V. Brandes, eds.),
ch. 4, Wien, New York: Springer, March 2009.
[5] G. Baier, T. Hermann, and U. Stephani, “Event-based sonification of eeg
rhythms in real time,” Clinical Neurophysiology, vol. 118, pp. 1377–
1386, June 2007.
[6] T. Hermann, G. Baier, U. Stephani, and H. Ritter, “Vocal sonification ofpathologic EEG features,” in Proceedings of the International Conference on Auditory Display (ICAD 2006) (T. Stockman, ed.), (London,
U.K.), pp. 158–163, International Community for Auditory Display(ICAD), Department of Computer Science, Queen Mary, University of
London, June 2006.
[7] T. Hermann, P. Meinicke, H. Bekel, H. Ritter, H. Müller, and S. Weiss,“Sonification for EEG Data Analysis,” in Proc. of the Int. Conf. on
Auditory Display, (Kyoto, Japan), pp. 37–41, International Community
for Auditory Display (ICAD), July 2002.
[8] T. Hermann and H. Ritter, “Listen to your data: Model-based sonification
for data analysis,” in Advances in intelligent computing and multimedia
systems (G. E. Lasker, ed.), (Baden-Baden, Germany), pp. 189–194, Int.
Inst. for Advanced Studies in System research and cybernetics, 08 1999.
[9] E. Niedermeyer and F. L. Da Silva, eds., Electroencephalography:
Basic Principles, Clinical Applications, and Related Fields. Lippincott
Williams & Wilkins, 5th ed., 2004.
[10] T. M. Rutkowski, F. Vialatte, A. Cichocki, D. Mandic, and A. K.
Barros, Knowledge-Based Intelligent Information and Engineering Systems, vol. 4253 of Lecture Notes in Artificial Intelligence, ch. Auditory
Feedback for Brain Computer Interface Management - An EEG Data
Sonification Approach, pp. 1232–1239. Springer Berlin & Heidelberg,
2006.
[11] H. Terasawa, J. Parvizi, and C. Chafe, “Sonifying ECOG seizure
data with overtone mapping: A strategy for creating auditory gestalt
from correlated multichannel data,” in Proceedings of the International
Conference on Auditory Display (ICAD 2012), (USA), ICAD, ICAD,
June 2012.
[12] P. L. Nunez, R. Srinivasan, A. F. Westdorp, R. S. Wijesinghe, D. M.
Tucker, R. B. Silberstein, and P. J. Cadusch, “EEG coherency: I:
statistics, reference electrode, volume conduction, laplacians, corticalimaging, and interpretation at multiple scales,” Electroencephalography
and Clinical Neurophysiology, vol. 103, no. 5, pp. 499 – 515, 1997.
[13] T. M. Rutkowski, A. Cichocki, and D. Mandic, “Information fusion
for perceptual feedback: A brain activity sonification approach,” in
Signal Processing Techniques for Knowledge Extraction and Information
Fusion (D. Mandic, M. Golz, A. Kuh, D. Obradovic, and T. Tanaka,eds.), pp. 261–273, Springer US, 2008.
[14] M. Kaminski, “Determination of transmission patterns in multichannel
data,” Philosophical Transactions of the Royal Society B: Biological
Sciences, vol. 360, no. 1457, pp. 947–952, 2005.