The Spatial Real and Virtual Sound Stimuli
Optimization for the Auditory BCI
Nozomu Nishikawa∗ , Yoshihiro Matsumoto∗ , Shoji Makino∗ , and Tomasz M. Rutkowski∗†▽∗ Multimedia Laboratory, TARA Center, University of Tsukuba, Tsukuba, Japan† RIKEN Brain Science Institute, Wako-shi, Japan

E-mail: tomek@tara.tsukuba.ac.jp (▽ corresponding author)

Abstract—The paper presents results from a project aiming
to create horizontally distributed surround sound sources andvirtual sound images as auditory BCI (aBCI) stimuli. The purpose is to create evoked brain wave response patterns depending
on attended or ignored sound directions. We propose to use amodified version of the vector based amplitude panning (VBAP)
approach to achieve the goal. The so created spatial soundstimulus system for the novel oddball aBCI paradigm allows us
to create a multi–command experimental environment with very
encouraging results reported in this paper. We also present results
showing that a modulation of the sound image depth changes also
the subject responses. Finally, we also compare the proposed
virtual sound approach with the traditional one based on realsound sources generated from the real loudspeaker directions.
The so obtained results confirm the hypothesis of the possibility
to modulate independently the brain responses to spatial types
and depths of sound sources which allows for the developmentof the novel multi-command aBCI.

I. I NTRODUCTION
In recent years, many researchers who study the brain waves
plug to the technological development of using electroencephalography (EEG) signals for engineering applications. The
research field has made a remarkable development. It is a factthat thanks to such application the disabled people could movesuccessfully a cursor or a wheelchair using their brain wave
patterns only. Thanks to the BCI technology it is expectedthat disabled people will receive the various services only bythinking or interacting with their environments [1].
The spatial auditory BCI (saBCI) is the interface that makes
use of the brain waves captured in response to spatial soundstimuli. In our research, we aim at the practical realizationand enhancement of the saBCI [2]. Figure 1 depicts our saBCI
system prototype that performs the following:∙ It creates the spatial sound stimuli in the octagonalsurround sound loudspeaker system using the modified
VBAP approach as well as the conventional real soundsources;∙ It captures and analyzes the EEG event related responses(ERP) to spatial sound stimuli;∙ Finally it classifies the responses in order to generate the
BCI application command (exemplary wheelchair in the
diagram). This paper does not include yet classification
results.
In the presented project we refer to the fact that a humanbrain given a sound stimulus generates the electric ERP pattern [3]. Attentional modulation adds a characteristic deflection

Fig. 1. Schematic drawing of the proposed auditory BCI system. The subject in
the center is surrounded by horizontally distributed loudspeakers. EEG signals
are processed by the BCI processing stages in synchrony with presented spatial
auditory stimuli.

(so called “aha response”) which could be utilized in the
saBCI paradigm development. This ERP feature depending onwhether the sound is attended (so called target) or not (nontarget) allows for further classification of subject’s choices [4],[5]. “The aha-response” to the attended target sound in comparison with the other non-target has the characteristic positive
deflection after 300ms from the onset and thus it is called “the𝑃 300 response.” An example of an averaged 𝑃 300 response(solid blue line - target), from our experiments, is depicted
in Figure 2 together with the ignored sound direction (dashed
red line - non-target). The properly classified, by a machinelearning application, 𝑃 300 response lets the BCI application
user to select a command. In this paper we only identify the
responses separability and the future research will result with

II. M ETHODS
86This section describes the stimuli generation methods, EEG
capture experimental protocol in offline saBCI mode, andthe ERP processing steps in order to elucidate and proveseparability of the brain 𝑃 300 responses for the future online
multi-command application with a classifier.

TARGET
nonTARGET

[μV]

4
2

A. VBAP Method for Generating of Spatial Virtual Sound
Sources in Application to saBCI

0

In the proposed saBCI paradigm the user receives the realand virtual sound source stimuli that are created using a MAX
environment [7]. The surround sound environment consist of
the eight loudspeakers distributed octagonally at a single meter
distance from the subject’s head as depicted in Figure 1.
We implement a custom VBAP [8] patch in MAX environment that controls the eight spatially distributed loudspeakers
in real time. The developed VBAP patch can be used toposition a virtual sound source with any arbitrary loudspeakerconfiguration. The amplitude panning concept implemented
in the VBAP method is formulated with vectors pointing
the desired sound source position in the sound field. In ourparticular configuration of a horizontal plane defined by theoctagonally distributed loudspeakers the virtual sound images
could be created only the pairs of neighboring loudspeakers.
The simplified pair-wise VBAP application reduces the soundamplitude panning to only two loudspeakers used at a time.
The proposed VBAP reformulation leads to simpler equations
for amplitude panning, and the use of the vectors makes thepanning computationally efficient. The VBAP implementationconsists of three objects (functions) in MAX environment:
define loudspeakers, vbap, and matrix∼ , as depicted in a flow
digram in Figure 3.
In our proposal of the simplified two-dimensional VBAP
method the two-channel loudspeaker configuration is reformulated as a two-dimensional vector base. The base is definedby unit length vectors

−2
−4
−6

0

0.2

0.4

0.6

[s]

Fig. 2. An example of averaged brain evoked responses to target (solid blueline) and non-target (dotted red line). The difference in ERP waves is mostly
pronounced around 300ms where usually 𝑃 300/aha-response occurs. The
zero second time point indicates stimulus onset.

final classification.
There are multiple optimization steps in a course of thesaBCI system development. The first step is an optimal
choice of the number of loudspeakers in order to create amulti–command system. In the state-of-the-are saBCI applications [5], [6] the number of commands depends on the numberof real loudspeakers, since they generate the sound sources.
We aim to increase the number by employing a concept ofa virtual sound sources or so called virtual sound images inour saBCI prototype. This will allow to decrease the numberof the necessary loudspeakers or to increase a number ofcommands in the current setups. The virtual sound image is
generated using multiple loudspeakers, not with a single realone. The VBAP method to generate virtual sound images is
described in the following sections together with the proposed
multi-command saBCI application. Additionally we propose
to utilize sound depth panning to further possibly increase
a number of commands in a limited loudspeaker number
environment.
We conduct a series of psychophysical and offline saBCI
EEG experiments in order to address the following questions:
1) Does the spatial sound stimuli delivery system influences
the results? In detail, is there any significant difference
between real and virtual sound images?
2) Does a type of sound stimuli timbre matter?
3) Does the sound image depth (modulated volume in ourcase) influences the responses?
The paper from now on is organized as follows. First we
introduce spatial sound creation and delivery system. Next
psychophysical and offline asBCI experiments are described.
Finally results discussion concludes the paper.

l1 = [𝑙1𝑥 𝑙1𝑦 ]𝑇

l2 = [𝑙2𝑥 𝑙2𝑦 ]𝑇 ,

and

(1)

which are pointing toward 𝑡ℎ𝑒 𝑙𝑜𝑢𝑑𝑠𝑝𝑒𝑎𝑘𝑒𝑟 #1 and #2,
respectively, as depicted in Figure 4. The superscript 𝑇 denotes
the matrix transposition. The 𝑙𝑖𝑥 and 𝑙𝑖𝑦 are the l𝑖 vector
coordinates on 𝑥 and 𝑦 axes, respectively. The unit-length
vector(2)p = [𝑝𝑥 𝑝𝑦 ]𝑇 ,pointing toward the desired virtual sound source can be treatedas a linear combination of the two neighboring loudspeaker
vectors as follows
p = 𝑔1 l1 + 𝑔2 l2 ,(3)where 𝑔1 and 𝑔2 are the gain factors, which are the nonnegative scalar variables to be optimized by VBAP in order toposition the virtual sound image. The equation (3) could bereformulated in a matrix form as𝑇

p = (gL12 ) ,(4)

Fig. 4. A concept of the simplified horizontal panning of sound source with
two loudspeakers based on the original VBAP [8] implementation.

squares of the gain factors shall always obey
𝑔12 + 𝑔22 = 1.

(6)

However, the total sound loudness could be set to a constantvalue 𝐶, that would reflect changing sound source depth
location as follows
𝑔12 + 𝑔22 = 𝑐

Fig. 3. A schematic figure of the VBAP method implementation.
𝑇

where g = [𝑔1 𝑔2 ] and 𝑳12 = [𝒍1 𝒍2 ] . The equation (4) canbe solved if 𝑳−1
12 exists, so the gain vectors could be obtained
as follows[]−1
𝑙1𝑥 𝑙1𝑦
=
[𝑝
𝑝
]
.
(5)
g = p𝑇 L−1
12
12
𝑙2𝑥 𝑙2𝑦−1
The inverse matrix L−1
12 satisfies L12 L12 = I, where I is theidentity matrix. The gain factors 𝑔1 and 𝑔2 calculated using
equation (5).

B. Increasing the Number of the saBCI Commands with the
Spatial Depth Adjustment of the Sound Sources
Until now we only discussed the spatial directional soundsources manipulation in order to create the multi-commandsaBCI paradigm. Human brain auditory system can also distinguish easily a change of sound source distance by comparing
their loudnesses. We propose to test additionally how thechanging sound distance or depth could help us to increase
a number of possible saBCI commands. In the previously
introduced VBAP method in equation (4) the loudspeakersound amplitudes are controlled with gain factors 𝑔1 and 𝑔2 ,respectively. In traditional VBAP implementation the sum of

0 ⩽ 𝑐 ⩽ 1.

(7)

So a depth setting for 𝑐 = 1 could be referred as a standard/reference setting. The smaller value of 𝑐, as in equation (7), will create a farther distance sound illusion. In our
experiments discussed in the following sections we use settingsof 𝑐 = {0.2, 1} to create sound sources with loudnessesof approximately ∼ 30dB or ∼ 70dB, respectively, in ourlaboratory’s soundproof studio.
III. T HE P SYCHOPHYSICAL E XPERIMENT WITH E IGHTL OUDSPEAKERS TO E VALUATE S PATIAL S OUND S TIMULI
Sound localization performed by human subjects is limited
by some features related to head and ear shapes. The mostcommon localization problem is related to front/back soundsource confusion, and consequently it is difficult for people todiscriminate the exactly frontal or rear positions. In order tocreate and test the auditory stimuli for their difficulty levels
we perform first the psychophysical experiment to measure
the behavioral responses before the final EEG experiment
in offline BCI setting. The spatial target directions resulting
in later behavioral responses in form of button presses shallbe classified as the more difficult or causing more cognitive
load [3]. Also a number of mistakenly recognized targets in arandom series in each trial identifies possible variable difficultyof the task. Our goal is to design a final spatial oddballparadigm which would require the same cognitive load forall directions in order to have similar 𝑃 300 response easy toclassify by machine learning algorithms later.

In the series psychophysical experiments described below
we compare the behavioral reaction times as modulated byspatial cognitive load difficulty in function of:∙ Various timbres of the sound stimuli;∙ Real or virtual sound images creation techniques (realloudspeakers or VBAP);∙ Various sound depth settings.
In all psychophysical experiments the sound sources are generated with the eight loudspeakers of our system as depicted
in Figure 5.
The experiments were conducted with agreement of thelocal institutional ethical committee guidelines. The subjects
were explained the psychophysical and EEG experimental
procedures in detail. All of them participated voluntarily.
In the first experimental session, all the sound sources
are generated from the real loudspeaker directions, thus thissession is labeled as “the real.” The second session utilizes the
proposed simplified version of VBAP, as in equation (4), in order to create virtual sound sources at the positions between theneighboring loudspeakers. This second experimental session islabeled as “the virtual.” For example in the virtual setting, asound from a direction of 45∘ is created actually with the two
loudspeakers placed at 0∘ and 90∘ , instead of the actual one
located there in the real experiment. Such setting allows us
to compare the real and virtual sound reproduction setting forexactly the same spatial directions.
A. Psychophysical Experiment Protocol
The procedure of the psychophysical experiment in a single
trial is composed of the following steps:
1) An instruction is given to the subject of which spatial
sound stimuli direction and which source depth setting(loudness) to attended in form of a sound delivered andfollowed by a break.

2) Next the subject listens to the 16 randomly ordereddirections with the same stimulus timbre played fromeach (each direction has two depth settings).
3) The subject responses immediately (as fast as possible)
by pressing a computer keyboard button after the target
direction sounds occurs in the sequence, as instructed in
the above step 1).
4) The above procedure is repeated for all the 16 target
directions - with single one in each trial.
In order not to overlap the psychophysical responses (usually around 500ms for our subjects) with the following stimuli
the stimuli onset asynchrony (SOA) is set to 1000ms withstimuli lasting for only the first 500ms. The followings are theexperimental method to determine the difficulty of each spatial
sound stimuli. After each trial consisting of 16 spatial soundsources (chance level = 1/16 = 6.25%) we evaluate subject’s
response correctness and the delay time. Also a comparison
among the directions and depth levels is carried.
Table I summarizes the details of the psychophysical experiment settings.
We conduct the experiments for the three kinds of stimulus
timbre in order to evaluate also the possible subject preferences. The results are discussed in the next section.
B. Psychophysical Experiment Results
The results of conducted psychophysical experiments arepresented in Tables II–IX. The analysis of sound depths
variability, summarized in Tables II, IV, VI, and VIII, resulted
with the lowest and no significant differences of response
times, as tested with Wilcoxon-pariwise-test for the means.
This result supports the hypothesis of the possibility to utilize
this feature for the saBCI paradigm design in real and virtual
sound reproduction settings.
On the other hand the results summarized in Tables III, V,
for real sound sources, and in Tables VII, IX for virtual ones,
resulted also in no significant differences for directions basedon the response time analyzes.
However a comparison of accuracy rates (AR) for real andvirtual sound sources in all Tables II–IX showed significant
differences with much lower values for virtual. This showsthat VBAP application still needs further research on thesound quality related to virtual image spatial spreads. The
TABLE IT HE PSYCHOPHYSICAL EXPERIMENT CONDITIONS .
Condition
Sound stimuli
Sound source
Response input
Sound directions

Fig. 5. The locations of the eight loudspeakers in our experiments. The frontal
to the subject loudspeaker at a position of 0∘ .

Volume levels
Subjects
Ttrials
Stimulus length
SOA

Parameterswhite noise, MIDI sound, effect soundeight channel loudspeaker systemcomputer keyboard
8(−135∘ , −90∘ , −45∘ , 0∘ , 45∘ , 90∘ , 135∘ , 180∘ )∼ 30dB and ∼ 70dB
612 (2 trials × 3 stimuli × 2 real/virtual-sounds)
500ms
1000msTABLE II
ACCURACY RATE OF PSYCHOPHYSICAL RESPONSE TO REAL SOUND
STIMULI WITH THE NEARER SOUND SOURCE SETTING (∼ 70 D B) AS A
FUNCTION OF VARIOUS STIMULI (AR = ACCURACY RATE , ART =AVERAGE RESPONSE TIME , 𝜎 = STANDARD DEVIATION ).
Stimulus typeWhite noise
MIDsoundEffect sound

AR [%]
91.0
93.2
82.0

ART [ms]
574
571
570

𝜎 [ms]103
102
104

TABLE III
ACCURACY RATE OF PSYCHOPHYSICAL RESPONSE TO REAL SOUND
STIMULI WITH THE NEARER SOUND SOURCE SETTING (∼ 70 D B) AS A
FUNCTION OF THE SPATIAL DIRECTION (AR = ACCURACY RATE , ART =AVERAGE RESPONSE TIME , 𝜎 = STANDARD DEVIATION ).
Sound directions
−135∘
−90∘
−45∘
0∘
45∘
90∘
135∘
180∘

AR [%]
94.4
91.0
94.4
91.6
100.0
88.8
88.8
86.1

ART [ms]
603
573
554
535
547
582
590
537

𝜎 [ms]
91.6
98.4
79.5
94.2
88.9
95.7
102.0
128.0

TABLE IV
ACCURACY RATE OF PSYCHOPHYSICAL RESPONSE TO REAL SOUND
STIMULI WITH THE NEARER SOUND SOURCE SETTING (∼ 30 D B) AS A
FUNCTION OF VARIOUS STIMULI (AR = ACCURACY RATE , ART =AVERAGE RESPONSE TIME , 𝜎 = STANDARD DEVIATION ).
Stimulus typeWhite noise
MIDsoundEffect sound

AR [%]
90.6
91.7
75.0

ART [ms]
582
539
562

𝜎 [ms]
117
105
99.4

TABLE V
ACCURACY RATE OF PSYCHOPHYSICAL RESPONSE TO REAL SOUND
STIMULI WITH THE NEARER SOUND SOURCE SETTING (∼ 30 D B) AS A
FUNCTION OF THE SPATIAL DIRECTION (AR = ACCURACY RATE , ART =AVERAGE RESPONSE TIME , 𝜎 = STANDARD DEVIATION ).
Sound directions
−135∘
−90∘
−45∘
0∘
45∘
90∘
135∘
180∘

AR [%]
91.6
91.6
83.3
80.5
88.8
77.7
88.8
83.3

ART [ms]
562
531
553
541
531
589
617
562

𝜎 [ms]
113
90.2
84.7
115
92.9
98.0
148
98.9

differences show that virtual sound images are more difficult
to localize so they shall not be mixed with the real sounds ina single experimental trials in the offline saBCI experiments
as presented in the next section.
IV. T HE O FFLINE SA BCI E XPERIMENT
Based on previously discussed very encouraging psychophysical experiment results showing no relation between

TABLE VI
ACCURACY RATE OF PSYCHOPHYSICAL RESPONSE TO VIRTUAL SOUND
STIMULI WITH THE NEARER SOUND SOURCE SETTING (∼ 70 D B) AS A
FUNCTION OF VARIOUS STIMULI (AR = ACCURACY RATE , ART =AVERAGE RESPONSE TIME , 𝜎 = STANDARD DEVIATION ).
Stimulus typeWhite noise
MIDsoundEffect sound

AR [%]
54.6
44.7
47.3

ART [ms]
612
571
623

𝜎 [ms]
115
115
113

TABLE VII
ACCURACY RATE OF PSYCHOPHYSICAL RESPONSE TO VIRTUAL SOUND
STIMULI WITH THE NEARER SOUND SOURCE SETTING (∼ 70 D B) AS A
FUNCTION OF THE SPATIAL DIRECTION (AR = ACCURACY RATE , ART =AVERAGE RESPONSE TIME , 𝜎 = STANDARD DEVIATION ).
Sound directions
−135∘
−90∘
−45∘
0∘
45∘
90∘
135∘
180∘

AR [%]
67.2
47.7
49.4
56.6
43.8
44.4
45.0
71.6

ART [ms]
630
641
595
583
634
619
605
611

𝜎 [ms]
135
92
149
103
125
106
104
142

TABLE VIII
ACCURACY RATE OF PSYCHOPHYSICAL RESPONSE TO VIRTUAL SOUND
STIMULI WITH THE NEARER SOUND SOURCE SETTING (∼ 30 D B) AS A
FUNCTION OF VARIOUS STIMULI (AR = ACCURACY RATE , ART =AVERAGE RESPONSE TIME , 𝜎 = STANDARD DEVIATION ).
Stimulus typeWhite noise
MIDsoundEffect sound

AR [%]
46.9
44.3
45.6

ART [ms]
603
607
625

𝜎 [ms]
109
132
90.9

TABLE IX
ACCURACY RATE OF PSYCHOPHYSICAL RESPONSE TO VIRTUAL SOUND
STIMULI WITH THE NEARER SOUND SOURCE SETTING (∼ 30 D B) AS A
FUNCTION OF THE SPATIAL DIRECTION (AR = ACCURACY RATE , ART =AVERAGE RESPONSE TIME , 𝜎 = STANDARD DEVIATION ).
Sound directions
−135∘
−90∘
−45∘
0∘
45∘
90∘
135∘
180∘

AR [%]
40.0
49.4
51.6
37.7
37.2
40.5
42.2
65.5

ART [ms]
603
605
597
586
668
607
648
620

𝜎 [ms]
78.1
108
127
121
98.9
111
126
82.1

subjects behavioral performance and spatial sound directions
we proceed to the offline saBCI paradigm EEG recordings
in which the brain responses will be measured to further
test our original hypothesis of the possibility to utilize realand proposed virtual spatial sound sources in the oddball
experimental paradigm. Now we verify whether the ERP
responses show any differences for directions and reproduction

The original contaminated ERP responses
15
10
amplitude [μV]system types. Since the previously conducted psychophysicalexperiment indicated no relation to sound timbres, in thefollowing analysis of EEG responses we average all types. In
order to do so we conduct the following experiments with thesame six volunteers and with accordance of the institutional
ethical committee guidelines.
The EEG signals are captured with eight dry electrode portable wireless system by g.tec (g.MOBIlab+ &g.SAHARA). The EEG signals are sampled at 256Hz rateand stored on a laptop computer running custom MATLAB
Simulink patch. The user interface and stimulus are createdin MAX.

5
0
−5
−10
−15
−20
−0.1

0

0.1
0.2
0.3
0.4
time after the stimuli offset [s]

0.5

0

0.1
0.2
0.3
0.4
time after the stimuli offset [s]0.5

A. The Offline saBCI EEG Experimental Protocol

B. EEG Signal Preprocessing and Analysis
Two Butterworth 5𝑡ℎ -order low– and high–pass filters aresubsequently applied with cutoff frequencies at 0.5𝐻z and
20Hz respectively to remove low frequency and DC-shiftinterferences. The low-pass filtering removes possible muscle
movement frequency artifacts. The preprocessed EEG signals
are next segmented to create event related epochs starting at−100ms from the stimulus onset and ending at 800ms after
it.
C. EEG Preprocessing to Clean the Signals from EyeMovements with Multivariate Empirical Mode Decomposition
There are many signal processing methods trying to remove eye– and muscle–movement interferences from EEG
signals. The most successful approaches belong the new
TABLE XT HE OFFLINE SA BCI EEG EXPERIMENTAL CONDITIONS .
Condition
Sound stimuli
Sound source
Response input
Sound directions
Volume levels
Subjects
Ttrials
Stimulus length
SOA

Parameterswhite noise, MIDI sound, effect soundeight channel loudspeaker system
8-electrodes EEG system
8(−135∘ , −90∘ , −45∘ , 0∘ , 45∘ , 90∘ , 135∘ , 180∘ )∼ 30dB and ∼ 70dB
612 (2 trials × 3 stimuli × 2 real/virtual-sounds)
500ms
1000ms4

amplitude [μV]

The offline saBCI experimental protocol for a single trial is
as follows:
1) First an auditory instruction is given to the subject towhich spatial sound stimuli to attended.
2) Next the subject listens to the 16 randomly ordered
stimuli, but this time no any behavioral response is given
and only counting of targets is advised to keep a focused
attention.
3) The above procedure is repeated for all target directionsand volumes separately, resulting in 16 trials for eachtimbre.
The detailed experimental setting are summarized in Table X.

2

0

−2

−4
−0.1

Fig. 6. An example of EMD ERP cleaning. The top panel depicts the originalcontaminated EEG ERP response with large drifts for both the in target (blue,
solid line) and non-target responses (red, dashed line). The cleaned version
is shown in lower panel, where there ERP related peaks are clearly visible.

developments based on empirical mode decomposition [9],[10], since they outperform ICA and blind source separation
based approaches [11]. There exist already multiple signal
processing approaches that extend the classical EMD [12]concept which constitutes the fully data adaptive technique
to decompose any non–linear or non–stationery signal into a
finite set of band-limited basis functions called intrinsic mode
functions (IMFs). Each of the decomposed IMFs carry the both
amplitude and frequency modulated oscillatory components.
The very recent development in the field is the multivariate
EMD (MEMD) [13], which is a more generalized extension of
the classical univariate EMD technique. The most interesting
part of that approach is that it can deal instantaneouslywith multivariate dataset, like EEG for example. The major
novelty proposed in [13] is based on possibility to process
the 𝑛-dimensional signals in the same number of spaces.
As a result at each decomposition step the 𝑛-dimensional
envelopes are generated by taking signal projections along
different directions in the same dimensional spaces. Next, as
in classical univariate EMD, the local mean is calculated as an
approximation of the integral of all envelopes along multiple
directions in the 𝑛-dimensional space. The authors in [13]

very elegantly resolve a problem related to a lack of formal
definition of maxima and minima in multidimensional domains
by utilizing a sampling concept based on low discrepancy
Hammersley sequence to generate the projections. In a next
step, the extrema could be interpolated using cubic splines.
The resulting signal envelopes are obtained from the projections along different directions in multidimensional spaces.
They are finally averaged to obtain the local mean of theoriginal multivariate EEG signal, as in our case. The whole
MEMD decomposition procedure of an input multivariate
signal 𝑠(𝑡) into a multidimensional set of IMFs, as in [14],
could be summarized as follows:
1) As the first step, generate the point-set based on the
Hammersley sequence for an uniform sampling of theEEG on an (𝑛 − 1) − 𝑑𝑖𝑚𝑒𝑛𝑠𝑖𝑜𝑛𝑎𝑙 − 𝑠𝑝ℎ𝑒𝑟𝑒;2) In the second step, calculate a projection {𝑝𝜃𝑘 (𝑡)}𝑇𝑡=1 of𝑇
the input EEG {𝑠(𝑡)}𝑡=1 along the direction vector 𝑋 𝜃𝑘 ,for all 𝑘 (the whole set of direction vectors) resulting in{𝑝𝜃𝑘 (𝑡)}𝐾
𝑘=1 set of projections;3) In the third step, find the time points {𝑡𝜃𝑖 𝑘 }𝐾
𝑘=1 representing the maxima in the set of projected signals{𝑝𝜃𝑘 (𝑡)}𝐾
𝑘=1 ;4) As the fourth step, perform interpolation [𝑡𝜃𝑖 𝑘 , 𝑠(𝑡𝜃𝑖 𝑘 )],for all values of 𝑘, in order to obtain the multivariate
envelope curves {𝑣 𝜃𝑘 (𝑡)}𝐾
𝑘=1 ;5) In the fifth step, for a set of 𝑘 direction vectors, calculate
the mean 𝑚(𝑡) of the envelope curves as:
𝐾𝑚(𝑡) =

1 ∑ 𝜃𝑘
𝑣 (𝑡)
𝑘

(8)

𝑘=1

6) Finally, extract/sift the “detail” 𝑑(𝑡) using 𝑑(𝑡) = 𝑋(𝑡)−𝑚(𝑡). Similarly as in classical univariate EMD approach,the “detail” 𝑑(𝑡) shall fulfill the stoppage criterion fora multivariate IMF, so apply the above procedure to𝑋(𝑡) − 𝑑(𝑡), otherwise apply it to 𝑑(𝑡).
Once the first multivariate IMF is identified, it is subtracted
from the input signal and the same process is applied to
the resulting signal yielding the second IMF, etc. In themultivariate case, similarly as in univariate one, the residue
corresponds to a signal whose projections do not containenough extrema to form a meaningful multivariate envelope.
The stopping criterion in MEMD is also similar to standard
EMD [12] one, only with a difference that the condition
for equality of the number of extrema and zero crossings is
not imposed, as extrema could not be properly defined for
multivariate signals. The resulting from MEMD decomposition
filter banks represent an array of band-pass filters designed
to filter, with possible spectral overlap, the input multivariate
signal into different frequency bands. The frequency bands are
the same for all IMFs which is a very interesting feature of
the MEMD method. In case of sequential implementation ofclassical univariate EMD, each channel would result with different number of IMFs, which would increase a complexity ofartifacts identification. The MEMD decomposition allows for
simple comparison of the adaptive subband filtering results andrejection only of the contaminated by eye/muscle-movement
IMFs.
For the eye/muscle-movement identification we apply an𝑝𝑒𝑎𝑘 − 𝑡𝑜 − 𝑝𝑒𝑎𝑘 thresholding over the the all IMFs. Those
IMFs which exceed 20𝜇V peak–to–peak value are marked as“contaminated” and removed from the following reconstruction. The result of the procedure is shown in Figure 6, wherethe very strong eye-blinks, completely damaging the averaged
ERPs, are successfully removed with the proposed method.
D. The Offline saBCI EEG Analysis Results
The averaged results of the conducted EEG experimentswith the six subjects in the offline saBCI mode are presented in
Figures 7–10. Each of those figures has eight panels comparing
the averaged responses to targets (solid–blue lines) and to
non–targets (dashed–red lines) for each EEG electrode used inour experiments. The presented ERPs are the grand averages
for all subjects and all sound timbres used in our experiments.
The comparison between the real (see Figures 7 and 8) andvirtual (see Figures 9 and 10) sound images show that strong
𝑃 300 responses (marked by black dotted–line squares) could
be obtained in both cases. This results show that the both
real and virtual sound reproduction systems yield similar
results of saBCI stimuli generation. Additionally we can see
from the results with farther sound (smaller loudness) in
Figures 8 and 10 that the 𝑃 300 response seems to be stronger
in amplitude comparing to closer distance (larger loudness)
results in Figures 7 and 9. This very interesting phenomenon
will be a target of our future research. The most importantmessage from our current experimental session is that the
proposed simplified VBAP application is a valid method tocreate the good saBCI stimuli. We could not observe any
dependence of the spatial sound locations on 𝑃 300 responses
which is also a good result for future saBCI applications.
V. C ONCLUSIONSThe purpose of the presented research was to check whether
virtual sound images creation procedures were useful for the
saBCI paradigm development. We have been researching a
possibility to increase the possible number of BCI commands
for future applications by keeping a limited number of theloudspeaker units. We conducted two experimental series
to test the hypotheses of real and virtual images equality
for saBCI paradigm design, sound depth manipulation, andwhether the various timbres modulate the psychophysical andbrain responses.
The very small, yet not significant, differences in psychophysical experiments were later not reproduced in EEG
experiments, in offline saBCI setting, which is a very goodsign for the future developments in saBCI research field. The
psychophysical experiments confirmed, based on the lack ofsignificant differences of pairwise statistical tests, that thedirections and sound reproduction systems (real or virtual)
result with the same behavioral responses.
The brain responses in offline saBCI setting further confirmed the original hypothesis of the similarity of real andFig. 7. EEG evoked responses to real sound stimuli. The blue–solid lines
represent the averaged responses to the target sounds, while the red–dashedlines to the averaged non-targets with closer depth setting (∼ 70dB). The
𝑃 300–response latencies are marked with black–dotted line squares for eachelectrode.

Fig. 8. EEG evoked responses to real sound stimuli. The blue–solid lines
represent the averaged responses to the target sounds, while the red–dashedlines to the averaged non-targets with farther depth setting (∼ 30dB). The
𝑃 300–response latencies are marked with black–dotted line squares for eachelectrode.

virtual sound settings. We reported also on the very interesting
ERP modulation depending on sound depth (loudness) which
resulted with slightly different, larger for the farther sound,𝑃 300 responses. This topic will be further investigated in our
future research.
In the future, we will develop and online saBCI application
based on the proposed simplified VBAP approach in order tocreate a more easy to use interfacing paradigm with possible
larger and independent from the loudspeakers quantity themultiple commands.
Additionally we plan to further expend the system touse fully three–dimensional sound field by adding elevation
variability, which shall further improve and enhance the systemcapability. Vertical sound localization is a bit more difficult toreproduce, yet it shall improve the interfacing comfort.
The presented results are a step forward in developmentof vision-free and auditory spatial brain computer interfacing

paradigm which shall in the future support and improvecomfort of life of locked–in patients.
ACKNOWLEDGMENTSFirst and foremost, we would like to thank God for being
my strength and guide in the writing of this thesis.
This research was supported in part by the Strategic Information and Communications R&D Promotion Programme no.
121803027 of The Ministry of Internal Affairs and Communication in Japan, and by KAKENHI, the Japan Society for the
Promotion of Science grant no. 12010738.
R EFERENCES[1] J. Wolpaw and E. W. Wolpaw, eds., Brain-Computer Interfaces: Principles and Practice. Oxford University Press, 2012.
[2] N. Nishikawa, “Spatial sound stimuli optimization for auditory localization based brain machine interface,” bachelor degree thesis, School ofInformatics, University of Tsukuba, February 2012.

Fig. 9. EEG evoked responses to virtual sound stimuli using VBAP. The
blue–solid lines represent the averaged responses to the target sounds, while
the red–dashed lines to the averaged non-targets with farther depth setting(∼ 70dB). The 𝑃 300–response latencies are marked with black–dotted line
squares for each electrode.

Fig. 10. EEG evoked responses to real sound stimuli. The blue–solid lines
represent the averaged responses to the target sounds, while the red–dashedlines to the averaged non-targets with farther depth setting (∼ 30dB). The
𝑃 300–response latencies are marked with black–dotted line squares for eachelectrode.

[3] E. Niedermeyer and F. L. Da Silva, eds., Electroencephalography:
Basic Principles, Clinical Applications, and Related Fields. Lippincott
Williams & Wilkins, 5 ed., 2004.
[4] T. M. Rutkowski, “Auditory brain-computer/machine interface
paradigms design,” in Haptic and Audio Interaction Design (E. Cooper,
V. Kryssanov, H. Ogawa, and S. Brewster, eds.), vol. 6851 of Lecture
Notes in Computer Science, pp. 110–119, Springer Berlin / Heidelberg,
2011.
[5] Z. Cai, H. Terasawa, S. Makino, T. Yamada, and T. M. Rutkowski,“Sound timbre and spatial location as informative cues in auditory BCI- brain evoked potential enhancement and involuntary eye movements
artifacts suppression approach,” in Proceedings of the Third APSIPA
Annual Summit and Conference (APSIPA ASC 2011), (Xi’an, China),
p. paper 241 (6 pages), APSIPA, October 18 - 21, 2011.
[6] M. Schreuder, B. Blankertz, and M. Tangermann, “A new auditory
multi-class brain-computer interface paradigm: Spatial hearing as an
informative cue,” PLoS ONE, vol. 5, p. e9813, 04 2010.
[7] http://cycling74.com/, “Max 6,” 2012.
[8] V. Pulkki, “Virtual sound source positioning using vector base amplitude
panning,” Journal of Audio Engineering Society, vol. 45, no. 6, pp. 456–
466, 1997.
[9] T. M. Rutkowski, A. Cichocki, T. Tanaka, D. P. Mandic, J. Cao, and A. L.

Ralescu, “Multichannel spectral pattern separation - an EEG processing
application,,” in Proceedings of the 2009 IEEE International Conferenceon Acoustics, Speech, and Signal Processing (ICASSP2009), (Taipei,
Taiwan), pp. 373–376, IEEE Press, 2009.
[10] M. K. I. Molla, T. Tanaka, and T. M. Rutkowski, “Multivariate EMD
based approach to EOG artifacts separation from EEG,” in Proceedingsof the 2012 IEEE International Conference on Acoustics, Speech, andSignal Processing (ICASSP2012), (Kyoto, Japan), pp. 653–656, IEEE,
March 25-30, 2012.
[11] M. K. I. Molla, M. R. Islam, T. Tanaka, and T. M. Rutkowski, “Artifact
suppression from EEG signals using data adaptive time domain filtering,”
Neurocomputing, vol. 97, pp. 297–308, November 15, 2012.
[12] N. Huang, Z. Shen, S. Long, M. Wu, H. Shih, Q. Zheng, N.-C. Yen,
C. Tung, and H. Liu, “The empirical mode decomposition and the Hilbert
spectrum for nonlinear and non-stationary time series analysis,” Proceedings of the Royal Society A: Mathematical, Physical and Engineering
Sciences, vol. 454, pp. 903–995, March 1998.
[13] N. Rehman and D. P. Mandic, “Multivariate empirical mode decomposition,” Proceedings of the Royal Society A: Mathematical, Physical andEngineering Science, vol. 466, no. 2117, pp. 1291–1302, 2010.
[14] N. Rehman and D. P. Mandic, “Filter bank property of multivariate empirical mode decomposition,” IEEE Transactions on Signal Processing,vol. 59, no. 5, pp. 2421–2426, 2011.