Head–related Impulse Response Cues for Spatial Auditory
Brain–computer Interface
Chisaki Nakaizumi1 , Shoji Makino1 , and Tomasz M. Rutkowski1,2,3,∗
Abstract— This study provides a comprehensive test of ahead–related impulse response (HRIR) cues for a spatial
auditory brain–computer interface (saBCI) speller paradigm.
We present a comparison with the conventional virtual soundheadphone–based spatial auditory modality. We propose andoptimize the three types of sound spatialization settings using
a variable elevation in order to evaluate the HRIR efficacy
for the saBCI. Three experienced and seven naive BCI users
participated in the three experimental setups based on ten
presented Japanese syllables. The obtained EEG auditory
evoked potentials (AEP) resulted with encouragingly good andstable P300 responses in online BCI experiments. Our casestudy indicated that users could perceive elevation in the
saBCI experiments generated using the HRIR measured froma general head model. The saBCI accuracy and informationtransfer rate (ITR) scores have been improved comparing
to the classical horizontal plane–based virtual spatial soundreproduction modality, as far as the healthy users in the current
pilot study are concerned.

I. I NTRODUCTION
BCI is a technology that uses brain neuronal signals
to operate a computer without any muscle movements.
Therefore, it is expected to provide a speller for disabled
people such as patients suffering from the amyotrophic
lateral sclerosis (ALS) [1]. Although currently a successful
visual BCI modality could provide a fast speller, the advanced patients who are in a locked–in state cannot use it
because they lose any intentional muscle control including
even eye blinks [2]. Auditory BCI can be an alternative
method because it does not require a good sight or eye
movements [1], [3]–[6]. We propose an alternative method toextend the previously published by our group spatial auditory
BCI (saBCI) paradigms [3], [6] by making use of a headrelated impulse response (HRIR) for the virtual spatial soundimages reproduction with headphones. Our research target
is the virtual sound saBCI using the HRIR–based spatial
cues to create the non–invasive and auditory stimulus–driven
paradigm, which does not require a long training. HRIR
appends interaural–intensity–differences (IID), interaural–time–differences (ITD), and spectral modifications to create
the spatial stimuli, while a vector–based amplitude panning∗ This research was supported in part by the Strategic Information andCommunications R&D Promotion Program (SCOPE) no. 121803027 of The
Ministry of Internal Affairs and Communication in Japan.
1 Chisaki Nakaizumi, Shoji Makino and Tomasz M. Rutkowski arewith Life Science Center of TARA and Department of Computer Science, University of Tsukuba, 1-1-1 Tennodai Tsukuba Ibaraki, Japan

tomek@bci-lab.info

http://bci-lab.info/

2 Tomasz M. Rutkowski is also with RIKEN Brain Science Institute,Wako-shi, Japan
3 Tomasz M. Rutkowski is the corresponding author.
∗ Tomasz M. Rutkowski was supported in part by YAMAHA Corporation.

978-1-4244-9270-1/15/$31.00 ©2015 IEEE

(VBAP) appends only the IID. The HRIR allows for more
precise and fully spatial virtual sound images positioning
utilizing even the user not own HRIR measurements [7].
In our previous study [8] we evaluated saBCI feasibility
with the HRIR–based spatial sound generator, and compared
it with a formerly reported vector–based–amplitude–panning(VBAP)–based spatial auditory experiments [6]. The above
study used only five Japanese vowels distributed horizontally
every 40◦ in front of a subject head. The pilot study resulted
with clear P300 responses and it has shown that HRIR
modality could improve spelling accuracy comparing to theconventional spatial sound generation methods. In the study
presented in this paper we introduce ten Japanese kanasyllables–based speller using the sound elevation features as
the second step. We also compare various sound elevation
settings. However, it is usually difficult to precisely perceive
various sound elevations using not the user own HRIR,
because the elevation perception is highly individual and it
is influenced by the shape of an auricle [7]. Unfortunately,
it would be very difficult to record a bedridden ALS–
patient own HRIR. Therefore, we propose to test the HRIR–
based saBCI efficacy using a general head model (KEMAR).
We conduct psychophysical and EEG experiments in online
saBCI ten Japanese kana syllables spelling experiments inorder to test our research hypothesis of the KEMAR HRIR–
based paradigm practical feasibility.
From now on the paper is organized as follows: next section describes methods and experimental set up; third section
reports results of the P300 response–based saBCI speller
classification accuracies and information transfer rate (ITR)
in comparison to a conventional method; finally, conclusions
and future research directions summarize the paper.
II. M ATERIALS AND M ETHODSIn order to prepare the saBCI stimuli to create an oddballparadigm generating the P300 responses [2], all spatial soundimages were created using HRIR of the general head model.
Ten Japanese syllables with a vowel ”a” were selected for assound stimuli in this project. The spoken syllables were takenfrom a public Japanese sound dataset of a female speech [9].
The monaural sound stimuli were spatially distributed using
a public domain KEMAR’ S HRTF DATABASE provided by
the MIT Media Lab [10]. In order to generate a stereo
sound placed at a spatial location at an azimuth of θ and anelevation of φ the following procedure was applied. Let hl,θ ,φand hr,θ ,φ be the minimum–phase impulse responses from the
KEMAR HRTF DATABASE measured at the chosen azimuth
θ and the elevation φ at the left (l) and right (r) ears. The

1071

Fig. 1. A diagram of the three proposed virtual spatial stimulus soundsettings. The left box depicts the horizontal only placement–based setting;the middle the elevation use–based case; and the right box the elevation use
with an additional frequency shifting set up.

stereo spatial sound delivered via headphones to the left andright ears respectively could be constructed, in time domainusing HRIR, as a two–dimensional signal composed of theleft xl (t) and right xr (t) headphone channels as follows,n−1

xl (t) = ∑ hl,θ ,φ (τ)s(t − τ),

(1)

τ=0
n−1

xr (t) = ∑ hr,θ ,φ (τ)s(t − τ),
τ=0where τ denotes sample time delay and n is the HRIR length
as obtained from the HRTF DATABASE [11]. The so createdspatial acoustic stimuli were delivered to the left and rightears of the user through the ear–fitting portable headphones
SENNHEISER CX 400II. Three proposed spatial settings
to evaluate the feasibility of the sound elevation using the
HRIR for the saBCI are presented in Figure 1. The first
spatial stimulus setting included only the horizontal soundimages’ placement of the ten saBCI commands. The soundimages were distributed at five directions every 45◦ on
the horizontal plane at the user ears’ level. Two types ofcommands were delivered from the same spatial directions.
The second stimulus setting included the sound elevation
variability. Ten stitmuli were distributed among the all different directions. Five commands were localized at elevationof 50◦ from the user ears’ level and every 45◦ horizontally.
The third setting included also the variable elevation andan additional option of a frequency modulation appended
to discriminate the sound images originating from various
elevations. In the psychoacoustics there is a well–known
property called a tonal bell causing a higher–frequency soundto be perceived as originating from a higher elevation [7].
To simulate the above effect we shifted in frequency domain
the stimuli, which shall be perceived at higher elevations,using the TANDEM-STRAIGHT method [12]. The soundstimuli to be perceived at elevation of 50◦ had shifted up
their fundamental frequencies by 24 Hz.
All of the experiments reported in this paper were performed in the Life Science Center of TARA, University ofTsukuba, Japan. Ten healthy users participated in our study.
They were comprised of seven naive and three experienced
BCI users. The average age was of 24.7 years old (standard
deviation 6.48 years old, three males and seven females).

The psychophysical and online EEG saBCI experiments wereconducted in accordance with The World Medical Association Declaration of Helsinki - Ethical Principles for Medical
Research Involving Human Subjects. The experimental procedures were approved and designed in agreement with the
ethical committee guidelines of the Faculty of Engineering,
Information and Systems at University of Tsukuba, Japan.
The psychophysical experiments were conducted to examine a perception of elevation and preferences for eachspatial sound setting. The users were instructed to respond bypressing the button as soon as possible after they perceived
the target stimulus as in the classical oddball paradigm [2]. Ina single experimental session 10 targets and 90 non-targets
were presented. Each experiment was comprised of three
sessions for every spatial sound setting. The stimulus duration was set to 300 ms and the inter–stimulus–interval (ISI)
to 700 ms. The online EEG experiments were conducted
to investigate whether P300 responses could be evoked in
the various spatial sound settings and to compare the saBCI
classification accuracies, as well as an efficaty of each set up.
The brain signals were collected by a bio–signal amplifiersystem g.USBamp by g.tec Medical Engineering GmbH,
Austria. The EEG signals were captured by sixteen activegel–based electrodes g.LADYbird attached to the following
head locations Cz, Pz, P3, P4, Cp5, Cp6, P1, P2, Poz,
C1, C2, FC1, FC2, and FCz as in the extended 10/10
international system [2]. The ground electrode was attachedon the forehead at the FPz location, and the reference on
the user’s left earlobe respectively. An in–house extended
BCI2000 [13] software was used for the saBCI experiments
to present stimuli and display online classification results.
A single EEG experimental session was comprised of
one training and two test runs for each spatial stimulus
setting. In the training run, EEG brainwaves were recorded
and next classifier parameters were calculated. In order tospell a single character, 15 targets and 135 non-targets were
presented for ERP response averaging. In the test runs, user
spelled two sets of the three words that the covered ten
targets (a–ta–ma; ha–ra; sa–wa–ya–ka–na; wa–ta; ha–na–
ya; a–ka–ra–sa–ma). For spelling single character, 5 target
and 45 non-target stimuli were presented continuously as for
single syllable (five ERPs averaging scenario). Experimentorder of every spatial sound setting and spelled words wererandomized for each subject. The sound stimulus duration
was set to 300 ms and the inter–stimulus–interval (ISI) to
150 ms. The EEG sampling rate was of 512 Hz and a notch
filter to remove electric power lines interface of 50 Hz wasapplied in a rejection band of 48 ∼ 52 Hz. The band–passfilter was set at 0.1 Hz and 60 Hz cutoff frequencies. The
acquired EEG brain signals were classified online by the in–house extended BCI2000 application using a stepwise linear
discriminant analysis (SWLDA) [14] classifier with features
drawn from the 0 ∼ 800 ms ERP interval, with removal of
the least significant input features, having p > 0.15, and withthe final discriminant function restricted to contain a maximum of 60 features. The participating users answered alsoquestionnaires, asking about which modality they preferred,1072

Fig. 2. Grand mean averaged EEG ERP responses together with standard
error bars for all the participating users in the study plotted for representativeelectrodes separately. Purple lines depict the grand mean averaged targets
with clear P300 responses, while blue traces are the non–targets. Eye blink
artifacts were removed with 80 µV thresholding.

after finishing psychophysical and EEG experiments.
III. R ESULTS
In the psychophysical experiments the users could perceive
the various elevation settings. The questionnaire answers
indicated that the frequency shifting supported the stimulus
discrimination. The elevation and frequency variable setting
resulted with the best perception of the spatial sound image
stimuli. Four out of six participants preferred elevation andfrequency–based spatial sound setting. The results of thesaBCI EEG experiment are depicted in Figure 2. Eachcolumn presents the grand mean averaged ERP results at
representative four electrodes for the three proposed spatial
sound settings. We confirmed the clear P300 responses inlatency ranges of 400 ∼ 1000 ms and their usability forthe subsequent classification. Figure 3 presents the classification accuracies of the P300 responses as obtained with
the SWLDA classifier. The theoretical chance level was of
10% in this study. All users scored with accuracies above
70% at the best. There were three users who resulted with
100% accuracies as the best in the reported experiments.
The average scores were obtained as mean values calculated
from the two saBCI test spelling sessions of all the users(training sessions were not included in the saBCI accuracy
calculations). The results are shown in Figure 4. There weresignificant differences between horizontal and elevation use;
elevation only versus elevation and frequency spatial soundsettings as calculated with the t–test (the significant level was
p < 0.05). The 70% of the users preferred elevation–based
spatial sound setting. The important outcome of the presented
study was that the users could clearly perceive elevation
variability using not their own HRIR filters. Although theresults of psychophysical experiments have shown that theFig. 3. The average saBCI spelling accuracies of all the users. The chancelevel was of 10%. The blue bars depict the saBCI accuracy of horizontal
sound images placement; green bars represent elevation–based results; andthe orange color the elevation and frequency settings, respectively. Th users#4, #5 and #10 were the experienced BCI users, while the remaining were
the naive participants.

users preferred elevation and frequency spatial sound setting,the online saBCI accuracies suggested that P300 responses
generated in elevation variability setting where the best.
In order to compare the developed HRIR–based saBCI
we calculated the information transfer rate (ITR) scoresconsidered as a major comparison measure among the BCI
paradigms [4]. ITR scores were calculated as follows:
IT R = V · R,(2)


R = log2 N + P · log2 P + (1 − P) · log2


1−P
,
N −1

where V was the classification speed in selections/minute; R
represented the number of bits/selection; N was the numberof classes (10 in this study), and P the classification accuracy.
The conditions contributing to ITR were in a trade–off
relationship with the task easiness. For example, the short ISI
could improve the ITR, but it could cause the task to be difficult. ITR could increase with larger number of commands,
higher accuracies, shorter ISI, and a smaller number of theaveraged trials. We also compared the ITR scores of thethree proposed modalities with our previous project results
and with the vector-based-amplitude-panning (VBAP)–based
spatial auditory approach, which was regarded as a conventional method [6]. The VBAP experiment was conducted
for two sessions and with 16 BCI–naive users [6]. The
electrode positions were the same as in the experiments
reported in this paper. The sound stimuli were presented withsmall ear–fitting headphones (SENNHEISER CX 400II)in the all the studies. The spatial locations of the stimulus
sound images, the number of commands, and ISI settings
are summarized in Table I. The ITR scores of the elevation
use spatial setting reached 14.92 bit/min. The other proposed
modalities scored above 8.5 bit/min and also exceeded ourprevious research results [8]. The headphone–based virtualsaBCI took one step forward. Although the HRIR basedsaBCI resulted with better accuracy scores, this results cannot

1073

Elevation
60%

78%

100%

Elevation and frequency
61%

The saBCI averaged accuracies"*" - statistical significance (p < 0.05)
Horizontaland'frequency
30%
30%
30%
40%
70%
60%
80%
80%
00%
60%
10%
50%
00%
00%
70%
70%
70%
50%
50%
60%
0.5%
.605

TABLE II
Spatial setting preferences

T HE MEAN SPELLING ACCURACY FOR VARIOUS HORIZONTAL AZIMUTHS

90%

Horizontal

80%

Sound mode

Elevation

70%

Horizontal
Elevation
Elevation andfrequency

Elevation and frequency

60%
50%
40%
30%

or
1

20%

78%

10%

70%

8

7184
7469

10

22.5 sec

Elevation andfrequency

Fig. 4. This averaged saBCI spelling accuracies for each of the tested
spatial sound settings. The theoretical chance level was of 10%. The blue
color depicts the accuracy of horizontal sound setting; green the elevation
variability; and orange the elevation and frequency case, respectively. The
right side pie chart depicts the user preferences of the three spatial soundsettings with the same color coding, respectively.

be compared for a statistical significance with the former
study due to the different subject groups. We also analyzed
spelling accuracy for the various horizontal azimuths asshown in Table II. Compared with the horizontal placement
modality, the remaining two spatial sound settings resulted
with reduced accuracy errors. The results have shown thatthe variable stimuli elevation helped the users to distinguishbetter the horizontal sound locations either.
IV. C ONCLUSIONSThe presented EEG results confirmed the P300 responses
feasibility among the experienced and naive saBCI users.
The proposed spatial sound stimulus settings for various
elevations obtained using HRIR were effective for saBCI
paradigm. Additionally, the short ISI did not distract theusers’ perception, rather it apparently sharpened it resultingwith the better saBCI accuracies.
The obtained ITRs resulted with better scores comparing
to our previous study using simple HRIR and VBAP–based
spatial sound virtualization.
Nevertheless, current study is not yet ready to competewith the faster visual BCI spellers. Furthermore, it is necessary to improve the ITR for a comfortable online saBCI–
based spelling. We plan to extend the proposed saBCI to
TABLE IT HE AVERAGED ACCURACIES AND ITR S OF THE PROPOSED , OUR
PREVIOUS HRIR PROJECT AND THE CONVENTIONAL VBAP METHODS

Horizontal
Elevation
Elev. and freq.

Averagedaccuracy
59.5%
78.0%
60.5%

ITR
(bit/min)
8.51
14.92
8.81

Horizontal

55.8%

1.79

Horizontal

45.6%

0.57

Spatial sound mode
The proposed method withthe use of various HRIRs;
10 commands; ISI=150 ms
Previous HRIR
5 commands; ISI=300 ms
Conventional VBAP;
5 commands; ISI=500 ms

90%

85%

100%

R EFERENCES

0%
Elevation95%

realize the full set Japanese kana characters–based speller,
and to design a more effective spatial sound placement using
more precise elevation settings.

20%

Horizontal

90%

10%

61%

60%

Sound azimuth–based saBCI mean accuracies
−90◦
−45◦
0◦
45◦
90◦
85%
80%
95% 90%90%90%
95%
95% 90%95%[1] A. Kübler, A. Furdea, S. Halder, E. M. Hammer, F. Nijboer, andB. Kotchoubey, “A brain–computer interface controlled auditory eventrelated potential (p300) spelling system for locked-in patients,” Annalsof the New York Academy of Sciences, vol. 1157, no. 1, pp. 90–100,
2009.
[2] J. Wolpaw and E. W. Wolpaw, Eds., Brain-Computer Interfaces:
Principles and Practice. Oxford University Press, 2012.
[3] T. M. Rutkowski, A. Cichocki, and D. P. Mandic, “Spatial auditory
paradigms for brain computer/machine interfacing,” in International
Workshop On The Principles and Applications of Spatial Hearing 2009
(IWPASH 2009) - Proceedings of the International Workshop, MiyagiZao Royal Hotel, Sendai, Japan, November 11-13, 2009, p. P5.
[4] M. Schreuder, B. Blankertz, and M. Tangermann, “A new auditory
multi-class brain-computer interface paradigm: Spatial hearing as an
informative cue,” PLoS ONE, vol. 5, no. 4, p. e9813, 04 2010.
[5] H. Higashi, T. M. Rutkowski, Y. Washizawa, A. Cichocki, andT. Tanaka, “EEG auditory steady state responses classification for
the novel BCI,” in Proceedings of the 33rd Annual International
Conference of the IEEE EMBS. Boston MA, USA: IEEE Press,
August 30 - September 3, 2011, pp. 4576–4579.
[6] M. Chang, N. Nishikawa, Z. R. Struzik, K. Mori, S. Makino,
D. Mandic, and T. M. Rutkowski, “Comparison of P300 responses
in auditory, visual and audiovisual spatial speller BCI paradigms,”
in Proceedings of the Fifth International Brain-Computer Interface
Meeting 2013. Asilomar Conference Center, Pacific Grove, CA USA:
Graz University of Technology Publishing House, Austria, June 3-7,
2013, p. Article ID: 156.
[7] J. Schnupp, I. Nelken, and A. King, Auditory Neuroscience - Making
Sense of Sound. MIT Press, 2010.
[8] C. Nakaizumi, K. Mori, T. Matsui, S. Makino, and T. M. Rutkowski,“Auditory brain-computer interface paradigm with head related impulse response-based spatial cues,” in Signal-Image Technology &Internet-Based Systems (SITIS), 2013 International Conference on.
IEEE, 2013, pp. 806–811.
[9] S. Amano, S. Sakamoto, T. Kondo, and Y. Suzuki, “Development offamiliarity-controlled word lists 2003 (FW03) to assess spoken-word
intelligibility in Japanese,” Speech Communication, vol. 51, no. 1, pp.
76–82, 2009.
[10] W. G. Gardner and K. D. Martin, “HRTF measurements of a KEMAR,”The Journal of the Acoustical Society of America, vol. 97, no. 6, pp.
3907–3908, 1995.
[11] V. R. Algazi, R. O. Duda, D. M. Thompson, and C. Avendano, “TheCIPIC HRTF database,” in 2001 IEEE Workshop on the Applicationsof Signal Processing to Audio and Acoustics. IEEE, 2001, pp. 99–102.
[12] H. Kawahara, M. Morise, T. Takahashi, R. Nisimura, T. Irino, and
H. Banno, “TANDEM-STRAIGHT: a temporally stable power spectral
representation for periodic signals and applications to interference-free
spectrum, f0, and aperiodicity estimation,” in Acoustics, Speech andSignal Processing, 2008. ICASSP 2008. IEEE International Conference on. IEEE, 2008, pp. 3933–3936.
[13] G. Schalk and J. Mellinger, A Practical Guide to Brain–Computer
Interfacing with BCI2000. Springer-Verlag London Limited, 2010.
[14] D. J. Krusienski, E. W. Sellers, F. Cabestaing, S. Bayoudh, D. J.
McFarland, T. M. Vaughan, and J. R. Wolpaw, “A comparison ofclassification techniques for the P300 speller,” Journal of NeuralEngineering, vol. 3, no. 4, p. 299, 2006.

1074