Proceedings of the Fifth International Brain-Computer Interface Meeting 2013

DOI:10.3217/978-3-85125-260-6-156

Comparison of P300 Responses in Auditory, Visual andAudiovisual Spatial Speller BCI Paradigms
M. Chang1, N. Nishikawa1, Z. R. Struzik2,3, K. Mori4, S. Makino1, D. Mandic5, T. M. Rutkowski1,21Life Science Center of TARA, University of Tsukuba, Japan; 2RIKEN Brain Science Institute, Wako-shi, Japan;
The University of Tokyo, Tokyo, Japan; 4Research Institute of National Rehabilitation Center for Persons withDisabilities, Tokorozawa, Japan; 5Imperial College London, London, UK3

Correspondence: T. M. Rutkowski, Life Science Center of TARA, University of Tsukuba, 1-1-1 Tennodai, Tsukuba, Ibaraki, Japan.
E-mail: tomek@tara.tsukuba.ac.jp

Abstract. The aim of this study is to provide a comprehensive test of three spatial speller settings, for the auditory,visual, and audiovisual paradigms. For rigour, the study is conducted with 16 BCI-naïve subjects in an experimental
set-up based on five Japanese hiragana characters. Auditory P300 responses give encouragingly longer target vs.
non-target latencies during the training phase, however, real-world online BCI experiments in the multimodal setting
do not validate this potential advantage. Our case studies indicate that the auditory spatial unimodal paradigm needs
further development in order to be a viable alternative to the established visual domain speller applications, as far asBCI-naïve subjects are concerned.
Keywords: EEG, P300, auditory evoked potentials, multimodal BCI

1. Introduction
Contemporary brain computer interface (BCI) paradigms rely mostly on unimodal approaches [Wolpaw andWolpaw, 2012]. Recently proposed solutions enhance the existing paradigms by adding spatial stimuli variability[Halder et al., 2010; Schreuder et al., 2010; Muller-Putz et al., 2006; van der Waal et al., 2012] in order to augment
the brain-computer interfacing comfort or to boost the information transfer rate (ITR) achieved by the users. We
tested this with 16 BCI-naïve subjects, in a simple five Japanese hiragana (a i u e o) spatial speller task to compare
the interfacing accuracy variability and the users' subjective comfort. In order to do so, we designed three spatial
tasks with visual letters and synthetic vowel representations originating from the congruent spatial directions using avirtual sound panning technique. We compare the results based on unimodal and bimodal spelling experiments anddraw conclusions for possible future research directions.

2. Material and Methods
In the experiments reported in this paper, 16 BCI-naïve subjects took part (mean age 21.81 with a standard
deviation of 0.75). All the experiments were performed at the Life Science Center of TARA, University of Tsukuba,
Japan. The online EEG BCI experiments were conducted in accordance with the WMA Declaration of Helsinki Ethical Principles for Medical Research Involving Human Subjects. The subjects of the experiments received amonetary gratification. The 200 ms long spatial unimodal (visual or auditory) or bimodal (audiovisual) stimuli werepresented from five distinct spatial locations. In the case of visual and audiovisual speller paradigms, large size
hiragana characters were flashed one at a time on a big computer display positioned in front of the subject (as in ausual oddball based P300 visual speller). In the case of the auditory modality, we designed a vector based soundamplitude panning application, which positioned a virtual sound image at a spatial location congruent with a visualletter (positioned at -45°, -22.5°, 0°, 22.5°, and 45° in front of the head). Each subject first conducted a shortpsychophysical test with a button press response to confirm understanding of the experimental set-up in eachmodality. During the online BCI experiments, the EEG signals were captured with a 16 active electrodes EEG
amplifier system, g.USBamp by g.tec. The electrodes were attached to the following head locations Cz, CPz, POz,Pz, P1, P2, C3, C4, O1, O2, T7, T8, P3, P4, F3, and F4, as in the 10/10 extended international system (seetopographic plot in Fig. 1). The ground and reference electrodes were attached at FCz and the earlobe respectively.
The recorded EEG signals were processed by the in-house enhanced BCI2000 application using a linear
discrimination analysis (LDA) classifier with features drawn from 0-600 ms event related potential (ERP) intervals.
The sampling frequency was set to 512 Hz, the high pass filter at 0.1 Hz, and the low pass filter at 100 Hz, with apower line interference notch filter set in the 48-52 Hz band. The inter-stimulus interval (ISI) was set to 500 ms andeach stimuli length was 200 ms. The subjects were instructed to spell five random sequences of hiragana letters,
which were presented visually, audibly or audiovisually in each session respectively. Each target was presented ten

Published by Graz University of Technology Publishing House, sponsored by

medical engineering GmbHArticle ID: 156

Proceedings of the Fifth International Brain-Computer Interface Meeting 2013

DOI:10.3217/978-3-85125-260-6-156

0.05

0.1

EEG electrode #610140ï5Targets vs. nonïtargets correlation
0.126010
ï0.1
14

ï0.04 ï0.02

0 100 300 500
time [ms]

(a)

00.02

0.040.06

Topographical plot of r at 428 ms

EEG electrode #Averaged targets [µV]20
ï2Averaged nonïtargets [µV]2261014

EEG electrode #ï52610140
ï2Targets vs. nonïtargets correlation
0.06
20.04
60.02
100
ï0.02
14
ï0.04
0 100 300 500
time [ms]EEG electrode #EEG electrode #EEG electrode #00Averaged nonïtargets [µV]52

EEG electrode #ï0.15 ï0.1 ï0.05

Topographical plot of r at 430 ms
52610
14ï0.1

ï0.050(b)

0.05

0.1

Averaged targets [µV]26101450
ï5Averaged nonïtargets52
610140ï5Targets vs. nonïtargets correlation
20.1
6010
ï0.1
14

EEG electrode #Averaged targets [µV]

Topographical plot of r at 449 ms

EEG electrode #times in a single spelling trial and the averages of ten ERPs were later used for the classification in order to make theexperiment easier for novices.

0 100 300 500
time [ms](c)

Figure 1. The averaged ERP responses of the 16 BCI-naïve subjects taking part in the experiment. Each panel (a), (b), and (c)presents topographical maps of target vs. non-target response correlation coefficient distributions at the mostdiscriminative latency. The averaged ERPs are also presented separately, together with correlation coefficient timeseries on the right side of each panel. Panel (a) presents visual speller results; (b) auditory; and (c) audiovisual. Theinteresting “longer” P300 averaged response deflection visualized in the top panel (b), and in the correlation timeseries, was not reflected in better accuracy scores (see Table 1). The EEG electrode on the vertical axes is shown in
the right panels. Electrode order is Cz, CPz, POz, Pz, P1, P2, C3, C4, O1, O2, T7, T8, P3, P4, F3, and F4.

3. Results and Discussion
The averaged ERP responses from the 16 BCI-naïve subjects are presented in Fig. 1 for the three testedmodalities. The results shown in panel (b) of the Fig. 1 are encouraging, since a more distinct (longer latencies)
P300 response is observed in the auditory unimodal paradigm. Unfortunately, this observation could not later beconfirmed in the form of correct spelling accuracies in the online BCI experiments reported in Table 1, where theclassical visual unimodal paradigm still resulted in the best scores. The topographic plots presented in Fig. 1 confirm
the classical P300 response scalp positions with more localized distributions for auditory unimodal experiments. The
preliminary, yet encouraging results presented call for more research on spatial auditory BCI paradigms in order
better to utilize the encouraging “longer” P300 response.
Table 1. Summary of the online BCI interfacing results with the 16 naïve subjects in auditory, visual and audiovisual modalities.
Modality

Mean accuracyStandard deviation

Auditory

52.5%

27.2%

Audiovisual

91.3%

12.6%

Visual

95.0%

11.5%

Acknowledgements
This research was supported in part by the Strategic Information and Communications R&D Promotion
Programme no. 121803027 of The Ministry of Internal Affairs and Communication in Japan, and by KAKENHI, theJapan Society for the Promotion of Science grant no. 12010738. We also acknowledge the technical support ofYAMAHA Sound & IT Development Division in Hamamatsu, Japan.
References
Halder S, Rea M, Andreoni R, Nijboer F, Hammer E, Kleih S, Birbaumer N, Kübler A. An auditory oddball brain–computer interface for binarychoices. Clin Neurophysiol, 121(4):516–523, 2010.
Schreuder M, Blankertz B, Tangermann M. A new auditory multi-class brain-computer interface paradigm: spatial hearing as an informative cue.
PLoS ONE, 5(4):e9813, 2010.
Wolpaw JR, Wolpaw EW. Brain-Computer Interfaces: Principles and Practice. Oxford University Press, 2012.

Published by Graz University of Technology Publishing House, sponsored by

medical engineering GmbH

Article ID: 156