Visual Motion Onset Augmented Reality Brain–computer Interface
Kensuke Shimizu, Takumi Kodama, Shoji Makino
Life Science Center of TARA
University of Tsukuba
Tsukuba, JapanTomasz M. Rutkowski1†‡?

† The University of Tokyo, Tokyo, Japan‡ Saitama Institute of Technology, Fukaya, Japan? RIKEN Brain Science Institute, Wako-shi, Japan

tomek@bci-lab.info
http://bci-lab.info/
Abstract— The paper presents a pilot study of a visual motion onset and augmented reality (AR) brain–computer interface (vmarBCI) paradigm. We also evaluate a BCI classification accuracy using a step–wise linear discriminant analysis method using different event related potential (ERP) averaging scenarios. We discuss thenovel visual motion in an AR set–up, which generates thevmarBCI stimuli. The AR set–up is created with a virtual
reality cardboard with a regular smartphone as a display.
Six visual motion patterns are presented to the user during the online vmarBCI experiments in an oddball style
paradigm allowing for brainwave “aha–responses” elucidation, within ERPs carrying P300 modulations. A subsequent classification accuracy comparisons are also discussed in online and offline vmarBCI studies. A research
hypothesis of the classification accuracy non–significant
differences among various numbers of ERP response averaging scenarios is confirmed.
Keywords— brain–computer interface (BCI); augmentedreality (AR); motion onset brain response.
I. INTRODUCTION
A brain–computer interface (BCI) is a modern neurotechnology employing the central nervous system (CNS) brain
signals (brainwaves) of a user (paralyzed or able–bodied) tocreate a new communication channel with others or to control external devices without depending on any muscle activity [1]. The BCI technology has provided a support already to patients’ life improvement who suffer from severe
paralysis due to diseases like an amyotrophic lateral sclerosis (ALS) [1]. The contemporary BCI applications rely
mostly on a static visual mode (no visual flow dynamics employed), which generates the most reliable event related potentials (ERP) so far [2]. Also many successful alternative
options have been developed recently to utilize spatial auditory [2, 3], tactile (somatosensory) [4, 5] or mixed [6] modes.
Meanwhile, the visual BCI still offers hard to beat communication options in comparison with the contemporary tactile
and auditory modes in case of locked–in syndrome (LIS) patients [7, 8]. We present results of a new study employing
an augmented reality (AR) environment realized with a headgear (a so–called “cardboard” set–up) as shown in Figure 1.
We propose to utilize a visual motion onset BCI paradigm [9]realized in a low cost and simple cardboard AR environment
with stereo vision display as shown in Figure 1c. The visual
1 The corresponding author.

onset motion and augmented reality BCI (vmarBCI) head
mounted display can generate various patterns to be appliedin the AR environment (see Figure 1c), and therefore it could
be adopted simply for those users with a good spatial vision.
The presented approach allows for creation of visually richand dynamic oddball paradigms [1, 9]. The goal of this pilot
study, with five users so far, is to compare and test a performance (a BCI classification accuracy) of the novel vmarBCIparadigm in comparison to the state of the art spatial tactile
and auditory modalities in function of various event–related
potential (ERP) averaging scenarios. Namely we take the averaging scenarios into account of ten, five, three, two and single ERPs processed. The pilot study results obtained withfive so far healthy users using a stepwise linear discriminant
analysis (SWLDA) classifier [10] and five different averaging
settings are analyzed and discussed in the paper.
From now on the paper is organized as follows. In the next
section we present methods developed in OpenVIBE [11]EEG experimental system together with Unity3D AR prototyping environment in order to capture, process and classify
the brainwave responses in the online proposed vmarBCI application. Online and offline EEG analysis results together
with conclusions summarize the paper.
II. METHODS
The visual motion AR stimuli are delivered through a cardboard mobile phone set–up as shown in Figures 1a and 1b.
Each visual motion stimulus pattern is generated in an oddball paradigm [1] style by OpenVIBE [11] scenario as shownin Figure 2. There are six oscillatory visual movement pattensdelivered in random order in order to elicit P300 brainwave
responses in an oddball style paradigm [1]. The visual motion
events are of 100 ms long with inter–stimulus intervals (ISI)of 300 ms as summarized in Table 1 with experimental condition details. During the online BCI EEG experiments, theuser wears the AR headgear attached with six moving patterns displayed randomly in the augmented vision environment programmed by our team in Unity3D as shown for a single box in Figure 1c. A smartphone camera is used to display
a real room with augmented graphical stimuli. The vmarBCI users respond mentally by confirming/counting only tothe instructed and spatially distributed visual motion patterns(see Figure 1c) while ignoring the others. The users are requested to spell sequences of six locations (six virtual digits
represented by the boxes in the AR).
The EEG vmarBCI experiments were conducted with ethical permission of Ethical Committee of RIKEN Brain Sci-

(a) The AR cardboard set–up #1 during the online BCI experiment with afirst user.

(b) The AR cardboard set–up #2 during the online BCI experiment with asecond user.

(c) The AR scene presented to the user within thecardboard with a room environment captured by afacing forward smartphone camera.

Figure 1: Users wearing the tested two different AR cardboard set–ups shown in panels (a) and (b) together with EEG caps
during the online vmarBCI experiments.
ence Institute and in accordance with The World Medical
Association Declaration of Helsinki - Ethical Principles for
Medical Research Involving Human Subjects. In the online vmarBCI experiments the EEG signals are captured with
an EEG amplifier g.USBamp by g.tec Medical Instruments,Austria. Eight EEG electrodes are attached to the head locations O1, O2, Cz, CPz, CP5, CP6, P3, and P4 as in 10/10
intentional system. A reference electrode is attached to a left
earlobe and a ground electrode on the forehead at FPz position respectively. The experimental details are summarized inTable 1.
The EEG signals are recorded and preprocessed online
by an in–house extended OpenVIBE application [11] withPython programmed network communication and classification units (see Figure 2). The EEG signals are segmented(“epoched”) as features drawn from ERP intervals of 0 ⇠800 ms (see Figure 3 with averaged results). The sampling
rate is set to 256 Hz, the high pass filter at 0.1 Hz, and the low
pass filter at 40 Hz, respectively. Each user performs four sessions of selecting the six patterns (a spelling of a sequence ofsix digits associated with each visual motion pattern). In online vmarBCI experiments each target is presented ten times
in a random series with remaining non–targets. We perform
also an offline analysis of the collected online EEG datasetsin order to test a possible influence of various ERP averaging scenarios by taking 10, 5, 3, 2 or single responses. Thestepwise linear discriminant analysis (SWLDA) [10] classifier is applied next in Python, with features drawn from the0 ⇠ 800 ms ERP intervals. A removal is performed of theleast significant input features, having p > 0.15, and with thefinal discriminant function restricted to contain a maximumof 60 features.
III. RESULTS
The results of the SWLDA in different averaging scenarios have been summarized in Table 2. The accuracy results
were dropping with lower ERP averaging settings, yet no statistically significant differences were observed among the results as evaluated with pairwise Wilcoxon rank sum tests. The
grand mean averaged ERP responses for rare targets (red) andnon–targets (blue) are depicted in Figure 3. The results allowed to draw a conclusion, based on the offline EEG analysis
of five users, that less averaging steps could be used in order
to boost the BCI interaction speed as suggested in Table 2.

Table 1: Conditions and details of the vmarBCI EEG experiment
Condition
Number of users
Visual motion stimulus lengthInter-stimulus-interval (ISI)
EEG recording system
Number of the EEG channels
EEG electrode positions
Reference electrode
Ground electrode
Sampling frequency
Classification method
Number of trials
Number of sessions

Detail5100 ms
300 msg.USBamp active wetelectrodes
8O1, O2, Cz, CPz,
CP5, CP6, P3, and P4
Behind the user’s left
earlobeOn the forehead (FPz)256 HzStepwise LDA (decimation filter by a factor 10)
10
5 (first session used fortraining)

IV. CONCLUSIONS
The aim of this study was to test the ERP responses variability
impact on the vmarBCI classification accuracy results. Theoffline results obtained with the SWLDA classifier did not
resulted with significant differences although lower averaging settings caused accuracy drops as summarized in Table 2.
One of the users could score the perfect accuracies with 10
trials averaging. All the obtained BCI accuracy results, withthe limited number of users so far (we continue hiring subjects), were above a theoretical chance level of 16.67%.
The preliminary results from the reported pilot study havebeen very promising for future online applications with users
suffering from neurodegenerative diseases as well as forhealthy people allowing for enriching the visual BCI stimuli presentation by introducing the spatial dynamic featuresin the AR set–up, as well as without significant classification
drops in relation to ERP averaging scenarios.
REFERENCES[1] J. Wolpaw and E. W. Wolpaw, Eds., Brain-Computer
Interfaces: Principles and Practice. Oxford University

Table 2: The vmarBCI accuracy results using ten, five,
three, two and single ERP averaging scenarios
User
number1
2
3
4
5
AverageUser
number12
345
AverageUser
number123
4
5
Average

Figure 2: The vmarBCI paradigm EEG acquisition and processing steps in OpenVIBE [11] environment. The yellow“Python processing” box was responsible to sending visualmotion onset trigger to the AR cardboard headgear shown inFigure 1.
Press, 2012.
[2] M. Chang, N. Nishikawa, Z. R. Struzik, K. Mori,
S. Makino, D. Mandic, and T. M. Rutkowski,“Comparison of P300 responses in auditory, visualand audiovisual spatial speller BCI paradigms,” inProceedings of the Fifth International Brain-Computer
Interface Meeting 2013. Asilomar Conference Center,
Pacific Grove, CA USA: Graz University of TechnologyPublishing House, Austria, June 3-7, 2013, p. Article
ID: 156. [Online]. Available: http://castor.tugraz.at/
doku/BCIMeeting2013/156.pdf[3] T. M. Rutkowski, A. Cichocki, and D. P. Mandic, “Spatial auditory paradigms for brain computer/machine interfacing,” in International Workshop On The Principles and Applications of Spatial Hearing 2009 (IWPASH 2009) - Proceedings of the International Workshop, Miyagi-Zao Royal Hotel, Sendai, Japan, November 11-13, 2009, p. P5.
[4] T. M. Rutkowski and H. Mori, “Tactile and boneconduction auditory brain computer interface forvision and hearing impaired users,” Journal ofUser
number1
2
3
4
5
AverageUser
number1
2
3
4
5
Average

Number of averaged ERPs = 10
Session#1#2#3
100.00% 100.00% 100.00%
100.00%83.33%
66.67%
66.67%
83.33%
50.00%
66.67%
50.00%
100.00%83.33%
66.67%
100.00%80.00%
Number of averaged ERPs = 5
Session#1#2#3
83.33%
83.33%
66.67%
66.67%
83.33%
75.00%
66.67%
33.33%
58.33%
66.67%
50.00%
91.67%
66.67%
66.67%
75.00%
66.67%
Number of averaged ERPs = 3
Session#1#2#372.22%66.67%
61.11%
55.56%
77.78%
55.56%
50.00%
44.44%
50.00%
44.44%
33.33%
72.22%
38.89%
50.00%
55.56%
54.17%
Number of averaged ERPs = 2
Session#1#2#3
60.00%
66.67%
60.00%
60.00%
60.00%
43.33%
43.33%
43.33%
30.00%
53.33%
33.33%
63.33%
50.00%
46.67%
46.67%
49.67%
Number of averaged ERPs = 1
Session#1#2#3
45.00%
46.67%
60.00%
48.33%
53.33%
50.00%
35.00%
31.67%
35.00%
45.00%
33.33%
55.00%
36.67%
35.00%
30.00%
42.42%#4
66.67%
83.33%
100.00%50.00%
83.33%#4
66.67%
66.67%
75.00%
25.00%
66.67%#444.44%
50.00%
50.00%
38.89%
72.22%#443.33%
43.33%
53.33%
40.00%
56.67%#443.33%
35.00%
45.00%
33.33%
51.67%

Neuroscience Methods, vol. 244, pp. 45 – 51,
2015. [Online]. Available: http://www.sciencedirect.
com/science/article/pii/S0165027014001265[5] H. Mori, Y. Matsumoto, Z. R. Struzik, K. Mori,
S. Makino, D. Mandic, and T. M. Rutkowski, “Multicommand tactile and auditory brain computer interface
based on head position stimulation,” in Proceedingsof the Fifth International Brain-Computer Interface
Meeting 2013. Asilomar Conference Center, PacificGrove, CA USA: Graz University of Technology
Publishing House, Austria, June 3-7, 2013, p. Article

Figure 3: Grand mean averaged ERP responses for rare targets (red) and non–targets (blue). The clear P300 responses to therare targets can be observed for EEG electrodes Cz, CPz, CP5, CP6, P3, and P4.
ID: 095. [Online]. Available: http://castor.tugraz.at/
doku/BCIMeeting2013/095.pdf[6] A. R. C. Donati, S. Shokur, E. Morya, D. S. F.
Campos, R. C. Moioli, C. M. Gitti, P. B. Augusto,S. Tripodi, C. G. Pires, G. A. Pereira, F. L. Brasil,S. Gallo, A. A. Lin, A. K. Takigami, M. A. Aratanha,
S. Joshi, H. Bleuler, G. Cheng, A. Rudolph, andM. A. L. Nicolelis, “Long-term training with a brainmachine interface-based gait protocol induces partial
neurological recovery in paraplegic patients,” Scientific
Reports, vol. 6, pp. 30 383 EP –, 08 2016. [Online].
Available: http://dx.doi.org/10.1038/srep30383[7] J. R. Patterson and M. Grabois, “Locked-in syndrome: areview of 139 cases.” Stroke, vol. 17, no. 4, pp. 758–64,
1986. [Online]. Available: http://stroke.ahajournals.org/content/17/4/758.abstract[8] M. van der Waal, M. Severens, J. Geuze, and P. Desain,“Introducing the tactile speller: an ERP-based brain–computer interface for communication,” Journal ofNeural Engineering, vol. 9, no. 4, p. 045002, 2012.
[Online]. Available: http://stacks.iop.org/1741-2552/9/i=4/a=045002[9] J. Pereira Junior, C. Teixeira, and T. M. Rutkowski,“Visual motion onset brain–computer interface,” arXiv,
Tech. Rep., July 2016. [Online]. Available: http://arxiv.org/abs/1607.02695[10] D. J. Krusienski, E. W. Sellers, F. Cabestaing,
S. Bayoudh, D. J. McFarland, T. M. Vaughan, and J. R.
Wolpaw, “A comparison of classification techniques
for the P300 speller,” Journal of Neural Engineering,
vol. 3, no. 4, p. 299, 2006. [Online]. Available:
http://stacks.iop.org/1741-2552/3/i=4/a=007[11] Y. Renard, F. Lotte, G. Gibert, M. Congedo, E. Maby,V. Delannoy, O. Bertrand, and A. Lécuyer, “Openvibe:
an open-source software platform to design, test, anduse brain-computer interfaces in real and virtual environments,” Presence: teleoperators and virtual environments, vol. 19, no. 1, pp. 35–53, 2010.