2013 International Conference on Signal-Image Technology & Internet-Based Systems

EEG Signal Processing and Classiﬁcation for theNovel Tactile–Force Brain–Computer Interface
Paradigm
Shota Kono1 , Daiki Aminaka1 , Shoji Makino1 , and Tomasz M. Rutkowski1,2,∗
1Life Science Center of TARA, University of Tsukuba, Tsukuba, Japan
2
RIKEN Brain Science Institute, Wako-shi, Japan∗
Email: tomek@tara.tsukuba.ac.jp
http://bci-lab.info/

Abstract—The presented study explores the extent to which
tactile–force stimulus delivered to a hand holding a force–
feedback joystick can serve as a platform for a brain–computerinterface (BCI). The four pressure directions are used to evoke
tactile brain potential responses, thus deﬁning a tactile–force
brain computer interface (tfBCI). We present brain signal processing and classiﬁcation procedures leading to successful online
interfacing results. Experimental results with seven subjects
performing online BCI experiments provide a validation of thehand location tfBCI paradigm, while the feasibility of the concept
is illuminated through remarkable information–transfer rates.
Index Terms—EEG, tactile BCI, brain signal processing, brain
somatosensory evoked response.

I. I NTRODUCTION
The state–of–the–art BCI is usually based on mental visualand motor imagery paradigms, which require substantial user
training and good eyesight from the subject [1]. Alternative
solutions have been proposed recently to make use of spatial
auditory [2] or tactile (somatosensory) modalities [3], [4], [5],[6], [7] to enhance brain-computer interfacing comfort. The
concept proposed and reported based on a conducted pilot
study in this paper further extends the previously reportedby the authors [6], [8] brain somatosensory (tactile) channel
to allow for tactile–force based stimulus application. The
rationale behind the use of the tactile–force tactile channel
is that it is usually far less loaded and more intuitive to learncomparing with auditory or even visual modality interfacing
applications. A very recent report [7] additionally has conﬁrmed superiority of the tactile BCI in comparison with visualand auditory modalities as tested with a locked–in syndrome(LIS) subject [9].
Another report [5] proposed to utilize as the tactile BCI a
Braille–code stimulator with 100 ms static force push stimulus
delivered to each of six ﬁngers to generate a somatosensory
evoked potential (SEP) response and the following P300
attentional modulation. The P300 response is a positive electroencephalogram SEP deﬂection starting at around 300 ms
and lasting for 200 − 300 ms after an expected stimulus in
an oddball (random) series of distractors [1]. Examples ofFig. 1. The visual instruction screen presented to the subjects during thepsychophysical experiment developed in MAX 6 [11].

averaged P300 responses are depicted in form of color coded
diagrams in Figure 5 and using time series plot red lines withstandard errors in Figure 6.
The P300 brain response is considered to be the mostreliable and easy to capture from EEG in majority of humansubjects. Thus, the P300 is commonly used in BCI applications [1], [10].
This paper reports on the novel successful application of
the tactile–force BCI. We present very encouraging results
obtained with seven healthy subjects of whom the majority
scored with 100% accuracy in online BCI experiments.
The rest of the paper from now on is organized as follows.
The next section introduces the materials and methods usedin the tactile–force BCI study. It also outlines the experiments
conducted. The results obtained in EEG online experimentswith seven healthy BCI subjects are then discussed. Finally,
conclusions are formulated and directions for future research
are outlined.

∗ Corresponding author.

978-1-4799-3211-5/13 $31.00 © 2013 IEEE
DOI 10.1109/SITIS.2013.132

812

II. M ATERIALS AND M ETHODSThe experiments in the reported study involved sevenhealthy subjects (six males and one female; mean age of
24.71 years, with a standard deviation of 7.5 years). All
the experiments were performed at the Life Science Centerof TARA, University of Tsukuba, Japan. The online EEG
BCI experiments were conducted in accordance with The
World Medical Association Declaration of Helsinki - EthicalPrinciples for Medical Research Involving Human Subjects.
The psychophysical and EEG recording for BCI paradigm
experimental procedures were approved by the Ethical Committee of the Faculty of Engineering, Information and Systems
at University of Tsukuba, Tsukuba, Japan. The participants
agreed voluntarily to take part in the experiments. The detailsof the tactile–force stimulus creation, psychophysical andEEG experimental protocols are described in the following
subsections.
A. Tactile–Force Stimulus
The tactile stimuli were delivered as movements generated
by a portable computer in MAX 6 [11] environment as depicted
in form of visual interface with instructions to the subject inFigure 1. Each tactile stimulus was generated via a ForceFeedback Joystick Driver for Java [12]. The stimuli weredelivered to the subject’s right palm via the FLIGHT FORCE
joystick by Logitech.
There were four stimulus tactile–force direction patternsdelivered in random order to the subject hand. The directions
were labeled as North, East, West, and South as depicted in
Figure 2. For example, the North directions stimulus interaction caused the joystick generate a forward tactile–force pressure on the subject’s hand holding it. Similarly the South, East,and West stimulus directions were causing backward, right, andleft tactile–force pressures on the subject hand respectively.
The joystick returned to the center position (no pressure) after
the each presented stimulus after the presentation time of
100 ms (see Tables I and II with experimental condition details
summarized).
During the both psychophysical and EEG experiments the
subject held the joystick handle using a dominant hand (rightin case of all the subjects participating in this study) and
responded (button press in psychophysical– and mental conﬁrmation/counting in case of EEG–experiment) only to the
instructed direction. The instruction which directions to attend
were presented visually using the same MAX 6 environment
program that created the stimulus and communicated it via theJava driver to joystick as depicted in Figure 1.

Fig. 2. The force–feedback (or tactile–force) joystick FLIGHT FORCE by
Logitech used in experiments reported in this paper. The tactile–force stimulus
was delivered to the subject’s dominant hand. Four movements, deﬁned as
North, East, South and West directions, were executed randomly from acomputer causing the joystick handle to move and push the subject’s handautomatically.

depicted in Figure 1. In the each psychophysical experiment
the subject was presented with 80 target and 240 non–target
directions as stimuli.
Each trial was composed of 100 ms tactile–force pressuresdelivered to subject hand in randomized order with an inter–stimulus–interval (ISI) of 900 ms. Every random sequence
thus contained a single target and three non–targets. A single
session was composed of the ten trials for each tactile–force
target. The choice of the relatively long ISI was justiﬁed bya slow behavioral response in comparison to the EEG evoked
potential, as described in the next section. The tactile–force
psychophysical experiment protocol details are summarized inTable I.
The behavioral response times were registered with the same

TABLE ITACTILE – FORCE PSYCHOPHYSICAL EXPERIMENT PROTOCOL CONDITIONSAND DETAILSB. Tactile–Force Psychophysical Experiment Protocol
The psychophysical experiment was conducted to investigate the stimulus tactile–force direction inﬂuence on thesubject behavioral response time and accuracy. The behavioral
responses were collected using a trigger button on the joystickhandle and the MAX 6 program. The subject was instructedwhich stimulus to attend in each session by an arrow on
the computer display pointing the direction of an target as

Condition

DetailNumber of subjects
Tactile stimulus lengthInter–stimulus–interval (ISI)
Stimulus generation

7
100 ms
900 msFLIGHT FORCE joystickby Logitech
10

Number of trials for each subject

813

TABLE II
C ONDITIONS AND DETAILS OF THE TF BCI EEG EXPERIMENT
Condition

DetailNumber of subjects
Tactile stimulus lengthInter–stimulus–interval (ISI)
EEG recording system

7
100 ms
300 msgUSBamp active wet EEGelectrodes system
16Cz, CPz, P3, P4, C3, C4, CP5,
CP6, P1, P2, POz, C1, C2, FC1,
FC2, FCz
Behind the subject’s left ear
On the forehead (FPz)FLIGHT FORCE joystickby Logitech
10

Number of the EEG channels
EEG electrode positions

Reference electrode
Ground electrodeStimulus generation
Number of trials for each subject

Fig. 3. Tactile–force interface psychophysical experiment results in form of a
confusion matrix of the grand mean averaged subject accuracy results. Rowsof the above matrix denote the instructed targets and columns the subject
response. A diagonal of the matrix visualizes the correct response, while theoff–diagonal values the subject errors. Numerical percentage values represent
the response accuracies. In the contacted experiments the mean errors were
marginal (below one percent). There were also no systematic errors observed(common mistakes between pairs of patterns), which further validated thetactile–force stimulus design.

MAX 6 program, also used for the stimulus generation andinstruction presentation as depicted in Figure 1. The goal of thepsychophysical experiment was to investigate the behavioral
response times and target recognition accuracies in order totest cognitive loads (tasks difﬁculties) generated by the four
various tactile–force stimuli. The results of the experiment are
discussed in the Section III-A.
C. EEG tfBCI Experiment Protocol

response in noisy EEG [10], [14].

In the BCI experiments EEG signals were captured with aportable EEG ampliﬁer system g.USBamp by g.tec Medical
Instruments GmbH, Austria. Sixteen active wet EEG electrodes were used to capture brain waves with event related
potentials (ERP) with attentional modulation elucidated by theso–called “aha–” or P300–response [1]. The EEG electrodes
were attached to the head locations Cz, CPz, P3, P4, C3,
C4, CP5, CP6, P1, P2, POz, C1, C2, FC1, FC2, and FCz,
as in 10/10 intentional system [13]. A reference electrode
was attached to a left mastoid and a ground electrode on aforehead at FPz position respectively. No electromagnetic orelectromyographic (EMG) interference was observed from themoving joystick. Details of the EEG experimental protocol aresummarized in Table II.
The EEG signals were recorded and preprocessed by a
BCI2000–based application [10], using a stepwise linear
discriminant analysis (SWLDA) classiﬁer [14] with features
drawn from the 0 − 800 ms ERP interval. The sampling rate
was set to 256 Hz, the high–pass ﬁlter at 0.1 Hz, and the low–
pass ﬁlter at 40 Hz. The ISI was 300 ms and each tactile–force
stimulus duration of 100 ms.
Instructions to the subject which tactile–force stimulus
direction to attend were presented visually as in the previous
psychophysical experiments using the MAX 6 program (seeFigure 1). Each target was presented 10 times in a random
series with the remaining 30 non–targets in a single intended
direction classiﬁcation step. A procedure of ten single ERP
responses averaging was used in order to enhance the P300

III. R ESULTS
This section presents and discusses results that we obtainedin the psychophysical and in the online tfBCI experiments.
The very encouraging results obtained in the psychophysicaland tfBCI paradigm experiment support the proposed conceptof the tactile–force modality.
A. Tactile–force Psychophysical Experiment Results
The psychophysical experiment accuracy results are summarized in Table III, depicted in form of a confusion matrix
in Figure 3, and as boxplot response time distributions inFigure 4, where the median response time and the range are
depicted for each direction as the boxplots (see also Figure 1
for the directions).
This result conﬁrmed the stimulus related cognitive load
similarity since the behavioral responses for all the directions
were basically the same (as resulted with non–signiﬁcant median differences from a pairwise Wilcoxon–test). This ﬁnding
validated the design of the following tfBCI EEG experiment,
since the four tactile–force patterns resulted with similar cognitive loads as conﬁrmed by the same accuracies in Table III
and Figure 3, as well as by response times depicted in Figure 4.
B. Online EEG Tactile–Force BCI Experiment Results
The results of the conducted online tfBCI paradigm EEG
experiment with the seven subjects are presented in Figure 5 in814

evoked responses at the highest and lowest ERP separability
latencies in the target vs. non–target averaging scenario. The
highest average difference was found at 434 ms (as calculatedby AUC), which perfectly represented the P300 response peak
as could be seen also in Figure 6, where target and non–target
responses are visualized separately for each electrode. Figure 6
presents also a very interesting post–P300 attentional modulation, which in the majority of chosen for our experiments
electrodes, had extended positive ERP modulation beyond theclassical P300 peak in a range exceeding the 300 − 600 ms
range up to 1000 ms.
The online tfBCI accuracies (as obtained with SWLDA classiﬁer) of the all seven participating subjects are summarized inTable IV. All the seven subjects scored well above the chancelevel of 25 %. Four out of the seven subjects reached 100%
accuracy based on the 10 ERP responses averaging, which isa very good outcome of the proposed tfBCI prototype. Basedon the obtained accuracies we calculated the ITR scores, inorder to allow a simple comparison of the proposed tfBCI
paradigm with other published approaches. The obtained IRTscores were in the range from 1.04 bit/min to 10.00 bit/min(see Table V). The ITR was calculated as follows,
IT R = V · R,

(1)

where V is the classiﬁcation speed in selections/minute (5 selections/minute in this case) and R stands for the number ofbits/selection calculated as,


1−P
, (2)
R = log2 N + P · log2 P + (1 − P ) · log2
N −1
where N represents the number of classes (four in this study);
and P the classiﬁcation accuracy (see Table IV). The ITR
scores obtained by the BCI subjects in our study have beensummarized in Table V. The results shall be considered as
good outcomes in comparison of the state–of–the–art BCI [1].

Fig. 4. Boxplots of the tactile–force psychophysical experiment responsetime distributions of the four North, East, South and West joystick directions.
The differences among median were not signiﬁcantly different (as tested withpairwise Wilcoxon statistical test). The boxplots depict also response timeinterquartile ranges (edges of the boxes) of the response time distributions,which almost completely cover each other in the above plot.

IV. C ONCLUSIONSThe psychophysical and EEG experiments case study havebeed presented with obtained results which conﬁrmed thepractical application feasibility of the novel four–commands
tactile–force BCI. The results from experiments with sevenhealthy subjects conﬁrmed our hypothesis of the tfBCI application validity.
The EEG experiment in the tfBCI paradigm has conﬁrmedthat tactile–force stimuli could be used easily (without any
prior training) and successfully with ITR scores ranging from1.04 bit/min to 10.00 bit/min in the online interfacing caseusing the SWLDA classiﬁer.
The results presented offer a step forward in the development of the novel neurotechnology application. The current
paradigm obviously needs still improvements and modiﬁcations to implement online with faster ISI and lower averaging
rate necessary to improve the EEG features separability. These
calls to determine the major lines of study for future research.
However, even in its current form, the proposed tfBCI can be
regarded as a practical solution for LIS patients (locked into

form of matrices depicting ERP latencies with P300 response
together with areas under the curve (AUC) feature separability
analyses. We also present averaged topographic plots of the

TABLE IIIP SYCHOPHYSICAL EXPERIMENT RESULTS ( NOTE , THIS IS NOT A BINARY
ACCURACY CASE YET THE ONE WITH A THEORETICAL CHANCE LEVEL OF
25%) IN TACTILE – FORCE INTERFACE TASK .

Subject number

The best psychophysical accuracy

#1
#2
#3
#4
#5
#6
#7
Average:

100%
100%
100%
95%
100%
100%
100%
99.3%

815

TABLE IV
T EN TRIALS AVERAGING CLASSIFICATION BCI ACCURACY ( NOTE , THIS IS
NOT BINARY P300 CLASSIFICATION RESULT BUT RESULTING SPELLINGRESULT WITH A THEORETICAL CHANCE LEVEL OF 25%) IN
TACTILE – FORCE TASK USING THE CLASSICAL SWLDA CLASSIFIER [14].
Subject number

Online BCI experiment SWLDA accuracy

#1
#2
#3
#4
#5
#6
#7
Average:

100%
75%
100%
100%
50%
100%
50%
82.1%

signal processing and machine learning (classiﬁcation) methods. Next we will test the paradigm with the LIS patients inneed for BCI technology.
TABLE VT EN TRIALS AVERAGING CLASSIFICATION ACCURACY BASED ITRRESULTS ( SEE TABLE IV).
Subject number

ITR scores

#1
#2
#3
#4
#5
#6
#7
Average:10.00 bit/min
3.96 bit/min
10.00 bit/min
10.00 bit/min
1.04 bit/min
10.00 bit/min
1.04 bit/min
6.58 bit/min

AUTHOR C ONTRIBUTIONSProgrammed the tactile–force stimulus generating and delivering interface: SK, DA, TMR. Performed the EEG experiments: SK, TMR. Analyzed the data: SK, TMR. Conceived theconcept of the tactile–force BCI: TMR. Supported the project:
SM. Wrote the paper: SK, TMR.

Fig. 5. Grand mean ERP and AUC scores leading to ﬁnal classiﬁcation resultsof the participating seven subjects. The top panels represent head topographic
plots of the target versus non–target area under the curve (AUC) scores (AUC
is a measure commonly used in machine learning intra-class discriminative
analysis and AUC > 0.5 conﬁrms usually features separability). The top
left panel represents a latency of the largest difference as obtained from thedata displayed in the bottom panel of the ﬁgure. The top right panel depicts
the smallest AUC latency. Those topographic plots also show the electrodepositions. All the electrodes received similar AUC values (red) supports theinitial electrode placement in the conducted tfBCI EEG experiments. The
second panel from top represents averaged EEG responses to the target
stimuli (P300 response in the range of 400-800 ms). The third panel fromtop represents averaged EEG responses to the non–target stimuli (no P300
response). Finally, the bottom panel depicts the AUC of target versus non–target responses (P300 response latencies could be again easily identiﬁed here
by red color–coded values).

ACKNOWLEDGMENT
This research was supported in part by the Strategic Information and Communications R&D Promotion Program no.
121803027 of The Ministry of Internal Affairs and Communication in Japan.
R EFERENCES[1] J. Wolpaw and E. W. Wolpaw, Eds., Brain-Computer Interfaces: Principles and Practice. Oxford University Press, 2012.
[2] T. M. Rutkowski, A. Cichocki, and D. P. Mandic, “Spatial auditory
paradigms for brain computer/machine interfacing,” in International
Workshop On The Principles and Applications of Spatial Hearing 2009
(IWPASH 2009) - Proceedings of the International Workshop, MiyagiZao Royal Hotel, Sendai, Japan, November 11-13, 2009, p. P5.
[3] G. Muller-Putz, R. Scherer, C. Neuper, and G. Pfurtscheller, “Steadystate somatosensory evoked potentials: suitable brain signals for braincomputer interfaces?” Neural Systems and Rehabilitation Engineering,IEEE Transactions on, vol. 14, no. 1, pp. 30–37, March 2006.

their own bodies despite often intact cognitive functioning),
who cannot use vision or auditory based interfaces due tosensory or other disabilities.
We plan to continue this line of the tactile–force BCI
research in order to further optimize the feature extraction,

816

Fig. 6. Grand mean averaged ERP of all participating subjects together. Each panel depicts responses from each electrode used in the study (see Table II for
details). The red lines depict targets and black non–targets. The clear P300 responses could be seen in the range of 300 − 600 ms. The further attentional
modulation of target responses extends till 1000 ms.

Berlin Heidelberg, 2013, vol. 7989, pp. 50–59. [Online]. Available:
http://arxiv.org/abs/1305.4319[9] F. Plum and J. B. Posner, The Diagnosis of Stupor and Coma. Philadelphia, PA, USA: FA Davis, 1966.
[10] G. Schalk and J. Mellinger, A Practical Guide to Brain–Computer
Interfacing with BCI2000. Springer-Verlag London Limited, 2010.
[11] (2012) Max 6. [Online]. Available: http://cycling74.com/[12] M. Wischenbart. (2010) Force–Feedback joystick driver for Java.
[Online]. Available: http://boat.lachsfeld.at/ffjoystick4java/[13] V. Jurcak, D. Tsuzuki, and I. Dan, “10/20, 10/10, and 10/5
systems revisited: Their validity as relative head-surface-based
positioning systems,” NeuroImage, vol. 34, no. 4, pp. 1600 – 1611,
2007. [Online]. Available: http://www.sciencedirect.com/science/article/
pii/S1053811906009724[14] D. J. Krusienski, E. W. Sellers, F. Cabestaing, S. Bayoudh, D. J.
McFarland, T. M. Vaughan, and J. R. Wolpaw, “A comparison ofclassiﬁcation techniques for the P300 speller,” Journal of NeuralEngineering, vol. 3, no. 4, p. 299, 2006. [Online]. Available:
http://stacks.iop.org/1741-2552/3/i=4/a=007[4] A.-M. Brouwer and J. B. F. Van Erp, “A tactile P300 braincomputer interface,” Frontiers in Neuroscience, vol. 4, no. 19,
2010. [Online]. Available: http://www.frontiersin.org/neuroprosthetics/
10.3389/fnins.2010.00019/abstract[5] M. van der Waal, M. Severens, J. Geuze, and P. Desain, “Introducing
the tactile speller: an ERP-based brain–computer interface forcommunication,” Journal of Neural Engineering, vol. 9, no. 4, p.
045002, 2012. [Online]. Available: http://stacks.iop.org/1741-2552/9/i=
4/a=045002[6] H. Mori, Y. Matsumoto, Z. R. Struzik, K. Mori, S. Makino, D. Mandic,
and T. M. Rutkowski, “Multi-command tactile and auditory brain
computer interface based on head position stimulation,” in Proceedingsof the Fifth International Brain-Computer Interface Meeting 2013.
Asilomar Conference Center, Paciﬁc Grove, CA USA: Graz Universityof Technology Publishing House, Austria, June 3-7, 2013, p. Article ID:
095.
[7] T. Kaufmann, E. M. Holz, and A. Kuebler, “Comparison of tactile,
auditory and visual modality for brain-computer interface use: A casestudy with a patient in the locked-in state,” Frontiers in Neuroscience,
vol. 7, no. 129, 2013. [Online]. Available: http://www.frontiersin.org/
neuroprosthetics/10.3389/fnins.2013.00129/abstract[8] H. Mori, Y. Matsumoto, V. Kryssanov, E. Cooper, H. Ogawa,
S. Makino, Z. Struzik, and T. M. Rutkowski, “Multi-command
tactile brain computer interface: A feasibility study,” in Haptic andAudio Interaction Design 2013 (HAID 2013), ser. Lecture Notes in
Computer Science, I. Oakley and S. Brewster, Eds. Springer Verlag

817