SCIS&ISIS 2014, Kitakyushu, Japan, December 3-6, 2014

Tactile Pressure Brain-computer Interface Using
Point Matrix Pattern Paradigm
Kensuke Shimizu1 , Hiromu Mori1 , Shoji Makino1 , and Tomasz M. Rutkowski1,2,∗
1Life Science Center TARA, University of Tsukuba, Tsukuba, Japan
2
RIKEN Brain Science Institute, Wako-shi, Japan∗
Email: tomek@tara.tsukuba.ac.jp
http://bci-lab.info/

Abstract—The paper presents a tactile pressure stimulusbased brain-computer interface (BCI) paradigm. 3 × 3 pressure
pins matrix stimulus patterns are presented to the subjects in
an oddball paradigm allowing for “aha-responses” generation toattended targets. A research hypothesis is confirmed with theresults with five subjects performing online BCI experiments.
One of the users could score with 100% accuracy in online ten
averages based BCI test. Three users scored above chance levels,
while one remained on the chance level border. The presented
pilot study experiments and EEG results confirm the effectivenessof the proposed tactile pressure stimulus based BCI.

The remainder of the paper is organized as follows. In
the next section, the experimental setup and the tactile pressure paradigm are described, together with EEG signal pre–processing steps. Next, analysis and optimization procedures of
the ERP P 300 response latencies for all experimental users aredescribed. Finally, classification and discussion of the tpBCI
paradigm with information transfer rate (ITR) results conclude
the paper, together with future research directions.

Keywords—BCI; tactile BCI; P300; EEG; brain signal processing;

The experiments in the reported study involved five healthyusers (five males; mean age of 26.2 years, with a standard deviation of 10.0 years old). All the experiments were performedat the Life Science Center of TARA, University of Tsukuba,
Japan. The online EEG BCI experiments were conducted inaccordance with The World Medical Association Declarationof Helsinki - Ethical Principles for Medical Research Involving
Human Subjects. The psychophysical and EEG recording for
BCI paradigm experimental procedures were approved by the
Ethical Committee of the Faculty of Engineering, Informationand Systems at University of Tsukuba, Tsukuba, Japan (experimental permission no. 2013R7). The participants agreed
voluntarily to take part in the experiments and signed informed
consents.

I.

I NTRODUCTION

A brain–computer interface (BCI) is a technology thatuses neurophysiological signals (brainwaves) of paralyzed or
locked–in patients to allow communication with others or acontrol of external devices without depending on any muscle
activity [1]. The BCI technology has contributed already topatients’ life improvement who suffer from severe paralysis
due to diseases like an amyotrophic lateral sclerosis (ALS) [1].
The majority of BCI applications is based on a visual modality,
which generates the most reliable event related potentials(ERP) so far [2]. However, ALS patients in the advanced stages
sometimes lose their ability to control intentionally even theireye gazes [3], and therefore they need other types of BCI for
communication.
Alternative solutions have been proposed recently to makeuse of spatial auditory [2] or tactile modalities [3], [4]. Meanwhile, the tactile BCI seems to offer the better communication
options in comparison with visual and auditory modalities incase of locked–in–syndrome (LIS) patients [5]. The paradigm
proposed in this paper is a BCI using tactile pressure stimulus
generated by solenoids, which we refer to in brief as tactile
pressure BCI (tpBCI). The tpBCI device can have various
stimuli patterns generated and body area stimulated, andtherefore it could be adopted to various patient symptoms.
The presented approach allows for faster and more precisedelivery of tactile pressure stimuli comparing to the previously
proposed vibrotactile stimulator–based approaches [3], [4] andit is not limited to finger tips only [6].
The goal of this study is to evaluate the performance of thenovel tpBCI paradigm proposed and developed by our team.
We present the concept of the novel tpBCI and results obtainedwith five healthy users tested in online BCI experiments.
978-1-4799-5955-6/14/$31.00 ©2014 IEEE

473

II.

M ATERIALS AND METHODS

The details of the tactile pressure stimulus device, as wellas the psychophysical and BCI EEG experimental protocols
are described in the following subsections.
A. Tactile Pressure Stimulus Device
The tactile stimuli were delivered as light pressure patterns
generated by a portable computer with a program developed byour team on the ARDUINO micro–controller board managed
from our visual programming application MAX 6 [7]. Each
TABLE I.

D ETAILS OF THE USERS PARTICIPATING IN THE
EXPERIMENTS

User number

Sex

Age [years old]

#1
#2
#3
#4
#5

male
male
male
male
male

22
24
21
20
44

Average age

26.2

Age standard deviation

10.0

SCIS&ISIS 2014, Kitakyushu, Japan, December 3-6, 2014
TABLE II.

S TIMULUS TACTILE PRESSURE LINEAR PATTERNS USED IN
PSYCHOPHYSICAL AND BCI EEG EXPERIMENTS . T HE ⊗ SYMBOLSDEPICT ACTIVE ( CREATING PRESSURE ) AND
THE PASSIVE SOLENOIDS( NO PRESSURE ).
Pattern Number

Linear Patten
⊗⊗⊗

#1

#2

⊗⊗⊗

#3
⊗⊗⊗

Fig. 1. The tactile pressure generator composed of the solenoids put on theuser’s dominant hand covering index, middle and ring fingers. These solenoids
created six pressure patterns as explained in Table II. On the right side of thephotograph the ARDUINO micro–controller board and the in–house developed
multichannel amplifier are also depicted.

tactile stimulus was generated via the tactile pressure generator
composed of nine solenoids arranged in the 3 × 3 matrix as
depicted in Figure 1. The voltage binary outputs from the
ARDUINO board were amplified by a multichannel amplifier
developed also by our team as depicted in the right side of
Figure 1.
There were six linear pattens of tactile pressure stimuli
delivered in random order to the user fingers as shown in
Table II. A symbol of ⊗ depicts stimulated position andnon-stimulated one. Three of them #1, #2, and #3 werehorizontal lines ordered from the top to bottom of user’s fingers
respectively. The remaining patterns #4, #5, and #6 werethe vertical lines in left to right position order. The solenoids
generated quick light pressures 100 ms long (see Table III withexperimental condition details summarized).
During the both psychophysical and EEG experiments, theusers positioned the tactile device on their dominant hand
hand’s fingers (right hand in case of all the users participatingin this study) with a glove for preventing any electric noise
originating from the device. The users responded (button pressin psychophysical and mental confirmation/counting in caseof EEG BCI experiment) only to the instructed pattern while
ignoring the others. The training instructions were presented
visually by means of the MAX 6 program designed by ourteam as depicted in form of an user interface display in Figure
2.

#4

#5

#6

⊗
⊗
⊗
⊗
⊗
⊗
⊗
⊗
⊗

session was composed of ten runs for each tactile pressure
target. The choice of the relatively long ISI was justified bya slow behavioral response in comparison to the EEG evoked
potentials, as described in the next section. The tactile pressure
psychophysical experiment protocol details are summarized inTable III.
The behavioral response times were registered with thesame MAX 6 program, also used for the stimulus generationand instruction presentation as depicted in Figure 2. The goalof the psychophysical experiment, investigating behavioral
response times and target recognition accuracy in order totest an even distribution of cognitive loads (tasks difficulties)
among the six tactile pressure stimuli, was reached and theresults are discussed in the next section.

B. Tactile Pressure Psychophysical Experiment Protocol
The psychophysical experiments were conducted to investigate the influence of tactile pressure stimulus on theuser behavioral response time and accuracy. The behavioral
responses were collected using a trigger button on the keyboard
and a MAX 6 program. The user was instructed which target
stimulus to attend in each session by a red pattern shape of ×symbols on the computer display as depicted in Figure 2 andsummarized in Table II.
Each trial was composed of 100 ms tactile pressure patternsdelivered to the user fingers in randomized order with an interstimulus-interval (ISI) of 900 ms. Every random sequence
thus contained a single target and five non–targets. A single
978-1-4799-5955-6/14/$31.00 ©2014 IEEE

474

Fig. 2.
The visual instruction screen presented to the users during thepsychophysical experiment programmed in MAX 6 [7]. The red × symbols
inform about the patten shape to be attended by a user in each experimentalrun.

SCIS&ISIS 2014, Kitakyushu, Japan, December 3-6, 2014
TABLE III.

TACTILE PRESSURE PSYCHOPHYSICAL EXPERIMENT

Confusion matrix of the psychophysical responses

CONDITIONS AND DETAILS

100

Condition

Detail

6 0.0%

2.0%

2.0%

0.0%

2.0%

94.1%

0.0%

Number of users
Tactile stimulus lengthInter-stimulus-interval (ISI)
Stimulus generator
Number of trials for each user

5
100 ms
900 ms3 × 3 pressure pin matrix
10

5 0.0%

0.0%

0.0%

2.0%

98.0%

0.0%

0.0%

80
70
instructed target

TABLE IV.

C ONDITIONS AND DETAILS OF THE TP BCI EEG
EXPERIMENT

Condition

DetailNumber of users
Tactile stimulus lengthInter-stimulus-interval (ISI)
EEG recording system
Number of the EEG channels
EEG electrode positions
Reference electrode
Ground electrodeStimulus generator
Number of trials for each user

5
100 ms
400 msg.USBamp active wet EEG electrodes system
8Cz, Cpz, P3, P4, C3, C4, CP5, and CP6
Behind the user’s left earlobe
On the forehead(FPz)
3 × 3 pressure pins matrix
10

0.0%

0.0%

96.0%

2.0%

0.0%

0.0%

3 2.1%

2.1%

93.8%

2.1%

0.0%

0.0%

0.0%

2 0.0%

100.0% 0.0%

0.0%

0.0%

0.0%

0.0%

60
50
40

20
1 96.2%
1

In the BCI experiments EEG signals were captured with aportable EEG amplifier system g.USBamp by g.tec Medical
Instruments, Austria. Eight active wet EEG electrodes wereused to capture brainwaves with event related potentials (ERP)
with attentional modulation elucidated by the so-called “aha” or P300-response. The EEG electrodes were attached tothe head locations Cz, Cpz, P3, P4, C3, C4, CP5, andCP6 as in 10/10 intentional system [8]. A reference electrode
was attached to a left earlobe and a ground electrode on
the forehead at FPz position respectively. The users put onpolyethylene gloves in order to limit any electric interference
possibly induced by the tactile pressure generator placed on
their hands. The users were also requested to limit theireye–blinks and body movements to avoid electromagnetic
and electromyographic interference. Details of the EEG experimental protocol are summarized in Table IV. The EEG
signals were recorded and preprocessed by a BCI2000-based
application [9], using a stepwise linear discriminant analysis(SWLDA) classifier [10] with features drawn ERP interval of
0 ∼ 800 ms. The sampling rate was set to 256 Hz, the highpass filter at 0.1 Hz, and the low pass filter at 40 Hz. The ISI
was 400 ms and each tactile pressure stimulus duration was
100 ms.
Each user performed three sessions of selecting the six
patterns (a spelling of a sequence of six digits associated witheach tactile pressure pattern). Each target was presented tentimes in a random series with the remaining non-targets. A
procedure of ten single ERP responses averaging was used toenhance the P300 responses.
R ESULTS

This section presents and discusses results that we obtainedin the psychophysical and in the online tpBCI experiments.
The very encouraging results obtained in the tpBCI paradigm
support the proposed new tactile interface concept.
978-1-4799-5955-6/14/$31.00 ©2014 IEEE

4 2.0%

30

C. EEG tpBCI Experiment Protocol

III.

90

475

0.0%

0.0%

2

3

0.0%

1.9%

1.9%

0.0%

4
5
subject response

6

N

10
0

Fig. 3.
Tactile pressure psychophysical experiment results in form of a
confusion matrix of the grand mean averaged user accuracy results. The
horizontal axis represents the instructed targets and vertical axis the user
responses. N in the responses means ”no–answer” condition, in which auser missed to push a keyboard. The off–diagonal responses and accuracies
represent the marginal errors made by the users.

A. Tactile pressure psychophysical experiment results
The psychophysical experiment accuracy results are depicted in form of a confusion matrix in Figure 3, and as boxplot
response time distributions in Figure 4, where the median
response times and the interquartile ranges are depicted foreach patterns respectively (see also Table II for the patterns).
This result confirmed the stimulus similarity since thebehavioral responses for all the patterns were basically thesame. This finding validated the design of the following
tpBCI EEG experiment, since the six tactile pressure patterns
resulted with similar cognitive loads as confirmed by the sameaccuracies and response times.
B. Online EEG tpBCI Experiment Results
The results of the conducted online tpBCI paradigm EEG
experiment with the five users are presented in Figure 5 inform of matrices depicting ERP latencies with P300 response
together with areas under the curve (AUC) feature separability
analyses. We also present averaged topographic plots of theevoked responses at the latencies of the highest and lowest
ERP separability in target vs. non-target scenario. The highest
average difference was found at 422ms (as calculated by AUC),
which perfectly represented the P300 response peak as could
be seen also in Figure 6, where target and non-target response
are visualized separately for each electrode. The online tpBCI
accuracies (as obtained with SWLDA classifier) of the all five
participating users are summarized in Table V. the four outof five users scored well above the chance level of 16%,
which is a good outcome of the proposed tpBCI prototype.
Based on the obtained accuracies we calculated to allow
simply comparison of the proposed tpBCI paradigm with otherpublished approaches, the ITR scores which were in the range

SCIS&ISIS 2014, Kitakyushu, Japan, December 3-6, 2014
TABLE V.
O NLINE BCI EXPERIMENT SWLDA ACCURACY RESULTS
FROM THREE EXPERIMENTAL SESSIONS CONDUCTED WITH EACH SUBJECT

Psychophysical experiment response times
1000
6

User number

Session #1

Session #2

Session #3

Average score

#1
#2
#3
#4
#5

50.0%
33.3%
50.0%
0.0%
100.0%

33.3%
50.0%
50.0%
16.7%
88.3%

66.7%
33.3%
66.7%
0.0%
83.3%

50%
38.9%
55.6%
5.6%
90.6%

Average

48.1%

900
5
800

TABLE VI.

T EN TRIALS AVERAGING – BASED CLASSIFICATION
ACCURACY ITR RESULTS FROM THREE EXPERIMENTAL SESSIONSCONDUCTED WITH EACH SUBJECT

4time [ms]700

User number

Session #1

Session #2

Session #3

Average score

#1
#2
#3
#4
#5

0.84 bit/min
0.24 bit/min
0.84 bit/min
0.00 bit/min
5.16 bit/min

0.24 bit/min
0.84 bit/min
0.84 bit/min
0.00 bit/min
3.59 bit/min

1.78 bit/min
0.24 bit/min
1.78 bit/min
0.00 bit/min
3.59 bit/min

0.84 bit/min
0.40 bit/min
1.13 bit/min
0.16 bit/min
3.83 bit/min

Average

0.76 bit/min

3
600

2
500

400

300

According to the results obtained from the online EEG BCI
experiments, a single user could score once with perfect accuracy and above 90% on average. Also one user balances below
chance level on average. The remaining three users scoredabove chance levels. The preliminary, yet encouraging results
support the initial research hypothesis of tactile pressure–basedstimulus validity for the BCI paradigms.

1

1

2

3
4
response numbers

5

0

6

Fig. 4. Psychophysical experiment response time probability distributionsof five users together summarized in form of distribution–shaped box plots
depicting also medians and interquartile ranges. Each number at the horizontal
axis represents the stimuli pattern. The horizontal axis represents the user
reaction time delays in milliseconds.

The current paradigm obviously needs still improvements
and modifications. However, even in its current form, theproposed tpBCI can be regarded as a alternative solution for
LIS patients, who cannot use vision or auditory modality.

from 0.16 bit/min to 3.83 bit/min (see Table VI). The ITR was
calculated as follows,

We plan to make the tactile pressure device smaller andmore portable in near future. The users shall be able to enjoy
the tpBCI more comfortably with possibly better accuracies.
A demo with a video of the proposed paradigm with single
trial averaging mode is available at [11].

IT R = V · R

AUTHOR C ONTRIBUTIONS(1)

where V is the classification speed in selections/minute (2selections/minute in this case) and R stands for the number ofbits/selection calculated as,
R = log2 N + P · log2 P + (1 − P ) · log2

1−R
N −1

ACKNOWLEDGMENTS(2)with N being a number of classes (six in this study); andP the classification accuracy (see Table V).
IV.

C ONCLUSIONSThe purpose of our research was to develop a six–commands–based tactile pressure BCI paradigm. We conducted psychophysical and online EEG BCI experiments toverify the efficiency of the proposed tactile pressure BCI withfive healthy subjects. Results obtained from psychophysicalexperiments confirmed the tpBCI paradigm design validity.
978-1-4799-5955-6/14/$31.00 ©2014 IEEE

Designed and performed the EEG experiments, as well as
analyzed the data: KS, HM, TMR. Conceived the concept of
the tactile pressure BCI: TMR. Supported the project: SM.
Wrote the paper: KS, TMR.

476

This research was supported in part by the Strategic
Information and Communications R&D Promotion Program(SCOPE) no. 121803027 of The Ministry of Internal Affairs
and Communication in Japan, and by KAKENHI, the Japan
Society for the Promotion of Science, grant no. 24243062.
We also acknowledge the technical support from
YAMAHA Sound & IT Development Division in Hamamatsu,
Japan.
R EFERENCES[1]

J. Wolpaw and E. W. Wolpaw, Eds., Brain-Computer Interfaces: Principles and Practice. Oxford University Press, 2012.

SCIS&ISIS 2014, Kitakyushu, Japan, December 3-6, 2014
Averaged targets

max(AUC) at 422 ms

0.54

[µV]

0.56

0.52

Cz
Pz
P3
P4
C3
C4
CP5
CP6

2
0
0

0.5

[µV]
0.56

0.48

900
4
2
0

300

600

900

Targets vs. non−targets AUC scores

AUC

0.52
0.5

600

Cz
Pz
P3
P4
C3
C4
CP5
CP6
0

0.54

300

Averaged non−targets

0.48

min(AUC) at 240 ms

4

0.56

Cz
Pz
P3
P4
C3
C4
CP5
CP6

0.54
0.52
0.5
0.48
0

300

600

900

time [ms]Fig. 5. Grand mean ERP and AUC score leading to final classification resultsof the all participating users. The left panels represent the head topographic
plots of the target versus non-target area under the curve (AUC), which
is a measure commonly used in machine learning intra-class discriminative
analysis. (AU C > 0.5 is usually assumed to be confirmation of features
separability). The first panel of the left side represents a head topographic plot
with a latency of the largest difference as obtained from the data displayed in
the bottom panel of the figure. The bottom right panel represents the smallest
AUC latency. Those topographic plots also show the electrode positions. The
fact that all the electrodes received similar AUC values supports the initial
electrode placement. The first panel of the right side represents averaged EEG
responses to the target stimuli (P300 response in the range of 300 ∼ 600 ms).
The second from the top right panel represents averaged EEG responses to thenon–target stimuli (no P300 response). Finally, the bottom right panel depicts
the AUC of target versus non–target responses (P300 response latencies could
be again easily identified here by purple color–coded values).

[2]

M. Chang, N. Nishikawa, Z. R. Struzik, K. Mori, S. Makino,
D. Mandic, and T. M. Rutkowski, “Comparison of P300 responses
in auditory, visual and audiovisual spatial speller BCI paradigms,”
in Proceedings of the Fifth International Brain-Computer Interface
Meeting 2013. Asilomar Conference Center, Pacific Grove, CA
USA: Graz University of Technology Publishing House, Austria,
June 3-7, 2013, p. Article ID: 156. [Online]. Available: http://castor.tugraz.at/doku/BCIMeeting2013/156.pdf

Fig. 6. Grand mean averaged ERP of all participating users. Each panel
depicts responses from electrodes used in the study (see Table IV for details).
The purple lines depict targets and blue non–targets
respectively with standard√
error bars (standard deviation divided by n, where n stands for a numberof averaged trials). The clear P300 responses could be seen in the range of300 ∼ 600 ms.

[3]

T. M. Rutkowski and H. Mori, “Tactile and bone–conduction auditory
brain computer interface for vision and hearing impaired users,” Journalof Neuroscience Methods, p. Available online 21 April 2014, 2014.
[Online]. Available: http://dx.doi.org/10.1016/j.jneumeth.2014.04.010

[7]

[4]

[5]

[6]

H. Mori, Y. Matsumoto, V. Kryssanov, E. Cooper, H. Ogawa,
S. Makino, Z. Struzik, and T. M. Rutkowski, “Multi-command
tactile brain computer interface: A feasibility study,” in Haptic andAudio Interaction Design 2013 (HAID 2013), ser. Lecture Notes in
Computer Science, I. Oakley and S. Brewster, Eds. Springer Verlag
Berlin Heidelberg, 2013, vol. 7989, pp. 50–59. [Online]. Available:
http://arxiv.org/abs/1305.4319
T. Kaufmann, E. M. Holz, and A. Kuebler, “Comparison of tactile,
auditory and visual modality for brain-computer interface use: A casestudy with a patient in the locked-in state,” Frontiers in Neuroscience,
vol. 7, no. 129, 2013. [Online]. Available: http://www.frontiersin.org/
neuroprosthetics/10.3389/fnins.2013.00129/abstract
M. van der Waal, M. Severens, J. Geuze, and P. Desain, “Introducing
the tactile speller: an ERP-based brain–computer interface forcommunication,” Journal of Neural Engineering, vol. 9, no. 4, p.
045002, 2012. [Online]. Available: http://stacks.iop.org/1741-2552/9/i=
4/a=045002

978-1-4799-5955-6/14/$31.00 ©2014 IEEE

477

[8]

[9]
[10]

[11]

http://cycling74.com/, “Max 6,” 2012. [Online]. Available: http://cycling74.com/
V. Jurcak, D. Tsuzuki, and I. Dan, “10/20, 10/10, and 10/5 systems
revisited: Their validity as relative head-surface-based positioning
systems,” NeuroImage, vol. 34, no. 4, pp. 1600 – 1611, 2007.
[Online]. Available: http://www.sciencedirect.com/science/article/pii/
S1053811906009724
G. Schalk and J. Mellinger, A Practical Guide to Brain–Computer
Interfacing with BCI2000. Springer-Verlag London Limited, 2010.
D. J. Krusienski, E. W. Sellers, F. Cabestaing, S. Bayoudh, D. J.
McFarland, T. M. Vaughan, and J. R. Wolpaw, “A comparison ofclassification techniques for the P300 speller,” Journal of NeuralEngineering, vol. 3, no. 4, p. 299, 2006. [Online]. Available:
http://stacks.iop.org/1741-2552/3/i=4/a=007“Virtual reality walk using tactile and bone-conduction auditory BCIs.”[Online]. Available: http://youtu.be/mLT-CpV5l20