# An introduction to Brain Computer Interface

Christelle Zielinski

► To cite this version:

Christelle Zielinski. An introduction to Brain Computer Interface. Master. Cognitive engineering, Aix-Marseille Université, Saint-Charles, France. 2022. [hal-03979161](https://hal.science/hal-03979161)

HAL Id: hal-03979161

https://hal.science/hal-03979161

Submitted on 8 Feb 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
An
introduction
to Brain
Computer
Interface

Christelle Zielinski

Centre de Ressources pour l'Expérimentation

Institute of Language, Communication and the Brain

UE ingeco Janv. 2022

<figure><img src="image_2.png" /></figure>
# An introduction to Brain Computer Interface

<figure><img src="image_3.png" /></figure>

## Brain-Computer Interfaces 1

### *Foundations and Methods*

Edited by Maureen Clerc
Laurent Bougrain and Fabien Lotte

<figure><img src="image_4.png" /></figure>

WILEY

Understanding & Improving Mental-Imagery Based
Brain-Computer Interface (Mi-Bci) User-Training:
towards A New Generation Of Reliable, Efficient &
Accessible Brain- Computer Interfaces

Camille Jeunet

► To cite this version:

Camille Jeunet. Understanding & Improving Mental-Imagery Based Brain-Computer Interface (Mi-Bci) User-Training: towards A New Generation Of Reliable, Efficient & Accessible Brain- Computer Interfaces. Psychology. Université de Bordeaux, 2016. English. NNT: 2016BORD0221. tel-01417606

<figure><img src="image_5.png" /></figure>

sensors

Review

EEG-Based Brain-Computer Interfaces Using
Motor-Imagery: Techniques and Challenges

<figure><img src="image_6.png" /></figure>

Natasha Padfield<fnref n="1" />, Jaime Zabalza<fnref n="1" />, Huimin Zhao<fnref n="2" />,<fnref n="3" />,*, Valentin Masero<fnref n="4" /> and Jinchang Ren<fnref n="1" />,<fnref n="5" />,•

<footnote n="1">1 Centre for Signal and Image Processing, University of Strathclyde, Glasgow G1 1XW, UK; natasha.padfield@strath.ac.uk (N.P.); j.zabalza@strath.ac.uk (J.Z.)</footnote>

<footnote n="2">2 School of Computer Sciences, Guangdong Polytechnic Normal University, Guangzhou 510665, China</footnote>

<footnote n="3">3 The Guangzhou Key Laboratory of Digital Content Processing and Security Technologies, Guangzhou 510665, China</footnote>

<footnote n="4">4 Department of Computer Systems and Telematics Engineering, Universidad de Extremadura, 06007 Badajoz, Spain; vmasero@unex.es</footnote>

<footnote n="5">5 School of Electrical and Power Engineering, Taiyuan University of Technology, Taiyuan 030024, China</footnote>

<footnote>* Correspondence: zhaohuimin@gpnu.edu.cn (H.Z.); jinchang.ren@strath.ac.uk (J.R.)</footnote>
# Brain Computer Interface

<figure><img src="image_7.png" /></figure>

Wolpaw, 2002: tools of communication and control that allow users to interact with their environment by means of their cerebral activity alone

* To consciously act and communicate without using our muscles

* Target population has been people with motor disabilities and preserved cognitive functions (assistive devices)

* Recent years: rehabilitation of motor or even cognitive functions for patients after CNS injury, brain state monitoring for healthy subjects, and new tools for studying human brain functions
# Historical events

<figure><img src="image_8.png" /></figure>

* **1875** Richard Caton (UK medical Dr), the pioneer in electrophysiology the brain produces electrical activity (electrodes in dog's brain)

* **1924** Hans Berger (German psychiatrist and neurologist), the father of the EEG – 1st human EEG, alpha wave description, alteration during epileptic seizures

<figure><img src="image_9.png" /></figure>

* **1960s** Joe Kamiya (Pr. of psychology, Berkeley) – 1st neurofeedback paradigm, training of participant to enter the alpha state

~~~ 1970 – development of computer sciences, simple app using neurofeedback ~~~

Some attempts by music composers to produce music in real time from brain activity

* **1973** Pr. Jacques Vidal (computer science, Los Angeles) – 1st description of the concept of BCI in “Towards direct brain-computer communication”

~~~ 90's – First real-time BCI designs ~~~

J. R. Wolpaw et al., 1991, “An EEG-based brain-computer interface for cursor control”
<figure><img src="image_10.png" /></figure>

Closed loop systems usually composed of 6 main stages
# Architecture

<figure><img src="image_11.png" /></figure>

Acquisition of raw data that reflect the user's brain activity
EEG widely used:

* ✓ non-invasive, high time resolution, potential for mobility, low cost

* × processing challenges: non-stationnary, external noise, signal artefacts
<figure><img src="image_12.png" /><figcaption>Figures 8 to 11 depict four examples for artifacts in EEG devices:</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>Fig. 8: 60Hz power line artifacts without (A) and with (B) an applied filter (from Marcuse et al., 2016, p. 34).</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>Fig. 9: The arrows indicate ECG (electrocardiography - heart muscle) artifacts (from Marcuse et al., 2016, p. 20).</figcaption></figure>

<figure><img src="image_15.png" /><figcaption>Fig. 10: Marked by the arrows are jaw muscle artifacts (from Marcuse et al., 2016, p. 27).</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>Fig. 11: These highly disruptive artifacts were produced by alternating tooth grinding (from Marcuse et al., 2016, p. 28).</figcaption></figure>

Cleaning up and reducing noise from raw data to extract the relevant information
# Architecture

<figure><img src="image_17.png" /></figure>

Ex. Band power in the mu and beta rhythms in C3 and C4

Extraction of discriminative and non-redundant information from the EEG to form a set of features on which classification can be carried out

Certain patterns of brain signals that can be categorized into task-specific brain state (ex. right hand movement)

The feature extraction method depends on the type of mental task
# Architecture

<figure><img src="image_18.png" /></figure>

<figure><img src="image_19.png" /></figure>

Associates a class to a set of features – machine learning
5

<figure><img src="image_20.png" /><figcaption>Applications</figcaption></figure>

<figure><img src="image_21.png" /><figcaption>Wheel chair</figcaption></figure>

<figure><img src="image_22.png" /><figcaption>Prosthesis</figcaption></figure>

<figure><img src="image_23.png" /><figcaption>IT applications
(text editors,
video games, etc.)</figcaption></figure>

Associates a command with the identified brain activity pattern / intention
Software / Hardware
# Architecture

<figure><img src="image_24.png" /></figure>

The brain activity pattern that have been recognized → user

Helps the user to learn to modulate brain activity to improve the control of
the BCI
Offline/Online steps

<figure><img src="image_25.png" /></figure>

(1) an offline calibration stage, during which the system's settings are determined

(2) an online operational stage, during which the system recognizes the user's brain activity patterns and translates them into application commands

BCI research community is currently searching for solutions to help avoid
the costly offline calibration stage
## Active / Reactive / Passive

<figure><img src="image_26.png" /></figure>

<figure><img src="image_27.png" /></figure>

### Active

* user actively employed by performing mental task to control the system (endogen origin) (ex. imagine hand movement)

* 2 main paradigms: Slow Cortical Potential based BCIs (SCP-BCI) and Mental-Imagery based BCIs (MI-BCI)
# BCIs properties

## Active / Reactive / Passive

<figure><img src="image_28.png" /><figcaption>a) Active BCI</figcaption></figure>

<figure><img src="image_29.png" /><figcaption>b) Reactive BCI</figcaption></figure>

## Reactive

* employs the user's brain reactions to given stimuli (exogen origin)

* Visual, auditory or sensory stimulation

* ERP-based (P300) and Steady-State Visually Evoked potentials (SSVEPs)
# BCI properties

## Active / Reactive / Passive

### Passive

* passively analyses the user's mental state in real time

* ex. their cognitive (workload), emotional (frustration) or motivational states, which can also be combined with other physiological and behavioral data

* an application/the environment can be adapted to the user's state (ex. user not motivated for an exercise, exercise can be modified)

* Cerebral rhythms...

<figure><img src="image_30.png" /></figure>
# BCIs properties

Active / Reactive / Passive

Synchronous / asynchronous

user can only control a BCI at specific times / at any time

Dependent / independent

depends on motor control (ex. user has to move his eyes to observe a stimuli) / without any movement at all

Invasive / non-invasive

Hybrid

combination of BCIs (ex. MI task + brain responses to stimuli SSVEP, to release the cognitive load); BCI + non-cerebral commands (muscular signals) or other interaction mechanisms (mouse)
# Markers examples / Active BCI

## Slow Cortical Potential

<figure><img src="image_31.png" /></figure>

* Voluntary variations of slow cortical potentials

* Cognitive ERP triggered from 300ms to several s after an internal or external event (expected) occurs - **internal**: people can train to modulate the SCP (positively or negatively with respect to the baseline) in a self-paced way

* Extensive training (~3 months) to learn to control SCPs

* First SCP-BCI enabling to control a spelling device: "A spelling device for the paralysed", Birbaumer et al., 1999
# Markers examples / Active BCI

## Mental-imagery task

* Mental tasks: motor-imagery of the limb, mental calculation or navigation

* Induce in specific cortical regions:

1. Event-Related Desynchronisation (ERD – while being performed)

2. Event-Related Synchronisation (ERS – once the user has stopped)

<figure><img src="image_32.png" /><figcaption>Ex. ERD/ERS during motor imagery tasks (Left/Right hand movements)</figcaption></figure>
# Markers examples / Active BCI

## Mental-imagery task

* ERD/ERS

* can "be considered to be due to a decrease or an increase in synchrony of the underlying neuronal populations" (Pfurstcheller et al)

* suggested to reflect cortical activation and deactivation

* Rather natural and intuitive to use, since their users "just" have to imagine a specific task to send control commands

* Enable asynchronous control and do not require valuable attentional resources // reactive BCI

* Co-adaptation user / machine - substantial training to find the correct imagination strategy that leads to the most distinct ERD/ERS

* Can currently only be used with a maximum of 3 or 4 different MI tasks to ensure maximal performances (recognition accuracy)
# Markers examples / Reactive BCI

## P300

* Landmark technologies

* Positive cortical potential around 300 ms after the occurrence of a rare and relevant event which is expected by the user (ERP) [parietal lobe]

<figure><img src="image_33.png" /></figure>

* No much user-training requiring and very reliable

* Results need to be integrated over several stimuli: computational time

* Considerable attentional resources, preventing use in interactive situation that requires high levels of attention (such as navigation and control)
# Markers examples / Reactive BCI

## SSVEP / SSSEP / ASSEP

* Looking at (or paying attention to) a **flashing stimulus** at a frequency f produces potential variations in the occipital lobe at a close frequency f (over-representation of the fovea)

* Detectable by a frequency analysis (Fourier) amplitude/phase.

<figure><img src="image_34.png" /></figure>

<figure><img src="image_35.png" /></figure>
# Markers examples / Reactive BCI

## SSVEP / SSSEP / ASSEP

* Stimulation frequencies between 3 and 40 Hz :

* < 3 Hz insufficient for a steady-state; brain responses = a succession of unitary responses

* > 40 Hz, the SSVEP's amplitude becomes too small to be detected from surface EEG signals.

* Easy to extract with basic processing of EEG signals

* Characteristics relatively independent of the subject, with the condition of specifically selecting electrode positions providing maximum amplitude

* Little affected by muscular artifacts such as blinking and facial muscles' EMG
# Signal processing

## Feature extraction (based on specific markers) and
Classification

<figure><img src="image_36.png" /></figure>

Hoffmann et al., 2006: "A good feature extraction method should reduce the dimension of the input data as much as possible while keeping all the information necessary for classification."
# Signal processing

<figure><img src="image_37.png" /></figure>

<figure><img src="image_38.png" /><figcaption>Fig. 8. Linear Discriminant Analysis (LDA) projection [26].</figcaption></figure>

<figure><img src="image_39.png" /><figcaption>Fig. 9. Linear and non-linear support vector machines</figcaption></figure>

<figure><img src="image_40.png" /></figure>

Classifier
accuracy

Deep learning: single signal processing block (group together the feature extraction, feature selection and classification tasks)
## A wide variety of the different pipeline structures

<table><caption>**Table 2.** A comparison of the different combinations BCI structures used in the literature, including features extracted, feature selection approach if used and classification method.</caption><thead><tr><td>Paper</td><td>Feature Extraction Method <sup>1</sup></td><td>Feature Selection Method <sup>2</sup></td><td>Classification Method <sup>3</sup></td><td>Classification Accuracy <sup>7</sup></td></tr></thead><tbody><tr><td>Rodríguez-Bermúdez & García-Laencina, 2012 [26]</td><td>AAR modelling, PSD</td><td>LARS/LOO-Press Criterion</td><td>LDA with regularization</td><td>62.2% (AAR), 69.4% (PSD)</td></tr><tr><td>Kevric & Subasi, 2017 [11]</td><td>Empirical mode decomposition, DWT, WPD <sup>4</sup></td><td>Kaiser criterion</td><td>k-NN</td><td>92.8% (WPD) <sup>6</sup></td></tr><tr><td>Zhou et al., 2018 [28]</td><td>Envelope analysis with DWT & Hilbert transform</td><td>None</td><td>RNN LSTM classifier</td><td>91.43%</td></tr><tr><td>Kumar et al., 2017 [47]</td><td>CSP & CSSP <sup>5</sup></td><td>None, FBCSP, DFBCSP, SFBCSP, SBLFB, DFBCSP-MI <sup>4</sup></td><td>SVM</td><td>Classification accuracy was not quoted.</td></tr><tr><td>Yu et al., 2014 [65]</td><td>CSP</td><td>PCA</td><td>SVM</td><td>76.34%</td></tr><tr><td>Baig et al., 2017 [3]</td><td>CSP</td><td>PSO, simulated annealing, ABC optimization, ACO, DE <sup>4</sup></td><td>LDA, SVM, k-NN, naive Bayes, regression trees <sup>4</sup></td><td>90.4% (PSO), 87.44% (simulated annealing), 94.48% (ABC optimization), 84.54% (ACO), 95% DE <sup>8</sup></td></tr></tbody></table>
# Graz motor imagery protocol

## I- training the system

Series of MI-task (ex. Left/Right)
[feature extraction to train a classifier]

<figure><img src="image_41.png" /></figure>

## II- training the user

MI-task with feedback.

User goal: to develope effective strategies to increase system performance

<figure><img src="image_42.png" /></figure>

<figure><img src="image_43.png" /></figure>

- Multiple sessions, each of which multiple sequences (4-6 runs - fatigue>6)

- One run:

10-20 trials per class (MI-task)

1 run lasting approx. 7 min
# User's brain and computer learning

Make the two components « understand each other »

<figure><img src="image_44.png" /></figure>

**Stage I :** the computer attempts to understand the command sent by the user, generally by extracting relevant information followed by classification (**signal processing, machine learning**)

**Stage II :** the user's turn to attempt to understand the meaning of the **feedback** generated by the computer, which indicates how the computer understood the command that is received
# BCIs improvement

## « Illiterate users »

Most current systems do not properly establish this mutual understanding

* 10 to 30% of users unable to control a BCI: "BCI illiteracy" or "BCI deficiency" phenomenon

* MI-BCI : even for not "illeterate", performance rather low

* 75% of classification accuracy for 2 class MI-BCIs

* 20% reach 80 to 100%

* 2 main factors

(1) Signal processing (Stage I) – many studies over the last years

(2) The user-training role: mostly neglected – inefficient training protocol (instructions, training tasks, feedback and training environment)

* Controlling an BCI requires acquisition of specific skills = to generate stable and distinct brain activity patterns while performing the tasks

* Understand how the human learn and how to adapt training process
# BCIs improvement

## Summary guidelines for designing more effective training protocols (C. Jeunet)

### Training task

- Progression (increasing difficulty), adaptive user-specific

- Self-paced and asynchronous sessions

- Preparatory training tasks (meditation)

### Feedback

Visual with emotional connotations (smileys) – tactile is promising (channel much less saturated in interactive situations) / increase the quantity and quality of information (topography of cerebral activity...)

### Training environment

- Improvement of user motivation & experience: gamified protocols (ex. ball-basket paradigm, spaceship that must avoid asteroids)

- Integration of virtual reality. Ex “Use the force” – allows users to levitate a spaceship by imagining moving their feet
# Usages

## Replacement

Can be a last resort for patients that have lost bodily functions

Communication (Speller visual or auditory spellers P300, SSVEP)

Environment control (Prosthesis)

Locomotion (Wheelchair)

<figure><img src="image_45.png" /></figure>

<figure><img src="image_46.png" /></figure>

<figure><img src="image_47.png" /></figure>
# Usages

## Restoration

Connection brain – limb that have been destroyed

## Neuroprosthesis

* motor-imagery task

* challenge: no tactile or proprioceptive feedback

* current efforts: adding stimulations in the somatosensory cortex or the skin to obtain the complete action-perception loop

<figure><img src="image_48.png" /></figure>

<figure><img src="image_49.png" /><figcaption>Australian Bionic Eye</figcaption></figure>

<figure><img src="image_50.png" /><figcaption>Bill Kochevar raising his arm for the first time in years (from Boseley, 2017)</figcaption></figure>
# Usages

## Improvement

Rehabilitation after a stroke (impaired motor control)

* BCI as a bridge in the action-loop

* Exercises with various feedbacks (gauge, virtual limb, robotised arm, or even functional electrical stimulation of the real limb)

* Neural plasticity

* Therapist can visualize patient's brain activity when performing attempted movement

<figure><img src="image_51.png" /></figure>
# Usages

## Enhancement

Mental state monitoring (passive BCIs)

More-or-less reliably in EEG signals: mental workload (pressure on the user cognitive resources, e.g. working memory), attention and, though less accurately, emotions (communication and personal development)

* Detection in the vigilance levels, loss in attention (car driver, air traffic controller) - auditive / visual notification

* Adaptation of application to improve user experience (ex. exercises as a function of attention / frustration)
# Usages

## Enhancement

Home automation

<figure><img src="image_52.png" /></figure>

<figure><img src="image_53.png" /></figure>

©Neurable

## Gaming and Virtual reality

* Select objects by paying attention to them (P300/SSVEP responses)

* Navigation in Virtual Environments VE (left/right/foot MI, or SSVEP on flickering control buttons overlayed to the game)

* Limited number of commands (1 to 3): BCI as a complementary input device of game pad, to trigger some actions

* alphaWoW (alpha World of Warcraft) game: avatar can turn from a brutal bear into a fragile elf depending on the player's relaxation level (alpha band)
Issues with BCI use

* Illiteracy (using an interactive, co-learning approach)

* Long-term use of BCIs requires repeated use of particular neural pathways -> possible health implications or change in brain functionality ?

Ex. long-term use of BCI-external actuators has led to restructuring of the brain's map of the body, with actuators perceived as an extension to the body

* Ethical standards

* establish liability in case of accidents using controlled apparatus

* deal with the appropriate use of bio-signal data

* privacy (influence on a user's mood – exploit user emotions / marketing or political agendas)
# Concluding remarks

<figure><img src="image_54.png" /></figure>

Designing a BCI is a complex and difficult task that requires knowledge of several disciplines (computer science, electrical engineering, signal processing, neuroscience, psychology)

<figure><img src="image_55.png" /></figure>

Many ongoing challenges

<figure><img src="image_56.png" /></figure>

Open community (software, hardware)

<figure><img src="image_57.png" /></figure>

Many projects for patients and healthy people

<figure><img src="image_58.png" /><figcaption>« Meet Milo, the brain controlled wheelchair »</figcaption></figure>

<figure><img src="image_59.png" /></figure>

©Emotiv

<figure><img src="image_60.png" /></figure>

©OpenBCI
# Concluding remarks

<figure><img src="image_61.png" /></figure>

Designing a BCI is a complex and difficult task that requires knowledge of several disciplines (computer science, electrical engineering, signal processing, neuroscience, psychology)

<figure><img src="image_62.png" /></figure>

Many ongoing challenges

<figure><img src="image_63.png" /></figure>

Open community (software, hardware)

<figure><img src="image_64.png" /></figure>

Many projects for patients and healthy people

<figure><img src="image_65.png" /><figcaption>« Meet Milo, the brain controlled wheelchair »</figcaption></figure>

<figure><img src="image_66.png" /><figcaption>©Emotiv</figcaption></figure>

<figure><img src="image_67.png" /><figcaption>©OpenBCI</figcaption></figure>

Are you ready?