# Controlling Assistive Machines in Paralysis Using Brain Waves and Other Biosignals

Authors: Paulo Rogério de Almeida Ribeiro; Fabricio Lima Brasil; Matthias Witkowski; Farid Shiman; Christian Cipriani; Nicola Vitiello; Maria Chiara Carrozza; Surjo Raphael Soekadar
Journal: Advances in Human-Computer Interaction
 (2013)

---

## Abstract

The extent to which humans can interact with machines significantly enhanced through inclusion of speech, gestures, and eye movements. However, these communication channels depend on a functional motor system. As many people suffer from severe damage of the motor system resulting in paralysis and inability to communicate, the development of brain-machine interfaces (BMI) that translate electric or metabolic brain activity into control signals of external devices promises to overcome this dependence. People with complete paralysis can learn to use their brain waves to control prosthetic devices or exoskeletons. However, information transfer rates of currently available noninvasive BMI systems are still very limited and do not allow versatile control and interaction with assistive machines. Thus, using brain waves in combination with other biosignals might significantly enhance the ability of people with a compromised motor system to interact with assistive machines. Here, we give an overview of the current state of assistive, noninvasive BMI research and propose to integrate brain waves and other biosignals for improved control and applicability of assistive machines in paralysis. Beside introducing an example of such a system, potential future developments are being discussed.


---

## Body

## 1. Introduction

The way humans interact with computers has changed substantially in the last decades. While, for many years, the input from the human to the machine was mainly managed through keystrokes, then later through hand movements using a computer mouse, other potential input sources have been opened up allowing more intuitive and effortless control, for example, based on speech [1], gestures [2], or eye movements [3], all depending on a functional motor system.As cardiovascular diseases increase and people live longer, an increasing number of people suffer from conditions that affect their capacity to communicate or limit their mobility [4], for example, due to stroke, neurodegenerative disorders, or hereditary myopathies. Motor disability can also result from traumatic injuries, affecting the central or peripheral nervous system or can be related to amputations of the upper or lower extremities. While these handicapped people would benefit the most from assistive machines, their capacity to interact with computers or machines is often severely impeded.Among the most important causes of neurological disabilities resulting in permanent damage and reduction of motor functions or the ability to communicate are stroke, multiple sclerosis (MS), spinal cord injury (SCI), brachial plexus injury (BPI), and neurodegenerative diseases, such as amyotrophic lateral sclerosis (ALS) or dementia [4].Stroke is the leading cause of long-term disability in adults and affects approximately 20 million people per year worldwide [5, 6]. Five millions remain severely handicapped and dependent on assistance in daily life [4]. Nearly 30% of all stroke patients are under the age of 65 [7]. Other diseases resulting in paralysis at such early age include MS, affecting more than 2.5 million people worldwide [8], or SCI with 12.1 to 57.8 cases per million [9, 10]. BPI, the disruption of the upper limb nerves leading to a flaccid paralysis of the arm, affects thousands of people every year [11]. Furthermore, every year there are approximately 2,000 new traumatic upper limb amputations in Europe [12].While there is major progress in the development of assistive apparatuses built for instance to compensate for a lost or paralyzed limb for example, lightweight and versatile prostheses or exoskeletons [13–16], intuitive and reliable control of such devices is an enormous challenge.Previous surveys on the use of artificial hands revealed that up to 50% of the amputees are not using their prosthetic hand regularly, mainly due to low functionality, poor cosmetic appearance, and low controllability [17].Since early on, the use of electromyographic (EMG) signals for prosthetic control, for example, from the amputee’s stump or contralateral chest muscles, was an important concept [18, 19]. However, its broader success is still limited due to many practical reasons that are valid for all assistive systems that depend on recording biosignals, primarily the effort and costs to provide good signal quality, a fast and effective calibration process, and, last but not least, the benefit of the system in the user’s everyday life. Furthermore, increasing the signal-to-noise ratio or the specificity of such recordings by means of techniques such as the electric nerve stimulation [20] is possible but increases the overall system complexity [21]. Adding sensory qualities during utilization of prosthetic devices increasing the bidirectional interaction between users and the machine improves the functionality of assistive systems [22]. Here, however, the same limitation applies as to the motor domain that the majority of such systems depend on an intact peripheral sensory system.Thus, the development and provision of assistive machines that are independent of the peripheral nervous system’s integrity represent a promising and appealing perspective, particularly, if controlled intuitively and without requiring extensive training to gain reliable control.


## 2. Brain-Computer and Brain-Machine Interfaces: A General Overview

Since it was discovered that brain waves contain information about cognitive states [23, 24] and can be functionally specific [25, 26], the idea to use such signals for direct brain control of assistive machines became a major driving force for the development of the so-called brain-computer or brain-machine interfaces (BCI/BMI) [27]. Such interfaces allow direct translation of electric or metabolic brain activity into control signals of external devices or computers bypassing the peripheral nervous and muscular system.As neural or metabolic brain activity can be recorded from sensors inside or outside the brain, BCI/BMI is categorized as invasive or noninvasive systems [28]. Other categorizations relate to the specific brain signal used for BCI/BMI control or the mode of operation (see Table 1).Table 1
Categories of brain-computer and brain-machine interfaces.
Based on:recording site of brain signals
Brain signal used
Recording technique
Invasive
Single spike
Single cell recordings
Multiunit activity
Multiunit arrays (MUA)
Local field potentials (LFP)
Electrocorticogram (ECoG)
Noninvasive
Electric brain potentials
Electroencephalography (EEG)
Neuromagnetic fields
Magnetoencephalography (MEG)
BOLD
Functional magnetic resonance imaging (fMRI)
Oxy/deoxyhemoglobin
Near-infrared spectroscopy (NIRS)
Based on:mode of operation
Active
Asynchronous control
Synchronous control
Reactive
N.A.
Passive
N.A.
Based on:purpose
Assistive/biomimetic
Restorative/biofeedback
Used for restoration of
Tested in the treatment of
Communication
Stroke
Paralysis
Chronic pain
Tinnitus
Dementia
Depression
SchizophreniaInvasively recorded brain signals that were successfully used for BCI/BMI control include single-spike or multiunit activity and local field potentials (LFP) [29]. These signals are necessarily recorded from inside the skull, while electric or magnetic brain oscillations reflecting pattern formation of larger cell assemblies’ activity [30] can also be recorded from outside the skull using electro- or magnetoencephalography (EEG/MEG). Each method offers access to specific unique properties of brain activity [31]. These noninvasive techniques allow, for example, detection and translation of slow cortical potentials (SCP), changes of sensorimotor rhythms (SMR), or event-related potentials (ERP), for example, the P300, translating them into control signals for external devices or computers. More recently, online interpretation of changes in metabolic brain activity [32, 33] was introduced for BCI/BMI application offering high spatial (in the range of mm), but low temporal, resolution (in the range of seconds). These systems use functional magnetic resonance imaging (fMRI) [32] or near-infrared spectroscopy (NIRS) [33, 34], both measuring changes in brain tissue’s blood-oxygenation-level dependent (BOLD) signals.In 1969, Fetz demonstrated that single neurons in precentral cortex can be operantly conditioned by delivery of food pellets [35]. Since then, operant conditioning of cortical activity was demonstrated in various paradigms [36], requiring, though, opening of the skull and insertion of electrodes into the brain with the risk of bleedings and infections [37, 38]. An intermediate, semiinvasive approach uses LFP recorded by epidural electrocorticography (ECoG) [29, 39]. LFP reflects neural activity of an area of up to 200 μm2 comprising hundreds of thousands of neurons with numerous local recurrent connections and connections to more distant brain regions [40], while brain oscillations recorded noninvasively (e.g., using EEG or MEG) contain information of millions of neurons [41].To control assistive devices or machines in paralysis, the following noninvasively recorded neurophysiologic signals were successfully used up to now: (1) slow cortical potentials (SCP) [42, 43], (2) sensorimotor rhythms (SMRs) and its harmonics [44, 45], and (3) event-related potentials (ERPs), for example, P300 [46].The use of SCP in BCI/BMI applications goes back to Birbaumer and his coworker’s work in the late 1970s showing that operant control of SCPs (slow direct-current shifts occurring event-related after 300 ms to several seconds) is possible while exhibiting strong and anatomically specific effects on behavior and cognition [47–49]. A tight correlation of central SCPs and blood-oxygen level-dependent (BOLD) signals in the anterior basal ganglia and premotor cortex was found [50] suggesting a critical role of the basal ganglia-thalamo-frontal network for operant control of SCP.In contrast to SCPs, SMRs are recorded over the sensorimotor cortex usually at a frequency between 8 and 15 Hz. In analogy to the occipital alpha and visual processing [51], the SMR (or rolandic alpha) shows a clear functional specificity, disappearing during planned, actual, or imagined movements [52]. Accordingly, a close association with functional motor inhibition of thalamocortical loops was suggested [53]. Depending on the context, the SMR is also called μ-rhythm [54] or rolandic alpha and was extensively investigated by the Pfurtscheller group in Graz [55] and the Wolpaw group in Albany [56, 57].Another well-established and tested BCI/BMI controller is the P300-based ERP-BCI introduced by Farwell and Donchin [58]. While SCP- and SMR-controls are learned through visual and auditory feedback often requiring multiple training sessions before reliable control is achieved, the P300-BCI needs no training at all. While, in the classical P300-ERP-BCI paradigm, the user focuses his attention to a visual stimulus, other sensory qualities such as tactile [59] or auditory stimuli [60, 61] were successfully implemented in ERP-BCI. Information rates of ERP-BCI can reach 20–30 bits/min: [62].In terms of operation mode, active, passive, and reactive BCI/BMI applications can be distinguished [63]. While active and reactive BCI/BMI require the user’s full attention to generate voluntary and directed commands, passive BCI/BMI relates to the concept of cognitive monitoring introducing the assessment of the users’ intentions, situational interpretations, and emotional states [64].In active BCI/BMI applications, two forms of control can be distinguished: synchronous and asynchronous control [65]. In synchronous control, translation of brain activity follows a fixed sequence or cue. The user is required to be fully attentive, while in asynchronous or uncued control, a specific brain signal is used to detect the user’s intention to engage in BCI/BMI control [65, 66].


## 3. Brain-Machine Interfaces in Neurorehabilitation of Paralysis

BMI used in neurorehabilitation follows two different strategies: while assistive or biomimetic BMI systems strive for continuous high-dimensional control of robotic devices or functional electric stimulation (FES) of paralyzed muscles to substitute for lost motor functions in a daily life environment [67–69], restorative or biofeedback BMI systems aim at normalizing of neurophysiologic activity that might facilitate motor recovery [70–74]. Insofar, restorative or biofeedback BMI can be considered as “training-tools” to induce use-dependent brain plasticity increasing the patient’s capacity for motor learning [44, 75].These two approaches derive from different research traditions and are not necessarily related to the invasiveness of the approach: in the early 80s of the last century, decoding of different movement directions from single neurons was successfully demonstrated [76]. Since then, reconstruction of complex movements from neuronal activity was pursued, using both invasive and noninvasive methods.Firing patterns acquired through single cell recordings from the motor cortex [77] or parietal neuronal pools [78] in animals were remarkably successful for reconstruction of movement trajectories. Monkeys learned to control computer cursors towards moving targets on a screen activating neurons in motor, premotor, and parietal motor areas. It was shown that 32 cells were sufficient to move an artificial arm and perform skillful reaching movements enabling a monkey to feed himself [67]. Learned control of movements based on single cell activity was also shown using neurons outside the primary or secondary motor representations [79]. In 2006, successful implantation of densely packed microelectrode arrays in two quadriplegic human patients was demonstrated, enabling them to use LFP in order to move a computer cursor in several directions [68]. Most recently, a study using two 96-channel intracortical microelectrodes placed in the motor cortex of a 52-year-old woman with tetraplegia demonstrated robust seven-dimensional movements of a prosthetic limb [80].In contrast to this work aiming at assistive appliance of invasive and noninvasive BMI technology, the development of restorative/biofeedback BMI systems is tightly associated with the development and successes of neurofeedback (NF) and its use to purposefully upregulate or downregulate brain activity—a quality that showed to have some beneficial effect in the treatment of various neurological and psychiatric disorders associated with neurophysiologic abnormalities [71]. In NF, subjects receive visual or auditory online feedback of their brain activity and are asked to voluntarily modify, for example, a particular type of brainwave. Successful modification becomes contingently rewarded. NF was successfully used in the treatment of epilepsy [81, 82], ADHD [83–85], chronic pain syndrome [86]. The rational to use this approach in the context of neurorehabilitation is based on data indicating that stroke patients with best motor recovery are the ones in whom ipsilesional cortical function is closer to that found in healthy controls [87]. A negative correlation between impairment and activation in ipsilesional M1 during hand motions has been documented [88]. Thus, a larger clinical study was performed at the University of Tübingen in Germany and the National Institute of Neurological Disorders and Stroke (NINDS, NIH) in USA with over 30 chronic stroke patients testing the hypothesis that augmentation of ipsilesional brain activity would improve motor recovery [89, 90]. In this study, all participating patients suffered from complete hand paralysis and were unable, for example, to grasp. The study showed that one month of daily ipsilesional BMI training combined with goal-directed physiotherapy resulted in significant motor improvements, while random BMI-feedback did not. Further analysis of neurophysiological parameters indicated that motor evoked potentials (MEP) from the ipsilesional hemisphere reflecting the integrity of the corticospinal tract could predict motor recovery of the trained patients [91]. Currently, further improvements of this training paradigm, for example, related to the feedback or specificity and effectiveness of training [44], for example, using electric brain stimulation to enhance neuroplasticity [92], are being tested.


## 4. Noninvasive Assistive Brain-Machine Interfaces in Paralysis

Both invasive and noninvasive BCI and BMI found their way into assistive systems, for example, allowing communication in locked-in patients [42] or restoration of movement in patients with paralysis [28, 93]. The Graz group was the first to use volitional SMR modulation for control of electric stimulation of a quadriplegic patient’s paralyzed hand [69, 94]. While the patient imagined a movement, the associated modulation of SMR was translated into functional electric stimulation (FES) of his upper limb muscles resulting in grasping motions. After this proof-of-concept study, numerous publications addressed the different aspects that are important to allow intuitive and seamless control of biomimetic devices [20] or FES [95] in a daily life environment [96]. While many challenges were successfully mastered in the last years, three major aspects were not satisfyingly solved yet: (1) intuitive, asynchronous BCI/BMI control, (2) 100% reliability, and (3) unambiguous superiority (in terms of information transfer rate, ITR, and necessary preparation effort) over the use of other biosignals (e.g., related to speech, gestures, or eye movements).These aspects do not apply to BCI use for communication in complete paralysis, for example, complete locked-in-state (CLIS) in ALS, as no asynchronous mode is necessary, reliability is secondary, and no other biosignals are available anymore [97].A system that is unreliable in daily does not only limit its practicality, but limits its practicality, but would be also associated with ethical difficulties [98, 99]. While there are good arguments suggesting that invasive BCI/BMI can provide a higher ITR [100], it is still unclear how much meaningful information, for example, for reconstruction of hand movements, can be extracted from noninvasively recorded brain signals [101]. Recently, work by Contreras-Vidal’s group at the University of Houston suggested that slow-frequency EEG (oscillations with a frequency of up to 4 Hz) might provide as much information as invasive recordings [102, 103], for example, for reconstruction of three-dimensional hand movements [103]. Currently, implementation of this approach in closed-loop paradigms is being pursued. Nevertheless, it is conceivable that the only viable solution to satisfyingly solve those three aspects will be the inclusion of other biosignals into a system merging different biosignal sources to detect user’s intentions and integrating this information into the current context of the user to further increase intuitive control and assure reliability of the system. Such systems that merge brain control with other biosignals were recently summarized under the term “brain-neural computer interaction” (BNCI) systems receiving notable funding through the 7th Framework Program for Research and Technological Development (FP7) of the European Union.Particularly promising in this context is integrating eye movements using electrooculography (EOG) or eye tracking into prosthetic control. At the University of Tübingen, a first prototype system was conceptualized that allows asynchronous BCI/BMI control while solving the reliability issue by using eye tracking, EOG, and computer vision-based object recognition. A computer equipped with a 3D camera recognizes objects placed on a table. The system detects when the user fixates any of the objects recognized as graspable, for example, a cup or ball. Once an object is fixated with the eyes, the BCI/BMI mode switches on, detecting whether the user wants to grasp the object. A robotic hand or exoskeleton (both developed by the BioRobotics Institute, Scuola Superiore Sant’Anna, Pisa, Italy) performs the grasping motion (Figure1). The motion becomes interrupted if the user does not fixate the object anymore as measured by eye tracking and EOG (see Figure 2). This assures that no action of the system depends exclusively on brain wave control that might be susceptible to inaccuracies. Such system, integrating perceptual and contextual computing developed in the field of human-computer interaction (HCI) research into BCI applications, promises to overcome many limitations of brain control alone, mainly the reliability issue, likewise broadening the repertoire of modern HCI research to infer user state and intention from brain activity.Figure 1
Organization of the University of Tübingen’ prototype system controlling assistive devices using brain waves and eye movements.Figure 2
Illustration of the processing chain for performing grasping motions of an assistive system using brain waves and eye movements. The grasping motion stops once the user does not fixate the object with his eyes anymore.As trauma or stroke can affect motor and body functions very differently in each individual, proper and fast calibration for inclusion into seamless BNCI control is often impeded. Thus, inclusion of eye movements is the most promising biosignal in this context so far. Particularly as visual interaction plays a key role when planning, executing, and adapting motor control. Beside electric biosignals such as EOG and EMG, other measures that can be used for BNCI control include magnetic, mechanic, optic, acoustic, chemical, and thermal biosignals. These biosignals, however, are more susceptible for artifacts and exhibit larger variability depending on the environmental conditions. Future research, however, might find novel ways to advantageously include such biosignals into BNCI control and application.The organisms’ behavior measurable in these various biosignals reflects conscious and unconscious processes that can be inferred and purposefully used for BNCI control. In case of eye movement control, changing fixation of an object can point to inattention, distraction, or volitional (conscious) act to interrupt unwanted output of the BNCI for example.Practicality of such approach is limited when, for instance, eyesight or eyeball control is impaired due to a disease or trauma. This can be the case in multiple sclerosis, traumatic brain injury, stroke, or neurodegenerative disorders such as ALS. ALS may lead to CLIS, where classical semantic conditioning might be the only way to sustain a communication channel [104] while inclusion or use of other biosignals seemed not particularly helpful [94]. Also, inclusion of other biosignals often increases preparation time for placing and calibrating the required biosensors further limiting practicality. This is particularly relevant when the system requires handicapped persons to place and handle the sensors in a home environment. Nevertheless, these technical limitations might dissolve in the course of near-future research and development.An important conceptual advantage of including other biosignals into BCI control relates to the improved reliability, which not only increases usability in daily life, but also the degree of self-efficacy, a dimension that should not be underestimated in acceptance of such technology, but also in the context of restorative/biofeedback BCI training for example. Here, the fact that a patient experiences full control of a completely paralyzed limb might facilitate overcoming “learned nonuse” and motivate the user to engage in behavioral physiotherapy [105].


## 5. Conclusion

BCI/BMI systems promise to enhance applicability of assistive technology in humans with a compromised or damaged motor system. While information transfer rates of noninvasive BCI/BMI are sufficient for communication, for example, in locked-in-state, versatile control of prosthetic devices using brain waves will require major research and development efforts to provide intuitive, asynchronous control sufficiently reliable in daily life environments. Many reasons suggest that using the combination of brain waves with other biosignals might entail many attractive solutions to control assistive, noninvasive technology even after severe damage of the central or peripheral nervous system.



---

*Source: 369425-2013-05-28.xml*