Real-Time State Estimation in a Flight Simulator Using fNIRS

Working memory is a key executive function for flying an aircraft. This function is particularly critical when pilots have to recall series of air traffic control instructions. However, working memory limitations may jeopardize flight safety. Since the functional near-infrared spectroscopy (fNIRS) method seems promising for assessing working memory load, our objective is to implement an on-line fNIRS-based inference system that integrates two complementary estimators. The first estimator is a real-time state estimation MACD-based algorithm dedicated to identifying the pilot’s instantaneous mental state (not-on-task vs. on-task). It does not require a calibration process to perform its estimation. The second estimator is an on-line SVM-based classifier that is able to discriminate task difficulty (low working memory load vs. high working memory load). These two estimators were tested with 19 pilots who were placed in a realistic flight simulator and were asked to recall air traffic control instructions. We found that the estimated pilot’s mental state matched significantly better than chance with the pilot’s real state (62% global accuracy, 58% specificity, and 72% sensitivity). The second estimator, dedicated to assessing single trial working memory loads, led to 80% classification accuracy, 72% specificity, and 89% sensitivity. These two estimators establish reusable blocks for further fNIRS-based passive brain computer interface development.

Related papers:

title: Executive Functions and Pilot Characteristics Predict Flight Simulator Performance in General Aviation Pilots
abstract: In general aviation, 85% of the crashes seem to be caused by pilots' errors (Li, Baker, Grabowski, & Rebok, 2001) and 46% of the crashes occur at airports (Li & Baker, 1999). It is important to determine if the same factors influence the flying performance and the landing decision making and to uncover which factors, among the pilot's cognitive status, personality traits, and experience, are the most predictive. We examined in 24 general aviation pilots the relationship between those factors and the flying performance and weather-related decision-making relevance. The cognitive assessment encompassed the three basic executive functions (Miyake et al., 2000), reasoning, and psychomotor velocity. The personal characteristics were age, flight experience, and level of impulsivity. Reasoning, updating in working memory, and flight experience were predictive of the flight performance. In addition, updating in working memory, flight experience, and level of impulsivity were linked with weather-related decision-making relevance.

title: Analysis of Problems in Routine Controller-Pilot Communication
abstract: Previous studies of controller-pilot communication have identified communication problems but not how often or why they occur. A more complete picture of these problems requires an understanding of how controllers and pilots successfully communicate during routine operations. We describe a field study of routine controller–pilot communication and the problems that disrupt this communication. Guided by a model of collaboration during communication, voice tapes of communication during terminal operations were coded and analyzed. Two kinds of communication problems were examined: (a) procedural deviations (e.g., partial readbacks) and (b) inaccuracies (e.g., incorrect readbacks). The results suggest that these problems are infrequent events in routine communication. However, they are associated with communication factors such as air traffic control message length and composition and traffic level. These relations suggest potential causes of problems that disrupt routine controller–pilot communication.

title: The magical number seven plus or minus two: some limits on our capacity for processing information.
abstract: First, the span of absolute judgment and the span of immediate memory impose severe limitations on the amount of information that we are able to receive, process, and remember. By organizing the stimulus input simultaneously into several dimensions and successively into a sequence or chunks, we manage to break (or at least stretch) this informational bottleneck. Second, the process of recoding is a very important one in human psychology and deserves much more explicit attention than it has received. In particular, the kind of linguistic recoding that people do seems to me to be the very lifeblood of the thought processes. Recoding procedures are a constant concern to clinicians, social psychologists, linguists, and anthropologists and yet, probably because recoding is less accessible to experimental manipulation than nonsense syllables or T mazes, the traditional experimental psychologist has contributed little or nothing to their analysis. Nevertheless, experimental techniques can be used, methods of recoding can be specified, behavioral indicants can be found. And I anticipate that we will find a very orderly set of relations describing what now seems an uncharted wilderness of individual differences. Third, the concepts and measures provided by the theory of information provide a quantitative way of getting at some of these questions. The theory provides us with a yardstick for calibrating our stimulus materials and for measuring the performance of our subjects. In the interests of communication I have suppressed the technical details of information measurement and have tried to express the ideas in more familiar terms; I hope this paraphrase will not lead you to think they are not useful in research. Informational concepts have already proved valuable in the study of discrimination and of language; they promise a great deal in the study of learning and memory; and it has even been proposed that they can be useful in the study of concept formation. A lot of questions that seemed fruitless twenty or thirty years ago may now be worth another look. In fact, I feel that my story here must stop just as it begins to get really interesting. And finally, what about the magical number seven? What about the seven wonders of the world, the seven seas, the seven deadly sins, the seven daughters of Atlas in the Pleiades, the seven ages of man, the seven levels of hell, the seven primary colors, the seven notes of the musical scale, and the seven days of the week? What about the seven-point rating scale, the seven categories for absolute judgment, the seven objects in the span of attention, and the seven digits in the span of immediate memory? For the present I propose to withhold judgment. Perhaps there is something deep and profound behind all these sevens, something just calling out for us to discover it. But I suspect that it is only a pernicious, Pythagorean coincidence.

title: Working memory
abstract: It is suggested that working memory comprises a system for the temporary storage and manipulation of information, forming an important link between perception and controlled action. Evidence is presented for a three-component model, comprising an attentional control system, the central executive, and two subsidiary slave systems. One of these the, the visuo-spatial sketch pad holds and manipulates spatial information, while the other, the phonological loop performs a similar function for auditory and speech-based information. Evidence is presented for the view that the phonological loop has evolved as a mechanism to facilitate the acquisition of language.

title: Interference Timing and Acknowledgement Response with Voice and Datalink Atc Commands
abstract: Datalink is a text system used to send messages between ATC and pilots. There are concerns related to changes in information processing demands and responses associated with executing speech and text ATC commands. The timing of interference and the acknowledgement response on command execution performance were examined during the processing of simulated ATC commands. Verbal and central executive (CE) interference tasks were presented before or after the acknowledgement. Participants received both speech and text commands, responded by a verbal or manual acknowledgement, and set the controls in a flight simulator. Results demonstrated an advantage for a manual acknowledgement with longer messages. CE as opposed to verbal interference prior to an acknowledgement had a greater negative effect that was exacerbated in the text condition. The findings are interpreted within the context of a working memory and multiple-resource perspective and implications are discussed with regard to communication processes in aviation.

title: The effects of information load and speech rate on younger and older aircraft pilots' ability to execute simulated air-traffic controller instructions.
abstract: In this applied study of memory for orally presented information, 15 younger and 15 older pilots heard recorded air-traffic controller (ATC) messages in the context of six simulated flights. The ATC messages varied in length (3 vs 4 items), speech rate (235 vs 365 wpm), and type of command (course commands consisting of headings and altitudes vs radio/transponder commands consisting of radio frequencies and transponder codes). Older pilots made more execution errors on average and the age difference was greater for the radio/transponder commands, which contained more unique digits than the course commands. Although longer message lengths and faster speech rates led to higher error rates, the increases were not more marked in the older group. Backward digit span was correlated with communication performance, but the older group's lower level of accuracy was not explainable in terms of differences in digit span.

title: Implementing Speech and Simulated Data Link Commands: The Role of Task Interference and Message Length
abstract: The present study investigated the ability of individuals to correctly execute different numbers of commands presented in speech and text format (simulating voice and data link communications). These commands were executed in the presence of different sources of task interference drawing upon visual, verbal, and central executive resources. The results showed that performance decreased as the number of commands in the message set increased from two to four, regardless of presentation format. Visual interference was less disruptive than both verbal and central executive interference. These findings have implications for the design of data link systems and suggest that during periods of high workload, communications should include fewer commands within a message set particularly when there is insufficient time to request clarification.

title: Cognitive ability, expertise, and age differences in following air-traffic control instructions.
abstract: Differences in cognitive ability and domain-specific expertise may help explain age differences in pilot performance. Pilots heard air-traffic controller messages and then executed them while "flying" in a simulator. Messages varied in length and speech rate. Age was associated with lower accuracy, but the expected Age x Message Difficulty interactions were not obtained. Expertise, as indexed by pilot ratings, was associated with higher accuracy; yet expertise did not reduce age differences in accuracy. The effect of age on communication task accuracy was largely explainable as an age-associated decrease in working memory span, which in turn was explainable as decreases in both speed and interference control. Results are discussed within frameworks of deliberate practice and cognitive mediation of age differences.

title: Towards a robust BCI: error potentials and online learning
abstract: Recent advances in the field of brain-computer interfaces (BCIs) have shown that BCIs have the potential to provide a powerful new channel of communication, completely independent of muscular and nervous systems. However, while there have been successful laboratory demonstrations, there are still issues that need to be addressed before BCIs can be used by nonexperts outside the laboratory. At IDIAP Research Institute, we have been investigating several areas that we believe will allow us to improve the robustness, flexibility, and reliability of BCIs. One area is recognition of cognitive error states, that is, identifying errors through the brain's reaction to mistakes. The production of these error potentials (ErrP) in reaction to an error made by the user is well established. We have extended this work by identifying a similar but distinct ErrP that is generated in response to an error made by the interface, (a misinterpretation of a command that the user has given). This ErrP can be satisfactorily identified in single trials and can be demonstrated to improve the theoretical performance of a BCI. A second area of research is online adaptation of the classifier. BCI signals change over time, both between sessions and within a single session, due to a number of factors. This means that a classifier trained on data from a previous session will probably not be optimal for a new session. In this paper, we present preliminary results from our investigations into supervised online learning that can be applied in the initial training phase. We also discuss the future direction of this research, including the combination of these two currently separate issues to create a potentially very powerful BCI.

title: A quantitative comparison of NIRS and fMRI across multiple cognitive tasks
abstract: Near infrared spectroscopy (NIRS) is an increasingly popular technology for studying brain function. NIRS presents several advantages relative to functional magnetic resonance imaging (fMRI), such as measurement of concentration changes in both oxygenated and deoxygenated hemoglobin, finer temporal resolution, and ease of administration, as well as disadvantages, most prominently inferior spatial resolution and decreased signal-to-noise ratio (SNR). While fMRI has become the gold standard for in vivo imaging of the human brain, in practice NIRS is a more convenient and less expensive technology than fMRI. It is therefore of interest to many researchers how NIRS compares to fMRI in studies of brain function. In the present study we scanned participants with simultaneous NIRS and fMRI on a battery of cognitive tasks, placing NIRS probes over both frontal and parietal brain regions. We performed detailed comparisons of the signals in both temporal and spatial domains. We found that NIRS signals have significantly weaker SNR, but are nonetheless often highly correlated with fMRI measurements. Both SNR and the distance between the scalp and the brain contributed to variability in the NIRS/fMRI correlations. In the spatial domain, we found that a photon path forming an ellipse between the NIRS emitter and detector correlated most strongly with the BOLD response. Taken together these findings suggest that, while NIRS can be an appropriate substitute for fMRI for studying brain activity related to cognitive tasks, care should be taken when designing studies with NIRS to ensure that: 1) the spatial resolution is adequate for answering the question of interest and 2) the design accounts for weaker SNR, especially in brain regions more distal from the scalp.

title: Optical brain monitoring for operator training and mental workload assessment
abstract: An accurate measure of mental workload in human operators is a critical element of monitoring and adaptive aiding systems that are designed to improve the efficiency and safety of human-machine systems during critical tasks. Functional near infrared (fNIR) spectroscopy is a field-deployable non-invasive optical brain monitoring technology that provides a measure of cerebral hemodynamics within the prefrontal cortex in response to sensory, motor, or cognitive activation. In this paper, we provide evidence from two studies that fNIR can be used in ecologically valid environments to assess the: 1) mental workload of operators performing standardized (n-back) and complex cognitive tasks (air traffic control--ATC), and 2) development of expertise during practice of complex cognitive and visuomotor tasks (piloting unmanned air vehicles--UAV). Results indicate that fNIR measures are sensitive to mental task load and practice level, and provide evidence of the fNIR deployment in the field for its ability to monitor hemodynamic changes that are associated with relative cognitive workload changes of operators. The methods reported here provide guidance for the development of strategic requirements necessary for the design of complex human-machine interface systems and assist with assessments of human operator performance criteria.

title: Anticipating human error before it happens: Towards a psychophysiological model for online prediction of mental workload
abstract: Mental workload is a key factor influencing the occurrence of human error; specifically in remotely-operated vehicle operations. Both low and high mental workload has been found to disrupt performance in a nonlinear fashion at a given task; however, research that has attempted to predict individual mental workload has met with little success. The objective of the present study is to investigate the potential of the dual-task paradigm and prefrontal cortex oxygenation as online measures of mental workload. Subjects performed a computerized object tracking task in which they had to follow a dynamic target with their aircraft. Task difficulty was manipulated in terms of processing load and difficulty of control: two critical sources of workload associated with remotely operating a vehicle. Mental workload was assessed by a secondary concurrent time production task and a functional near infrared spectrometer. Results show that the effects of task difficulty differ across measures of mental workload. This pattern of behavioural and neurophysiologic results suggests that the empirically-based selection of an appropriate secondary task for the measure of mental workload is critical as its sensitivity may vary considerably depending on task factors.

title: Verbal working memory load affects prefrontal cortices activation: evidence from a functional NIRS study in humans
abstract: Working memory (WM) refers to the temporary maintenance of information that is no longer accessible in the environment, and the manipulation of this information for subsequent use. PET and functional MRI studies suggest that prefrontal cortex (PFC) is involved in WM. Here, we report a functional near-infrared spectroscopy (NIRS) study on the PFC activation caused by a WM task, a verbal n-back task. During performance of the task, concentration changes of oxy-Hb (HbO2), deoxy-Hb (Hb), and total-Hb (HbT) in subjects’ prefrontal cortex were monitored by a 24-channel functional NIRS imager. The behavioral performances (accuracy and response time) were recorded simultaneously. Results revealed that as memory load increased, subjects showed poorer behavioral performance as well as monotonously increasing magnitudes of the activations in the left ventrolateral PFC (VLPFC) and bilateral dorsolateral PFC (DLPFC). In addition, the analysis of comparison between subjects showed that certain relations likely exist between the cerebral activation and the performance parameters for an individual subject: lower accuracy is accompanied by longer response time and further activation. Such means that the subject with difficulty in solving a problem will demonstrate more significant hemodynamic changes compared with the subject without difficulty in solving the same problem.

title: This is your brain on interfaces: enhancing usability testing with functional near-infrared spectroscopy
abstract: This project represents a first step towards bridging the gap between HCI and cognition research. Using functional near-infrared spectroscopy (fNIRS), we introduce tech-niques to non-invasively measure a range of cognitive workload states that have implications to HCI research, most directly usability testing. We present a set of usability experiments that illustrates how fNIRS brain measurement provides information about the cognitive demands placed on computer users by different interface designs.

title: Using near infrared spectroscopy and heart rate variability to detect mental overload
abstract: Mental workload is a key factor influencing the occurrence of human error, especially during piloting and remotely operated vehicle (ROV) operations, where safety depends on the ability of pilots to act appropriately. In particular, excessively high or low mental workload can lead operators to neglect critical information. The objective of the present study is to investigate the potential of functional near infrared spectroscopy (fNIRS) - a non-invasive method of measuring prefrontal cortex activity - in combination with measurements of heart rate variability (HRV), to predict mental workload during a simulated piloting task, with particular regard to task engagement and disengagement. Twelve volunteers performed a computer-based piloting task in which they were asked to follow a dynamic target with their aircraft, a task designed to replicate key cognitive demands associated with real life ROV operating tasks. In order to cover a wide range of mental workload levels, task difficulty was manipulated in terms of processing load and difficulty of control - two critical sources of workload associated with piloting and remotely operating a vehicle. Results show that both fNIRS and HRV are sensitive to different levels of mental workload; notably, lower prefrontal activation as well as a lower LF/HF ratio at the highest level of difficulty, suggest that these measures are suitable for mental overload detection. Moreover, these latter measurements point toward the existence of a quadratic model of mental workload.

title: Design of Differential Near-Infrared Spectroscopy Based Brain Machine Interface
abstract: Near-Infrared Spectroscopy (NIRS) is a non-invasive technology for measuring brain activity. Recently, the number of research papers on Brain Machine Interface (BMI) based on NIRS technology is increasing. NIRS is a safe and convenient technique but its measurement results are unstable. To improve reliability of NIRS-based BMI, methods to extract stable data from NIRS signals are necessary. This paper describes a reliable NIRS-based BMI system we have developed. The feasibility of the method was demonstrated through generating motion of a humanoid robot

title: Brain-computer interface using a simplified functional near-infrared spectroscopy system.
abstract: A brain-computer interface (BCI) is a device that allows a user to communicate with external devices through thought processes alone. A novel signal acquisition tool for BCIs is near-infrared spectroscopy (NIRS), an optical technique to measure localized cortical brain activity. The benefits of using this non-invasive modality are safety, portability and accessibility. A number of commercial multi-channel NIRS system are available; however we have developed a straightforward custom-built system to investigate the functionality of a fNIRS-BCI system. This work describes the construction of the device, the principles of operation and the implementation of a fNIRS-BCI application, 'Mindswitch' that harnesses motor imagery for control. Analysis is performed online and feedback of performance is presented to the user. Mindswitch presents a basic 'on/off' switching option to the user, where selection of either state takes 1 min. Initial results show that fNIRS can support simple BCI functionality and shows much potential. Although performance may be currently inferior to many EEG systems, there is much scope for development particularly with more sophisticated signal processing and classification techniques. We hope that by presenting fNIRS as an accessible and affordable option, a new avenue of exploration will open within the BCI research community and stimulate further research in fNIRS-BCIs.

title: Classification of prefrontal activity due to mental arithmetic and music imagery using hidden Markov models and frequency domain near-infrared spectroscopy.
abstract: Near-infrared spectroscopy (NIRS) has recently been investigated as a non-invasive brain-computer interface (BCI). In particular, previous research has shown that NIRS signals recorded from the motor cortex during left- and right-hand imagery can be distinguished, providing a basis for a two-choice NIRS-BCI. In this study, we investigated the feasibility of an alternative two-choice NIRS-BCI paradigm based on the classification of prefrontal activity due to two cognitive tasks, specifically mental arithmetic and music imagery. Deploying a dual-wavelength frequency domain near-infrared spectrometer, we interrogated nine sites around the frontopolar locations (International 10-20 System) while ten able-bodied adults performed mental arithmetic and music imagery within a synchronous shape-matching paradigm. With the 18 filtered AC signals, we created task- and subject-specific maximum likelihood classifiers using hidden Markov models. Mental arithmetic and music imagery were classified with an average accuracy of 77.2% +/- 7.0 across participants, with all participants significantly exceeding chance accuracies. The results suggest the potential of a two-choice NIRS-BCI based on cognitive rather than motor tasks.

title: Classification of functional near-infrared spectroscopy signals corresponding to the right- and left-wrist motor imagery for development of a brain–computer interface
abstract: This paper presents a study on functional near-infrared spectroscopy (fNIRS) indicating that the hemodynamic responses of the right- and left-wrist motor imageries have distinct patterns that can be classified using a linear classifier for the purpose of developing a brain-computer interface (BCI). Ten healthy participants were instructed to imagine kinesthetically the right- or left-wrist flexion indicated on a computer screen. Signals from the right and left primary motor cortices were acquired simultaneously using a multi-channel continuous-wave fNIRS system. Using two distinct features (the mean and the slope of change in the oxygenated hemoglobin concentration), the linear discriminant analysis classifier was used to classify the right- and left-wrist motor imageries resulting in average classification accuracies of 73.35% and 83.0%, respectively, during the 10s task period. Moreover, when the analysis time was confined to the 2-7s span within the overall 10s task period, the average classification accuracies were improved to 77.56% and 87.28%, respectively. These results demonstrate the feasibility of an fNIRS-based BCI and the enhanced performance of the classifier by removing the initial 2s span and/or the time span after the peak value.

title: Classification of prefrontal and motor cortex signals for three-class fNIRS–BCI
abstract: Functional near-infrared spectroscopy (fNIRS) is an optical imaging method that can be used for a brain-computer interface (BCI). In the present study, we concurrently measure and discriminate fNIRS signals evoked by three different mental activities, that is, mental arithmetic (MA), right-hand motor imagery (RI), and left-hand motor imagery (LI). Ten healthy subjects were asked to perform the MA, RI, and LI during a 10s task period. Using a continuous-wave NIRS system, signals were acquired concurrently from the prefrontal and the primary motor cortices. Multiclass linear discriminant analysis was utilized to classify MA vs. RI vs. LI with an average classification accuracy of 75.6% across the ten subjects, for a 2-7s time window during the a 10s task period. These results demonstrate the feasibility of implementing a three-class fNIRS-BCI using three different intentionally-generated cognitive tasks as inputs.

title: Distinguishing Difficulty Levels with Non-invasive Brain Activity Measurements
abstract: Passive brain-computer interfaces are designed to use brain activity as an additional input, allowing the adaptation of the interface in real time according to the user's mental state. The goal of the present study is to distinguish between different levels of game difficulty using non-invasive brain activity measurement with functional near-infrared spectroscopy (fNIRS). The study is designed to lead to adaptive interfaces that respond to the user's brain activity in real time. Nine subjects played two levels of the game Pacman while their brain activity was measured using fNIRS. Statistical analysis and machine learning classification results show that we can discriminate well between subjects playing or resting, and distinguish between the two levels of difficulty with some success. In contrast to most previous fNIRS studies which only distinguish brain activity from rest, we attempt to tell apart two levels of brain activity, and our results show potential for using fNIRS in an adaptive game or user interface.

title: Sensing cognitive multitasking for a brain-based adaptive user interface
abstract: Multitasking has become an integral part of work environments, even though people are not well-equipped cognitively to handle numerous concurrent tasks effectively. Systems that support such multitasking may produce better performance and less frustration. However, without understanding the user's internal processes, it is difficult to determine optimal strategies for adapting interfaces, since all multitasking activity is not identical. We describe two experiments leading toward a system that detects cognitive multitasking processes and uses this information as input to an adaptive interface. Using functional near-infrared spectroscopy sensors, we differentiate four cognitive multitasking processes. These states cannot readily be distinguished using behavioral measures such as response time, accuracy, keystrokes or screen contents. We then present our human-robot system as a proof-of-concept that uses real-time cognitive state information as input and adapts in response. This prototype system serves as a platform to study interfaces that enable better task switching, interruption management, and multitasking.

title: Towards a Near Infrared Spectroscopy-Based Estimation of Operator Attentional State
abstract: Given the critical risks to public health and safety that can involve lapses in attention (e.g., through implication in workplace accidents), researchers have sought to develop cognitive-state tracking technologies, capable of alerting individuals engaged in cognitively demanding tasks of potentially dangerous decrements in their levels of attention. The purpose of the present study was to address this issue through an investigation of the reliability of optical measures of cortical correlates of attention in conjunction with machine learning techniques to distinguish between states of full attention and states characterized by reduced attention capacity during a sustained attention task. Seven subjects engaged in a 30 minutes duration sustained attention reaction time task with near infrared spectroscopy (NIRS) monitoring over the prefrontal and the right parietal areas. NIRS signals from the first 10 minutes of the task were considered as characterizing the ‘full attention’ class, while the NIRS signals from the last 10 minutes of the task were considered as characterizing the ‘attention decrement’ class. A two-class support vector machine algorithm was exploited to distinguish between the two levels of attention using appropriate NIRS-derived signal features. Attention decrement occurred during the task as revealed by the significant increase in reaction time in the last 10 compared to the first 10 minutes of the task (p<.05). The results demonstrate relatively good classification accuracy, ranging from 65 to 90%. The highest classification accuracy results were obtained when exploiting the oxyhemoglobin signals (i.e., from 77 to 89%, depending on the cortical area considered) rather than the deoxyhemoglobin signals (i.e., from 65 to 66%). Moreover, the classification accuracy increased to 90% when using signals from the right parietal area rather than from the prefrontal cortex. The results support the feasibility of developing cognitive tracking technologies using NIRS and machine learning techniques.

title: Temporal classification of multichannel near-infrared spectroscopy signals of motor imagery for developing a brain–computer interface
abstract: There has been an increase in research interest for brain-computer interface (BCI) technology as an alternate mode of communication and environmental control for the disabled, such as patients suffering from amyotrophic lateral sclerosis (ALS), brainstem stroke and spinal cord injury. Disabled patients with appropriate physical care and cognitive ability to communicate with their social environment continue to live with a reasonable quality of life over extended periods of time. Near-infrared spectroscopy is a non-invasive technique which utilizes light in the near-infrared range (700 to 1000 nm) to determine cerebral oxygenation, blood flow and metabolic status of localized regions of the brain. In this paper, we describe a study conducted to test the feasibility of using multichannel NIRS in the development of a BCI. We used a continuous wave 20-channel NIRS system over the motor cortex of 5 healthy volunteers to measure oxygenated and deoxygenated hemoglobin changes during left-hand and right-hand motor imagery. We present results of signal analysis indicating that there exist distinct patterns of hemodynamic responses which could be utilized in a pattern classifier towards developing a BCI. We applied two different pattern recognition algorithms separately, Support Vector Machines (SVM) and Hidden Markov Model (HMM), to classify the data offline. SVM classified left-hand imagery from right-hand imagery with an average accuracy of 73% for all volunteers, while HMM performed better with an average accuracy of 89%. Our results indicate potential application of NIRS in the development of BCIs. We also discuss here future extension of our system to develop a word speller application based on a cursor control paradigm incorporating online pattern classification of single-trial NIRS data.

title: Real-time imaging of human brain function by near-infrared spectroscopy using an adaptive general linear model
abstract: Near-infrared spectroscopy is a non-invasive neuroimaging method which uses light to measure changes in cerebral blood oxygenation associated with brain activity. In this work, we demonstrate the ability to record and analyze images of brain activity in real-time using a 16-channel continuous wave optical NIRS system. We propose a novel real-time analysis framework using an adaptive Kalman filter and a state-space model based on a canonical general linear model of brain activity. We show that our adaptive model has the ability to estimate single-trial brain activity events as we apply this method to track and classify experimental data acquired during an alternating bilateral self-paced finger tapping task.

title: Designing a passive brain computer interface using real time classification of functional near-infrared spectroscopy
abstract: Passive brain-computer interfaces consider brain activity as an additional source of information, to augment and adapt the interface instead of controlling it. We have developed a software system that allows for real time brain signal analysis and machine learning classification of affective and workload states measured with functional near-infrared spectroscopy (fNIRS) called the online fNIRS analysis and classification (OFAC). Our system reproduces successful offline procedures, adapting them for real-time input to a user interface. Our first evaluation compares a previous offline analysis with our online analysis. While results show an accuracy decrease, they are outweighed by the new ability of interface adaptation. The second study demonstrates OFAC's online features through real-time classification of two tasks, and interface adaptation according to the predicted task. Accuracy averaged over 85%. We have created the first working real time passive BCI using fNIRS, opening the door to build adaptive user interfaces.

title: Brainput: enhancing interactive systems with streaming fnirs brain input
abstract: This paper describes the Brainput system, which learns to identify brain activity patterns occurring during multitasking. It provides a continuous, supplemental input stream to an interactive human-robot system, which uses this information to modify its behavior to better support multitasking. This paper demonstrates that we can use non-invasive methods to detect signals coming from the brain that users naturally and effortlessly generate while using a computer system. If used with care, this additional information can lead to systems that respond appropriately to changes in the user's state. Our experimental study shows that Brainput significantly improves several performance metrics, as well as the subjective NASA-Task Load Index scores in a dual-task human-robot activity.

title: Dynamic difficulty using brain metrics of workload
abstract: Dynamic difficulty adjustments can be used in human-computer systems in order to improve user engagement and performance. In this paper, we use functional near-infrared spectroscopy (fNIRS) to obtain passive brain sensing data and detect extended periods of boredom or overload. From these physiological signals, we can adapt a simulation in order to optimize workload in real-time, which allows the system to better fit the task to the user from moment to moment. To demonstrate this idea, we ran a laboratory study in which participants performed path planning for multiple unmanned aerial vehicles (UAVs) in a simulation. Based on their state, we varied the difficulty of the task by adding or removing UAVs and found that we were able to decrease error by 35% over a baseline condition. Our results show that we can use fNIRS brain sensing to detect task difficulty in real-time and construct an interface that improves user performance through dynamic difficulty adjustment.

title: Investigating the Need for Modelling Temporal Dependencies in a Brain-Computer Interface with Real-Time Feedback Based on near Infrared Spectra
abstract: Near infrared (NIR) spectroscopy is an emerging non-invasive brain-computer interface (BCI) modality that measures changes in haemoglobin concentrations in neurocortical tissue. Previous NIR spectroscopy studies have not employed real-time feedback with online classification, a combination which would allow users to alter their mental strategy on the fly. In particular, it is unclear whether or not the temporal dependencies of haemodynamic changes ought to be considered in online classification. To answer this quest ion, this study contrasted online classification of prefrontal haemodynamics using NIR spectra processed using two approaches: an artificial neural network (ANN) that considered instantaneous samples of oxy- and deoxy-haemoglobin concentrations (i.e. ignored temporal dependencies) and a hidden Markov model-based (HMM) classifier which modelled a temporal sequence of concentrations (i.e. embodied temporal dependencies). Both classifiers were implemented for online operation with immediate visual feedback via a monitor showing a vertical bar the height of which was contingent on the classifier's output. Ten subjects participated in two study sessions each, one with each type of classifier. Participants were cued to raise and lower the bar in alternating 20s intervals using mental fast singing and focused breathing, respectively. Only the ANN classifier facilitated online classification rates greater than chance (P = 0.0289). The influence of physiological noise on online classification of prefrontal haemodynamics was deemed to be minimal via offline analysis of concurrently measured respiration and blood pulse. Nine of the ten participants reported using the feedback to alter their activation strategy. Mental fatigue, task repetitiveness and the lack of ambient lighting were identified as factors compromising performance in half the participants. The inferior performance of the HMM classifiers suggests that modelling of the temporal dynamics of haemoglobin concentration changes may not be necessary in an online NIR-BCI. Further study of online NIR-BCIs with instantaneous feedback is warranted.

title: Dynamic topographical pattern classification of multichannel prefrontal NIRS signals: II. Online differentiation of mental arithmetic and rest.
abstract: Near-infrared spectroscopy (NIRS) has recently gained attention as a modality for brain-computer interfaces (BCIs), which may serve as an alternative access pathway for individuals with severe motor impairments. For NIRS-BCIs to be used as a real communication pathway, reliable online operation must be achieved. Yet, only a limited number of studies have been conducted online to date. These few studies were carried out under a synchronous paradigm and did not accommodate an unconstrained resting state, precluding their practical clinical implication. Furthermore, the potentially discriminative power of spatiotemporal characteristics of activation has yet to be considered in an online NIRS system.



In this study, we developed and evaluated an online system-paced NIRS-BCI which was driven by a mental arithmetic activation task and accommodated an unconstrained rest state. With a dual-wavelength, frequency domain near-infrared spectrometer, measurements were acquired over nine sites of the prefrontal cortex, while ten able-bodied participants selected letters from an on-screen scanning keyboard via intentionally controlled brain activity (using mental arithmetic). Participants were provided dynamic NIR topograms as continuous visual feedback of their brain activity as well as binary feedback of the BCI's decision (i.e. if the letter was selected or not). To classify the hemodynamic activity, temporal features extracted from the NIRS signals and spatiotemporal features extracted from the dynamic NIR topograms were used in a majority vote combination of multiple linear classifiers.



An overall online classification accuracy of 77.4 ± 10.5% was achieved across all participants. The binary feedback was found to be very useful during BCI use, while not all participants found value in the continuous feedback provided.



These results demonstrate that mental arithmetic is a potent mental task for driving an online system-paced NIRS-BCI. BCI feedback that reflects the classifier's decision has the potential to improve user performance. The proposed system can provide a framework for future online NIRS-BCI development and testing.

title: BCI meeting 2005-workshop on BCI signal processing: feature extraction and translation
abstract: This paper describes the outcome of discussions held during the Third International BCI Meeting at a workshop charged with reviewing and evaluating the current state of and issues relevant to brain-computer interface (BCI) feature extraction and translation. The issues discussed include a taxonomy of methods and applications, time-frequency spatial analysis, optimization schemes, the role of insight in analysis, adaptation, and methods for quantifying BCI feedback.

title: Enhanced performance by a hybrid NIRS–EEG brain computer interface
abstract: Noninvasive Brain Computer Interfaces (BCI) have been promoted to be used for neuroprosthetics. However, reports on applications with electroencephalography (EEG) show a demand for a better accuracy and stability. Here we investigate whether near-infrared spectroscopy (NIRS) can be used to enhance the EEG approach. In our study both methods were applied simultaneously in a real-time Sensory Motor Rhythm (SMR)-based BCI paradigm, involving executed movements as well as motor imagery. We tested how the classification of NIRS data can complement ongoing real-time EEG classification. Our results show that simultaneous measurements of NIRS and EEG can significantly improve the classification accuracy of motor imagery in over 90% of considered subjects and increases performance by 5% on average (p<0:01). However, the long time delay of the hemodynamic response may hinder an overall increase of bit-rates. Furthermore we find that EEG and NIRS complement each other in terms of information content and are thus a viable multimodal imaging technique, suitable for BCI.

title: Hemodynamic characteristics for improvement of EEG-BCI performance
abstract: Although brain computer interfaces (BCI) based on electroencephalographic (EEG) signals have been studied increasingly over decades, their performance is still limited in two main aspects. First, the shorter the EEG epochs, the more difficult the detection of a BCI command. Second, BCI commands are often misclassified while the subject is not performing any tasks by not focusing on a command (offset condition) because the EEG characteristics of the offset condition are not unique. In order to circumvent these limitations, the hemodynamic fluctuations in the brain during stimulation with steady-state visual evoked potentials (SSVEP) were measured using near infrared spectroscopy (NIRS) simultaneously with EEG. The offset condition is distinguished from the onset condition (focusing on a command) with extracted NIRS features through the design of low-pass filter. BCI command estimates were based on EEG SSVEP response. Simultaneous evaluation of EEG and NIRS was shown to improve the SSVEP classification, notably including the offset condition as an independent class, using a novel offset condition estimation approach. For 13 subjects, wrong classification for 9 classes with inclusion of offset condition were decreased. This proposed bimodal approach including the offset condition detection may render current BCI systems more reliable.

title: Speeded Near Infrared Spectroscopy (NIRS) Response Detection
abstract: The hemodynamic response measured by Near Infrared Spectroscopy (NIRS) is temporally delayed from the onset of the underlying neural activity. As a consequence, NIRS based brain-computer-interfaces (BCIs) and neurofeedback learning systems, may have a latency of several seconds in responding to a change in participants' behavioral or mental states, severely limiting the practical use of such systems. To explore the possibility of reducing this delay, we used a multivariate pattern classification technique (linear support vector machine, SVM) to decode the true behavioral state from the measured neural signal and systematically evaluated the performance of different feature spaces (signal history, history gradient, oxygenated or deoxygenated hemoglobin signal and spatial pattern). We found that the latency to decode a change in behavioral state can be reduced by 50% (from 4.8 s to 2.4 s), which will enhance the feasibility of NIRS for real-time applications.

title: Moving Average Convergence Divergence filter preprocessing for real-time event-related peak activity onset detection : Application to fNIRS signals
abstract: Real-time solutions for noise reduction and signal processing represent a central challenge for the development of Brain Computer Interfaces (BCI). In this paper, we introduce the Moving Average Convergence Divergence (MACD) filter, a tunable digital passband filter for online noise reduction and onset detection without preliminary learning phase, used in economic markets analysis. MACD performance was tested and benchmarked with other filters using data collected with functional Near Infrared Spectoscopy (fNIRS) during a digit sequence memorization task. This filter has a good performance on filtering and real-time peak activity onset detection, compared to other techniques. Therefore, MACD could be implemented for efficient BCI design using fNIRS.

title: Intersubject variability of near-infrared spectroscopy signals during sensorimotor cortex activation.
abstract: We investigate the intersubject signal variability of near-infrared spectroscopy (NIRS), which is commonly used for noninvasive measurement of the product of the optical path length and the concentration change in oxygenated hemoglobin (DeltaC'oxy) and deoxygenated hemoglobin (DeltaC'deoxy) and their sum (DeltaC'total) related to human cortical activation. We do this by measuring sensorimotor cortex activation in 31 healthy adults using 24-measurement-position near-infrared (NIR) topography. A finger-tapping task is used to activate the sensorimotor cortex, and significant changes in the hemisphere contralateral to the tapping hand are assessed as being due to the activation. Of the possible patterns of signal changes, 90% include a positive DeltaC'oxy, 76% included a negative DeltaC'deoxy, and 73% included a positive DeltaC'total. The DeltaC'deoxy and DeltaC'total are less consistent because of a large intersubject variability in DeltaC'deoxy; in some cases there is a positive DeltaC'deoxy. In the cases with no positive DeltaC'oxy in the contralateral hemisphere, there are cases of other possible changes for either or both hemispheres and no cases of no change in any hemoglobin species in either hemisphere. These results suggest that NIR topography is useful for observing brain activity in most cases, although intersubject signal variability still needs to be resolved.

title: Differences in the hemodynamic response to event-related motor and visual paradigms as measured by near-infrared spectroscopy
abstract: Several current brain imaging techniques rest on the assumption of a tight coupling between neural activity and hemodynamic response. The nature of this neurovascular coupling, however, is not completely understood. There is some evidence for a decoupling of these processes at the onset of neural activity, which manifests itself as a momentary increase in the relative concentration of deoxyhemoglobin (HbR). The existence of this early component of the hemodynamic response function, however, is controversial, as it is inconsistently found. Near infrared spectroscopy (NIRS) allows quantification of levels of oxyhemoglobin (HbO(2)) and HbR during task performance in humans. We acquired NIRS data during performance of simple motor and visual tasks, using rapid-presentation event-related paradigms. Our results demonstrate that rapid, event-related NIRS can provide robust estimates of the hemodynamic response without artifacts due to low-frequency signal components, unlike data from blocked designs. In both the motor and visual data the onset of the increase in HbO(2) occurs before HbR decreases, and there is a poststimulus undershoot. Our results also show that total blood volume (HbT) drops before HbO(2) and undershoots baseline, raising a new issue for neurovascular models. We did not find early deoxygenation in the motor data using physiologically plausible values for the differential pathlength factor, but did find one in the visual data. We suggest that this difference, which is consistent with functional magnetic resonance imaging (fMRI) data, may be attributable to different capillary transit times in these cortices.

title: Single-trial classification of NIRS signals during emotional induction tasks: towards a corporeal machine interface
abstract: BackgroundCorporeal machine interfaces (CMIs) are one of a few available options for restoring communication and environmental control to those with severe motor impairments. Cognitive processes detectable solely with functional imaging technologies such as near-infrared spectroscopy (NIRS) can potentially provide interfaces requiring less user training than conventional electroencephalography-based CMIs. We hypothesized that visually-cued emotional induction tasks can elicit forehead hemodynamic activity that can be harnessed for a CMI.MethodsData were collected from ten able-bodied participants as they performed trials of positively and negatively-emotional induction tasks. A genetic algorithm was employed to select the optimal signal features, classifier, task valence (positive or negative emotional value of the stimulus), recording site, and signal analysis interval length for each participant. We compared the performance of Linear Discriminant Analysis and Support Vector Machine classifiers. The latency of the NIRS hemodynamic response was estimated as the time required for classification accuracy to stabilize.ResultsBaseline and activation sequences were classified offline with accuracies upwards of 75.0%. Feature selection identified common time-domain discriminatory features across participants. Classification performance varied with the length of the input signal, and optimal signal length was found to be feature-dependent. Statistically significant increases in classification accuracy from baseline rates were observed as early as 2.5 s from initial stimulus presentation.ConclusionNIRS signals during affective states were shown to be distinguishable from baseline states with classification accuracies significantly above chance levels. Further research with NIRS for corporeal machine interfaces is warranted.

title: Technical Analysis: Power Tools for Active Investors
abstract: Unlike most technical analysis books, Gerald Appel's Practical Power Tools! offers step-by-step instructions virtually any investor can use to achieve breakthrough success in the market. Appel illuminates a wide range of strategies and timing models, demystifying even advanced technical analysis the first time. Among the models he covers: NASDAQ/NYSE Relative Strength, 3-5 Year Treasury Notes, Triple Momentum, Seasonality, Breadth-Thrust Impulse, and models based on the revolutionary MACD techniques he personally invented. Appel covers momentum and trend of price movement, time and calendar cycles, predictive chart patterns, relative strength, analysis of internal vs. external markets, market breadth, moving averages, trading channels, overbought/oversold indicators, Trin, VIX, major term buy signals, major term sell signals, moving average trading channels, stock market synergy, and much more. He presents techniques for short-, intermediate-, and long-term investors, and even for mutual fund investors.

title: Using MazeSuite and Functional Near Infrared Spectroscopy to Study Learning in Spatial Navigation
abstract: MazeSuite is a complete toolset to prepare, present and analyze navigational and spatial experiments1. MazeSuite can be used to design and edit adapted virtual 3D environments, track a participants' behavioral performance within the virtual environment and synchronize with external devices for physiological and neuroimaging measures, including electroencephalogram and eye tracking. Functional near-infrared spectroscopy (fNIR) is an optical brain imaging technique that enables continuous, noninvasive, and portable monitoring of changes in cerebral blood oxygenation related to human brain functions2-7. Over the last decade fNIR is used to effectively monitor cognitive tasks such as attention, working memory and problem solving7-11. fNIR can be implemented in the form of a wearable and minimally intrusive device; it has the capacity to monitor brain activity in ecologically valid environments. Cognitive functions assessed through task performance involve patterns of brain activation of the prefrontal cortex (PFC) that vary from the initial novel task performance, after practice and during retention12. Using positron emission tomography (PET), Van Horn and colleagues found that regional cerebral blood flow was activated in the right frontal lobe during the encoding (i.e., initial naïve performance) of spatial navigation of virtual mazes while there was little to no activation of the frontal regions after practice and during retention tests. Furthermore, the effects of contextual interference, a learning phenomenon related to organization of practice, are evident when individuals acquire multiple tasks under different practice schedules13,14. High contextual interference (random practice schedule) is created when the tasks to be learned are presented in a non-sequential, unpredictable order. Low contextual interference (blocked practice schedule) is created when the tasks to be learned are presented in a predictable order. Our goal here is twofold: first to illustrate the experimental protocol design process and the use of MazeSuite, and second, to demonstrate the setup and deployment of the fNIR brain activity monitoring system using Cognitive Optical Brain Imaging (COBI) Studio software15. To illustrate our goals, a subsample from a study is reported to show the use of both MazeSuite and COBI Studio in a single experiment. The study involves the assessment of cognitive activity of the PFC during the acquisition and learning of computer maze tasks for blocked and random orders. Two right-handed adults (one male, one female) performed 315 acquisition, 30 retention and 20 transfer trials across four days. Design, implementation, data acquisition and analysis phases of the study were explained with the intention to provide a guideline for future studies.

title: Modeling of influence of frontal sinus on NIRS signal of brain activation
abstract: In the brain activation measurements by near infrared spectroscopy (NIRS), the partial optical path length in the brain, which is an index of the sensitivity of the NIRS signal to the brain activation, is strongly affected by the thickness and the structure of the superficial tissues. In this study, we investigate the influence of the frontal sinus on the NIRS signal of the brain activation. The light propagation in a simplified head model including a void region mimicking the frontal sinus is predicted by Monte Carlo simulation to investigate the influence of the frontal sinus on the partial optical path length in the brain and the mean optical path length in the head. The frontal sinus strongly affects the light propagation in the head. The partial optical path length for small source-detector separation tends to be increased by the presence of the frontal sinus whereas that for large source-detector separation is decreased by the influence of the frontal sinus.

title: Support-Vector Networks
abstract: The support-vector network is a new learning machine for two-group classification problems. The machine conceptually implements the following idea: input vectors are non-linearly mapped to a very high-dimension feature space. In this feature space a linear decision surface is constructed. Special properties of the decision surface ensures high generalization ability of the learning machine. The idea behind the support-vector network was previously implemented for the restricted case where the training data can be separated without errors. We here extend this result to non-separable training data.High generalization ability of support-vector networks utilizing polynomial input transformations is demonstrated. We also compare the performance of the support-vector network to various classical learning algorithms that all took part in a benchmark study of Optical Character Recognition.

title: Building Predictive Models in R Using the caret Package
abstract: The caret package, short for classification and regression training, contains numerous tools for developing predictive models using the rich set of models available in R. The package focuses on simplifying model training and tuning across a wide variety of modeling techniques. It also includes methods for pre-processing training data, calculating variable importance, and model visualizations. An example from computational chemistry is used to illustrate the functionality on a real data set and to benchmark the benefits of parallel processing with several types of models.

title: Putting the Brain to Work: Neuroergonomics Past, Present, and Future
abstract: The authors describe research and applications in prominent areas of neuroergonomics.  Because human factors/ergonomics examines behavior and mind at work, it should include the study of brain mechanisms underlying human performance.  Neuroergonomic studies are reviewed in four areas: workload and vigilance, adaptive automation, neuroengineering, and molecular genetics and individual differences.  Neuroimaging studies have helped identify the components of mental workload, workload assessment in complex tasks, and resource depletion in vigilance. Furthermore, real-time neurocognitive assessment of workload can trigger adaptive automation. Neural measures can also drive brain-computer interfaces to provide disabled users new communication channels. Finally, variants of particular genes can be associated with individual differences in specific cognitive functions.  Neuroergonomics shows that considering what makes work possible — the human brain — can enrich understanding of the use of technology by humans and can inform technological design.  Applications of neuroergonomics include the assessment of operator workload and vigilance, implementation of real-time adaptive automation, neuroengineering for people with disabilities, and design of selection and training methods.

title: Frontal lobes and human memory: insights from functional neuroimaging.
abstract: The new functional neuroimaging techniques, PET and functional MRI (fMRI), offer sufficient experimental flexibility and spatial resolution to explore the functional neuroanatomical bases of different memory stages and processes. They have had a particular impact on our understanding of the role of the frontal cortex in memory processing. We review the insights that have been gained, and attempt a synthesis of the findings from functional imaging studies of working memory, encoding in episodic memory and retrieval from episodic memory. Though these different aspects of memory have usually been studied in isolation, we suggest that there is sufficient convergence with respect to frontal activations to make such a synthesis worthwhile. We concentrate in particular on three regions of the lateral frontal cortex--ventrolateral, dorsolateral and anterior--that are consistently activated in these studies, and attribute these activations to the updating/maintenance of information, the selection/manipulation/monitoring of that information, and the selection of processes/subgoals, respectively. We also acknowledge a number of empirical inconsistencies associated with this synthesis, and suggest possible reasons for these. More generally, we predict that the resolution of questions concerning the functional neuroanatomical subdivisions of the frontal cortex will ultimately depend on a fuller cognitive psychological fractionation of memory control processes, an enterprise that will be guided and tested by experimentation. We expect that the neuroimaging techniques will provide an important part of this enterprise.

title: An integrative theory of prefrontal cortex function.
abstract: The prefrontal cortex has long been suspected to play an important role in cognitive control, in the ability to orchestrate thought and action in accordance with internal goals. Its neural basis, however, has remained a mystery. Here, we propose that cognitive control stems from the active maintenance of patterns of activity in the prefrontal cortex that represent goals and the means to achieve them. They provide bias signals to other brain structures whose net effect is to guide the flow of activity along neural pathways that establish the proper mappings between inputs, internal states, and outputs needed to perform a given task. We review neurophysiological, neurobiological, neuroimaging, and computational studies that support this theory and discuss its implications as well as further issues to be addressed

title: The Hybrid BCI
abstract: Nowadays, everybody knows what a hybrid car is. A hybrid car normally has two engines to enhance energy efficiency and reduce CO2 output. Similarly, a hybrid brain-computer interface (BCI) is composed of two BCIs, or at least one BCI and another system. A hybrid BCI, like any BCI, must fulfill the following four criteria: (i) the device must rely on signals recorded directly from the brain; (ii) there must be at least one recordable brain signal that the user can intentionally modulate to effect goal-directed behaviour; (iii) real time processing; and (iv) the user must obtain feedback. This paper introduces hybrid BCIs that have already been published or are in development. We also introduce concepts for future work. We describe BCIs that classify two EEG patterns: one is the event-related (de)synchronisation (ERD, ERS) of sensorimotor rhythms, and the other is the steady-state visual evoked potential (SSVEP). Hybrid BCIs can either process their inputs simultaneously, or operate two systems sequentially, where the first system can act as a “brain switch”. For example, we describe a hybrid BCI that simultaneously combines ERD and SSVEP BCIs. We also describe a sequential hybrid BCI, in which subjects could use a brain switch to control an SSVEP-based hand orthosis. Subjects who used this hybrid BCI exhibited about half the false positives encountered while using the SSVEP BCI alone. A brain switch can also rely on hemodynamic changes measured through near-infrared spectroscopy (NIRS). Hybrid BCIs can also use one brain signal and a different type of input. This additional input can be an electrophysiological signal such as the heart rate, or a signal from an external device such as an eye tracking system.

title: Temporal classification of multichannel near-infrared spectroscopy signals of motor imagery for developing a brain–computer interface
abstract: There has been an increase in research interest for brain-computer interface (BCI) technology as an alternate mode of communication and environmental control for the disabled, such as patients suffering from amyotrophic lateral sclerosis (ALS), brainstem stroke and spinal cord injury. Disabled patients with appropriate physical care and cognitive ability to communicate with their social environment continue to live with a reasonable quality of life over extended periods of time. Near-infrared spectroscopy is a non-invasive technique which utilizes light in the near-infrared range (700 to 1000 nm) to determine cerebral oxygenation, blood flow and metabolic status of localized regions of the brain. In this paper, we describe a study conducted to test the feasibility of using multichannel NIRS in the development of a BCI. We used a continuous wave 20-channel NIRS system over the motor cortex of 5 healthy volunteers to measure oxygenated and deoxygenated hemoglobin changes during left-hand and right-hand motor imagery. We present results of signal analysis indicating that there exist distinct patterns of hemodynamic responses which could be utilized in a pattern classifier towards developing a BCI. We applied two different pattern recognition algorithms separately, Support Vector Machines (SVM) and Hidden Markov Model (HMM), to classify the data offline. SVM classified left-hand imagery from right-hand imagery with an average accuracy of 73% for all volunteers, while HMM performed better with an average accuracy of 89%. Our results indicate potential application of NIRS in the development of BCIs. We also discuss here future extension of our system to develop a word speller application based on a cursor control paradigm incorporating online pattern classification of single-trial NIRS data.

title: Classifying human operator functional state based on electrophysiological and performance measures and fuzzy clustering method
abstract: The human operator’s ability to perform their tasks can fluctuate over time. Because the cognitive demands of the task can also vary it is possible that the capabilities of the operator are not sufficient to satisfy the job demands. This can lead to serious errors when the operator is overwhelmed by the task demands. Psychophysiological measures, such as heart rate and brain activity, can be used to monitor operator cognitive workload. In this paper, the most influential psychophysiological measures are extracted to characterize Operator Functional State (OFS) in automated tasks under a complex form of human–automation interaction. The fuzzy c-mean (FCM) algorithm is used and tested for its OFS classification performance. The results obtained have shown the feasibility and effectiveness of the FCM algorithm as well as the utility of the selected input features for OFS classification. Besides being able to cope with nonlinearity and fuzzy uncertainty in the psychophysiological data it can provide information about the relative importance of the input features as well as the confidence estimate of the classification results. The OFS pattern classification method developed can be incorporated into an adaptive aiding system in order to enhance the overall performance of a large class of safety–critical human–machine cooperative systems.

title: Formal Detection of Attentional Tunneling in Human Operator–Automation Interactions
abstract: The allocation of visual attention is a key factor for the humans when operating complex systems under time pressure with multiple information sources. In some situations, attentional tunneling is likely to appear and leads to excessive focus and poor decision making. In this study, we propose a formal approach to detect the occurrence of such an attentional impairment that is based on machine learning techniques. An experiment was conducted to provoke attentional tunneling during which psycho-physiological and oculomotor data from 23 participants were collected. Data from 18 participants were used to train an adaptive neuro-fuzzy inference system (ANFIS). From a machine learning point of view, the classification performance of the trained ANFIS proved the validity of this approach. Furthermore, the resulting classification rules were consistent with the attentional tunneling literature. Finally, the classifier was robust to detect attentional tunneling when performing over test data from four participants.

title: Reinforcement Learning: A Survey
abstract: This paper surveys the field of reinforcement learning from a computer-science perspective. It is written to be accessible to researchers familiar with machine learning. Both the historical basis of the field and a broad selection of current work are summarized. Reinforcement learning is the problem faced by an agent that learns behavior through trial-and-error interactions with a dynamic environment. The work described here has a resemblance to work in psychology, but differs considerably in the details and in the use of the word "reinforcement." The paper discusses central issues of reinforcement learning, including trading off exploration and exploitation, establishing the foundations of the field via Markov decision theory, learning from delayed reinforcement, constructing empirical models to accelerate learning, making use of generalization and hierarchy, and coping with hidden state. It concludes with a survey of some implemented systems and an assessment of the practical utility of current methods for reinforcement learning.

title: Circadian and sleep/wake dependent aspects of subjective alertness and cognitive performance
abstract: Circadian and sleep/wake dependent processes underlying variations in subjective alertness and cognitive performance were assessed in a constant routine protocol and in a protocol in which the sleep/wake cycle was uncoupled from the output of the endogenous circadian pacemaker. In the latter protocol, the contribution of a sleep/wake dependent process and a circadian process to alertness and performance were separated by folding the data at either the period of the sleep/wake cycle or at the period of the endogenous circadian body temperature rhythm. This analysis revealed that prior wakefulness within a range of 0–18 h significantly reduced alertness and performance and that the circadian rhythm of core body temperature paralleled the circadian rhythm of alertness and performance. During the first 16 h of the constant routine protocol, which coincided with the subjects' habitual period of wakefulness, alertness and performance remained at a stable level. The latter finding was explained by assuming that during our usual waking day the circadian system counteracts the detrimental effects of increasing duration of prior wakefulness.

title: Meta-classification: Combining Multimodal Classifiers
abstract: Combining multiple classifiers is of particular interest in multimedia applications. Each modality in multimedia data can be analyzed individually, and combining multiple pieces of evidence can usually improve classification accuracy. However, most combination strategies used in previous studies implement some ad hoc designs, and ignore the varying expertise of specialized individual modality classifiers in recognizing a category under particular circumstances. In this paper we present a combination framework called meta-classification, which models the problem of combining classifiers as a classification problem itself. We apply the technique on a wearable experience collection system, which unobtrusively records the wearer's conversation, recognizes the face of the dialogue partner, and remember his/her voice. When the system sees the same person's face or hears the same voice, it can then use a of the last conversation to remind the wearer. To identify a person correctly from a mixture of audio and video streams, classification judgments from multiple modalities must be effectively combined. Experimental results show that combining different face recognizers and speaker identification aspects using the meta-classification strategy can dramatically improve classification accuracy, and is more effective than a fixed probability-based strategy. Other work in labeling weather news broadcasts showed that meta-classification is a general framework that can be applied to any application that needs to combine multiple classifiers without much modification.

title: A Meta-classifier Approach for Medical Diagnosis
abstract: Single classifiers, such as Neural Networks, Support Vector Machines, Decision Trees and other, can be used to perform classification of data for relatively simple problems. For more complex problems, combinations of simple classifiers can significantly improve performance. There are several combination methods, like Bagging and Boosting that combine simple classifiers. We propose, here, a new meta-classifier approach which combines several different combination methods, in analogy to the combination of simple classifiers. The meta-classifier approach is employed in the implementation of a medical diagnosis system and evaluated using three benchmark diagnosis problems as well as a problem concerning the classification of hepatic lesions from computed tomography (CT) images.

title: Intersession Consistency of Single-Trial Classification of the Prefrontal Response to Mental Arithmetic and the No-Control State by NIRS
abstract: Near-infrared spectroscopy (NIRS) has been recently investigated for use in noninvasive brain-computer interface (BCI) technologies. Previous studies have demonstrated the ability to classify patterns of neural activation associated with different mental tasks (e.g., mental arithmetic) using NIRS signals. Though these studies represent an important step towards the realization of an NIRS-BCI, there is a paucity of literature regarding the consistency of these responses, and the ability to classify them on a single-trial basis, over multiple sessions. This is important when moving out of an experimental context toward a practical system, where performance must be maintained over longer periods. When considering response consistency across sessions, two questions arise: 1) can the hemodynamic response to the activation task be distinguished from a baseline (or other task) condition, consistently across sessions, and if so, 2) are the spatiotemporal characteristics of the response which best distinguish it from the baseline (or other task) condition consistent across sessions. The answers will have implications for the viability of an NIRS-BCI system, and the design strategies (especially in terms of classifier training protocols) adopted. In this study, we investigated the consistency of classification of a mental arithmetic task and a no-control condition over five experimental sessions. Mixed model linear regression on intrasession classification accuracies indicate that the task and baseline states remain differentiable across multiple sessions, with no significant decrease in accuracy (p = 0.67). Intersession analysis, however, revealed inconsistencies in spatiotemporal response characteristics. Based on these results, we investigated several different practical classifier training protocols, including scenarios in which the training and test data come from 1) different sessions, 2) the same session, and 3) a combination of both. Results indicate that when selecting optimal classifier training protocols for NIRS-BCI, a compromise between accuracy and convenience (e.g., in terms of duration/frequency of training data collection) must be considered.

title: Towards Zero Training for Brain-Computer Interfacing
abstract: Electroencephalogram (EEG) signals are highly subject-specific and vary considerably even between recording sessions of the same user within the same experimental paradigm. This challenges a stable operation of Brain-Computer Interface (BCI) systems. The classical approach is to train users by neurofeedback to produce fixed stereotypical patterns of brain activity. In the machine learning approach, a widely adapted method for dealing with those variances is to record a so called calibration measurement on the beginning of each session in order to optimize spatial filters and classifiers specifically for each subject and each day. This adaptation of the system to the individual brain signature of each user relieves from the need of extensive user training. In this paper we suggest a new method that overcomes the requirement of these time-consuming calibration recordings for long-term BCI users. The method takes advantage of knowledge collected in previous sessions: By a novel technique, prototypical spatial filters are determined which have better generalization properties compared to single-session filters. In particular, they can be used in follow-up sessions without the need to recalibrate the system. This way the calibration periods can be dramatically shortened or even completely omitted for these ‘experienced’ BCI users. The feasibility of our novel approach is demonstrated with a series of online BCI experiments. Although performed without any calibration measurement at all, no loss of classification performance was observed.