Electrophysiological Feedback in Adaptive Human Computer Interfaces

This paper explores the feasibility of using EEG in the context of Stimulus Rich Reactive Interfaces (SRRIs), as a source of feedback on the cognitive state of the user. We have run experiments to evaluate the utility of two potential EEG measures of whether a stimulus has been perceived: 1) reduced EEG power in the alpha band at posterior brain areas and 2) a P3-like positive deflection over parietal areas. Such measures would enable re-presentation of a critical stimulus that has been missed. This paper considers whether, in the context of SRRIs, these measures can be reliably extracted online, i.e. in real-time. To determine this, we have investigated the extent to which online extraction of these measures predicts target report. Our results are positive, and suggest that a combination of multiple approaches can be used to improve the reliability of single trial discrimination. We also discuss possible ways in which such a system could be implemented and integrated into a head-mounted wearable display that is electrically isolated from the surroundings by optical input and output. Introduction One of the fundamental problems involved in real-time processing is information overload. A system has to pick and choose which stimuli to attend, before it has expended the resources to process them. The human brain has evolved a sophisticated attentional mechanism to pick and choose what information is selected for further processing. As a consequence, the designer of a computer interface is not always able to ensure that what the interface knows is important (such as a low altitude warning on an aircraft) is selected by the human users attention system. HCI design has learned a great deal of tricks to increase the inherent salience of input signals to a human user (using warning lights and sounds, for example), but accidents still occur due to critical information that is ignored. If described using computer networking as a metaphor, this situation is akin to transmitting information over a communication channel without acknowledgement of receipt. The computer is transmitting information to the user with the intent that it be received, but the system has no way of verifying that this transmission was successfully completed. No well-designed computer-computer interface operates in this way. Acknowledgement of packet receipt is a core element of any modern networking technology. The goal of this research is to record an acknowledgement signal from scalp recorded brain waves. Electrophysiological neuroscience has identified a number of electrical signals produced by the brain when information is encoded into working memory that could serve this role. In this paper, we explore a novel use of recorded brain signals, to enhance the reliability of information transmission from computer to user. Brain Computer Interface: The direct brain to computer interface (e.g. Vidal 1973, Levine, et al 2000, Beverina, Palmas, Silvoni, Piccione, & Giove 2000) has been one of the most useful applications to have arisen from electrophysiological neuroscience. By recording brain waves from human subjects with motor impairments, it has been possible to give the severely disabled a means of communication with the world (e.g. Cilliers & Van Der Kouwe 1993) Using these techniques, a subject can passively view a matrix of characters and guide a computer interface that is recording and analyzing EEG signals to the intended words he or she wishes to communicate. For some people, the electrical signal produced by their brain is 1 The full reference for this article is, B. Wyble, P. Craston and H. Bowman. "Electrophysiological feedback in adaptive human computer interfaces. " Technical Report 8-06, Computing Lab, University of Kent at Canterbury, September 2006. the only means by which they can consciously communicate with the world. The Brain-ComputerInterface (BCI) field is now quite mature, using cutting edge technology to maximize the accuracy and speed of the information that can be read from the disabled subjects brain (Meinicke, Kasper, Hoppe, Heumann & Ritter 2003; Hochberg, et al 2006) Work in the BCI field has almost exclusively focused on human-to-computer information transfer. However, recording brain signals can also serve to enhance the reliability of information flow in the opposite direction, an application that is virtually unexplored. We will describe a system that can use brainwaves to warn a computer that its user may have missed a critical piece of information. This warning will allow the computer to re-present missed information until it is perceived. The algorithms described in this report are simpler than those commonly used in BCI research. The intent of our approach is to develop a device with has the following characteristics: o Implementable with currently available electrical components o Small enough to be enclosed within a helmet o Easily shielded from nearby interference o Power consumption sufficiently low to run on lightweight battery power o Extremely rapid response (response within less than 1 second of target onset) o Easy to set up These restrictions presently rule out both sophisticated waveform analyses and multi-electrode arrays. This paper focuses on whether an uncomplicated analysis of data recorded from a single pair of electrodes can provide useful information to the computer about whether a target was detected. We will focus on two elements of the human EEG signal as indicators of target perception: the P3 component of the Event Related Potential (ERP) and changes in the power of oscillations near 10 hz, known as the Alpha power band. The P3 Component: When one records EEG from the human scalp, the signal measured is deflected by ongoing cognitive operations. In the EEG literature, these deflections are referred to as components, which are observed in the Event Related Potential (ERP) that emerges from averaging together a large number of trials, time locked to the onset of a salient stimulus. ERP components can be manipulated experimentally; hence, they have been associated with particular cognitive processes. Whereas the early part of the waveform is associated with sensory processing of target stimuli, the later part is associated with high-level processing of a stimulus. This component is typically called the P3 (i.e., the third positive peak of the ERP, also referred to as the P300 due to its typical latency of 300ms post-stimulus). Although some researchers have identified a frontally located P3a component, which is elicited by infrequent but task-irrelevant stimuli, we focus on the P3b, which has its maximum over parietal electrode sites. The P3b (called P3 from here on) is present for stimuli that are both infrequent and relevant to the task (Squires et al. 1975), As the P3 is only observed in the ERP for target stimuli that are detected by the subject (Vogel et al. 1998), it can be assumed to be an indication of an item being encoded into working memory (Donchin, 1981). Depending on the amount of noise in the signal, one normally has to average across a considerable number of trials to obtain a clean ERP waveform. However, the P3 component is often large enough to be detected even in the raw EEG. Of course, one cannot draw conclusions about P3 latency and shape from raw P3s; however, they are often clear enough to be detected on a trial-by-trial basis. The algorithm used in our approach focuses on these raw P3s. Alpha Waves: During scalp EEG recording, alpha oscillations (in the 8-12 hz band) are prominent in posterior areas of the brain and this is especially true when the subjects eyes are closed. Alpha activity is so large that it can easily be discerned in the raw, unfiltered signal and is generally much larger than the P3 component, within an individual trial. However, because alpha activity is poorly phase locked to external events, it is not visible in most event triggered average plots, as employed in ERPs. However, if alpha activity is computed within individual waveforms, using FFTs, strong changes in this oscillation can be observed in relation to cognitive events. One prominent relationship between oscillations and mental activity is that alpha power tends to diminish at the onset of a cognitive event, such as detecting a target, encoding something into memory, or initiating a movement. This is called an ERD or Event Related Desynchronization (see Pfurtshellter & da Silva 1999 for a review). It is thought that alpha activity may reflect an area of cortex that is being held in idling state. Experiments and Data: To test our ideas, we used data from another experiment, which presented targets embedded in a stream of distractors. We were able to record prominent P3 ERPs as well as alpha waves. This EEG data serves as a test of the ability of online algorithms to determine wether a target was seen or missed. The experiment involved Rapid Serial Visual Presentation (RSVP), a presentation method commonly used in the visual attention field (Weichselgartner & Sperling 1987). In RSVP, a subject fixates on the center of the screen and views a stream of constantly changing stimuli, which contains one or more targets. In the experiments reported here, subjects viewed a stream of digits in search for any uppercase letter, which was reported at the end of a trial. The subjects did not know when the letter would appear, and stimuli changed every 47 ms. EEG data were compared with target detection on corresponding trials, to gauge the ability of EEG to discriminate between target seen and target missed trials for a random group of 12 subjects. Method Participants: Twelve underand postgraduate university students (mean age 24.1, Std. Dev. 2.9; 6 male and 6 female; 11 rightand 1 left-handed) provided written consent and received 10 GBP for participation. Participants were free from neurological disorders and had normal or corrected-to-normal vision. The study was approved by the local ethics c