# Evaluation of a P300-Based Interface for Smart Home Control

Cristian-Cezar Postelnicu, Alexandra Covaci, Alina Ninett Panfir, Doru Talaba

► To cite this version:

Cristian-Cezar Postelnicu, Alexandra Covaci, Alina Ninett Panfir, Doru Talaba. Evaluation of a P300-Based Interface for Smart Home Control. 3rd Doctoral Conference on Computing, Electrical and Industrial Systems (DoCEIS), Feb 2012, Costa de Caparica, Portugal. pp.179-186, 10.1007/978-3-642-28255-3_20. [https://inria.hal.science/hal-01365584](https://inria.hal.science/hal-01365584)

HAL Id: hal-01365584

https://inria.hal.science/hal-01365584

Submitted on 13 Sep 2016

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Evaluation of a P300-Based Interface for Smart Home Control

Cristian-Cezar Postelnicu¹, Alexandra Covaci¹, Alina Ninett Panfir¹, Doru Talaba¹

¹ Transilvania University of Brasov, Romania,
Department of Product Design and Robotics,
{cristian-cezar.postelnicu, alexandra.covaci, alina.panfir, talaba}@unitbv.ro

**Abstract.** This paper evaluates a hybrid neurophysiological interface for controlling a virtual smart home. Electrooculography (EOG) and Electroencephalography (EEG) techniques are used for recording biosignals from the user's body. The signals are further analyzed and translated into commands for controlling a virtual smart home. The purpose of the study was the evaluation of the implemented interface and the rejection of a set of possible erroneous commands generated by the interface. From a main menu the user is able to select a mask that will reveal a second menu with supplementary commands. The user has the option to cancel the propagation of an erroneous selected mask in the system by selecting another item from the main menu.

**Keywords:** Electrooculography, Electroencephalography, Brain Computer Interface, P300.

## 1 Introduction

A human-computer interface based on neurophysiological signals, measures, analyzes and translates the electrical activity from the body in commands used for controlling external devices such as a cursor on a computer screen, robots or virtual environments [1]. A system that uses EEG signals for controlling spelling applications [2], [3], [6], [15], [16], virtual environments [14] or a wheelchair [4] is named Brain-Computer Interface (BCI). These systems were mainly developed for people with severe neuromuscular diseases such as brainstem stroke or amyotrophic lateral sclerosis (ALS), but can also be valuable for healthy persons by providing them a supplementary communication channel.

The EOG technique, which measures the resting potential of the retina, was previously used for controlling a virtual robotic arm [9], a virtual keyboard [5] or a wheelchair [7]. This technique models the eye as a dipole with a positive pole at the cornea and a negative pole at the retina. The signals are recorded by two pairs of electrodes placed around the eyes at reference positions [10].

A smart home was previously controlled by brainwaves in [11], [12] and [13]. A set of 7 masks: TV, phone, music, light, temperature, go to, move were created for controlling the devices and the navigation inside the house. Three subjects participated in the experiment and they were given tasks such as: go to the living
room, switch on the TV and select a specific channel. The control of the movement inside the house is made on the principle of goal-oriented actions, which means that the user has to select only the destination, and the system will automatically move to that location. Thus, the user does not have to execute all the required actions for going at a desired place inside the house, resulting in a simplified interaction method between the user and the environment. This interaction method is extremely valuable for disabled persons that cannot communicate or interact with other persons through normal pathways such as voice or gestures. The system uses the P300 wave for commands selection. It represents an amplitude peak in the EEG signals at around 300ms after a rare stimulus that was presented to the user. It appears in the signal only for the desired command, for the others the EEG signals having normal amplitude.

In this paper, we focus on the evaluation of a paradigm for controlling a virtual smart home by using the EOG and EEG signals for selecting commands from a predefined set. In the “Half Single Display Paradigm” and “System Architecture” sections are presented the details of the proposed interaction method and the “Results” section lists the values obtained from the conducted experiment. We conclude by presenting further improvements for the proposed interaction paradigm and possible research directions.

## 2 Technological Innovation for Value Creation

A novel P300 stimulus presentation paradigm is presented and evaluated. The paradigm assumes a virtual separation of symbols contained by a P300 matrix in two regions, thus the time required for a command selection being reduced to half. It also offers the possibility to generate commands in a cascade manner by creating a menu with main commands (masks), while a second menu will reveal commands for a selected mask.

The proposed paradigm was applied for selecting commands in order to control a smart home. Similar studies were made on control of a smart home [11], [12] and [13], and the results suggested that the P300-based paradigm interaction has a strong potential for real environments control. Compared with the other studies in the present paper the proposed paradigm offers a solution for avoiding a set of possible errors. For example the classification algorithm can generate a command that was not desired by the user (the classifier does not have 100% accuracy all the time), and further the user must select another command from an undesired menu. Thus, by using our paradigm the user has the ability to cancel a wrong generated command by selecting another menu item (the command actually desired by the user). The paradigm is mainly developed for disabled persons that can communicate with other persons or assistive devices only by means of their biosignals. A smart home presents a strong potential for creating a set of autonomous functions for their living, and the virtual environment is a valuable tool for initial tests.
## 3 Developed System

Starting from the smart home developed in [13], we propose a modified P300 matrix interface that avoids a main issue that might appear in the original interface. Thus, in the original interface there is the possibility to generate a wrong command from the ‘go to’ mask e.g. go to the living room while the user was actually thinking of activating a different mask, e.g. the ‘TV‘ mask (see Fig. 1). In the original interface, the user selects a mask, revealing its specific commands that are displayed below or next to the P300 main matrix. Starting from this point the application will flash all the icons for generating another command [13]. For the ‘go to’ mask, the reported average accuracy was of 94.43%, and it sustains our hypothesis that wrong commands can be generated by the application [13].

We propose the use of a similar interface, but for example if the application identifies by mistake a wrong mask the user will have the opportunity to cancel that mask. Thus, by using EOG signals the application can automatically identify a wide area, e.g. the P300 main matrix (see Fig. 1) or the P300 mask model, where the user gazes at. For example, if the application identifies by mistake a wrong mask then by gazing at the main menu (it contains the basic functions and the masks) the application will only try to select a new command from this menu. The wrong selected mask will be canceled before sending further wrong commands, and a new item will be selected from the main menu. For the present study the left area is considered the main menu and the right area is defined for the functions under the masks.

<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Left – the P300 matrix for main menu; the ‘Speller’ mask is currently flashed; right – the commands under the Go to mask; the user is currently gazing at the main menu</figcaption></figure>
## 3.1 Half Single Display Paradigm

The Single Display Paradigm (SDP) was first introduced in [15], and it flashes a single item at a time. Results suggested that the SDP produces higher P300 peaks compared with the row-column paradigm (RCP), resulting in higher accuracy rates [15]. A recent study proved that the accuracy of the P300 speller application considerably depends on gaze direction [17]. It results that a system that would use both the P300 interface and the user's gaze might increase its accuracy.

Based on the SDP and EOG, the paradigm used in this paper splits a typical P300 matrix in two separate regions: “left” and “right” – the half SDP (HSDP). The paradigm is based on a hybrid configuration that used EOG and EEG signals. The selection of a region is made automatically by tracking the user's gaze. By analyzing the EOG signals the system will ‘know’ in which region the user is gazing at. It results that a supplementary matrix of commands can be added in the system without increasing the time required for a command selection (see Fig. 1). An EOG calibration is performed when the application starts in order to make the correlation between the user's gaze and the area on the computer's monitor. A predefined sequence of visual stimuli is presented to the user. The stimuli are drawn at fixed positions on the computer's monitor at ±2.5°, ±5°, ±10° and ±15° (degrees of visual angle). Values are computed with respect to the distance of 50cm were the users are placed in front of the monitor during the experiment. We used a model that assumes linearity by intervals because the amplitude values corresponding to the saccades do not vary perfectly linearly for the entire interval. After the calibration is performed, the application will continue with its normal flow.

The effective selection of a command is performed by analyzing the EEG signals. The P300 peak amplitudes are searched within the corresponding buffers after all the flashes are performed for a command selection. After a buffer is selected, its corresponding command is executed. For each flash of one item its corresponding buffer contains 700ms of information from the EEG signals. One main restriction that is implemented in our paradigm assumes that the same item cannot be flashed twice during the 700ms interval.

## 3.2 System Architecture

Our system records EEG and EOG patient's biosignals, analyzes and translates them into commands for controlling the virtual smart house (see Fig. 2). The application for acquisition, processing, analyzing and classifying the biosignals is implemented in C++, while the application that renders the virtual smart home was developed in XVR (eXtreme Virtual Reality). The application for the smart home is hidden (the user cannot see it) because we were mainly focused on the evaluation of the interaction menu. The commands sent by the users are recorded, and after the test is done the user is able to see the application that presents the house and the commands that were selected by him step by step.

An EOG calibration is required when the application starts in order to know where the user gazes at. The origin of the system is considered between the edge of the main menu and the secondary menu (commands under the mask – when no mask is
selected there is no menu on the right side). Once the gazing area is identified (3s are
allocated for this task – the area identification is computed automatically) the P300
stimuli presentation task starts. After all the stimuli are presented the classification
process is started, a command is identified, and the application repeats the area
identification process. The “EOG search area” phase is also performing a
recalibration after each two commands are identified. The recalibration assumes that
the user must gaze at the origin of the system for 2s, while all the commands are
hidden. After the recalibration is done the application ‘knows’ exactly were the user
gazes at a specific time. Thus, the system can always identify exactly the “left” or
“right” areas all the time.

<figure><img src="image_3.png" /><figcaption>**Fig. 2.** System architecture. Left - the C++ application flow with the acquisition, filtering and classification blocks; right – the XVR application that renders the virtual smart home</figcaption></figure>

**3.3 Experimental Setup**

For the present study, a g.USBamp (g.tec Medical Engineering GmbH, Austria) system was used for recording the EOG and EEG signals. The EOG recording was made by using electrodes placed in a bipolar configuration [9]. Two signals are recorded for the EOG (horizontal and vertical eye movements) and eye saccades are used for identifying the user’s gaze point, while the blinks and winks are rejected. The EOG signals were bandpass filtered between 0.1Hz and 30Hz; also a notch filter was applied to suppress the line noise. The EEG signals were recorded using a single electrode placed at Cz location according to 10/20 international system [8]. All signals were sampled at 256Hz and the EEG signals were bandpass filtered between 0.5Hz and 30Hz.
The users were placed in front of a computer monitor at a distance of around 50cm. The stimuli from the main menu were flashed for 78.125ms and the dark time was approximately 43ms (42.96875ms), each icon being flashed for 12 times. It results a total time of ≈21.8s [(78ms+43ms) * 15 commands * 12 times] required for selecting a command from the main menu. For the commands under the masks different flash and dark times were chosen depending on the number of available commands. Basically, for a higher number of commands lower flash and dark time values are chosen, and respectively for a lower number of commands higher values are defined.

For each item in the current active area, a buffer is assigned and it is filled with the EEG signals starting after each corresponding flash. A simple algorithm for P300 peaks classification was implemented. It searches for a typical P300 wave [1] in the buffers corresponding to the flashed stimuli and chooses the buffer that contains the signal that 'looks' the most similar with the typical P300 wave.

## 3.4 Results

Four subjects took part in a series of experiments; three subjects had previous experience with EOG or EEG systems. They were asked to follow a simple test that consisted of 5 tasks each one with 5 commands. For example one task was: select TV – Channel 1 – Light – Go to – bedroom. In this example “Light” was considered as a wrong command, although the user was asked to select it. The wrong command was inserted in order to validate the paradigm and whether the user can cancel a wrong selected mask. The application was issuing a new command only if the previous command was correctly selected.

The maximum achieved accuracy was of 92% for one user, while the average accuracy for all subjects was of 73%. Three users were able to complete 2 or 3 tasks with 100% and all of them could correctly select at least a command from the interface. A correct or wrong selection was considered only for the first attempt of selecting that command. The application didn't count the latter attempts for that command although the user was 'forced' to continue until he had successfully selected the correct command.

The interface was also analyzed for the EOG classification accuracy, and for all commands identified by the application the area where the users were gazing at was correctly identified.

<table><caption>**Table 1.** Results obtained for each user (for the average and for each task is mentioned the number of successful commands)</caption><thead><tr><th></th><th>User 1</th><th>User 2</th><th>User 3</th><th>User 4</th></tr></thead><tbody><tr><td>Task 1</td><td>4</td><td>3</td><td>4</td><td>3</td></tr><tr><td>Task 2</td><td>5</td><td>2</td><td>3</td><td>4</td></tr><tr><td>Task 3</td><td>4</td><td>5</td><td>1</td><td>3</td></tr><tr><td>Task 4</td><td>5</td><td>4</td><td>2</td><td>4</td></tr><tr><td>Task 5</td><td>5</td><td>5</td><td>4</td><td>3</td></tr><tr><td>Average</td><td>4.6</td><td>3.8</td><td>2.8</td><td>3.4</td></tr><tr><td>Accuracy (%)</td><td>92</td><td>76</td><td>56</td><td>68</td></tr><tr><td>Average accuracy (%)</td><td></td><td>73</td><td></td><td></td></tr></tbody></table>
## 4 Conclusions and Future Work

The results proved that the users could control the virtual house with good accuracies. The proposed paradigm avoids propagation of errors from a previous command to a next one. The method represents a valuable solution for testing applications which will be further tested under real environments.

The goal of the study was the evaluation of the HSDP for a virtual reality application and the avoidance of errors propagation. Maximum accuracies couldn't be achieved for all the tasks because the classification algorithm for the P300 was a simple one. By using linear discriminant analysis (LDA) we are expecting for higher accuracies as it was previously proved in other tests [14]. The users were able to cancel a wrong command by selecting another one from the main menu, although the maximum accuracy wasn't achieved for all the given tasks.

Future work is related to development of a real environment and integration of the P300-based interaction paradigm within. The possible research activities include the development of a wheelchair for navigation purposes, and the user will select commands from a monitor attached to the wheelchair. Thus, the user will have the ability to navigate in the real smart home only by thoughts and in a goal-oriented fashion. The HSDP has a strong potential for the P300 speller applications and we will perform a comparative study with the currently available paradigms for this type of application: SDP, RCP or the checkerboard paradigm [14], [15] and [16].

**Acknowledgments.** This paper is supported by the Sectoral Operational Programme Human Resources Development (SOP HRD), financed from the European Social Fund and by the Romanian Government under the contract number POSDRU/88/1.5/S/59321.

## References

1. Wolpaw, J.R., Birbaumer, N., McFarland, D.J., Pfurtscheller, G. Vaughan, T.M.: Brain-computer interfaces for communication and control. J. Clin. Neurophysiol. 113(6), 767--791 (2002)

2. Donchin, E., Spencer, K.M., Wijesinghe, R.: The Mental Prosthesis: Assessing the Speed of a P300-Based Brain-Computer Interface. IEEE Trans. Rehab. Eng. 8, 174--179 (2000)

3. Blankertz, B., Dornhege, G., Krauledat, M., Schroder, M., Williamson, J., Murray-Smith, R., Muller, K.R.: The Berlin brain-computer interface presents the novel mental typewriter hex-o-spell. In: Proceedings of the 3rd International Brain-Computer Interface Workshop and Training Course, pp. 108--109. Verlag der Technischen Universitat Graz (2006)

4. Vanacker, G., Millan, J. del R., Lew, E., Ferrez, P.W., Galan Moles, F., Philips, J., Van Brussel, H., Nuttin, M.: Context-Based Filtering for Assisted Brainactuated Wheelchair driving. Computational Intelligence and Neuroscience, vol. 2007, Hindawi Publishing Corporation (2007)

5. Dhillon, H.S., Singla, R., Rekhi, N.S., Jha, R.: EOG and EMG based virtual keyboard: A brain-computer interface. In: 2nd IEEE International Conference on Computer Science and Information Technology, pp. 259--262. IEEE Press (2001)
6. Blankertz, B., Krauledat, M., Dornhege, G., Williamson, J., Murray-Smith, R., Muller, K.R.: A Note on Brain Actuated Spelling with the Berlin Brain-Computer Interface. Universal Access in Human-Computer Interaction. Ambient Interaction. LNCS, vol. 4555, pp. 759--768. Springer, Heidelberg (2007)

7. Barea, R., Boquete, L., Mazo, M., Lopez, E.: System for Assisted Mobility Using Eye Movements. IEEE Trans. on Neural Systems and Rehab. Eng. 10(4), 209--218 (2002)

8. Jasper, H.: Ten-twenty Electrode System of the International Federation. Electroencephalography. J. Clin. Neurophysiol. 10, 371--375 (1958)

11. Postelnicu, C.-C., Talaba, D., Toma, M.-I.: Controlling a Robotic Arm by Brainwaves and Eye Movement, IFIP Advances in Information and Communication Technology: Technological Innovation for Sustainability. 349, 157--164 (2011)

12. Brown, M., Marmor, M., Vaegan, Zrenner, E., Brigell, M., Bach, M.: ISCEV Standard for Clinical Electro-oculography (EOG). Documenta Ophthalmologica. 113(3), 205-212 (2006)

13. Edlinger, G., Holzner, C., Groenegress, C., Guger, C., Slater, M.: Goal-Oriented Control with Brain-Computer Interface. In: FAC'09 Proc. of the 5th Int. Conf. on Foundations of Augmented Cognition. 5638, 732--740 (2009)

14. Edlinger, G., Holzner, C., Guger, C.: A Hybrid Brain-Computer Interface for Smart Home Control, In: Proc. of the 14th Int. Conf. HCI 2011. 6272, 417--426 (2011)

15. Guger, C., Holzner, C., Groenegress, C., Edlinger, G., Slater, M.: Control of a Smart Home with a Brain-Computer Interface, 4th International Brain-Computer Interface Workshop and Training Course 2008, Graz, Austria, University of Technology Graz (2008)

17. Guger, C., Daban, S., Sellers, E., Holzner, C., Krausz, G., Carabalona, R., Gramatica, F., Edlinger, G.: How many people are able to control a P300-based brain-computer interface (BCI)? Neuroscience Letters. 462 (1), 94--98 (2009)

18. Guan, C., Thulasidas, M., Wu, J.: High performance P300 speller for brain-computer interface, Proc. IEEE Int. Workshop on Biomed. Circ. Syst., S3/5/INV - S3/13-16 (2004)

19. Townsend, G.T., LaPallo, B.K., Boulay, C.B., Krusienski, D.J., Frye, G.E., Hauser, C.K., Schwartz, N.E., Vaughan, T.M., Wolpaw, J.R., Sellers, E.W.: A novel P300-based brain-computer interface stimulus presentation paradigm: moving beyond rows and columns, Clin. Neurophysiol. 121 (7), 1109--1120 (2010)

20. Brunner, P., Joshi, S., Briskin, S., Wolpaw, J.R., Bischof, H., Schalk, G.: Does the 'P300' speller depend on eye gaze? J. Neural Eng. 7 (5), 9pp (2010)