# Using Eye Blinking for EOG-Based Robot Control

Mihai Duguleana, Gheorghe Leonte Mogan

► To cite this version:

Mihai Duguleana, Gheorghe Leonte Mogan. Using Eye Blinking for EOG-Based Robot Control. First IFIP WG 5.5/SOCOLNET Doctoral Conference on Computing, Electrical and Industrial Systems (DoCEIS), Feb 2010, Costa de Caparica, Portugal. pp.341-348, 10.1007/978-3-642-11628-5_37. hal-01060757

HAL Id: hal-01060757

https://inria.hal.science/hal-01060757

Submitted on 17 Nov 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Using Eye Blinking for EOG-Based Robot Control

Mihai Duguleana and Gheorghe Mogan

Transylvania University of Brasov, Product Design and Robotics Department,
Bulevardul Eroilor, nr. 29, Brasov, Romania
{Mihai.Duguleana, Mogan}@unitbv.ro

**Abstract.** This paper proposes a new approach to real-time robot controlling by integrating an Electrooculography (EOG) measuring device within human-robot interaction (HRI). Our study focuses on controlling robots using EOG for fulfilling elementary robot activities such as basic motor movements and environment interaction. A new EOG-based HRI paradigm has been developed on the specific defined problem of eye blinking. The resulted model is tested using biometric capturing equipment. We propose a simple algorithm for real-time identification and processing of signals produced by eyes during blinking phases. We present the experimental setup and the results of the experiment. We conclude by listing further research issues.

**Keywords:** electrooculography, robot, control, human-robot interaction, eye blink.

## 1 Introduction

Robot control is one of the key fields in Robotics. Efficient alternative controlling capabilities are important for motor-disabled persons, elderly people and patients suffering from several types of Sclerosis or other diseases that affect hand and facial muscles. Using information from multiple sensor devices will also increase the quality of HRI when applying sensor-fusion.

Human eye movements have been recently used with success in human-computer interaction by several researchers [1], [3], [6] and [7]. Most of the applications based on this kind of feedback are conducted in fields like Virtual Reality, Robotics and Biomedicine.

Eye movements can be recorded in 3 ways: with magnetic coils, using video processing or using EOG [6]. Video processing and EOG are most used techniques, however, although video processing requirements are very easy to fulfill (just a quality camera), EOG tends to produce more quality outputs in terms of speed and error. While in the video-based approach, the camera scans for the eyes of the subject and measures the movement using powerful yet time consuming image processing libraries, in the EOG approach, a simple low cost device can use one or two pairs of electrodes to measure the resting potential of the retina. This potential is the result of eyeball movement within the conducting environment of the skull and it varies proportional to the amplitude of eye movements [1].
Several other researchers have been conducting related studies to the subject of our paper. In order to use EOG, some researchers have been able to reduce noise in EOG recordings by filtering the heart beating pollution from the output signal and by eliminating the drift that appears randomly when the subject is moving his eyes [6], to minimize the signal differences between different testing subjects or to eliminate certain signals [4] and to control wheelchairs and other mobile robots [1].

Different types of signals have been identified for each of the following actions: blinking, horizontal and vertical saccadic motions or winking [7].

There are also available comparisons between different EOG devices and their efficiency or between measuring methods such as EOG, EEG and EMG [3]. The research question which we approach is whether robust robot control in real time using EOG can be achieved.

In this paper, we focus on developing a simple motor controlling application for commanding the grasping modules of a humanoid robot using EOG. We propose a new algorithm for blinking detection in real-time EOG applications. Furthermore, we describe in the “Experimental Setup” section the equipment we have used for conducting our research and conclude in the last section with implementation issues and further research problems.

## 2 Contribution to Technological Innovation

Although there have been several attempts to clearly delimitate a good interaction model between humans and robots using EOG information, most of the literature focuses on offline control and saccadic eye movements processing [2]. Most of the experiments are based on EOG devices that require skin preparation (skin scrubbing) and the appliance of conducting gel between the electrode surface and the skin of the subject. We present a system that uses dry electrodes and wireless technology for communicating with the processing unit, thus speeding up the testing process and increasing the flexibility of this solution.

We foresee the implementation of a fully functional EOG-based device that could be used in industrial environment for integrating people with disabilities into the work field or for achieving more precise robot control as one of the practical technological applications that can be developed based on the results presented this paper.

### 2.1 EOG-based Robot Commanding Paradigm

Having eye blinking the only form of input increases the robustness of the system. We propose a new commanding paradigm for real-time control with eye blinking in an EOG-based HRI (see Fig. 1).

The EOG device mounted on the subject's head sends amplified data to the computer unit. Considering the continuous attribute of the experiment, we split the application into 2 threads. In the signal processing module, the data is processed and the result is sent to the state handler. Several filters (like noise, drift and other biometric algorithms) are contributing to robust eye blinking identification.
Depending on the state of the robot and on the previous data received by the state handler, this module accesses predefined commands of the robot. The handler consists of 2 states: the waiting state and the active state. One may choose to let the robot finishing its previous tasks before engaging into other new activities, or to override these with the new received command. As data is continuously received from the signal processing module, some commands that are quickly assessed may have a certain initial value that may change its meaning when receiving new information.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Real-time eye blinking EOG-based HRI paradigm.</figcaption></figure>

In the first part of this section we describe the experimental equipment used for completing our study. We then present the algorithm used for detecting eye blinking. We end this section with a critical analysis of experiment results.

## 2.2 Experimental Setup

We are using for this experiment an EOG solution from Starlab Company. The wireless electro-physiology sensor system (presented in the left side of Fig. 2) is mounted on the subject's head; 2 of the electrodes are placed on the sides, the other 2 above each eye and the Driven Right Leg (DRL) circuit (the reference) on the right ear of the subject (center Fig. 2). DRL circuit is particularly important because it
influences the quality of measurements. We are using for DRL conductive gel to
increase the common-mode rejection ratio (CMRR) and to improve the noise
immunity [5].

When blinking, humans use their facial muscles. Because of this, the EOG blinking
signal is very strong, thus allowing us to mount the electrodes in approximate
positions without having to worry about data loss.

The EOG wireless device sends data to a USB modem connected to our computer.
The Enobio application auto-filters the signal and uses Channels 2 and 3 for providing
EOG information.

The computer that handles the entire experiment is also connected in the same
network with a humanoid robot called NAO and produced by Aldebaran Company
(right side of Fig. 2). We will command the hand grippers of the robot, differentiating
between left and right motors by blinking once or blinking two times.

<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Experiment setup. Left side – EOG device with 4 electrodes and wireless transmitter (*Enobio equipment*), center – Enobio electrodes setup (*head scheme*), right side – humanoid robot (*Nao*).</figcaption></figure>

**2.3 Processing Blink Signal**

Considering a normal EOG wave signal, the event of a blink triggers a peak
occurrence in the amplitude values recorded by Enobio software, far higher than any
other possible noise that the EOG measurement might have if the blink wouldn't
exist. As seen in Fig. 3, an average blink lasts between 100ms and 200ms (these
values vary depending on the subject's bio-attributes, fatigue level, eye health and
other factors) and reaches up to a 400µV (a 250-300µV in average) maximum before
returning to the mean voltage level that was recorded before blinking.

Using the above observations, we may detect blinks by adding a preset minimum
threshold which should filter most of the noise and false peaks. Unfortunately, there
are some cases in which the signal wave experiences a drift - sometimes the voltage
increases with similar values like the ones recorded from blinks. We can further filter
these cases by setting a maximum threshold and calculating difference between the
minimum values recorded before and after peak occurrence. When implementing
these restrictions in real time, we apply the following algorithm written in pseudo-
code:
```c
while receiving_data
{
    get_data;
    decode_data;
    if check_time_from_last_blink = ok
    {
        if is_minimum_since_last_t_ms
        {
            get
            get
            if maximum > min_thres and maximum <
            max_thres
            {
                get_min_before_time_of_maxim;
                if
                    compare_min_before_time_of_maxim
                    with_current_data = ok
                {
                    BLINK;
                    retain_time_of_blink;
                }
            }
        }
    }
}
```

The algorithm searches for amplitude points recorded at the end of a peak, after a certain period from the last acknowledged blink, to ensure that we do not count twice the same recording. If local minimum is found, we sequentially check for the maximum value recorded within a proper time interval, compare it with minimum (240µV) and maximum thresholds (450µV), search for the minimum value recorded before the occurrence of maximum and compare it with current data to eliminate drift errors. In the event of a valid result (i.e. a maximum 100 µV difference between the 2 minimums), we compute the command and retain the occurrence time.

<figure><img src="image_4.png" /><figcaption>Fig. 3. Blink signals recorded in EDF format.</figcaption></figure>
In order to differentiate between one and multiple blinks, we will use the time
retained from the last acknowledged blink. Two or more blinks occur continuously,
one wave after another. We expect i.e. to record two blinks within a maximum 500ms
timeframe. We will set this interval to be the waiting period after which we decide if
the subject has blinked once or twice.

## 2.4 Discussion of Results

We compare the data received from Enobio device and processed by our application with the data saved by Enobio in EDF format, available the end of each session. The differences are presented in Fig. 4.

The main problem faced by real-time computing is the processing time. Intel processors (as any other processor) execute instructions once every few milliseconds, depending on their clock frequency and their interruption rate. We can try to increase the interrupt rate granularity using Microsoft Windows Multimedia SDK (for Win 32 systems), thus improving the preciseness of our system, but this comes with a severe power cost. The algorithm used for data decoding and processing also requires a few milliseconds, as well as receiving data from the wireless modem. Summing up all these times, our system (Intel Core 2 Duo T9300, 2.50 GHz, 2GB DDR2 SDRAM, Window Vista) works with data received every 15 to 20ms. This limitation has serious impact on the recorded wave. There are cases when blinks are missed, or data is seriously affected.

<figure><img src="image_5.png" /><figcaption>**Fig. 4.** Comparing recorded results (*first graph*) with EDF stored results (*bottom graph*). In this experiment session, the first three blinks are properly recorded and processed. The following two blinks are missed due processor delay. The next 3 blinks are also properly assessed. The continuous blinking at the end of the session is missed by the application.</figcaption></figure>

After 20 measuring sessions on a single subject (313 recorded blinks), we can
calculate the average blink recognition rate to be 56.23%. The double blink
recognition rate (28 recorded double blinks) is 42.85%. The results are displayed in
Fig. 5.
## 3 Conclusions and Further Work

Integrating the proposed EOG-based HRI paradigm for real-time robot controlling in real life can be both useful and time-saving. The main result is that people with severe movement-disabilities will be able to have an efficient interaction with environment. Although the implementation of the described paradigm requires bypassing issues like the need of complete redesign of the living environment, control with eye blinking can be achieved. After extending the symbolic functions presented in our experiment, persons with paralyzed facial muscles will be able to use robots to fulfill a wide variety of tasks.

<figure><img src="image_6.png" /><figcaption>**Fig. 5.** Analyzing the recognition rates for single and double blinks, over a dataset of 20 test sessions. On the OX axis is displayed the session number and on the OY axis, the number of occurrences.</figcaption></figure>

The experiment results depend with the system on which we run the application. Further work will be conducted into performing the experiment on different hardware and software platforms.

The computation times need to be lowered as much as possible. Further work will be conducted into improving the communication between the EOG equipment and the processing unit, optimizing the source code of the application and reconsidering the thresholds values to more precise ones, obtained after statistically analyzing data from multiple sessions.

**Acknowledgments.** This paper is supported by the Sectoral Operational Programme Human Resources Development (SOP HRD), financed from the European Social Fund and by the Romanian Government under the contract number POSDRU/6/1.5/S/6 and by the national research grant ID_775/2009. The experiments were conducted at Italian Institute of Technology (IIT), Italy.
References

1. Barea, R., Boquete, L., Lopez, E., Mazo, M.: Guidance of a Wheelchair Using Electrooculography. In: Proceeding of the 3rd International Multiconference on Circuits, Systems, Communications and Computers (1999)

2. Choudhury, S.R., Venkataramanan S., Nemade H.B, Sahambi J.S.: Design and Development of a Novel EOG Biopotential Amplifier. International Journal of Bioelectromagnetism, vol. 7(1), pp. 271--274, Hosei University, Tokyo, Japan (2005)

3. Grave de Peralta Menendez, R., Noirhomme, Q., Cincotti, F., Mattia, D., Aloise, F., Andino, S.G.: Modern Electrophysiological Methods for Brain-Computer Interfaces. Computational Intelligence and Neuroscience Journal, vol. 2007, Hindawi Publishing Corporation (2007)

4. Manoilov, P., Borodzhieva, A.: An Algorithm for Eye-Blinking Artefacts Duration Determination, In: Proceedings of the Fourth International Bulgarian-Greek Conference COMPUTER SCIENCE'2008, Part 1, pp. 846--850, Kavala, Greece (2008)

5. Starlab Enobio, http://starlab.es/products/enobio

6. Usakli, A.B., Gurcan, S., Aloise F., Vecchiato Babiloni, F.: On The Use of Electrooculogram For Efficient Human Computer Interfaces. In: Computational Intelligence and Neuroscience Journal, article in press, 2009. Available at: http://www.hindawi.com/journals/cin/aip.html

7. Youngmin, K., Nakju, L.D., Youngil, Y., Wan, K.C.: Robust Discrimination Method of the Electrooculogram Signals for Human-Computer Interaction Controlling Mobile Robot. Intelligent Automation and Soft Computing Journal, vol. 13(3), TSI Press, USA (2007)