# Eye Location and Eye State Detection in Facial Images Using Circular Hough Transform

Ömer Faruk Söylemez, Burhan Ergen

► To cite this version:

Ömer Faruk Söylemez, Burhan Ergen. Eye Location and Eye State Detection in Facial Images Using Circular Hough Transform. 12th International Conference on Information Systems and Industrial Management (CISIM), Sep 2013, Krakow, Poland. pp.141-147, 10.1007/978-3-642-40925-7_14. hal-01496060

HAL Id: hal-01496060

https://inria.hal.science/hal-01496060

Submitted on 27 Mar 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
EYE LOCATION AND EYE STATE
DETECTION IN FACIAL IMAGES USING
CIRCULAR HOUGH TRANSFORM

Omer Faruk Soylemez¹ and Burhan Ergen²

¹ Dicle University, Diyarbakir, Turkey,
osoylemez@dicle.edu.tr,

² Firat University, Elazig, Turkey,
bergen@firat.edu.tr

**Abstract.** Recently, eye states are used as inputs to various applications such as facial expression recognition systems, human-computer interaction and driver fatigue detection systems. Especially with the prominence of human computer interaction, eye state detection has drawn great attention in the past decade. In this study, an eye state detection system based on Circular Hough Transform (CHT) has been offered. Initially, face and eye images are extracted from given gray-level images. After some preprocessing steps, existence of circular iris structure is searched within the extracted eye image using CHT. Existence of circular iris structure is searched within the eye image with the help of circular Hough transform. Eyes are decided as open if iris could identified as a circle.

**Keywords:** Eye state detection, Circular Hough transform

# 1 Introduction

Detection of eye location and states has received a great deal of attention because eyes are the most important part of human face. Today, many applications are using eye states as inputs. Facial expression recognition, human-computer interaction and driver fatigue detection systems could be given as examples to such applications. In facial expression recognition systems, subjects mood could be estimated by utilizing eye openness along with other facial features [1]. In driver fatigue detection systems, eye blinking count, time taken by a blink and frequency of eye blinks are combined together to form an alert and warn the driver in case of a driver fatigue level increase [2]. In human computer interaction, by tracing eye movement and focus, computers could acknowledge and process user requests [3]. By employing these kinds of applications in academic and commercial fields, eye state detection has drawn great attention in the past decade. In the field of eye state recognition, studies are employing parameters such as iris, eye perimeter, eye edge and eye lids. Wang and Yang [4] used chain code tracing for extracting edges of iris and then used Circular Hough Transform (CHT) in order to detect iris circle. Tian, Kanade and Cohn [5] took
advantage of Gabor wavelets and neural networks in order to detect eye states. Seneratne et al. [6] compared eye blocks intensity values using support vector machines (SVM) and Naive Bayes (NB) classification methods to overcome eye state detection problem. Xu, Zeng and Wang [7] tried to detect eye states by using Adaboost based cascade classifier and a histogram property which is named as Local Binary Pattern (LBP). In this study, we carry out a practice to detect eye states in facial images. Initially, faces and eyes are detected and extracted from intensity images. After some preprocessing steps, edges of the iris are used to detect a circular structure referring to state of the eyes. The detection of a circular structure is achieved by means of CHT.

## 2 Proposed Method

Our proposed method for eye state detection is composed of three steps. Viola-Jones [9] cascade object detector which gives higher recognition rates on real time object detection has been used for face and eye detection states. Eye regions are estimated sequel to extraction of faces and eyes from given images. CHT is applied to the images consisting of eye region at the end, in order to detect eye states. BioID face database [8] which consists of 1521 gray scale images that are taken from 23 different subjects under varying circumstances is used in our study.

### 2.1 Face and Eye Detection

Cascade classifier model which is proposed by Ojala T. [10] was executed on 1010 images that provide adequate lighting levels. Face locations were correctly identified on 916 images. Eyes are searched within those locations by the help of Viola-Jones object detection framework just like as face detection. Cascade classifier model which proposed by Marco [11] is used for detecting eye pairs. To detect each eye separately; cascade classifier model which proposed by Shiqi [12] is used. Since our method is based on the intensity of dark region formed by iris and pupil, employing an eye image with the absence of eyebrows became more suitable for detection purposes. Therefore, we employ eye pair images for eye detection instead of separately detected eye images for their minimal eyebrow containment. Cascade classifier was performed on total of 916 images. Eye pairs were detected successfully in 899 images.

### 2.2 Eye State Detection

In order for an eye to carry out seeing task, a black point which is called pupil has to receive light. Lights received from pupil are focused on to rearmost layer of the eye which is called retina layer. Image that is formed on the retina layer is transmitted to visualization center of the brain via optical nerves and thus visual process completes. Iris and pupil are two circles that share the same center. In accordance with this relation half opening of iris means pupil is also
half open and so seeing is probable. We have used this information in our study to assume an eye as open. Our system starts with equalizing contrast of the acquired eye image with the histogram equalization. As a result of this process, iris and pupil, which have a lower intensity value compared to sclera (white of the eye) are going to be easily separated from the sclera and the rest of the eye. Afterwards, gray-scale thresholding is applied to the resulting eye image. With the aid of thresholding, iris and pupil had become greatly separated from the rest of the eye. Even so, some unwanted features such as shades, eyelashes and stains could may have reside on the image. Therefore we removed those little residues after obtaining thresholded image. Yet another great factor that hampers our method is the lights which are reflected from the pupil. These lights are forming a high intensity area (brights) inside the low intensity area which is formed by pupil and iris. As this white area has the ability to misdirect our edge detection output, it has been dealt with morphological techniques. After all of these processes, iris and pupil are completely extracted from the rest of the eye and are ready to be served into next step. Edges of the concerned image are detected with use of canny edge detector. Afterwards, presence of circular shapes are searched with the aid of circular Hough Transform. 492 eye images were used for eye state detection. Those images are selected within 899 eye pairs that are obtained from the previous step and they meet the minimum resolution case for applicability of CHT. In 473 of the images eyes were open and in the other 19 of the images eyes were closed. Open and closed eyes are classified with %94.5 and %63.2 accuracy respectively.

# 3 Experimental Results

In this section, we will analyze our proposed method by means of error matrix, correct classification rate and kappa statistics. Afterwards, we will present some test results that are obtained with our proposed method.

## 3.1 Confusion Matrix

Confusion matrix is a tool to measure the performance of a classification system. Each row of the matrix shows number of occurrences of an estimated class, while each column of the matrix shows the occurrence of a real class. A confusion matrix is shown in Table 1, which is used to classify a two class system. As stated above; TP (resp. TN) represents the number of instances of class a (resp. class b) well classified by the system. P=TP+FP (resp. N=FN+TN) represents the total of estimated occurrences of class a (resp. class b). p=TP+FN (resp. n=FP+TN) represents the total of real occurrences of class a (resp. class b). T is the sum of occurrences of both classes.

## 3.2 Correct Classification Rate

Correct classification rate (CCR) which is also known as success rate, is obtained by dividing the number of correctly classified samples to the number of total
<table><caption>**Table 1.** Confusion matrix of a classification system</caption><thead><tr><th>Real Class/Estimated Class A</th><th>B</th><th>Total</th></tr></thead><tbody><tr><td>a</td><td>True Positive (TP)</td><td>False Negative (FN) p</td></tr><tr><td>b</td><td>False Positive (FP)</td><td>True Negative (TN) n</td></tr><tr><td>Total</td><td>Positive (P)</td><td>Negative (N)</td><td>Total (t)</td></tr></tbody></table>

samples as shown in Equation 1.

$$CCR = \frac{TP + TN}{T} \qquad (1)$$

## 3.3 Cohen's Kappa Coefficient

Cohen's Kappa coefficient ($\kappa$) [13] measures the agreement between two raters who each classify N items into C mutually exclusive categories. The equation for Cohen's Kappa coefficient is shown in Equation 2.

$$\kappa = \frac{P_0 - P_e}{1 - P_e} \qquad (2)$$

Where $P_0$ is observed agreement proportion that refers to CCR and $P_e$ represents random agreement proportion given by Equation 3.

$$P_e = \frac{1}{T^2} [(P * p) + (N * n)] \qquad (3)$$

$\kappa$ coefficient varies between -1 and 1. When it is equal to 1, it is considered that agreement between the observers on the studied system is perfect. Inversely when it is equal to -1, it is considered that there is a total disagreement on the studied system among the observers. Table 2 shows kappa statistics values along with their interpretations.

## 3.4 Experimental Results

All applications are executed on Matlab software environment by using the system having the following specification: Intel Core2 duo T5600 Dual Core Processor clocked at 1.83 GHz, 533 MHz 4 GB RAM. Our software took 0.14 seconds to localize face, 0.06 seconds to localize eye pair and cropping eyes, 0.014 seconds for preprocessing steps and another 0.014 seconds for circular Hough transform. Our software took total of 230 ms in order to process a frame which means 4 fps. Table 3 shows statistical values of the proposed system. Kappa statistic is found as 0.40 which shows that our system works on a fair agreement level. Eye state detection success rates are given in Table 4. %94.5 of the open eyes and %63.1 of the closed eyes are classified correctly. CCR, which is obtained by dividing correctly classified samples to all samples is calculated as %93.
<table><caption>**Table 2.** Kappa statistics values along with their interpretations</caption><thead><tr><th>Kappa Statistics</th><th>Interpretation</th></tr></thead><tbody><tr><td>κ > 0.8</td><td>Almost Perfect Agreement</td></tr><tr><td>0.8 ≥ κ > 0.6</td><td>Substantial Agreement</td></tr><tr><td>0.6 ≥ κ > 0.4</td><td>Moderate Agreement</td></tr><tr><td>0.4 ≥ κ > 0.2</td><td>Fair Agreement</td></tr><tr><td>0.2 ≥ κ > 0.0</td><td>Slight Agreement</td></tr><tr><td>0 ≥ κ</td><td>No Agreement</td></tr></tbody></table>

<table><caption>**Table 3.** Statistical values of the system</caption><thead><tr><th>TP</th><th>TN</th><th>FP</th><th>FN</th><th>T</th><th>CCR</th><th>Kappa</th></tr></thead><tbody><tr><td>447</td><td>26</td><td>12</td><td>7</td><td>492</td><td>0.93</td><td>0.40</td></tr></tbody></table>

<table><caption>**Table 4.** Eye state detection success rates</caption><thead><tr><th>Real/Detected</th><th>Open</th><th>Closed</th><th>Total</th></tr></thead><tbody><tr><th>Open</th><td>447(%94,5)</td><td>26(%5,5)</td><td>473(%96,1)</td></tr><tr><th>Closed</th><td>7(%36,9)</td><td>12(%63,1)</td><td>19(%3,9)</td></tr><tr><th>Total</th><td>454(%92,2)</td><td>38(%7,8)</td><td>492(%100)</td></tr></tbody></table>
Samples to the eye detection classes which are TP, TN, FP and FN are shown at Figure 1. Red circles are indicating a valid retina and pupil are found and marks where they reside.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Examples to detection classes.</figcaption></figure>

4 Conclusion

With the development of computer vision techniques, face detection and eye
state detection has become important for many applications. In this study, an
eye state detection system has been proposed. In order to detect faces from given
images, Viola-Jones face detector has been used. Faces had been detected suc-
cessfully in 916 of 1010 images. Sequel to face detection, eyes pairs are detected
by utilizing Viola-Jones eye pair detector. 899 eye pairs were detected in 916
face images. Right and left eyes are acquired by cropping eye pairs. For eye state
detection, contrast of the eye image -which was obtained in the previous stage-
is increased. Then gray level thresholding is applied to this image. Later, the
residual areas which are not used in the detection process are eliminated. Holes
inside this new image are filled. After this step, edges are extracted via canny
edge detection algorithm. Finally, circular shapes are searched within images by
using circular Hough transform. Latest image set was composed of 492 images
which contains images that provide proper resolution and illumination for eye
state detection process to function. 94.5 percent of the open eye images and 63.1 of closed eye images are identified correctly. Gray level images are used in this study. Studies that employ colored images could produce better results. Eye state detection could also be possible in low light environment by using capture devices that utilize infrared rays. Eye state detection problem could be solved more efficiently by applying these types of approaches. Eye state information, is used as input to many applications nowadays. With the development of faster and more accurate eye state detection methods there is no doubt that eye state information could utilize itself in broader utilization.

## References

1. Sandbach, G., Zafeiriou, S., Pantic, M., Yin, L.: Static and dynamic 3D facial expression recognition: A comprehensive survey. Image and Vision Computing (2012)

2. Wang, Q., Yang, J., Ren, M., Zheng, Y.: Driver fatigue detection: a survey. In Intelligent Control and Automation, 2006. WCICA 2006. The Sixth World Congress on (Vol. 2, pp. 8587-8591). IEEE (2006)

3. Jacob, R. J., Karn, K. S.: Eye tracking in human-computer interaction and usability research: Ready to deliver the promises. Mind, 2(3), 4 (2003)

4. Wang, Q., Yang, J. Y.: Eye location and eye state detection in facial images with unconstrained background, Journal of Information and Computing Science, Vol. 1, No. 5, pp. 284-289 (2006)

5. Tian, Y. L., Kanade, T., Cohn, J. F.: Eye-state action unit detection by gabor wavelets, International Conference on Multimodal Interfaces 2000, LNCS 1948, Vol. 1948, pp. 143-150 (2000)

6. Seneratne, R., Hardy, D., Vanderaa, B., Halgamuge, S., Liu, D., et al.: Driver Fatigue Detection by Fusing Multiple Cues, Lecture Notes In Computer Science Vol. 4492, pp. 801-809 (2007)

7. Xu, C., Zheng, Y., Wang, Z.F.: Efficient eye states detection in realtime for drowsy driving monitoring system, IEEE Conf. on ICIA, vol. 1-4, pp. 170-174 (2008)

8. http://www.bioid.com/downloads/software/bioid-face-database.html [Online], 27.02.2012.

9. Viola, P., Michael J.J.: Rapid Object Detection using a Boosted Cascade of Simple Features, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Volume: 1, pp.511-518 (2001)

10. Timo, O., Matti, P., Topi, M.: Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns. In IEEE Transactions on Pattern Analysis and Machine Intelligence. Volume 24, Issue 7, pp. 971-987 (2002)

11. Marco, C., Oscar, D., Cayetano, G., Mario, H., ENCARA2: Real-time detection of multiple faces at different resolutions in video streams. In Journal of Visual Communication and Image Representation, 18.2: pp. 130-140 (2007)

12. http://yushiqi.cn/research/eyedetection [Online], 27.02.2012.

13. Fleissa, J. L., Cohen, J., Everitt, B.S.: Large sample standard errors of kappa and weighted kappa, Psychological Bulletin, vol.72, Issue 5, Pages 323-327 (November 1969)