# Differentiating Conscious and Unconscious Eyeblinks for Development of Eyeblink Computer Input System

Shogo Matsuno, Minoru Ohyama, Kiyohiko Abe, Shoichi Ohi, Naoaki Itakura

<figure><img src="image_1.png" /></figure>

## To cite this version:

Shogo Matsuno, Minoru Ohyama, Kiyohiko Abe, Shoichi Ohi, Naoaki Itakura. Differentiating Conscious and Unconscious Eyeblinks for Development of Eyeblink Computer Input System. 1st and 2nd International Workshop on Usability- and Accessibility-Focused Requirements Engineering (UsARE 2012 / UsARE 2014), Jun 2012, Zurich, Switzerland. pp.160-174, 10.1007/978-3-319-45916-5_10 . hal-01631309

HAL Id: hal-01631309

https://inria.hal.science/hal-01631309

Submitted on 9 Nov 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# Differentiating Conscious and Unconscious Eyeblinks for Development of Eyeblink Computer Input System

Shogo Matsuno<fnref n="1" />, Minoru Ohyama<fnref n="2" />, Kiyohiko Abe<fnref n="3" />, Shoichi Ohi<fnref n="2" />, Naoaki Itakura<fnref n="1" />

**Abstract.** In this paper, we propose and evaluate a new conscious eyeblink differentiation method, comprising an algorithm that takes into account differences in individuals, for use in a prospective eyeblink user interface. The proposed method uses a frame-splitting technique that improves the time resolution by splitting a single interlaced image into two fields—even and odd. Measuring eyeblinks with sufficient accuracy using a conventional NTSC video camera (30 fps) is difficult. However, the proposed method uses eyeblink amplitude as well as eyeblink duration as distinction thresholds. Further, the algorithm automatically differentiates eyeblinks by considering individual differences and selecting a large parameter of significance in each user. The results of evaluation experiments conducted using 30 subjects indicate that the proposed method automatically differentiates conscious eyeblinks with an accuracy rate of 83.6% on average. These results indicate that automatic differentiation of conscious eyeblinks using a conventional video camera incorporated with our proposed method is feasible.

**Keywords:** eyeblink, eye gaze input, voluntary eyeblink, eyeblink waveform, input interface.

## 1 Introduction

In general, eyeblinks can be classified as voluntary, reflex, or spontaneous. A voluntary eyeblink occurs consciously, a reflex eyeblink occurs as a result of external factors such as sound and/or light stimuli, and a spontaneous eyeblink is one that occurs unconsciously [1]. If a system was able to distinguish when a user has blinked with a conscious desire to enter information, then we would be able to control a computer

<footnote n="1">¹ Graduate School of Informatics and Engineering, The University of Electro-Communications, 1-5-1 Chofugaoka, Chofu, Tokyo 182-8585, Japan
{m1440004}@edu.cc.uec.ac.jp</footnote>

<footnote n="2">² Department of Information Environment, Tokyo Denki University, 2-1200 Muzai Gakuendai, Inzai, Chiba 270-1382, Japan
{ohyama,ohi}@mail.dendai.ac.jp</footnote>

<footnote n="3">³ College of Science and Engineering, Kanto Gakuin University, 1-50-1 Mutsuurahigashi Kanazawa-ku, Yokohama, Kanagawa 236-8501, Japan
{abe}@kanto-gakuin.ac.jp</footnote>
device. In other words, computer control using eyeblinks could be realized if a method that automatically distinguishes conscious eyeblinks from unconscious eyeblinks was available.

The results of psychology experiments have shown that the occurrence of eyeblinks is associated with cognitive status. Using this knowledge, a system that measures the state of exhaustion of drivers has been developed [2]. Further, studies have been conducted in an effort to determine whether it can be used as a communication support and assistance system for severely crippled persons such as amyotrophic lateral sclerosis (ALS) patients [3, 4, 5]. Systems using eyeblink as an input switch and otherwise combining it with eye gaze in an input interface to operate equipment have also been proposed [6, 7, 8, 9, 10, 11]. However, in most systems, because eyeblinks occur at high speeds, accurate and dedicated equipment is required to measure them. In addition, these systems usually employ a fixed threshold or special operations.

Our aim is to develop an eyeblink input system that can be installed on conventional information devices, such as smartphones and smart glasses [12, 13, 14]. Using image analysis [10, 11], we previously obtained and examined shape feature parameters in an eyeblink waveform (i.e., the waveform representing the time evolution of the eyeblink process) and observed differences between conscious and unconscious eyeblinks among subjects [15]. In this paper, we propose a new automatic conscious eyeblinks differentiation method, and report on the results of evaluation experiments conducted using the proposed method and algorithm with 30 subjects.

## 2 Related Work

Conventional eyeblink input systems are classified into two basic types. The first type uses input based on pre-established time values (for example, when a user closes his/her eyes for more than 200 ms) [16, 17]. In this case, a dynamic threshold value is used for each type of eyeblink because eyeblinks show wide individual differences. A false input may occur if the threshold is fixed because the input time is user-dependent; a user might unconsciously produce considerably short or long eye movements. The second type of input system examines special eye movements, such as double eyeblinks and winks [18, 19]. However, these systems require the user to perform conscious, and occasionally complex, actions; therefore, users have to practice in order to be proficient at using these systems. In addition, the unusual eyeblinks required can cause user stress, especially when the systems are used over a long period [1], [9].

In an effort to overcome these problems, eyeblink input interfaces that incorporate more natural eyeblinks are being studied. However, a user who does not display a noticeable difference in shape feature parameters between voluntary and spontaneous eyeblinks must be conditioned and encouraged by such a system for it to accurately measure voluntary eyeblinks [7, 8, 9, 10]. Conversely, the system proposed in this paper uses a messaging system—for example, it announces to a user, “you blinked correctly at the perceived signal”—to decrease user stress and to amplify the difference in the shape feature parameters. This system most closely approximates an actual eyeblink interface
because it is expected that the user is conscious of the input when blinking, even if no user training has been conducted.

<table><caption>**Table 1.** Strengths and weaknesses of previous works.</caption><thead><tr><th>Fixed-length threshold [16, 17]</th><th>Concepts</th><th>Special eye movements [18, 19]</th></tr></thead><tbody><tr><td>Easy</td><td>Inputting</td><td>Have to practice</td></tr><tr><td>Necessary</td><td>Calibration</td><td>May be necessary</td></tr><tr><td>A bit too much</td><td>Get exhausted</td><td>Much</td></tr><tr><td>High</td><td>Requisite measuring accuracy</td><td>Low</td></tr><tr><td>Difficult</td><td>Estimate of intention</td><td>Easy</td></tr></tbody></table>

## 3 Characteristics of Eyeblinks Waveforms

We distinguish between conscious and unconscious eyeblinks by considering the fact that the duration of a conscious eyeblink is longer than that of an unconscious eyeblink [11], [16]. However, an eyeblink is a rapid motion that completes a series of operations on the order of a few hundred milliseconds; over and above that, individual differences are substantial. Consequently, because the time resolution of conventional video cameras is low, when measured with these cameras, significant differences in the eyeblink duration are not observed. Eyeblinks vary widely by individual, but in most cases, during a conscious eyeblink, the eyelids close completely. In addition, variation in terms of the eyeblink waveform is relatively small in each individual. Therefore, we focused on the following parameters: closing-phase amplitude, opening-phase amplitude, and eyeblink duration, as discussed in a previous study [9]. Figure 1 shows a model of an eyeblink waveform in which the closing-phase amplitude *Acl* is defined as the height of the closing-phase starting point *Ps* to the minimum point *Pmin*. *Pmin* is defined as the point where the eye-opening area is smallest; that is, from the closing-phase end point *Psb* to the opening-phase starting point *Peb*. Similarly, the opening-phase amplitude *Aop* is defined as the height of the minimum point *Pmin* to the opening-phase end point *Pe*. Finally, the eyeblink duration *Dur* is defined as the field count from *Ps* to *Pe*.

<figure><img src="image_3.png" /><figcaption>**Fig. 1.** Model of eyeblink waveform.</figcaption></figure>
# 4 Automatic Measurement of an Eyeblink Waveform

If the time evolution of the eyeblink process could be accurately measured, it would be possible to express an eyeblink as a waveform. Individual eyeblinks must be measured and then analyzed for automatic differentiation of eyeblink types. The typical techniques used to sample eyeblink waveforms are electrooculogram (EOG) and image analysis. The EOG method involves placing an electrode on the skin near the eyeball. Eyeblink waveforms are then collected by recording changes in the cornea-retina potential. This technique was proposed for automatic detection of conscious eyeblinks until recently [20]. However, the EOG method requires a unique apparatus to process ocular potential, and the user must have an electrode attached to his/her skin. Therefore, the EOG method is unsuitable for a simple interface. Moreover, extraneous noise from a living body can cause interference. By contrast, image analysis examines pictures of eyeblinks captured by a video recorder. It has become popular because it requires no bodily contact and is manageable and adaptable. However, eye movements are difficult to capture with a video camera that has a standard aspect ratio (NTSC) because an eyeblink is a high-speed operation. Therefore, in this paper, we incorporate an algorithm used in previous research [10] that detects changes in eye aperture. The algorithm samples at 1/60 s using interlaced NTSC video images further divided into field images. Figure 2 shows the processing flow for detecting changes in the eye-aperture area.

<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Overview of frame-splitting method and binarization.</figcaption></figure>

When image analysis is used, the first step is to analyze video images of the area surrounding the eye in order to assess changes in eye aperture using binarization based on flesh color. Figure 3 shows an example of changes that occur in the eye-aperture area. The data shown in Fig. 3 include changes in the eyeblink waveform. The next step applies smoothing differentiation between the split field area and the next split field. Coordinates that reveal the maximum area difference value and the minimum area difference value are then determined using a second differentiation.
<figure><img src="image_5.png" /><figcaption>Fig. 3. Changes in eye-opening area.</figcaption></figure>

However, this step in the analysis involves excessive noise resulting from small movements in the vicinity of the eye, such as from an eyelid. Therefore, we remove three coordinate classes of extreme value (maximum, minimum, and few-moving) using the k-means method. We determine the start and end of an eyeblink waveform using its maximum and minimum values because one eyeblink waveform contains only one maximum and one minimum value. Minimum values exist in the opening phase and maximum values exist in the closing phase. Data are obtained from one eyeblink waveform according to these factors. If the obtained maximum and minimum values are observed in succession as two points, the point closer to the field of temporal axes is used. An eyeblink start field is calculated by differentiating between field areas in the direction opposite to that of the temporal axes from the maximum value's field. In this field, the threshold $Th_1$ becomes positive for the first time. By contrast, the eyeblink end field is calculated by the difference between the field areas in the forward direction of the temporal axes from the minimum value's field. In this field, the threshold $Th_1$ becomes negative for the first time. The threshold $Th_1$ is then determined by the following equation:

$$Th_1 = f(n) - f(n + 1)$$

where n is the attention field and f(n) is the eye-opening area in the n field. Figure 4 shows an example of the detected eyeblink waveform.
<figure><img src="image_6.png" /><figcaption>Fig. 4. Example of an eyeblink waveform.</figcaption></figure>

An eyeblink waveform measured by means of image analysis can be applied to the model in Fig. 4. The eyeblink duration is represented as field numbers from the eyeblink starting point to the eyeblink end point. The eyeblink amplitude is represented as changes in the eye-aperture area. The point *Pmin*, at which an area is minimized, is determined based on the model (Fig. 1) in theory. However, it might not be determined by an actual measurement (Figure 5). Therefore, *Pmin* is defined as the average of the eye-aperture field areas that are less than threshold *Th*₂ in one eyeblink waveform. Threshold *Th*₂ is determined from the following equation:

$$Th_2 = \frac{A_{max} - A_{min}}{10} + A_{min}$$

where *Amax* and *Amin* are the maximum and minimum, respectively, of the eye-opening area of the eyeblink waveform. In addition, the closing-phase amplitude is calculated based on the difference between the area of the eyeblink starting field and point *Pmin*. Similarly, the opening-phase amplitude is calculated based on the difference between the area of the eyeblink end field and point *Pmin*.

<figure><img src="image_7.png" /><figcaption>Fig. 5. Example of a difficult-to-decide minimum point.</figcaption></figure>
## 5 Automatically Differentiating Conscious and Unconscious Eyeblinks

In this section, we examine the differentiation of eyeblinks on the basis of the parameters of the extracted eyeblink waveform using the method outlined in Section 4. It has been reported that the duration of a conscious eyeblink is longer than that of an unconscious eyeblink [21]. However, in many cases, distinguishing between the two types of eyeblinks using this information is difficult because the difference in the duration of eyeblinks cannot be measured if the time resolution of the moving image is low. Therefore, the proposed method improves the distinction accuracy by combining the duration and amplitude. There are many cases in which differences between eyeblinks are not found because amplitude values are more sensitive to individual differences than duration values. However, we have already confirmed the following in preliminary experiments. Specifically, approximately one-half of all subjects in our experiment had a significant difference in both the duration and amplitude, and the other half had significant differences in either one of duration or amplitude. We also administered a t-test to the subjects using a 1% standard deviation between conscious and unconscious parameters. And Table 2 shows the details of the results obtained.

<table><caption>**Table 2.** Results of preliminary experiment [21].</caption><thead><tr><td>Parameter type</td><td>Subject number</td><td>Rate</td></tr></thead><tbody><tr><td>Both parameters</td><td>23</td><td>46%</td></tr><tr><td>Duration only</td><td>12</td><td>24%</td></tr><tr><td>Amplitude only</td><td>11</td><td>22%</td></tr><tr><td>No difference</td><td>4</td><td>8%</td></tr><tr><td>Total</td><td>50</td><td>100%</td></tr></tbody></table>

A significant difference of 24% (12 subjects) is evident in eyeblink duration. For eyeblink amplitude, the difference is 22% (11 subjects). For both parameters, the significant difference is 46% (23 subjects). Finally, no significant difference is apparent in 8% of the subjects (4 subjects). In other words, a significant difference in shape feature parameters between voluntary and spontaneous eyeblinks is seen in a minimum of 92% of the subjects. Moreover, the results of examination of individual parameters reveal the following. The total percentage of subjects who show a significant difference in eyeblink duration is 70%. The total percentage of subjects who display significant differences in eyeblink amplitude is 68%. Finally, the total percentage of subjects who show significant differences in both parameters is 46%.

Figure 6 shows a histogram that summarizes the distribution of the average value of the eyeblink duration of the 50 subjects by eyeblink type. Conversely, the histogram in Fig. 7 summarizes the distribution of the average value of the eyeblink amplitudes of the 50 subjects by eyeblink type.
<figure><img src="image_8.png" /><figcaption>Fig. 6. Duration difference in each group [21].</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 7. Amplitude difference in each group [21].</figcaption></figure>

We perform automatic differentiation using a threshold to distinguish the larger differences among the measured feature parameters between conscious and unconscious eyeblinks in every subject. In the proposed method, a normalization process is first applied to each feature parameter based on the average value of conscious eyeblinks to decide the differentiation threshold of each subject. As shown in Table 1, the trends in the parameters of subjects can be classified into three groups:
Groups A, B, and C. Group A comprises subjects who exhibit significant differences in eyeblink duration only. Group B comprises subjects who exhibit significant differences in eyeblink amplitude only. Group C comprises subjects who exhibit significant differences in both parameters. The method then selects the parameter difference of the larger side among the duration and amplitude after normalization. For instance, in a scenario where there is a particular difference in duration, the threshold, $Th_3$, used to distinguish conscious eyeblinks, is determined from the following equation:

$$Th_3 = \frac{Tdv - Tds}{2} + Tds$$

where $Tdv$ is the average duration of conscious eyeblinks and $Tds$ is the average duration of unconscious eyeblinks. In this instance, an eyeblink is distinguished as a conscious eyeblink if the duration exceeds $Th_3$. Further, an eyeblink is distinguished as unconscious if it falls below $Th_3$.

On the other hand, in the case where there is a particular difference in amplitude, the threshold, $Th_4$, used to distinguish conscious eyeblinks is determined from the following equation:

$$Th_4 = \frac{Adv - Ads}{2} + Ads$$

where $Adv$ is the average amplitude of conscious eyeblinks and $Ads$ is the average amplitude of unconscious eyeblinks. In this case, an eyeblink is distinguished as a conscious eyeblink if the duration exceeds $Th_4$. Conversely, it is distinguished as unconscious if it falls below $Th_4$.

The proposed method determines thresholds $Th_3$ and $Th_4$ via the above method during calibration, before actual measurements are conducted. Subsequently, it automatically distinguishes eyeblinks as conscious or unconscious based on the feature parameter.

# 6 Evaluation Experiments

In this section, we discuss the results obtained on employing the eyeblink waveform measurement and distinction algorithm outlined above and measuring the eyeblink waveform of 30 subjects (22 men and 8 women with ages in the range of 20–29 years; all without disabilities) to analyze the periodicity of the shape feature parameters of eyeblinks.

## 6.1 System Outline

The hardware comprising our experimental system included a Sony HDR-HC9 digital camcorder for obtaining eye images, and a personal computer for image and eyeblink waveform analysis. Although the camera could capture high-definition (HD) pictures,
standard-definition (SD) pictures were used in the experiments. The system is intended
to be mounted on wearable and smart devices. Furthermore, an experimental system was
developed as a prototype.

Ordinary indoor lighting (incandescent lighting) was used when capturing moving images. A pair of light-emitting diodes (LEDs) was placed symmetrically on both sides of the camera and at a distance of approximately 60 cm directly in front of the face of the subject. The back of the subject's head was lightly supported with a stabilizing device to prevent it from shaking. The video camera was placed in front of and below the subject's head at a distance of approximately 20 cm. The camera was then used to magnify and obtain pictures of the area surrounding the subject's left eye. Because the image format was set for SD video, the resolution was 720 × 480 pixels with a 16:9 aspect ratio and refresh rate of 30 fps (NTSC). These experiments were performed on the naked eye; therefore, eyeglasses were not allowed during filming.

## 6.2 Experimental Method

The subjects were given the following instructions during filming:

* Pay attention to the silver dot mark located on the upper part of the camcorder. (The mark was placed at this location by us.)

* When you hear the signal, “blink well” always.

* You do not have to resist any unconscious urge to blink.

The “blink well” instruction was meant to increase the difference in the shape of the feature parameters between conscious and unconscious eyeblinks. In other words, the signal was a means of encouraging subjects to be strongly conscious of their voluntary blinks. The signal was sounded randomly at intervals of 4 to 10 s using a digital timer. Images were captured for an overall total of 90 s during the course of the experiment. The first 20 s was used for calibration. This experiment does not use a control group because it was a conscious property the eyeblink immediately after sounding.

After measuring the eyeblink video image, we measured and analyzed the individual
eyeblink waveform using moving images. The calibration determined the distinction
threshold by using 20 s at the beginning of the moving image based on the method
described in Section 4 to decide the feature parameter to use as the distinction threshold
by normalization and comparison in each subject. Differentiation of conscious and the
unconscious eyeblinks was then performed in the subsequent 70 s of moving images,
using the obtained $Th_3$ and $Th_4$ thresholds. At this point, the system distinguished only
eyeblink waveforms that had been successfully detected automatically.

At the conclusion of the experiment, the subjects were asked to complete
questionnaires and/or comment about their experiences during the experiment. The
items in question were age, gender, sleep time during the previous night, health
condition (five levels: one (bad) through five (good)), task difficulty (five levels: one
(easy) through five (difficult)), confidence in achieving the task (five levels: one (low)
through five (high)), and personal interpretation of “blink well.”
## 6.3 Real-time Measurement Experiment

Table 3 provides data on the subjects that show a significant difference in the measurements between conscious and unconscious blinks. Representative results of the experiment in relation to measured conscious and unconscious blinks, including the average values of the durations of blinks, the closing-phase amplitude, and the opening-phase amplitude, are shown. The right side of the table shows the results after normalization and the selected feature parameter.

Using the amplitude ratio of the closing phase to the opening phase for parameters is complicated because the ratio of the closing-phase to the opening-phase amplitude was, in all cases, found to contain a minimum of one large parameter. Therefore, we redefined the average value of two amplitudes as the eyeblink amplitude. In Table 3, the tendency for variation in individual differences between conscious and unconscious eyeblinks is as follows.

Let us now analyze those subjects who either did not show significant differences or exhibited only some differences in shape feature parameters. The number of unconscious eyeblinks was found to be limited. Two reasons explain this. The first is the fact that few eyeblinks actually occurred, which may be because the subjects were under stress during the experiment. The second is that eyeblinks registered movements that were too small to be accurately detected. Therefore, this study might promote future research in eyeblink detection accuracy.
<table><caption>**Table 3.** Results of the extracted parameters.</caption><thead><tr><th rowspan="2">Subjects</th><th colspan="3">Conscious Eyeblinks</th><th colspan="3">Unconscious Eyeblinks</th><th rowspan="2">Normalization</th></tr><tr><th>Counts</th><th>Duration (ms)</th><th>Amplitude (pixel)</th><th>Counts</th><th>Duration (ms)</th><th>Amplitude (pixel)</th><th>Selected</th></tr></thead><tbody><tr><td>1</td><td>4</td><td>841</td><td>6799</td><td>7</td><td>650</td><td>6789</td><td>Duration</td></tr><tr><td>2</td><td>5</td><td>590</td><td>7906</td><td>6</td><td>335</td><td>4875</td><td>Duration</td></tr><tr><td>3</td><td>5</td><td>580</td><td>7054</td><td>16</td><td>338</td><td>5725</td><td>Duration</td></tr><tr><td>4</td><td>3</td><td>755</td><td>14111</td><td>5</td><td>393</td><td>12195</td><td>Duration</td></tr><tr><td>5</td><td>5</td><td>706</td><td>13009</td><td>4</td><td>511</td><td>10558</td><td>Duration</td></tr><tr><td>6</td><td>2</td><td>325</td><td>18785</td><td>3</td><td>288</td><td>11858</td><td>Amplitude</td></tr><tr><td>7</td><td>5</td><td>1053</td><td>19852</td><td>4</td><td>450</td><td>17100</td><td>Duration</td></tr><tr><td>8</td><td>5</td><td>660</td><td>16718</td><td>3</td><td>553</td><td>14435</td><td>Duration</td></tr><tr><td>9</td><td>5</td><td>560</td><td>15216</td><td>5</td><td>256</td><td>10850</td><td>Duration</td></tr><tr><td>10</td><td>5</td><td>553</td><td>15552</td><td>4</td><td>316</td><td>11326</td><td>Duration</td></tr><tr><td>11</td><td>5</td><td>400</td><td>9370</td><td>4</td><td>278</td><td>6620</td><td>Duration</td></tr><tr><td>12</td><td>5</td><td>463</td><td>7829</td><td>4</td><td>341</td><td>6918</td><td>Duration</td></tr><tr><td>13</td><td>5</td><td>686</td><td>5883</td><td>9</td><td>445</td><td>3763</td><td>Amplitude</td></tr><tr><td>14</td><td>5</td><td>733</td><td>10139</td><td>2</td><td>308</td><td>6817</td><td>Duration</td></tr><tr><td>15</td><td>5</td><td>576</td><td>9919</td><td>3</td><td>288</td><td>7077</td><td>Duration</td></tr><tr><td>16</td><td>5</td><td>390</td><td>11880</td><td>6</td><td>350</td><td>10942</td><td>Duration</td></tr><tr><td>17</td><td>5</td><td>623</td><td>11268</td><td>1</td><td>366</td><td>11213</td><td>Duration</td></tr><tr><td>18</td><td>4</td><td>400</td><td>14121</td><td>3</td><td>376</td><td>8291</td><td>Amplitude</td></tr><tr><td>19</td><td>4</td><td>625</td><td>9160</td><td>14</td><td>331</td><td>7166</td><td>Duration</td></tr><tr><td>20</td><td>4</td><td>595</td><td>12904</td><td>19</td><td>263</td><td>7481</td><td>Duration</td></tr><tr><td>21</td><td>5</td><td>530</td><td>13494</td><td>17</td><td>336</td><td>10860</td><td>Duration</td></tr><tr><td>22</td><td>5</td><td>376</td><td>11559</td><td>18</td><td>345</td><td>10951</td><td>Duration</td></tr><tr><td>23</td><td>5</td><td>553</td><td>5644</td><td>5</td><td>486</td><td>5631</td><td>Duration</td></tr><tr><td>24</td><td>5</td><td>386</td><td>7059</td><td>21</td><td>256</td><td>6553</td><td>Duration</td></tr><tr><td>25</td><td>5</td><td>606</td><td>5564</td><td>20</td><td>400</td><td>5095</td><td>Duration</td></tr><tr><td>26</td><td>5</td><td>686</td><td>9257</td><td>13</td><td>385</td><td>7822</td><td>Duration</td></tr><tr><td>27</td><td>5</td><td>856</td><td>8673</td><td>15</td><td>436</td><td>7948</td><td>Duration</td></tr><tr><td>28</td><td>5</td><td>530</td><td>7122</td><td>8</td><td>279</td><td>5119</td><td>Duration</td></tr><tr><td>29</td><td>5</td><td>500</td><td>8699</td><td>6</td><td>435</td><td>6885</td><td>Amplitude</td></tr><tr><td>30</td><td>4</td><td>436</td><td>6418</td><td>3</td><td>216</td><td>3385</td><td>Duration</td></tr></tbody></table>

Table 4 provides the results of automatic distinction rate. Representative results of the experiment are displayed in relation to measured conscious and unconscious blinks. The table shows counts of detected conscious eyeblinks Vi and unconscious eyeblinks Si, distinction error of conscious eyeblinks Ev, and unconscious eyeblinks Es, distinction accuracy rate of conscious eyeblinks Cv and unconscious eyeblinks Cs, and total accuracy rate Ct. The accuracy rate values Cv, Cs, and Ct are determined from the following equations:
$$C_v = \frac{Vi - Ev}{Vi} \times 100$$

$$C_s = \frac{Si - Es}{Si} \times 100$$

$$C_t = \frac{(Vi + Si) - (Ev + Es)}{Vi + Si} \times 100$$

These equations for accuracy rate are adopted from [11].

Using our proposed method, the average rate of successful differentiating of conscious eyeblink is approximately 72.7% for the experimental sample of 30 subjects. While, the average rate of successful differentiating of unconscious eyeblink is approximately 90.3%. Thus, the average accuracy rate of total is 83.6%. In unconscious eyeblinks are high identification rate, however in conscious eyeblinks are lower as compared to the unconscious rate. At this point, we believe that this passed differentiating of conscious blink is not a major problem. If these passed differentiating occur, the input can be attempted again through an intentional repetition of the conscious eyeblink. Therefore, we think the accuracy rate of unconscious is more important than conscious rate. In addition, there are often subjects of only a low accuracy rate of either conscious or unconscious. Because we used a simple algorithm in this experiment (e.g. subject 1, 3, 5, and more...) intend to improve the accuracy of differentiating by using a combination of two parameters.

Following the experiments, we interviewed the subjects and discovered that some subjects did not perform eyeblinks consciously when signals were given because their unconscious eyeblinks occurred at the same rate. On the basis of the results of these interviews, we plan to revise future instructions to promote more clarity. In addition, the classification of eyeblink types can be improved based on those subjects who did not show significant differences.
<table><caption>**Table 4.** Results of automatic distinction rate of conscious eyeblinks.</caption><thead><tr><th rowspan="2">Subjects</th><th colspan="2">Counts of eyeblink</th><th colspan="2">Distinction error</th><th colspan="2">Distinction rate (%)</th><th rowspan="2">All</th></tr><tr><th>Conscious</th><th>Unconscious</th><th>Conscious</th><th>Unconscious</th><th>Conscious</th><th>Unconscious</th></tr></thead><tbody><tr><td>1</td><td>11</td><td>12</td><td>4</td><td>1</td><td>63.6</td><td>91.7</td><td>78.3</td></tr><tr><td>2</td><td>10</td><td>11</td><td>0</td><td>0</td><td>100.0</td><td>100.0</td><td>100.0</td></tr><tr><td>3</td><td>10</td><td>39</td><td>8</td><td>4</td><td>20.0</td><td>89.7</td><td>75.5</td></tr><tr><td>4</td><td>10</td><td>12</td><td>0</td><td>0</td><td>100.0</td><td>100.0</td><td>100.0</td></tr><tr><td>5</td><td>10</td><td>10</td><td>7</td><td>0</td><td>30.0</td><td>100.0</td><td>65.0</td></tr><tr><td>6</td><td>10</td><td>15</td><td>0</td><td>5</td><td>100.0</td><td>66.7</td><td>80.0</td></tr><tr><td>7</td><td>10</td><td>12</td><td>0</td><td>3</td><td>100.0</td><td>75.0</td><td>86.4</td></tr><tr><td>8</td><td>10</td><td>7</td><td>5</td><td>0</td><td>50.0</td><td>100.0</td><td>70.6</td></tr><tr><td>9</td><td>9</td><td>11</td><td>1</td><td>0</td><td>88.8</td><td>100.0</td><td>95.0</td></tr><tr><td>10</td><td>10</td><td>10</td><td>0</td><td>0</td><td>100.0</td><td>100.0</td><td>100.0</td></tr><tr><td>11</td><td>9</td><td>15</td><td>2</td><td>3</td><td>77.7</td><td>80.0</td><td>79.2</td></tr><tr><td>12</td><td>10</td><td>13</td><td>2</td><td>3</td><td>80.0</td><td>76.9</td><td>78.3</td></tr><tr><td>13</td><td>10</td><td>17</td><td>1</td><td>0</td><td>90.0</td><td>100.0</td><td>96.3</td></tr><tr><td>14</td><td>10</td><td>12</td><td>0</td><td>0</td><td>100.0</td><td>100.0</td><td>100.0</td></tr><tr><td>15</td><td>9</td><td>6</td><td>5</td><td>0</td><td>44.4</td><td>100.0</td><td>66.7</td></tr><tr><td>16</td><td>10</td><td>12</td><td>7</td><td>0</td><td>30.0</td><td>100.0</td><td>68.2</td></tr><tr><td>17</td><td>10</td><td>7</td><td>4</td><td>0</td><td>60.0</td><td>100.0</td><td>76.5</td></tr><tr><td>18</td><td>10</td><td>3</td><td>1</td><td>0</td><td>90.0</td><td>100.0</td><td>92.3</td></tr><tr><td>19</td><td>7</td><td>29</td><td>3</td><td>0</td><td>57.1</td><td>100.0</td><td>91.7</td></tr><tr><td>20</td><td>10</td><td>43</td><td>1</td><td>12</td><td>90.0</td><td>72.1</td><td>75.5</td></tr><tr><td>21</td><td>10</td><td>49</td><td>3</td><td>4</td><td>70.0</td><td>91.8</td><td>88.1</td></tr><tr><td>22</td><td>10</td><td>23</td><td>1</td><td>8</td><td>90.0</td><td>65.2</td><td>72.7</td></tr><tr><td>23</td><td>10</td><td>17</td><td>5</td><td>0</td><td>50.0</td><td>100.0</td><td>81.5</td></tr><tr><td>24</td><td>7</td><td>58</td><td>3</td><td>5</td><td>57.1</td><td>91.4</td><td>87.7</td></tr><tr><td>25</td><td>9</td><td>47</td><td>7</td><td>5</td><td>22.2</td><td>89.4</td><td>78.6</td></tr><tr><td>26</td><td>10</td><td>35</td><td>2</td><td>7</td><td>80.0</td><td>80.0</td><td>80.0</td></tr><tr><td>27</td><td>10</td><td>21</td><td>4</td><td>3</td><td>60.0</td><td>85.7</td><td>77.4</td></tr><tr><td>28</td><td>10</td><td>16</td><td>2</td><td>1</td><td>80.0</td><td>93.8</td><td>88.5</td></tr><tr><td>29</td><td>10</td><td>15</td><td>0</td><td>3</td><td>100.0</td><td>80.0</td><td>88.0</td></tr><tr><td>30</td><td>11</td><td>10</td><td>0</td><td>2</td><td>100.0</td><td>80.0</td><td>90.5</td></tr><tr><td>Average</td><td></td><td></td><td></td><td></td><td>72.7</td><td>90.3</td><td>83.6</td></tr></tbody></table>

## 7 Conclusion

In this paper, we proposed a method for automatic differentiation of conscious eyeblinks. A method that can automatically differentiate between conscious and unconscious eyeblinks is an important prerequisite for developing an input interface for eyeblinks. The results of the evaluation experiment conducted using the proposed
method show that it is possible to automatically distinguish eyeblinks with higher
accuracy than in previous studies if there is a small difference in the eyeblink duration.
The proposed method shows that it is possible using a frame-splitting method even in
environments that use a low time resolution video camera. The results of our evaluation
experiment conducted with 30 different subjects indicate that the average accuracy is
83.6%. We required to fix head lightly and to detach glasses from subjects. This is a
problem at actual use. We believe that this problem can be solve by image processing
using motion vector. Consequently, typical information devices will able to control
using eyeblinks, only installing software based on proposed method.

In the future, we plan to develop a real-time computer input system based on
proposed measuring system. We also plan to improve this method to increase the
detection accuracy and investigate methods by which this system can be incorporated
into mobile devices. And we want to validate racial and cultural difference influence to
eyeblinks.

Acknowledgment

We thank Mr. Hironobu Sato of the Kanto Gakuin University College of Science and Engineering for his advice, comments and help in this study. This work was supported by JSPS KAKENHI Grant Number 24700598.

References

1. Matteo, B., Rocco, A., Gregori, B., Belvisi, D., Ottaviani, D., et al.: Voluntary, spontaneous and reflex blinking in patients with clinically probable progressive supranuclear palsy. Brain 132(2), 502–510 (2009).

2. Majaranta, P., Raiha, K.-J.: Twenty years of eye typing: Systems and design issues. In: Proc. of Symposium on ETRA 2002, ACM. pp. 15–22 (2002).

3. Richard, C, S., Heidi, H, K.: Adaptive one-switch row-column scanning. In: IEEE Trans. Rehabilitation Engineering. Vol.7, no.4, pp. 464-473 (1999)

4. Melanie, B., Andrew, T.: Indirect Text entry using one or two keys. In: Proc. 8th International ACM SIGACCESS Conf. Computer and Accessibility, pp. 18-25 (2006).

5. Kiyohiko, A., Shoichi, O., Minoru, O.: An eye-gaze input system using information on eye movement history. In: Proc. of the 4th international conference on Universal access in Human-Computer Interaction: Ambient Interaction, pp. 721–729 (2007).

6. Diogo, P., Maria, D, G, P., Amy, W., Khai, N, T.: Filteryedping: Design challenges and user performance of dwell-free eye typing. ACM Trans. Accessible Computing, vol. 6, no. 1, Article. 3 (2015)

7. Naoaki, I., Takumi, O., Kazutaka, S.: Investigation for calculation method of eye-gaze shift from electro-oculograph amplified by AC coupling with using eye-gaze input interface. IEICE Transactions on Information and Systems, J90-D(10), pp. 2903–2913 (2007)

8. Dekun, G., Naoaki, I., Tota, M., Kazuyuki, M.: Improvement of eye gesture interface system. In: Proc. of the 16th Asia Pacific Symposium of Intelligent and Evolutionary Systems. no.3, pp. 1–3 (2012).
9. Tanabe, K.: Eyeblink activity during identification of Katakana characters viewed through a restricted visual field. IEICE Trans. Fundam. Electron. Commun. Comput. Sci., E87-A(8), 2189-2191 (2004).

10. Abe, K., Ohi, S., Ohyama, M.: Automatic method for measuring eye blinks using split-interlaced images. Human-Computer Interaction, Part I, pp. 3-11 (2009).

11. Abe, K., Sato, H., Matsuno, S., Ohi, S., Ohyama, M.: Automatic classification of eye blink types using a frame-splitting method. In: 10th International Conference, EPCE 2013, Held as Part of HCI International 2013, Part II, pp. 117-124 (2013).

12. Miluzzo, E., Wang, T., Campbell, A. T.: EyePhone: Activating mobile phones with your eyes. In: Proc. of the Second ACM SIGCOMM Workshop on Networking, Systems and Applications on mobile handhelds. pp. 15-20 (2010)

13. Mayberry, A., Hu, P., Marlin, B., Salthouse, C., Ganesan, D.: iShadow: Design of a wearable, real-time mobile gaze tracker. In: Proc. of the 12th Annual International Conference on Mobile Systems, Applications, and Services. ACM. pp. 82-94 (2014)

14. Shogo, M., Kota, A., Naoaki, I., Tota, M., Kazuyuki, M.: Computer input system using eye glances. In: Proc. the 17th International Conference on Human-Computer Interaction, Part I. pp. 425-432 (2015)

15. Matsuno, S., Ohyama, M., Abe, K., Sato, H., Ohi, S.: Automatic discrimination of voluntary and spontaneous eyeblinks. Use of the blink as a switch interface. In: The Sixth International Conference on Advances in Computer-Human Interactions, pp. 433-439 (2013).

16. Grauman, K., Betke, M., Gips, J., Bradski, G. R.: Communication via eye blinks—Detection and duration analysis in real time. In: Proc. IEEE CS Conf. Computer Vision and Pattern Recognition (CVPR 2001), vol. 1, pp. 1010-1017 (2001).

17. Mackenzie I.S., Ashtiani, B.: BlinkWrite: Efficient text entry using eye blinks. Universal Access in the Information Society, 10(1), pp. 69-80 (2011).

18. Missimer, E., Betke, M.: Blink and wink detection for mouse pointer control. In: Proc. 3rd International Conference on Pervasive Technologies Related to Assistive Environments. ACM, No. 23 (2010).

19. Gorodnichy, D.: Second-order change detection, and its application to blink-controlled perceptual interfaces. In: Proc. of the International Association of Science and Technology for Development Conference on Visualization, Imaging and Image Processing, pp. 140-145 (2003).

20. Tetuya, O., Hironori, K., Masashi, K.: Development of an input operation of the communication tool using voluntary eye blink. In: Papers of Technical Meeting on Medical and Biological Engineering, IEE Japan, vol. 6, pp. 1-4 (2006).

21. Matsuno, S., Ohyama, M., Abe, K., Ohi, S., Itakura, N.: Analysis of trends in the occurrence of eyeblinks for an eyeblink input interface. In: Proc. IEEE Conf. UsARE2014 in conjunction with RE 2014, pp. 25-31 2014.