SUGISAKAMASANORI

Journal of Robotics, Networking and Artificial Life
Vol. 9(2); September (2022), pp. 93–98
ISSN (Online):2352-6386; ISSN (Print):2405-9021
https://alife-robotics.org/jrnal.html

Research Article

Mouse Cursor Control System using Gaze and Eye Openingand Closing
Masayoshi Tabuse1, Mana Fukumoto2, Yasunari Yoshitomi1, Taro Asada1

Graduate School of Life and Environmental Science, Kyoto Prefectural University, 1-5 Shimogamohangi-cho, Sakyo-ku, Kyoto 606-8522, Japan
Department of Informatics and Environment Science, Kyoto Prefectural University, 1-5 Shimogamohangi-cho, Sakyo-ku, Kyoto 606-8522, Japan1
2

ARTIC LE INFO
Article History
Received 25 November 2021
Accepted 24 April 2022

Keywords
Mouse cursor control system
Eye gaze
Eye opening and closing

ABSTRACT

It is necessary to support of computer operation for a physically disabled person. One of thepossible physical movements of the physically disabled person is Eye movement. Eye gaze anddetection of eye opening and closing of a person makes it possible to operate a computer. In oursystem, a web camera, Gaze pointer, Dlib C++ library and OpenCV library are used torecognize eye gaze and detect eye opening and closing. Changing the eye gaze, we can move amouse cursor. Recognizing closed eye, we can carry out an operation of mouse click. In thispaper, we evaluated the effect on operability due to the eye gaze and detection of eye openingand closing.

© 2022 The Author. Published by Sugisaka Masanori at ALife Robotics Corporation Ltd
This is an open access article distributed under the CC BY-NC 4.0 license(http://creativecommons.org/licenses/by-nc/4.0/).

pointer moves to the face direction and the system
clicks the mouth in the case of the opening mouth andthe closing eye. However in the previous study weassume that the user can turn his face and open andclose his mouth and eyes so that the system cannot beused by severe amyotrophic lateral sclerosis (ALS)
patients.
So, it is necessary to develop a mouse cursor control
system that can be used by patients with severe ALS.
In this research, we developed a mouse cursor control
system only by eye movements, and evaluated its
performance.

1. Introduction
In recent years, a computer input system without themouse and keyboard has been developed as a support
system of computer operation for a physically disabledperson. However, many of them are contact typesystems that must be attached to the body which
imposes a physical burden on the user. In addition,
many non-contact type system are special sensors andexpensive.
Therefore, in the previous study [1], we proposed a
non-contact system which operates the mouse cursor
using an inexpensive web camera. The system
recognizes the direction of the face and the opening
and closing of the mouth and eyes so that the mouse
Corresponding Author’s E-mail: tabuse@kpu.ac.jp

2. The communication means
In ALS, it is important to consider communicationmeans as the symptoms progress. If a patient can movehis arms and fingers, he can talk in writing, and if he93

5. Movement of the mouse cursor

can move his mouth, he can communicate using shaperecognition of mouth. If he can move his eyes, he cancommunicate using a transparent letter board. However,communication means using shape recognition ofmouth and a transparent letter board always require thehelp of a helper. On the other hand, if he can move hisbody or eyes, it is possible to communicate by
selecting an appropriate computer input system. This
study targets people with advanced symptoms whohave residual eye movements.

In this research, we use a web camera and GazePointersoftware [2] to estimate the eye gaze and move themouse cursor based on the eye gaze. Before using thesystem, the user calibrates eye gaze.
6.

Mouse click

6.1. Judgment of gaze state
The system gets the coordinate M0 of the mouse cursor
at time t = 0 and sets this coordinate to the point G0. If
the coordinates M1 of the mouse cursor gets by thesystem at the next time t = 1 are within the rectangulararea R0 of 114 × 106 centered on G0, the system sets
R1 with the midpoint of the coordinates M1 and G0 as
the point G1. When the coordinate Mt of the mouse
cursor at time t is in the rectangular area Rt-1 (Fig. 2)
for about 2 seconds, it is judged as "gaze state" and thecoordinates of Gt are determined to be the gazecoordinates. If Mt is not in the rectangular area Rt-1,
initialize the time and continue the calculation (Fig. 3).

3. Comparison with the previous study
In the previous study [1], we assumed that the user
could turn his face and open and close his mouth andeyes. In this study, it is assumed that the user cannotchange the direction of his face, cannot move hismouth but he can move his eyes. The movement of themouse cursor depends on the direction of his eyes andclick process is performed by gazing. In addition, we
make it possible to change click to double-click bychanging the input mode by closing his eye.
4. The proposed systemThe movement of the mouse cursor depends on his eye
gaze and click process is performed by gazing for 2
seconds or longer. To prevent unintended clickprocessing, the click function will be stopped if botheyes are closed for 4 seconds or longer. The proposedsystem can be divided into three elements: "movementof mouse cursor", "mouse click processing", and"controlling mouse click function". Figure 1 shows aflowchart of the outline of the system.

Fig.2 The coordinate Mt isin the region Rt-1

Fig.3 The coordinate Mt isout the region Rt-1

6.2. Process of Mouse click
If the state is the "gaze state", the coordinate of thepoint Gt is the coordinate of the gaze and the system
Moves the mouse cursor to the point Gt to perform
click processing.
6.3. Judgment of eye opening and closing
The judgment of opening and closing eyes is used by
the method described in [1]. First, an open source C ++library Dlib [3] is used to detect facial feature points(68 points in total).
The eye area is determined by the coordinates of thefeature points of both eyes, as shown in Fig. 4. TheFig 1. Flowchart of the outline of the system.
94

6.4. Controlling the mouse click

range of x-axis is the difference of x-coordinate
between the left and right, and the range of y-axis is
the difference of y-coordinate between the top andbottom. The RGB image of the eye area is extracted
from a captured image and converted from the RGB
image to a grayscale image. Next, we determine athreshold value is obtained by the discriminant analysis
method [4], and binarization is performed for the eye
area image. From the binarized image, we make thehistogram of black pixels. The vertical length he of theeye is the maximum on the histogram, and thehorizontal length we of the eye is the range of xcoordinate in the eye area, as shown in Fig. 5. The
ratio Re is obtained from he and we by equation (1). If
Re is above the threshold value, it is determined thatthe eyes are open, and if Re is below the thresholdvalue, it is determined that the eyes are closed. Fromthe previous study [1], the threshold is set to 0.3.

Re =heweWhen the mouse click function is activated and one
eye is closed for about 1 second, the double click andsingle click functions are switched. Also, if both eyes
are closed for about 4 seconds, the mouse click
function will be stopped. On the contrary, when themouse click function is stopped and both eyes areclosed for about 4 seconds, the mouse click function isactivated. Figure 6 shows a flowchart for controllingthe mouse click function.

(1)

Fig. 6 Flowchart of controlling the mouse click
7.

Fig. 4. Feature points of eye extracted using Dlib.

Experiments

7.1. Operation screen
Figure 7 shows the operation screen when using thesystem. The user's face is displayed in the upper left
and upper right of the screen. The upper left screen
displays the eyes gaze detection, and the upper rightdisplays the eye opening and closing judgment.

Fig. 5. The vertical and horizontal length and histogram ofopened eye (left) and closed eye (right).

Fig. 7 Operation screen
7.2. Experiment contents
We conducted the following two evaluation
experiments comparing the proposed system with thepreviously reported system [1] for 10 subjects in their
20s (male: A to F, female: G to J).
95

・Evaluation experiment on mouse cursor movement
and mouse click・ Evaluation experiment on character input
Before the experiment, the subjects explained theoperation method of each system and the contents ofthe experiment, and then practiced for 2 minutes.
When using the proposed system, calibration wasperformed for each experiment. After the experiment, aquestionnaire was conducted.
7.3. Evaluation of mouse cursor movement andmouse click
The subjects use each system to double-click the icon(Notepad) installed on the desktop, and measured therequired time (Experiment 1). Also, for the operationof double-clicking five icons installed on the desktop,
the required time was measured for each system(Experiment 2), and the difference in the required time
between single operation and continuous operation wasinvestigated.

Fig. 9 Time required for Experiment 1
was shorter when the proposed system was used except
for 3 subjects. Under experiment 2, the time required
was shorter when the proposed system was used for allsubjects. The time required for the three subjects who
used the proposed system under the experiment 1 waslonger than that when the proposed system was used
under the experiment 2. This is because the proposedsystem, which is accustomed to moving the mouse
cursor with the eye gaze while performing continuous
operations, gradually shortens the time required tomove the mouse cursor, and as a result, the mouse
cursor moves faster.

7.4. Evaluation of character input
Using the Windows screen keyboard (Fig. 8), wemeasured the time it took to enter "kyoto" in Notepad,
and evaluated the difference in operability in character
input of each system (Experiment 3).

Fig. 8 Character input
8.

Results8.1. Evaluation of mouse cursor movement andmouse click
Figure 9 shows the time required for each subject
under experiment 1 and the average value of 10subjects, and Figure 10 shows the time required foreach subject under experiment 2 and the average valueof 10 subjects. Under experiment 1, the time required

Fig. 10 Time required for Experiment 2
8.2. Evaluation on character input
Figure 11 shows the time required for 10 subjects toenter "kyoto" in each system and the average value of96

10 subjects. The time required was shorter when theproposed system was used for all subjects. It isconsidered that this is because, as in the above
experiment, the user became accustomed to moving
the mouse cursor with the eye gaze while performing
continuous operations, and as a result, the required
time was shortened.

Table 2 Evaluation results of all subjects

9. Conclusion
In this research, we developed a system that detects theeye gaze and the eye opening and closing, moves themouse cursor, and clicks the mouse, and evaluated itsperformance. Comparing the time required for theoperation, the time was shorter overall when theproposed system was used than the previous reported
system [1]. It can be said that it is particularly useful torequire continuous mouse operations such as character
input.
A future task is to reduce the error in the noncontact state with respect to the problem that theposition of the mouse pointer is inaccurate due to thedisplacement of the head.
In addition, in order to further improve usability, wewill add a function that allows the user to freelychange the gaze time at the time of clicking, anddevelop easier character input system with the eyegaze.

Fig. 11 Time required for Experiment 3
8.3. Experimental questionnaire
Table 1 shows the contents of the 5-step evaluationquestionnaire conducted after the experiment, where
System①is the previous reported system and System② is the proposed system, and Table 2 shows theevaluation results and averages of each question itemof all subjects. In Q1 and 3, the average value of theproposed system was 0.5 higher than that in thepreviously reported system [1], and in Q2 and 4, theaverage value of the proposed system was 0.6 higher
than that of the previously reported system [1]. In thefree description field,
・It was difficult to fix my face, so that I thought itwould be nice to have something like a chin rest.
・Since system ② can input characters faster, I feltthat the operability was good after some practice.
・I thought that the operability of System ① would
not change even if I got used to it. It took time to getused to system ②, but I think that the accuracy will
improve once you get used to it.

References
1. Masayoshi Tabuse, Manase Mizobe, Yasunari
Yoshitomi, Taro Asada, “Mouse Cursor Control System
Using Facial Movements”, ICAROB2020, 394-397(2020).
2. Muhammad U Ghani et al, “GazePointer: A Real Time
Mouse Pointer Control Implementation Based On Eye
Gaze Tracking”, Journal of Multimedia Processing andTechnologies Vol.5 No.2, 64-75 (2014).
3. Dlib C++ Library, http://dlib.net/
4. Nobuyuki Otsu, “A Threshold Selection Method from
Gray-Level Histograms”, IEEE Transactions onSystems, Man, and Cybernetics, Vol.9, 62-69 (1979).

Table 1 Questionnaire

97Authors Introduction
Dr. Masayoshi Tabuse
He received his M.S. and Ph.D.
degrees from Kobe University in1985 and 1988 respectively. From
June 1992 to March 2003, he had
worked in Miyazaki University.
Since April 2003, he has been in
Kyoto Prefectural University. His
current research interests aremachine learning, computer visionand natural language processing. IPSJ and IEICE member.
Miss Mana Fukumoto
She received her B.S. degree fromKyoto Prefectural University in 2021.

Dr. Yasunari Yoshitomi
He received his B.E, M.E. and Ph.D.
degrees from Kyoto University in 1980,
1982 and 1991, respectively. He works
as a Professor at the Graduate School ofLife and Environmental Sciences of
Kyoto Prefectural University. His
specialties are applied mathematics andphysics, informatics environment,intelligent informatics. IEEE, HIS,ORSJ, IPSJ, IEICE, SSJ, JMTA and IIEEJ member.
Dr. Taro AsadaHe received his B.S., M.S. and Ph.D.
degrees from Kyoto Prefectural
University in 2002, 2004 and 2010,
respectively. He works as an Associate
Professor at the Graduate School ofLife and Environmental Sciences ofKyoto Prefectural University. His
current research interests are humaninterface and image processing. HIS, IIEEJ member.

98