Eyeball locating method and system

An eyeball locating system includes a display panel, an application program and a matrix optical sensor. The matrix optical sensor is disposed in the display panel. The application program controls the matrix optical sensor, executing an eyeball locating method to measure the movement of an eyeball including an eye white and a pupil. The method includes: providing a default graph showing the change of light energy; defining a default characteristic value of the pupil according to the default graph; shining light on the eye white and the pupil, a part of the light reflected by the eye white and the pupil to form a reflecting light; detecting the reflecting light energy to form a measured graph showing the change of reflected light energy; calculating a measured characteristic value according to the measured graph; and comparing the default with the measured characteristic values to determine the change of the pupil.

BACKGROUND OF THE INVENTION

(1) Field of the Invention

The present invention relates to an eyeball locating method and system, particularly to an eyeball locating method and system using an optical sensor for determining the light reflected by the surface of the eyeball.

(2) Description of the Prior Art

Computer makes people's daily life more and more convenient, but for most disabled men, it is rather inconvenient to handle computer.

To help the disabled men to handle computer, several control methods of the man-machine interface system have been developed in the world, such as hand joint control, voice control, electromyogram (EMG) behavior, shoulder control, gassing and so on. The common disadvantage among these control methods is that they are not capable of executing complex control behaviors, and they are also unable to control the computer conveniently and real-timely because the control signal needs complex identification and longer processing period.

To conquer these limits, the technology of using the movement of eyeball to control the computer emerges as the times require. The eyeball tracking system is most common, which is capable of controlling the cursor or keyboard of the computer by detecting the movement of the eyeball in real time. At present, the eyeball tracking system has piezoelectric type, optical type and magnetic type according to the detection principles.

The piezoelectric type is detecting the movement direction of the eyeball by the change of eye pressure, which is pasting the piezoelectric sensor around the eyes, converting the eye pressure into electrical signal by the piezoelectric sensor and measuring the electrical signal. However this system influences the measurement of the electrical signal easily due to sweat.

The magnetic type is measuring the movement of the eyes by forming magnetic field around the eyeball.

The optical type is capturing an eyeball image, coordinating the present target position through algorithm of image analysis, sending the results to the personal computer and driving mouse to execute control instruction.

Refer toFIG. 1for the head-mounted optical eyeball tracking system100. The eyeball tracking system100has a charge-coupled device (CCD) image detector120, a screen140and a frame160. The CCD image detector120is electrically connected to the screen140and disposed on the frame160together with the screen140. When an user200wears the frame160, the CCD image detector120and the screen140are fixed near the eyes of the user200by the frame160. The screen140displays a plurality of the location points (not shown) for the user200. When the user200watches one of the location points on the screen140, the CCD image detector120captures his pupil image and performs binary processing to get the position of the pupil.

Furthermore, the CCD image detector120is connected to a computer180. The CCD image detector120collects and analyzes the pupil image real-timely, and converts it into a control order to command the cursor to handle the computer180.

However, the user200of the eyeball tracking system100needs to wear the frame160to increase success rate. The difficulty of the optical detection is when capturing the pupil image, the contrast ratio between the pupil and the eye white is too low to measure, especially under the condition of glasses obstruct, outer light disturb or eyeball pathological changes. In conclusion, the conventional eyeball tracking system100is limited in use and has low resolution, which makes it hard to apply in the view or browsing equipments or general medical occasion.

SUMMARY OF THE INVENTION

The present invention is to provide an eyeball locating method and system capable of rising the successful rate of eyeball locating, and being used more conveniently.

For achieving one, some or all of the above mentioned object, an eyeball locating method is provided as an embodiment of the present invention. The eyeball locating method is used to measure the movement of an eyeball including an eye white and a pupil, the method includes the steps of: providing a default graph showing the change of light energy; defining a default characteristic value of the pupil according to the default graph showing the change of light energy; shining light on the eye white and the pupil wherein at least a part of the light is reflected by the eye white and the pupil, so as to form a reflecting light; detecting the energy of the reflecting light, so as to form a measured graph showing the change of reflected light energy; calculating a measured characteristic value according to the measured graph showing the change of reflected light energy; and comparing the default characteristic value with the measured characteristic value to determine the change of the pupil.

In one embodiment, the default characteristic value is the default contrast ratio between the eye white and the pupil, and the measured characteristic value is the measured contrast ratio between the eye white and the pupil.

In one embodiment, the default characteristic value is the default width of the pupil, and the measured characteristic value is the measured width of the pupil.

In above, the step of comparing the default characteristic value with the measured characteristic value to determine the change of the pupil includes determining the width, displacement or area change of the pupil. The step of determining the displacement change of the pupil includes determining a vertical displacement, a horizontal displacement or a near-far displacement of the pupil.

In one embodiment, the eyeball locating method further includes the steps of: providing a matrix optical sensor for shining the light and detecting the energy of the reflected light; measuring the time difference between the light emitted from the matrix optical sensor and the reflecting light coming back to the matrix optical sensor; and calculating the distance between the matrix optical sensor and the eyeball according to the light speed and the time difference.

An eyeball locating system is provided as an embodiment of the present invention. The eyeball locating system includes a display panel, a matrix optical sensor, an application program. The matrix optical sensor is disposed in the display panel, for shining light on the eyeball. Wherein, at least a part of the light is reflected by the eye white and the pupil to form a reflecting light, and the matrix optical sensor detects the energy of the reflecting light. The application program provides a default graph showing the change of light energy to define a default characteristic value of the pupil, and controls the matrix optical sensor, so as to form a measured graph showing the change of reflected light energy according to the energy of the reflecting light. The application program calculates a measured characteristic value according to the measured graph showing the change of reflected light energy and compares the default characteristic value with the measured characteristic value to determine the change of the pupil.

In one embodiment, the matrix optical sensor is selected from the group consisting of a CCD sensor, a CMOS (complementary metal-oxide semiconductor) sensor and an infrared sensor.

In one embodiment, the default or measured characteristic value is the contrast ratio between the eye white and the pupil, or a width, a displacement, an area of the pupil.

In above, both the default graph showing the change of light energy and the measured graph showing the change of reflected light energy show the relationship between the light reflectivity and time.

Accordingly, the embodiments of the present invention compares the default characteristic value with the measure characteristic value based on the default graph showing the change of light energy for determining the change of the pupil more accurately.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Refer toFIG. 2for an eyeball locating system300to measure the movement of an eyeball400. The eyeball locating system300includes a display panel320, a first matrix optical sensor340, a second matrix optical sensor360and an application program380. Referring toFIG. 3, the eyeball400has an eye white420and a pupil area440which has a pupil442and an iris444.

Referring toFIG. 2, the first matrix optical sensor340and the second matrix optical sensor360are disposed separately at two sides322and324of the display panel320. In this embodiment, both the first matrix optical sensor340and the second matrix optical sensor360are the components providing optical signal emission and induction, shining light on the surface of the eyeball400respectively. At least a part of the light is reflected by the eye white420and the pupil442, so as to form a reflecting light. Additionally, the first matrix optical sensor340and the second matrix optical sensor360each detect the energy of the reflecting light from itself. It should be noted that the first matrix optical sensor340and the second matrix optical sensor360in this embodiment are used to determine the energy of the reflecting light accurately, but the form or number of the matrix optical sensor is not limited. In other words, applying single matrix optical sensor, or an independent optical emitter with an optical sensor may also achieve the optical signal emission and induction.

The application program380provides a default graph showing the change of light energy, which may be used to define a default characteristic value of the pupil442, such as the default contrast ratio of the eye white420and the pupil442, the width, the area and the displacement of the pupil442.

In addition, the application program380converts the energy of the reflecting light detected by the first matrix optical sensor340and the second matrix optical sensor360into numerical value by a microprocessor (not shown), so as to form the first and the second graphs showing the change of reflected light energy, and calculates a measured characteristic value of the pupil442, such as the measured contrast ratio of the eye white420and the pupil442, the width, the area and the displacement of the pupil442. By comparing the default characteristic value with the measured characteristic value, the application program380is able to determine the change of the pupil442.

Refer toFIG. 3for the relationship between the default graph C showing the change of light energy and the eye white420, the iris444and the pupil442. In a preferable embodiment, the longitudinal axis of the graph C is optical reflectivity and the lateral axis is time. As shown inFIG. 3, the curves from the time t1to t2and the time t5to t6are the default curves showing the optical reflectivity change of the eye white420. The curve from the time t2to t5is the default curve showing the optical reflectivity change of the pupil area440. The curves from the time t2to t3and the time t4to t5are the default curves showing the optical reflectivity change of the iris444. The curve from the time t3to t4is the default curve showing the optical reflectivity change of the pupil442.

As for the definition of the default and measured characteristic values, the example is taken below.

After the contrast ratio of the eye white420and the pupil442is defined, the areas below the optical reflectivity change curves from the time t1to t2and the time t3to t4may be calculated separately, and then the ratio of the two areas is obtained. Another method to define the contrast ratio of the eye white420and the pupil442is comparing the height difference of the peaks of the optical reflectivity change curves from the time t1to t2and the time t3to t4, for example, the height difference b inFIG. 3represents the contrast ratio of the eye white420and the pupil442.

Each time interval of the graph C showing the change of light energy has a conversion relationship with the width, area or displacement of the pupil442. For instance, assuming the scanning speed of the first matrix optical sensor340or the second matrix optical sensor360is constant and the part scanned from t3to t4is the pupil442, the width of the pupil442may be calculated by the scanning speed and the time interval t3-t4. Thus, these default characteristic values may be defined according to the graph C showing the change of light energy.

Refer toFIG. 4toFIG. 7for the embodiment of the first or second graph showing the change of reflected light energy. The first matrix optical sensor340and the second matrix optical sensor360are used in practice to measure the eye white420and the pupil442. The reflectivity of the eye white420is high, so the signal of the reflecting light from the eye white420is strong, while the reflectivity of the pupil442is low, so the signal of the reflecting light from the pupil442is weak. Accordingly, the first or second graph showing the change of reflected light energy is obtained. By comparing the waves of the graphs C1, C2, C3, C4showing the change of reflected light energy with the default graph C showing the change of light energy, the eyeball400may be located. For convenient comparison with the graph C, all the longitudinal axes of the graphs C1, C2, C3, C4are optical reflectivity and all the lateral axes are time.

Referring toFIG. 4, comparing the default graph C showing the change of light energy (broken line) and the graph C1showing the change of reflected light energy, it appears that the current time intervals t1-t2and t5-t6do not change, but the time intervals t2-t3′ and t4′-t5are shorter than the default time intervals t2-t3and t4-t5, while the time interval t3′-t4′ is longer than the default time interval t3-t4. It represents that at this time the eyeball400is not running, but the iris444is shrunk and the pupil442is enlarged. InFIG. 4, the mark444′ stands for the shrunk iris and mark442′ for the enlarged pupil. In contrary, if the time intervals t2-t3′ and t4′-t5are longer than the default time intervals t2-t3and t4-t5, and the time interval t3′-t4′ is shorter than the default time interval t3-t4, it represents that the iris444is released and the pupil442is shrunk.

Referring toFIG. 5, comparing the default graph C showing the change of light energy with the graph C2showing the change of reflected light energy, it appears that when the time interval t1-t2′ is shorter than the default t1-t2′, but the t5′-t6is longer than the default t5-t6, it represents that the eyeball400is running and the pupil442is moving to the left side of the image. In contrary, if the time interval t1-t2′ is longer than the default t1-t2′, but t5′-t6is shorter than the default t5-t6, it represents that the pupil442is moving to the right side of the image. In addition, if the time intervals t2′-t3″, t3″-t4″, t4″-t5′ are different from the time intervals t2-t3, t3-t4, t4-t5in the default graph C showing the change of light energy, it represents that the pupil442is enlarged or shrunk.

Referring toFIG. 6, the graph C3showing the change of reflected light energy is the result of the default graph C showing the change of light energy shifting left. Under this circumstance, the time points T1, T2, T3, T4, T5, T6depart from the default time points t1, t2, t3, t4, t5, t6, which means the eyeball400moves left, such as the head turning or moving; the same principle is suitable for judging the eyeball400moving right. As shown inFIG. 4orFIG. 5, each time interval within t1-t6changes along with the iris442changing or the eyeball400turning.

Referring toFIG. 7, the graph C4showing the change of reflected light energy shows when the default graph C showing the change of light energy is enlarged, the time points T1′, T2′, T3′ shift to the left of the default time points t1, t2, t3, and the time points T4′, T5′, T6′ shift to the right of the default time points t4, t5, t6. It represents the eyeball400moves back and forth, such as the face moving back and forth. As shown inFIG. 4orFIG. 5, each time interval within t1-t6changes along with the iris442changing or the eyeball400turning.

It may be speculated according to the above description that the embodiment of the present invention is also suitable for the eyeball400running up and down.

According toFIG. 7, the movement of the eyeball400running back and forth may be detected, thus the distance change between the eyeball400and the first matrix optical sensor340or the second matrix optical sensor360may be defined and the backlight of the display panel320and the font size on the image may be adjusted.

FIG. 8provides another method to measure the distance between the eyeball400and the first matrix optical sensor340or the second matrix optical sensor360. Referring toFIG. 8, the first matrix optical sensor340or the second matrix optical sensor360each shines a measuring light on the surface of a left eyeball400L and a right eyeball400R, and the light is reflected by the left eyeball400L and the right eyeball400R to the first matrix optical sensor340or the second matrix optical sensor360to form a time difference. According to light speed and the time difference, the distance DBLbetween the first matrix optical sensor340and the left eyeball400L, the distance DBRbetween the first matrix optical sensor340and the right eyeball400R, the distance DALbetween the second matrix optical sensor360and the left eyeball400L, and the distance DARbetween the second matrix optical sensor360and the right eyeball400R are calculated.

InFIG. 8, the distance W between the first matrix optical sensor340and the second matrix optical sensor360is a known value. The distances DAL, DAR, DBL, DBRbetween the first matrix optical sensor340or the second matrix optical sensor360and the left eyeball400L or the right eyeball400R are measured by the above method. With these data and trigonometric functions, the vertical distances eL, eRbetween the left eyeball400L, the right eyeball400R and the display panel320may be calculated, thus the backlight of the display panel320and the font size on the image are adjusted accordingly.

In addition, in an embodiment, when eyes are closed, it is hard to distinguish the eye white420area and the pupil442area from the graph showing the change of reflected light energy measured by the first matrix optical sensor340and the second matrix optical sensor360. When eyes are open, the eye white420area and the pupil442area may be distinguished from the graph showing the change of reflected light energy measured again, thus whether the eyes are blinking and the blinking frequency may also be detected.

The first matrix optical sensor340and the second matrix optical sensor360may convert the light into electric charge, then into digital signal. In a preferable embodiment, the first matrix optical sensor340and the second matrix optical sensor360may be, but not limited to a CCD (charge coupled device) sensor, a CMOS (complementary metal-oxide semiconductor) sensor, an infrared sensor, a weak laser sensor or a digital camera module. Its detecting manner is, but not limited to photo, scanning or interlacing type. In addition, single matrix optical sensor is enough to perform the function of the eyeball locating system300, while multiple matrix optical sensors may improve the accuracy.

The eyeball locating system300is used in the electrical device with the display panel320, such as digital camera, projector, automated teller machine (ATM), media or medical instrument. For example, dispose a matrix optical sensor of the eyeball locating system300, such as the first matrix optical sensor340, inside the display panel320of the electrical device and set up the application program380in its memory (not shown). After the first matrix optical sensor340detects the reflecting light from the surface of the eyeball400, the application program380calculates the graph showing the change of reflected light energy and the characteristic value by the microprocessor of the electrical device, so as to handle the electrical device.

The possible application of the embodiment of the present invention is further illustrated as follows: the movement of the eyeball400controls the cursor and the blinking defines the shortcut. The blinking speed and number define shortcuts, such as: blinking right eye for confirmation and blinking left eye for cancellation. The intensity of the backlight is adjusted by determining the size of the pupil442. The change of the pupil442is measured by determining the wave, so as to adjust the brightness of the backlight. If no user, the embodiment of the present invention may use to detect the environment, transfer special advertisement and message or even close the computer.

As shown inFIG. 9, the eyeball locating system300has an eyeball locating method, including: providing a default graph C showing the change of light energy (S1); defining a default characteristic value of the pupil442according to the default graph C showing the change of light energy (S2); shining light on the eye white420and the pupil442and making at least a part of the light be reflected by the eye white420and the pupil442, so as to form a reflecting light (S3); detecting the energy of the reflecting light from the eye white420and the pupil442(S4), so as to form a measured graph C1, C2, C3or C4showing the change of reflected light energy (S5); calculating a measured characteristic value of the pupil442according to the measured graph C1, C2, C3or C4showing the change of reflected light energy (S6); and comparing the default characteristic value with the measured characteristic value (S7) to determine the change of the pupil442(S8) according to the difference between the default characteristic value and the measured characteristic value.

In an embodiment, the step of comparing the default characteristic value with the measured characteristic value to determine the change of the pupil442includes determining the width, displacement or area change of the pupil442. Furthermore, the step of determining the displacement change of the pupil442includes determining the vertical displacement, horizontal displacement and near-far displacement of the pupil442.

In an embodiment, the time difference between the light traveling from the matrix optical sensor340or360to the surface of the eyeball400and it being reflected to the matrix optical sensor340or360from the surface of the eyeball400is measured. According to the light speed and the time difference, the distance between the matrix optical sensor340or360and the eyeball400is calculated.

The eyeball locating system300is used to avoid touching products. Besides closer to the special requirement of general user, it is more suitable for the disabled men and to avoid the bacterial diseases.