Gaze tracking method and gaze tracking device using ihe same

A gaze tracking method includes the following steps. Firstly, a to-be-analyzed facial image is captured. Then, whether the to-be-analyzed facial image conforms to a customized 3D face model is determined If not, a customized 3D face model of the to-be-analyzed facial image is created. If yes, an eye area image of the to-be-analyzed facial image is obtained. Then, according to the customized 3D face model and the to-be-analyzed facial image, head posture information is obtained. Then, an eye camera coordinate value referenced to a camera coordinate system is obtained. Then, the eye camera coordinate value is converted into an eye frame coordinate value referenced to a display frame coordinate system. Then, according to the eye frame coordinate value, the head posture information and an eyeball radius, an eyeball center point coordinate value is obtained, and accordingly a gaze coordinate value is obtained.

This application claims the benefit of Taiwan application Serial No. 110147911, filed Dec. 21, 2021, the subject matter of which is incorporated herein by reference.

TECHNICAL FIELD

The disclosure relates in general to a gaze tracking method and a gaze tracking device using the same.

BACKGROUND

Gaze tracking systems have been widely applied in medical, retail, human-machine interface and game fields due to improvement and parity of technology. However, in order to achieve high precision, most systems include quite complex and personalized correction procedures and thus it is quite inconvenient to use. Therefore, how to propose a new technology that could improve the aforesaid problems is the goal of continuous efforts of those skilled in the art.

SUMMARY

According to an embodiment, a gaze tracking method is provided. The gaze tracking method includes the following steps: capturing, by a camera, a to-be-analyzed image, wherein the to-be-analyzed image includes a to-be-analyzed facial image of a face of a photographed person; determining whether the to-be-analyzed facial image of the to-be-analyzed image conforms to a customized 3D face model; establishing the customized 3D face model of the to-be-analyzed facial image through a customized 3D face modeling process if the to-be-analyzed facial image of the to-be-analyzed image does not conform to the customized 3D face model; obtaining an eye region image of the to-be-analyzed facial image if the to-be-analyzed facial image of the to-be-analyzed image conforms to the customized 3D face model; obtaining a head posture information of a head of the photographed person according to the customized 3D face model and the to-be-analyzed facial image; obtaining an eye camera coordinate value of an eye characteristic point of the eye region image, wherein the eye camera coordinate value is referenced to a camera coordinate system; converting the eye camera coordinate value into an eye frame coordinate value, wherein the eye frame coordinate value is referenced to a display frame coordinate system; obtaining an eyeball center point coordinate value of the photographed person according to the eye frame coordinate value, the head posture information and an eyeball radius; and obtaining a gaze projection point coordinate value of the photographed person according to the eye frame coordinate value and the eyeball center point coordinate value.

According to another embodiment, a gaze tracking device is provided. The gaze tracking device includes a display, a camera, a determination unit, a modeling unit and an analysis unit. The camera is disposed on the display and configured for capturing a to-be-analyzed image, wherein the to-be-analyzed image includes a to-be-analyzed facial image of a face of a photographed person. The determination unit is configured for determining whether the to-be-analyzed facial image of the to-be-analyzed image conforms to a customized 3D face model. The modeling unit is configured for establishing the customized 3D face model of the to-be-analyzed facial image through a customized 3D face modeling process if the to-be-analyzed facial image of the to-be-analyzed image does not conform to the customized 3D face model. The analysis unit is configured for obtaining an eye region image of the to-be-analyzed facial image if the to-be-analyzed facial image of the to-be-analyzed image conforms to the customized 3D face model; obtaining a head posture information of a head of the photographed person according to the customized 3D face model and the to-be-analyzed facial image; obtaining an eye camera coordinate value of an eye characteristic point of the eye region image, wherein the eye camera coordinate value is referenced to a camera coordinate system; converting the eye camera coordinate value into an eye frame coordinate value, wherein the eye frame coordinate value is referenced to a display frame coordinate system; obtaining an eyeball center point coordinate value of the photographed person according to the eye frame coordinate value, the head posture information and an eyeball radius; and obtaining a gaze projection point coordinate value of the photographed person according to the eye frame coordinate value and the eyeball center point coordinate value.

DETAILED DESCRIPTION

Referring toFIGS. 1 to 3C.FIG. 1is a schematic diagram of a gaze tracking device100according to an embodiment of the present disclosure, FIG.2is a schematic diagram of a to-be-analyzed camera image M1captured by the camera120ofFIG. 1, andFIGS. 3A to 3Care schematic diagrams of the photographed U1and the camera120ofFIG. 1at different viewing angles.

As shown inFIGS. 1 and 2, the gaze tracking device100includes a display110, a camera120, a determination unit130, a modeling unit140and an analysis unit150. At least two of the determination unit130, the modeling unit140and the analysis unit150could be integrated into a single unit. Alternatively, at least one of the determination unit130, the modeling unit140and the analysis unit150could be integrated into a processor or a controller. The determination unit130, the modeling unit140and/or the analysis unit150are, for example, software, firmware or hardware. At least one of the determination unit130, the modeling unit140and/or the analysis unit150is a physical circuit formed by, for example, a semiconductor process.

The camera120is disposed on the display110and is configured for capturing a to-be-analyzed image M1, and the to-be-analyzed image M1includes a to-be-analyzed facial image MF1of a face of the photographed U1. The determination unit130is configured to determine whether the to-be-analyzed facial image MF1conforms to a customized three-dimensional (3D) face model CF. The modeling unit140is configured for establish the customized 3D face model CF of the to-be-analyzed facial image MF1through a customized 3D face modeling process if the to-be-analyzed facial image MF1does not conform to the customized 3D facial model CF. The analysis unit150is configured to (1) obtain an eye region image MF1eof the to-be-analyzed facial image MF1if the to-be-analyzed facial image MF1conforms to the customized 3D facial model CF; (2). obtain a head posture information of a head of the to-be-analyzed facial image MF1(for example, a up-and-down tilt angle ωxof the head of the photographed U1inFIG. 3Aand a left-and-right tilt angle ωyof the head of the photographed U1in inFIG. 3Baccording to the customized 3D face model CF and to-be-analyzed facial image MF1; (3). obtain at least one eye camera coordinate value (for example, the eye camera coordinate value of a pupil center point U1P71and the eye camera coordinate value of an eye center point U1P72of a left eye inFIG. 2, and the eye camera coordinate value of a pupil center point U1P81and the eye camera coordinate value of an eye center point U1P82of a right eye inFIG. 2), wherein the eye camera coordinate value refers to a camera coordinate system Xc-Yc-Zc; (4). convert the eye camera coordinate value of the eye into an eye frame coordinate value of the eye, wherein the eye frame coordinate value refers to a display frame coordinate system Xs-Ys-Zs (the display frame coordinate system Xs-Ys-Zs are shown inFIGS. 3A to 3C); (5). obtain at least one eyeball center point coordinate value of the photographed U1(for example, a coordinate of an eyeball center point U1P73of the left eye of and a coordinate of an eyeball center point U1P83of the right eye) according to the eye frame coordinate value and an eyeball radius REBof the eye; and (6). obtain a gaze (or line-of-sight) projection point Vg(the gaze projection point Vg is shownFIG. 3) of the photographed according to the eye camera coordinate value of the eye and the coordinate value of the eye center point.

After obtaining the gaze projection point Vg, the display110could display the gaze projection point Vgto display the latest gaze dynamic of the photographed U1.

Since the customized 3D face model CF is a customized face model for the photographed person U1, the gaze projection point Vgof the to-be-analyzed facial image MF1obtained by the gaze tracking device100has high precision and accuracy (that is, the gaze projection point Vgobtained by the gaze tracking device100is equivalent to an actual gaze point of the photographed person U1), and if the photographed person U1has not established the customized 3D face model CF, the gaze tracking device100would also actively establish the customized 3D face model CF exclusive to the photographed person U1.

The following describes the gaze tracking method of the gaze tracking device100.

Referring toFIGS. 4 and 5A to 5C.FIG. 4is a flowchart of a gaze tracking method of the gaze tracking device100ofFIG. 1, andFIGS. 5A to 5Care the schematic diagrams of processes of establishing the to-be-analyzed facial image MF1ofFIG. 2.

In step S110, as shown inFIG. 2, the camera120captures the to-be-analyzed image M1, wherein the to-be-analyzed image M1includes the to-be-analyzed facial image MF1of the face of the photographed person U1(the photographed person U1is shown inFIG. 3A).

In step S120, the determination unit130determines whether the to-be-analyzed facial image MF1of the to-be-analyzed image M1conforms to the customized 3D face model CF. If the to-be-analyzed facial image MF1of the to-be-analyzed image M1conforms to the customized 3D face model CF, the process proceeds to step S130to obtain the gaze projection point Vgof the to-be-analyzed facial image MF1. If the to-be-analyzed facial image MF1of the to-be-analyzed image M1does not conform to the customized 3D face model CF, the process proceeds to step S140to establish the customized 3D face model CF of the to-be-analyzed facial image MF1.

For example, as shown inFIG. 2, the determination unit130could analyze several characteristic points of the to-be-analyzed facial image MF1, such as 68 characteristic points, at least including a left eye corner point U1P1, a right eye corner point U1P2, a middle point U1P3between two eyebrows, a nose tip point U1P4, a left mouth corner point U1P5, a right mouth corner point U1P6, a pupil center point U1P71and an eye center point U1P72of the left eye and a pupil center point U1P81and an eye center point U1P82of right eye ofFIG. 2, etc., wherein whether the to-be-analyzed facial image MF1conforms to the customized 3D face model CF is to determined according to the left eye corner point U1P1, the right eye corner point U1P2, the middle point U1P3between two eyebrows, the nose tip point U1P4, the left mouth corner point U1P5and the right mouth corner point U1P6. If the left eye corner point U1P1, the right eye corner point U1P2, the middle point U1P3between two eyebrows, the nose tip point U1P4, the left mouth corner point U1P5and the right mouth corner point U1P6of the to-be-analyzed facial image MF1do not meet all the six characteristic points of the customized 3D face model CF, and it means that the customized 3D face model CF has not yet been established by the photographed person U1, and the process proceeds to step S140to establish the customized 3D face model CF exclusive to the photographed person U1. If the left eye corner point U1P1, the right eye corner point U1P2, the middle point U1P3between two eyebrows, the nose tip point U1P4, the left mouth corner point U1P5and the right mouth corner point U1P6of the to-be-analyzed facial image MF1meet all the six characteristic points of the customized 3D face model CF, it means that the photographed person U1has previously established the customized 3D face model CF, and the process proceeds to step S130to obtain the gaze projection point Vg of the photographed person U1.

The following describes the customized 3D face modeling process (step S140).

The customized 3D face modeling process could be implemented by various implementations, one of which is described below with the processes ofFIG. 4.

In step S141, the modeling unit140obtains several characteristic points of the to-be-analyzed facial image MF1, for example, the modeling characteristic points P1to P6ofFIG. 5A, wherein the modeling characteristic point P1is the left eye corner point of the to-be-analyzed facial image MF1, the modeling characteristic point P2is the right eye corner point, the modeling characteristic point P3is the middle point between two eyebrows, the modeling characteristic point P4is the nose tip point, and the modeling characteristic point P5is the left mouth corner point and the modulo characteristic point P6is the right mouth corner point. The coordinate values of the modeling characteristic points P1to P6are merely examples, and the coordinate values are also different for different photographed person with different facial geometry. Furthermore, the modeling characteristic points are not limited to the aforementioned facial features, but could be characteristic points of any part of the face.

In step S142, as shown inFIG. 5B, the modeling unit140adjusts the coordinate value of a reference one of the modeling characteristic points P1-P6to a reference coordinate value. For example, the modeling unit140uses the modeling characteristic point P4(the nose tip point) as a reference one and adjusts the coordinate value of the reference one to the reference coordinate value, for example, (0, 0).

In step S143, as shown inFIG. 5B, the modeling unit140normalizes the coordinate value of each of the others of the modeling characteristic points P1-P6according to the reference coordinate values. For example, the modeling unit140adjusts the coordinate values of the other modeling characteristic points P1-P3and P5-P6of the modeling characteristic points P1-P6with the reference coordinate value as the center. Before and after the adjusting, the relative distance between the modeling characteristic points P1-P6remains fixed (unchanged). The coordinate values of the normalized modeling characteristic points P1-P3and P5-P6are shown inFIG. 5B, but such numerical values are not meant to be for limiting.

In step S144, as shown inFIG. 5C, the modeling unit140adjusts the coordinate value of each of the normalized modeling characteristic points P1-P3and P5-P6into a modeling coordinate value according to an adjustment factor N1. For example, the modeling unit140obtains the product of the coordinate value of each of the normalized modeling characteristic points P1-P3and P5-P6and the adjustment factor N1and uses the value of the product as the modeling coordinate value. The adjusted modeling coordinate values of the modeling characteristic points P1-P3and P5-P6are shown inFIG. 5C, but the values are only examples and such numerical values are not meant to be for limiting. In the present embodiment, the adjustment factor N1is, for example, an integer, such as an integer between 0.1 and 1.0, such as 0.6 or other values. The value of the adjustment factor N1is determined according to the resolution and/or the relative distance of the camera, and it is not limited by the embodiments of the present disclosure.

In an embodiment, the modeling unit140obtains the corresponding adjustment factor N1according to the relative distance d between the photographed person U1and the display110. The value of the adjustment factor N1could be proportional to the relative distance d. The unit of the adjustment factor N1is, for example, length/per pixel (for example, mm/pixel). When the relative distance d becomes larger, it means that the actual size (corresponding to the photographed person) represented by each pixel becomes larger, the value of the adjustment factor N1also becomes larger to reflect that the pixel size increases with the increase of the relative distance. Conversely, when the relative distance d becomes smaller, it means that the actual size (corresponding to the photographed person) represented by each pixel becomes smaller, the value of the adjustment factor N1also becomes smaller to reflect that the pixel size reduces with the reduction of the relative distance.

In addition, the aforementioned relative distance d could be detected by the camera120. The camera120is, for example, a Time of Flight (ToF) camera. Alternatively, the camera120includes at least two image sensors, and these image sensors could use conventional image processing technique to calculate the detected image signal to obtain the relative distance d. Alternatively, the camera120could obtain the relative distance d through the principle of pinhole imaging and the focal length of the camera.

In step S145, the modeling unit140stores the reference coordinate value of the reference one and the modeling coordinate value of each of the other modeling characteristic points P1-P3and P5-P6of the modeling characteristic points P1-P6as the customized 3D face model CF exclusive to the photographed person U1of the to-be-analyzed facial image MF1. The customized 3D face model CF could be stored in database B (shown inFIG. 1).

The process of obtaining the gaze projection point (step S130) will be described below.

The process of obtaining the gaze projection point could be implemented by various implementations, one of which is described below with reference to step S130inFIG. 4.

In step S131, the analysis unit150obtains the region of interest (ROI) of the face to-be-analyzed image MF1, for example, the eye region image MF1eofFIG. 2.

In step S132, as shown inFIG. 2, the analysis unit150obtains the eye camera coordinate value of at least one eye characteristic point of the eye region image MF1e. The eye characteristic points are, for example, the pupil center point and the eye center point, such as the pupil center point U1P71and the eye center point U1P72of the left eye and the pupil center point U1P81and the eye center point U1P82of the right eye. The eye center point here represents the center point of the eyeball surface of the eye region image MF1e. When the eye looks straight, the pupil center point is substantially coincident with the eye center point. When the eye oblique views, the pupil center point and the eye center point are staggered (that is, not coincides with each other). In addition, the pupil center point U1P71of the left eye has the coordinate value (xEP′,yEP′), the eye center point U1P72of the left eye has the coordinate value (xEC′,yEC′), and the pupil center point U1P81of the right eye has the coordinate value (xEP′,yEP′), and the eye center point U1P82of the right eye has the coordinate value (xEC′,yEC′).

In step S133, as shown inFIG. 2, the analysis unit150obtains the head posture information of the head of the photographed person U1according to the customized 3D face model CF (shown inFIG. 1) and the to-be-analyzed facial image MF1exclusive to the photographed person U1, wherein the head posture information includes, for example, the up-and-down tilt angle ωxand the left-and-right tilt angle ωyof the head of the photographed person U1ofFIG. 3A, wherein the up-and-down tilt angle ωxis an included angle of the head relative to a horizontal axis (for example, axis Xs), or a rotation angle of the head around the axis Xs, and the left-and-right tilt angle ωyis an included angle of the head relative to a vertical axis (for example, axis Ys), or a rotation angle of the head around the axis Ys. Furthermore, when the relative positions between the six characteristic points U1P1to U1P6of the to-be-analyzed facial image MF1and the six modeling characteristic points P1to P6of the corresponding customized 3D face model CF change, it means that the head posture of the photographed person U1changes. The analysis unit150analyzes the difference by using the analysis technology of 3D head match based on the customized 3D face model, and obtains the head posture information of the head of the photographed person U1. However, as long as the up-and-down tilt angle ωxand the left-and-right tilt angle ωycould be obtained, the embodiment of the present disclosure does not limit the adopted algorithm and/or image processing technology.

In step S134, the analysis unit150converts the eye camera coordinate values of the eye characteristic points into the eye frame coordinate values, wherein the eye frame coordinate values refer to the display frame coordinate system Xs-Ys-Zs (the display frame coordinate system Xs-Ys-Zs is shown inFIG. 3A). Furthermore, the analysis unit150could convert the eye camera coordinate values of the eye characteristic points into the eye frame coordinate values by using the following formulas (1) and (2), wherein the eye frame coordinate values refer to the display frame coordinate system Xs-Ys-Zs, so that the coordinate system calculated by the following formulas (3) to (4B) could be referred to the display frame coordinate system Xs-Ys-Zs.

Step S134could include the following steps S1341and S1342.

In step S1341, the analysis unit150obtains (or generates) a correction matrix H corresponding to the relative distance d according to the relative distance d between the photographed person U1and the camera120(the method for obtaining the correction matrix H will be described later). The correction matrix H is a coordinate transformation matrix of the camera coordinate system Xc-Yc-Zc and the display frame coordinate system Xs-Ys-Zs, and it could be obtained by using any suitable mathematical method (for example, vector coordinate transformation technology). The values of the elements h11to h33of the correction matrix H depend on the relative distance d, and the values of the elements h11to h33are not constant (non-fixed value).

In step S1342, the analysis unit150converts the eye camera coordinate values of the eye characteristic points into the eye frame coordinate values according to the correction matrix H. For example, after the correction matrix H is determined (or obtained), the eye camera coordinate value (xEP′,yEP′) of the pupil center point is substituted into formula (1) to obtain the eye frame coordinate value (xEP,yEP) of the pupil center point, and the eye camera coordinate value (xEC′,yEC′) of the eye center point is substituted into formula (2) to obtain the eye frame coordinate value (xEC,yEC) of the eye center point. The left eye and the right eye respectively have the eye frame coordinate value of the pupil center point and the eye frame coordinate value of the eye center point. Furthermore, the analysis unit150substitutes the eye camera coordinate value of the pupil center point U1P71and the eye camera coordinate value of the eye center point U1P72of the left eye into the following formulas (1) and (2) respectively, to obtain the corresponding eye frame coordinate values, and substitutes the eye camera coordinate value of the pupil center point U1P81and the eye camera coordinate value of the eye center point U1P82of the right eye into the following formulas (1) and (2) respectively, to obtain the corresponding eye frame coordinate values.

In addition, the eye frame coordinate value of the pupil center point could be written as (xEP,yEP,zEP), wherein xEPand yEPare obtained from formula (1), and zEPis the value of the relative distance d, and the eye frame coordinate value of the eye center point could be written as (xEC,yEC,zEC), wherein xECand yECare obtained from formula (2), and zECis the value of the relative distance d.

In step S135, the analysis unit150obtains an eyeball center point coordinate value (xEB,yEB,zEB) of the photographed person U1according to the eye frame coordinate value, the head posture information and the eyeball radius REB, as shown in the following formula (3). The eyeball radius REBof general adult eye is, for example, about 12 mm; however, such exemplification is not meant to be for limiting. The positive and negative values of formula (3) are defined by the swinging direction of the head. The eyeball center point coordinate value (xEB,yEB,zEB) is referenced to the display frame coordinate system Xs-Ys-Zs (the display frame coordinate system Xs-Ys-Zs is shown inFIGS. 3A to 3C).
(xEB,yEB,zEB)=(xEC±|REBcos ωxsin ωy|,yEC±|REBcos ωysin ωx|,REBcos ωxcos ωy+zEC)  (3)

In step S136, the analysis unit150obtains a gaze projection point coordinate value (xIP,yIP) of the photographed person U1according to the eye frame coordinate value (xEP,yEP,zEP) and the eyeball center point coordinate value (xEB,yEB,zEB). The gaze projection point coordinate value (xIP,yIP) is the coordinate value of the coordinate point where he photographed person's gaze is projected on a display surface110s(z=0). As shown inFIG. 3B, VgRis the gaze projection point of the gaze VLRof the right eye, and VgLis the gaze projection point of the gaze VLLof the left eye, wherein the gaze is the line connecting the center point of the pupil and the center point of the eyeball.

For example, the analysis unit150could substitute the eye frame coordinate values (xEP,yEP,zEP) and the eyeball center point coordinate value (xEB,yEB,zEB) of the pupil center point into the following formulas (4A) and (4B). The left eye and the right eye each has a corresponding group (xIP,yIP), the gaze projection point coordinate value VgRof the right eye is written as (xIP,yIP)R, and the gaze projection point coordinate value VgLof the left eye is written as (xIP,yIP)L.

The analysis unit150could obtain one gaze projection point coordinate value Vg(shown inFIG. 3B) according to the gaze projection point coordinate value VgR(xIP,yIP)Rof the right eye and the gaze projection point coordinate value VgL(xIP,yIP)Lof the left eye. For example, the analysis unit150could obtain one gaze projection point coordinate value Vgby taking an average value of the gaze projection point coordinate value VgR(xIP,yIP)Rand the gaze projection point coordinate value VgL(xIP,yIP)L.

Then, the display110could display the gaze projection point coordinate value Vg.

The following is an example to describe the method of obtaining the correction matrix H. Before obtaining the correction matrix H, a number of initial coordinate values of a number of object camera coordinate points DC1to DC4and a number of initial coordinate values of a number of object display coordinate points DS1to DS4are obtained first.

The following describes the process of obtaining the initial coordinate values of the object camera coordinate points and the initial coordinate values of the object display coordinate points.

Referring toFIGS. 6 to 7D.FIG. 6is a flow chart of obtaining the initial coordinate values of the object camera coordinate points and the initial coordinate values of the object display coordinate points according to an embodiment of the present disclosure,FIG. 7Ais a schematic diagram of a correction object D according to an embodiment of the present disclosure,FIG. 7Bis a schematic diagram of a relative position of the correction object D and the camera120ofFIG. 7A,FIG. 7Cis a schematic diagram of an object camera image M1′ captured by the camera120ofFIG. 1, andFIG. 7Dis a schematic diagram of the correction object D ofFIG. 7Arelative to the display surface110sof the display110.

In step S210, as shown inFIG. 7B, when the correction object D is separated from the camera120by a correction distance ds, the camera120captures the object camera image M1′ (the object camera image M1′ is shown inFIG. 7C) of the correction object D. As shown inFIG. 7A, the correction object D has a known length W1and width W2, wherein the length W1could be equal to or smaller than a resolution (which is the difference between a boundary point S2and a boundary point S1of the display surface110sofFIG. 7D) of the display110in length, and the width W2could be equal to or smaller than a resolution (which is the difference between a boundary point S4and a boundary point S1of the display surface110sofFIG. 7D) of the display110in width. As shown inFIG. 7C, the object camera image M1′ has a number of boundary points C1to C4. The boundary points C1to C4are, for example, the four corners of the object camera image M1′, wherein the boundary point C1has a coordinate value (C1x, C1y), the boundary point C2has a coordinate value (C2x, C2y), the boundary point C3has a coordinate value (C3x, C3y), and the boundary point C4has coordinate values (C4x, C4y). In an embodiment, the analysis unit150uses one of the boundary points as the origin, for example, the boundary point C1as the origin, and the coordinate of the origin could be defined as (0, 0).

In step S220, the analysis unit150obtains several object camera coordinate points DC1to DC4of the correction object D in the object camera image M1′ from the camera120, wherein the object camera coordinate points DC1to DC4are, for example, the four corners of the correction object image MDC in the object camera image M1′. The object camera coordinate point DC1has the initial coordinate value (DC1x, DC1y), the object camera coordinate point DC2has the initial coordinate value (DC2x, DC2y), the object camera coordinate point DC3has the initial coordinate value (DC3x, DC3y), and the object camera coordinate point DC4has the initial coordinate value (DC4x, DC4y). The initial coordinate values of the object camera coordinate points DC1to DC4are values relative to the origin (for example, the boundary point C1) of the object camera image M1′.

Since the resolution of the camera120is known (or given), the coordinate values (for example, (C1x, C1y), (C2x, C2y), (C3x, C3y) and (C4x, C4y)) of the boundary points C1to C4are also known. Since the dimensions (for example, the length W1and the width W2) of the correction object D are known, the initial coordinate values (for example, (DC1x, DC1y), (DC2x, DC2y), (DC3x, DC3y) and (DC4x, DC4y)) of the object camera coordinate point DC1to DC4are also known.

In step S230, as shown inFIG. 7D, the analysis unit150obtains several object display coordinate points DS1to DS4of the correction object D relative to the display surface110sof the display110.

As shown inFIG. 7D, the display surface110shas a number of the boundary points S1to S4, wherein the boundary points S1to S4are, for example, four corners of the display surface110s, the boundary point S1has the coordinate value (S1x, S1y), and the boundary point S2has the coordinate value (S2x, S2y), the boundary point S3has the coordinate value (S3x, S3y), and the boundary point S4has the coordinate value (S4x, S4y). In an embodiment, the analysis unit150uses one of the boundary points as the origin, for example, the boundary point S1as the origin, and the origin is defined as (0, 0).

Since the resolution of the display110is known, the coordinate values (for example, (S1x, S1y), (S2x, S2y), (S3x, S3y) and (S4x, S4y)) of the boundary points S1to S4are also known. Since the dimensions of the correction object D (for example, the length W1and the width W2) are known, the initial coordinate values (for example, (DC1x, DC1y), (DC2x, DC2y), (DC3x, DC3y) and (DC4x, DC4y)) of the object camera coordinate points DC1to DC4are also known.

When the correction distance ds changes, the coordinate values of the object camera coordinate points DC1to DC4also change accordingly. When the correction object D becomes farther from the camera120(the correction distance ds inFIG. 7Bbecomes larger), for the object camera image M1′ ofFIG. 7C, the correction object image MDC becomes smaller, and the correction coordinate values of the object camera coordinate points DC1to DC4change accordingly. When the correction object D becomes closer to the camera120(the correction distance ds ofFIG. 7Bbecomes smaller), the correction object image MDC of the object camera image M1′ ofFIG. 7becomes larger, and the correction coordinate values of the object camera coordinate points DC1to DC4change accordingly.

In the present embodiment, the analysis unit150could obtain, under the first distance d1(the correction distance), the initial coordinate values of the object camera coordinate points (first object camera coordinate points) DC1to DC4by using the aforementioned method, and the object camera coordinate points are respectively represented as (DC1x,d1, DC1y,d1), (DC2x,d1, DC2y,d1), (DC3x,d1, DC3y,d1) and (DC4x,d1, DC4y,d1), and obtain, under the second distance d2(the correction distance), the initial coordinate values of the object camera coordinate points (second object camera coordinate point) DC1to DC4, and the object camera coordinate points are respectively represented as (DC1x,d2, DC1y,d2), (DC2x,d2, DC2y,d2), (DC3x,d2, DC3y,d2) and (DC4x,d2, DC4y,d2).

The initial coordinate values (DC1x,d1, DC1y,d1), (DC2x,d1, DC2y,d1), (DC3x,d1, DC3y,d1) and (DC4x,d1, DC4y,d1) of the object camera coordinate points DC1to DC4under the first distance, the initial coordinate values (DC1x,d2, DC1y,d2), (DC2x,d2, DC2y,d2), (DC3x,d2, DC3y,d2) and (DC4x,d2, DC4y,d2) of the object camera coordinate points DC1to DC4under the second distance and the initial coordinate values (DS1x, DS1y), (DS2x, DS2y), (DS3x, DS3y) and (DS4x, DS4y) of the object display coordinate points could be stored in the analysis unit150or another storage unit (not shown) for the generation of the correction matrix H.

The procedure for obtaining the correction matrix H will be described below.

Referring toFIG. 8.FIG. 8is a flowchart of obtaining the correction matrix (or “geometric correction matrix”) H in step S134ofFIG. 4.

In step S134A, the camera120or other distance measuring (or ranging) device obtains the relative distance d between the photographed person U1and the camera120(the relative distance d is shown inFIG. 3A).

In step S134B, the analysis unit150determines whether the relative distance d is smaller than the first distance d1. If the relative distance d is smaller than the first distance d1, the process goes to step S134C; if not, the process proceeds to step S134D. In step S134D, the analysis unit150determines whether the relative distance d is greater than the second distance d2. If the relative distance d is greater than the second distance d2, the process proceeds to step S134E; if not, the process proceeds to step S134F.

In step S134F, the analysis unit150obtains the correction matrix H according to the initial coordinate values of the object display coordinate points DS1to DS4and the object camera coordinate point DC1to DC4under the relative distance d.

The values of the elements h11to h33of the correction matrix H in the formula (6) depend on the initial coordinate values of the object display coordinate points DS1to DS4and the correction coordinate values of the object camera coordinate point DC1to DC4under the relative distance d, not constant (non-fixed value). In the formula (6), Cx and Cy represent the coordinate values of any pixel of the to-be-analyzed image M1(the to-be-analyzed image M1is shown inFIG. 2). The coordinate value (Cx, Cy) of any pixel of the to-be-analyzed image M1could be converted into the coordinate value (Sx, Sy) on the to-be-analyzed display image M2through the correction matrix H.

An effective working area is provided between the second distance d2and the first distance d1. Within this effective working area, the eye tracking device could generate a corresponding correction matrix according to the distance between the user and the camera, so as to increase the precision and the accuracy of eye tracking. In an embodiment, when the relative distance d is between the first distance d1and the second distance d2, the analysis unit150could obtain the correction coordinate values of the object camera coordinate points DC1to DC4under the relative distance d by using, for example, an interpolation method or an extrapolation method. For example, the Xs-axis coordinate value x_DCnand the Ys-axis coordinate value y_DCnof the correction coordinate values (x_DCn,y_DCn) of the object camera coordinate points DC1to DC4could be obtained through the following formulas (7A) and (7B) respectively. The subscript “n” represents different object camera coordinate point. For example, the subscript “1” represents the object camera coordinate point DC1, the subscript “2” represents the object camera coordinate point DC2, and so on.

After obtaining the correction coordinate values (x_DCn,y_DCn) of the object camera coordinate point DC1to DC4, the analysis unit150obtains the correction matrix H corresponding to the relative distance d according to the corrected coordinate values (x_DCn,y_DCn) of the object camera coordinate point DC1to DC4and the initial coordinate values of the object display coordinate points DS1to DS4. The correction matrix H could be obtained by using the technique including, for example, a suitable mathematical method or a software function; however, such exemplification is not meant to be for limiting. In an embodiment, the correction matrix H could be obtained, for example, by using the function cv.findHomography(ptc, pts), wherein the parameter ptc represents the correction coordinate values (x_DCn,y_DCn) of the object camera coordinate points DC1to DC4, and the parameter pts represents the initial coordinate values of the object display coordinate points DS1to DS4. The function cv.findHomography could obtain or generate the transformation matrix between two planes, but as long as the transformation matrix between the two planes could be obtained, it is not limited to using the function cv.findHomography.

In step S134C, due to the relative distanced being smaller than the first distance d1, regardless of the difference between the relative distance d and the first distance d1, the analysis unit150sets the value of the relative distance d as the value of the first distance d1and obtains (or generates) the correction matrix H by using the method same as or similar to step S134F. In step S134E, due to the relative distance d being greater than the second distance d2, regardless of the difference between the relative distance d and the second distance d2, the analysis unit150sets the value of the relative distance d as the value of the second distance d2, and obtains (or generates) the correction matrix H by using the method same as or similar to step S134F.

As shown in the above formula (6), the coordinate value (Cx, Cy) of any pixel of the to-be-analyzed image M1(shown inFIG. 2) is converted into the coordinate value (Sx, Sy) on the to-be-analyzed display image M2through the correction matrix H, and thus the to-be-analyzed facial image presented by the to-be-analyzed display image M2(shown inFIG. 3C) of the display110does not change with the change of the relative distance d. Furthermore, even if the relative distance d increases or decreases, the to-be-analyzed facial image in the display image M2displayed on the display110could remain substantially unchanged.

In addition, based on the property of the camera120, the to-be-analyzed image M1(shown inFIG. 2) is inevitably deformed (for example, a trapezoid with a narrow top and a wide bottom). However, through the correction of the correction matrix H in the disclosed embodiment, the four coordinate points DC5to DC8(the coordinate points DC5to DC8are shown inFIG. 2) of a region R1(the region R1is shown inFIG. 2) of the deformed to-be-analyzed image M1could be converted to the four coordinate points DS5to DS8(the coordinate points DS5to DS8are shown inFIG. 3B) of a region R2(the region R2is shown inFIG. 3C) of the to-be-analyzed display image M2, wherein the region R1could be a trapezoid, and the region R2could be a rectangle. The coordinate points DC5to DC8are, for example, the four corners of the region R1, and the coordinate points DS5to DS8are, for example, the four corners of the region R2. As long as an appropriate position of the region R1is selected, the corresponding region R2could be the size of the entire display surface of the display110. As shown inFIGS. 2 and 3C, in the present embodiment, the area R1is smaller than the area R2, and thus another photographer located in the area other than the area R1of the to-be-analyzed image M1will not be displayed on the to-be-analyzed display image M2. However, even if another photographer located in the area other than the area R1of the to-be-analyzed image M1will not be displayed on the to-be-analyzed display image M2, the gaze tracking device100could still analyze the to-be-analyzed facial image outside the area R1by using the method same as or similar to that of the to-be-analyzed facial image MF1, and it will not be repeated here.

Tables 1 to 4 below are the property comparison table showing the precision and the accuracy for the head swing of the tested subject (dummy) based on the eye tracking device100using the customized 3D face model in the present disclosure and the eye tracking device using the conventional 3D face model. The better the precision and accuracy performance is, the more accurate and precise the gaze projection point Vgis.

As shown in Table 1 above, according to the test results on the premise of not wearing the mask, in comparison to using the conventional 3D face model, the gaze tracking device100of the embodiment of the present disclosure has the higher individual accuracy in the left-and-right direction, the higher individual accuracy in the up-and-down direction and the higher Integration accuracy based on the customized 3D face model.

As shown in Table 2 above, according to the test results on the premise of wearing the mask, in comparison to using the conventional 3D face model, the gaze tracking device100of the embodiment of the present disclosure has the higher individual accuracy in the left-and-right direction, the higher individual accuracy in the up-and-down direction and the higher Integration accuracy based on the customized 3D face model.

As shown in Table 3 above, according to the results of the accuracy test (the accuracy detected under the test dummy moving up and down by 1 degree and swinging by 3 degrees left and right), in comparison to using the conventional 3D face model, the gaze tracking device100of the embodiment of the present disclosure has the higher accuracy in middle point, the left point, the right point, the up point (upper point) and the down point (lower point), wherein the left point, the right point, the up point and the down point are respectively relative to a center point, and the center point does not limit the position in the dummy's face.

As shown in Table 4 above, according to the results of the precision test (the precision detected under the test dummy moving up and down by 1 degree and swinging by 3 degrees left and right), in comparison to using the conventional 3D face model, the gaze tracking device100of the embodiment of the present disclosure has the higher precision in middle point, the left point, the right point, the up point (upper point) and the down point (lower point), wherein the left point, the right point, the up point and the down point are respectively relative to a center point, and the center point does not limit the position in the dummy's face.

To sum up, the embodiments of the present disclosure provide a gaze tracking device and a gaze tracking method thereof. The gaze tracking device could determine whether the photographed person has created the customized 3D face model exclusive to the photographed person. If so, the photographed person's gaze is tracked based on the customized 3D face model. If not, the customized 3D face model exclusive to the photographed person is established. Due to the established customized 3D face model being exclusive to the photographed person, the gaze tracking device could obtain the accurate and precise head posture information of the photographed person for obtaining the accurate and precise gaze projection point.