Office Action 3: Notice of Allowance (2024-09-11)

CLAIMS:
AMENDMENTS TO THE CLAIMS

This listing of claims will replace all prior versions, and listings of claims in the Application.

SVG {EMBED HERE} 0.110.0431.(Currently amended) An eye-gaze tracking apparatus, comprising:
a plurality of foreground cameras each arranged to capture at least a portion of a target view area of a user; andan eye-gazSVG {EMBED HERE} 0.080.14 estimation module two near-eye infrared viewing cameras each arranged to capture an image an eye of the user, so as to estimate a position of a pupil of the eye based on the image of the eye;
and a calibration module;wherein the plurality of foreground cameras comprise a wide-angle camera having a 1000 diagonal FOV covering an entire target area to estimate a wider view of the user and a magnified camera having a 600 diagonal FOV associated with a "focus" view or within a specific range or portion within entire the target area to obtain finer foreground images of the specific area;
wherein the foreground cameras are adapted to capture images with a same sampling frequency and a same maximum resolution of the captured images;
wherein the calibration module is adapted to carry out the steps of:arranging four markers to be set in fixed positions of the foreground and be distributed in four corners of a matching area,
obtaining a rectangular area consisted of the four markers both in images obtained from the wide-angle camera and the magnified camera which are matching areas
obtaining a gaze estimation position in images firstly by the magnified camera and then obtaining the matching gaze estimation position in images of the wide-angle camera according to a relative position of the gaze estimation position in a rectangle formed by the four markers;
wherein the wide-angle camera and the magnified camera are matched by mapping a gaze estimation position of the magnified camera within a matching area to a foreground image of the wide-angle camera in a calibration; and
wherein the foreground cameras and the eye-gaze cameras are adapted to capture images of both the target view of a user as well as the images of the eyes of the user simultaneously to determine eye-gaze positions of the user.

2\. (Cancelled).

3\. (Currently amended) The eye-gaze tracking apparatus in accordance with claim \[\[2\]\]1, wherein the plurality of foreground cameras are adapted to capture a video stream of the users FOV at 30Hz with a highest image resolution of 1920\*1080 pixel, and the two near-eye infrared viewing cameras are adapted to capture user's eye images at 200\*200 pixels at 200Hz.

4\. (Currently amended) The eye-gaze tracking apparatus in accordance with claim \[\[1\]\]3, wherein each of the near-eye infrared viewing camera has a package size of 10\*45\*7 mm and are adapted to capture images at 860 nm wavelength.

5\. (Original) The eye-gaze tracking apparatus in accordance with claim 4, further comprising at least one infrared LED arranged to illuminate the eye of the user.

6\. (Currently amended) The eye-gaze tracking apparatus in accordance with claim \[\[1\]\]5, further comprising \[\[a\]\]the calibration module arranged to facilitate a
calibration of the eye-gaze estimation module according to characteristics of the eye of the user.

7\. (Original) The eye-gaze tracking apparatus in accordance with claim 6, wherein the calibration module is arranged to provide a plurality of displayed markers such that the eye-gaze estimation module is arranged to map a plurality of calibration positions of the pupil of the eye to the plurality of displayed markers overlaid at different positions in the target view area of a user.

8\. (Original) The eye-gaze tracking apparatus in accordance with claim 6, wherein the calibration module includes a plurality of physical markers such that the eye-gaze estimation module is arranged to map a plurality of calibration positions of the pupil of the eye to the plurality of physical markers disposed at different positions in the target view area of the user.

9\. (Previously amended) The eye-gaze tracking apparatus in accordance with (Previously amended) The eye-gaze tracking apparatus in accordance with wherein the calibration module is arranged to facilitate calibration of the eye-gaze estimation module with respect to different target view areas of a user associated with different FOVs of the plurality of foreground cameras, and using a combined gaze estimation method based on convolutional neural network and a geometric model to get an accurate gaze position estimation.

10\. (Original) The eye-gaze tracking apparatus in accordance with claim 1, wherein the eye-gaze estimation module includes a neural network arranged to process the image of the eye captured by the at least one eye-gaze camera so as to estimate an eye-gaze angle of the user associated with the position of the pupil of the eye.

11\. (Currently amended) A method of eye-gaze tracking, comprising the steps of:carrying out a calibration method by a calibration module;
capturing at least a portion of a target view area of a user using a plurality of foreground cameras; and

capturing an image \[\[an\]\]eacheye of the user using a near-eye infrared viewing cameras, so as to estimate a position of a pupil of the eye based on the image of the eye;
 wherein the plurality of foreground cameras comprise awide-angle camera having a 1000 diagonal FOV covering an entire target area to estimate a wider view of the user and a magnified camera having a 60° diagonal FOV associated with a "focus" view or within a specific range or portion in the target area to obtain finer foreground images of the specific area;
wherein the foreground cameras are adapted to capture images with a same sampling frequency and a same maximum resolution of the captured images;
wherein the calibration method comprises the steps of:arranging four markers to be set in fixed positions of the foreground and be distributed in four corners of a matching area,
obtaining a rectangular area consisted of the four markers both in images obtained from the wide-angle camera and the magnified camera which are matching areas obtaining a gaze estimation position in images firstly by the magnified camera and then obtaining the matching gaze estimation position in images of the wide-angle camera according to a relative position of the gaze estimation position in a rectangle formed by the four markers;
wherein the wide-angle camera and the magnified camera are matched by mapping a gaze estimation position of the magnified camera within a matching area to a foreground image of the wide-angle camera in a calibration; and
wherein the foreground cameras and the eye-gaze cameras are adapted to capture images of both the target view of a user as well as the images of the eyes of the user simultaneously to determine eye-gaze positions of the user.

12\. (Cancelled).

13\. (Currently amended) The method of claim 1 \[\[2\]\]1, wherein the plurality of foreground cameras are adapted to capture a video stream of the users FOV at 30Hz with a highest image resolution of 1920\*1080 pixel, and the two near-eye infrared viewing cameras are adapted to capture user's eye images at 200\*200 pixels at 200Hz.

14\. (Currently amended) The method of claim 1 \[\[1\]\]3, wherein each of the near-eye infrared viewing camera has a package size of 10\*45\*7 mm and are adapted to capture images at 860 nm wavelength.

15\. (Previously amended) The method of claim 14, further comprising the step of illuminating the eye of the user using at least one infrared LED.

16\. (Original) The method of claim 11, further comprising the step of calibrating an eye-gaze estimation module according to characteristics of the eye of the user, wherein the eye-gaze estimation module is arranged to estimate the position of the pupil of the eye.

17\. (Original) The method of claim 16, wherein the step of calibrating the eye-gaze estimation module comprises the steps of:providing a plurality of displayed markers overlaid at different positions in the target view area of a user; and
mapping a plurality of calibration positions of the pupil of the eye to the plurality of displayed markers.

18\. (Original) The method of claim 16, wherein the step of calibrating the eye-gaze estimation module comprises the steps of:disposing a plurality of physical markers at different positions in the target view area of the user; and
mapping a plurality of calibration positions of the pupil of the eye to the plurality of physical markers.

19\. (Previously amended) The method of claim 16, wherein the eye-gaze estimation module is calibrated with respect to different target view areas of a user associated with different FOVs of the plurality of foreground cameras and using a combined gaze estimation method based on convolutional neural network and a geometric model to get an accurate gaze position estimation.

20\. (Original) The method of claim 11, further comprising the step of processing the image of the eye captured by the at least one eye-gaze camera using a neural network, so as to estimate an eye-gaze angle of the user associated with the position of the pupil of the eye.

OFFICE ACTION:
DETAILED ACTION
Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection.  Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114.  Applicant's submission filed on 08/15/2024 has been entered.
EXAMINER’S AMENDMENT
An examiner’s amendment to the record appears below. Should the changes and/or additions be unacceptable to applicant, an amendment may be filed as provided by 37 CFR 1.312. To ensure consideration of such an amendment, it MUST be submitted no later than the payment of the issue fee.
Authorization for this examiner’s amendment was given in an interview with Ms. Kimberly Peaslee (64,311) on 08/27/2024.
The application has been amended as follows:
(Currently amended) An eye-gaze tracking apparatus, comprising:
a plurality of foreground cameras each arranged to capture at least a portion of a target view area of a user; and
an eye-gaze estimation module comprising two near-eye infrared viewing cameras each arranged to capture an image of an eye of the user, each estimate a position of a pupil of the eye based on the image of the eye of the user; and
a calibration module;
wherein the plurality of foreground cameras comprises a wide-angle camera having a 100° diagonal FOV covering an entire target area to estimate a wider view of the user and a magnified camera having a 60° diagonal FOV to provide comprising a portion of the entire
wherein the plurality of foreground cameras provide captured images with a same sampling frequency and a same maximum resolution
wherein the calibration module is adapted to carry out the steps of:
arranging four markers a foreground such that they create ;
obtaining a rectangular area marked by the four markers the captured images obtained from the wide-angle camera and the magnified camera which are the matching area[[s]];
obtaining a gaze estimation position in images captured by the magnified camera and then obtaining a matching gaze estimation position in images captured by s in [[a]]the rectangle formed by the four markers;
wherein the wide-angle camera and the magnified camera are matched by mapping [[a]]the gaze estimation position of the magnified camera within the[[a]] matching area to a foreground image of the wide-angle camera in a calibration; and
wherein the foreground cameras and the eye-gaze cameras are adapted to capture images of both the target view of a user and images of the eyes of the user simultaneously to determine eye-gaze positions of the user.
(Cancelled).
(Currently Amended) The eye-gaze tracking apparatus in accordance with claim 1, wherein the plurality of foreground cameras are adapted to capture a video stream of the user’s FOV at 30Hz with a highest image resolution of 1920*1080 pixel, and the two near-eye infrared viewing cameras are adapted to capture user’s eye images at 200*200 pixels at 200Hz.
(Currently amended) The eye-gaze tracking apparatus in accordance with claim 3, wherein each of the near-eye infrared viewing cameras has a package size of 10*45*7 mm and each are adapted to capture images at 860 nm wavelength.
(Original) The eye-gaze tracking apparatus in accordance with claim 4, further comprising at least one infrared LED arranged to illuminate the eye of the user.
(Previously amended) The eye-gaze tracking apparatus in accordance with claim 5, further comprising the calibration module arranged to facilitate a calibration of the eye-gaze estimation module according to characteristics of the eye of the user.
(Currently Amended) The eye-gaze tracking apparatus in accordance with claim 6, wherein the calibration module is arranged to provide a plurality of displayed markers such that the eye-gaze estimation module is arranged to map a plurality of calibration positions of the pupil of the eye of the user to the plurality of displayed markers overlaid at different positions in the target view area of a user.
(Currently Amended) The eye-gaze tracking apparatus in accordance with claim 6, wherein the calibration module includes a plurality of physical markers such that the eye-gaze estimation module is arranged to map a plurality of calibration positions of the pupil of the eye of the user to the plurality of physical markers disposed at different positions in the target view area of the user.
(Previously amended) The eye-gaze tracking apparatus in accordance with claim 6, wherein the calibration module is arranged to facilitate calibration of the eye-gaze estimation module with respect to different target view areas of a user associated with different FOVs of the plurality of foreground cameras, and using a combined gaze estimation method based on convolutional neural network and a geometric model to get an accurate gaze position estimation.
(Currently Amended) The eye-gaze tracking apparatus in accordance with claim 1, wherein the eye-gaze estimation module includes a neural network arranged to process the image of the eye captured by the two eye-gaze cameras
(Currently amended) A method of eye-gaze tracking, comprising the steps of:
carrying out a calibration method by a calibration module;
capturing at least a portion of a target view area of a user using a plurality of foreground cameras; and
capturing an image of each eye of the user using a respective near-eye infrared viewing camera[[s]], of the user based on the image of the eye;
wherein the plurality of foreground cameras comprises a wide-angle camera having a 100° diagonal FOV covering an entire target area to estimate a wider view of the user and a magnified camera having a 60° diagonal FOV associated with a “focus view comprising a portion of [[in]] the entire target area
wherein the plurality of foreground cameras are adapted to capture images with a same sampling frequency and a same maximum resolution
wherein the calibration method comprises the steps of:
arranging four markers a foreground such that they create four corners of a matching area[[,]];
obtaining a rectangular area marked by the four markers the matching area[[s]];
obtaining a gaze estimation position in images captured by the magnified camera and then obtaining a[[the]] matching gaze estimation position in images captured by [[of]] the wide-angle camera according to [[a]] relative positions of the gaze estimation position in [[a]]the rectangle formed by the four markers;
wherein the wide-angle camera and the magnified camera are matched by mapping [[a]]the gaze estimation position of the magnified camera within a matching area to a foreground image of the wide-angle camera in a calibration; and
wherein the foreground cameras and the eye-gaze cameras are adapted to capture images of both the target view of a user and the images of the eyes of the user simultaneously to determine eye-gaze positions of the user.
(Cancelled).
(Currently amended) The method of claim 11, wherein the plurality of foreground cameras are adapted to capture a video stream of the user’s FOV at 30Hz with a highest image resolution of 1920*1080 pixel, and the two near-eye infrared viewing cameras are adapted to capture user’s eye images at 200*200 pixels at 200Hz.
(Currently amended) The method of claim 13, wherein each of the two near-eye infrared viewing cameras has a package size of 10*45*7 mm and are adapted to capture images at 860 nm wavelength.
(Previously amended) The method of claim 14, further comprising the step of illuminating the eye of the user using at least one infrared LED.
(Original) The method of claim 11, further comprising the step of calibrating an eye-gaze estimation module according to characteristics of the eye of the user, wherein the eye-gaze estimation module is arranged to estimate the position of the pupil of the eye.
(Original) The method of claim 16, wherein the step of calibrating the eye-gaze estimation module comprises the steps of:
providing a plurality of displayed markers overlaid at different positions in the target view area of a user; and
mapping a plurality of calibration positions of the pupil of the eye to the plurality of displayed markers.
(Original) The method of claim 16, wherein the step of calibrating the eye-gaze estimation module comprises the steps of:
disposing a plurality of physical markers at different positions in the target view area of the user; and
mapping a plurality of calibration positions of the pupil of the eye to the plurality of physical markers.
(Previously amended) The method of claim 16, wherein the eye-gaze estimation module is calibrated with respect to different target view areas of a user associated with different FOVs of the plurality of foreground cameras and using a combined gaze estimation method based on convolutional neural network and a geometric model to get an accurate gaze position estimation.
(Currently Amended) The method of claim 11, further comprising the step of processing the image of the eye captured by the two eye-gaze cameras using a neural network,
Allowable Subject Matter
Claims 1, 3-11, and 13-20 are allowed.
The following is an examiner’s statement of reasons for allowance: The best prior art of record, i.e., Sztuk et al (Publication number: US 2023/0119935) in view of Jacques (Patent number; US 9,185,352) do not specifically show the limitations of “arranging four markers to be set in fixed positions of the a foreground such that they create and be distributed in four corners of a matching area, obtaining a rectangular area consisted of marked by the four markers both in the captured images obtained from the wide-angle camera and the magnified camera which are the matching area; obtaining a gaze estimation position in images firstly captured by the magnified camera and then obtaining the a matching gaze estimation position in images captured by of the wide-angle camera according to a relative position of the gaze estimation positions in the rectangle formed by the four markers; wherein the wide-angle camera and the magnified camera are matched by mapping the gaze estimation position of the magnified camera within the matching area to a foreground image of the wide-angle camera in a calibration; and wherein the foreground cameras and the eye-gaze cameras are adapted to capture images of both the target view of a user as well as the and images of the eyes of the user simultaneously to determine eye-gaze positions of the user.”
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee.  Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”
Conclusion

APPLICANT REBUTTAL: REMARKS
This amendment is in response to the Final Office Action dated May 16, 2024. To further the prosecution of this Application, Applicants submit the following remarks and amended claims. Additionally, Applicant submits a Request for Continued Examination. The claims as now presented are believed to be in allowable condition.
In the Claims:
Please amend claims 1, 3, 4, 6, 11, 13 and 14 in which the claim additions are shown by underlining and/or the claim deletions are shown by either strikeout or brackets. Please cancel claims 2 and 12. Please enter the amended claims into the record of this case.
Prior art publication date
The Applicant observed that the priority date of the current application is September 13, 2022. The publication date of the citation Sztuk et al. '953 is April 20, 2023. The applicant respectfully disagrees with the citation Sztuk et al. '953 qualifying as a printed publication before the effective filing date of the claimed Invention. Further, the Applicant objects to the combination of Jacques with the unpublished Sztuk et al for the rejections under 35 U.S.C. §103 as the combination would not be obvious before the effective filing date of the present application.
35 U.S.C. § 103 - Obviousness
The Office has rejected Claims 1, 2, 4-6, 9, 11, 12,14-16, and 19 under 35 U.S.C. 103 as being unpatentable over Sztuk et al (Publication number: US 2023/0119935) in view of Jacques (Patent number: US 9,185,352); Claims 7-8 and 17-18 under 35 U.S.C. 103 as being unpatentable over Sztuk et al in view of Jacques in view of Bastani et al (Publication number: US 2021/0166341); Claims 3 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Sztuk et al in view of Jacques in view of Trail et al (Patent number: US 10,928635); Claims 10 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Sztuk et al in view of Jacques in view of Bathiche et al (Publication number: US
2022/0221932); and Claims 9 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Sztuk et al in view of Jacques in view of YU et al (Publication number: US 2021/0319420). The Applicant acknowledges and respectfully traverses the raised obviousness rejections in view of the above amendments and the following remarks.
Assuming, arguendo, that Sztuk et al is a proper refence, the Applicant proposes to amend Claim 1 and Claim 11 to include the feature:
"wherein the plurality of foreground cameras comprise a wide-angle camera having a 100° diagonal FOV covering an entire target area to estimate a wider view of the user and a magnified camera having a 600 diagonal FOV associated with a "focus"
view or within a specific range or portion in the target area to obtain finer
foreground images of the specific area;
wherein the foreground cameras are adapted to capture images with a same
sampling frequency and a same maximum resolution of the captured images".
Support for these amendments to Claim 1 and Claim 11 are based on paragraphs [0047] and [0048] of the description.
Further, Claim 1 and Claim 11 include the feature:
"wherein the calibration module is adapted to carry out the steps of:
arranging four markers to be set in fixed positions of the foreground and be
distributed in four corners of a matching area,
obtaining a rectangular area consisted of the four markers both in images obtained
from the wide-angle camera and the magnified camera which are matching areas
obtaining a gaze estimation position in images firstly by the magnified camera and then obtaining the matching gaze estimation position in images of the wide-angle camera according to a relative position of the gaze estimation position in rectangle
formed by the four markers;
wherein the wide-angle camera and the magnified camera are matched by mapping a gaze estimation position of the magnified camera within a matching area to a foreground image of the wide-angle camera in a calibration".
This feature of the amended Claim 1 and Claim 11 is based on paragraphs [0060] and [0064] of the description.
The Applicant further proposes to amend Claim 3 and Claim 13 to include the feature of "the plurality of foreground cameras are adapted to capture a video stream of the users FOV at 30Hz with a highest image resolution of 1920* 1080 pixel, and the two near- eye infrared viewing cameras are adapted to capture user's eye images at 200*200 pixels at 200Hz". The amendments to Claim 3 and Claim 13 are based on paragraphs [0051] and [0052] of the description.
The Applicant proposes to amend Claim 4 and Claim 14 to include the feature of "near-eye infrared viewing cameras has a package size of 10*45*7 mm and are adapted to capture images at 860 nm wavelength". The amended Claim 4 and Claim 14 are based on paragraphs [0052] of the description.
Now turning to the cited art, Jacques discloses a device with a first camera mounted to the user and the lens of the first camera is positioned in a manner that faces away from the user so that the camera collects scene information generally associated with the field of view of the user. A second camera is mounted to the user and the lens of the second camera is positioned in a manner so as to be in close proximity to and face toward an eve of the user so that the camera collects image information of the eve. The system is calibrated such that a determination is made as to the direction of the gaze of the eve based on the image information collected of the eve by the second camera. Using the scene information as input, the system determines a location of the identifying region of the field of view. When the direction of the gaze of the user's eve tracked by the second camera is within the identifying region, input may subsequently be provided by the user through movements of the eve to the computing device".
However, the Applicant cannot find anywhere in Jacques that teaches, suggests or even motivates a person skilled in the art that the foreground cameras and the eye-gaze cameras are adapted to capture images of both the target view of a user as well as the images of the eyes of the user
simultaneously
to determine eye-gaze positions of the user.
The Applicant direct the Occie to paragraph [0048] of Sztuk et al which states that
"the FOVs can be rearranged to lap or overlap by moving the orientation of a camera or widening or narrowing the FOV by adjusting a lens assembly of the
camera"
A plurality of adjustable cameras is different from having a wide-angle camera and a magnified camera.
Sztuk et al does not teach or direct a person skilled in the art of a combination of a wide-angle camera and a magnified camera. As such, it is impossible for Sztuk et al to teach or direct the person skilled in the art of at least the feature:
diagonal FOV covering an entire target area to estimate a wider view of the user and a magnified camera having a 60° diagonal FOV associated with a "focus" view or within a specific range or portion in the target area to obtain finer foreground
images of the specific area;
wherein the foreground cameras are adapted to capture images with a same
sampling frequency and a same maximum resolution of the captured images"
Similarly, Jacques also does not disclose or direct a person skilled in the art the above feature.
Therefore, it is impossible for Sztuk et al, Jacques or any combination of prior art to further teach or direct a person skilled in the art to the calibration method with a plurality of foreground cameras. In particular, the calibration method comprising the step of
"arranging four markers to be set in fixed positions of the foreground and be
distributed in four corners of a matching area,
obtaining a rectangular area consisted of the four markers both in images obtained
from wide-angle camera and the magnified camera which are matching areas
obtaining a gaze estimation position in images firstly by the magnified camera and then obtaining the matching gaze estimation position in images of the wide-angle camera according to a relative position of the gaze estimation position in rectangle
formed by the four markers"
It is also impossible for a person skilled in the art to derive in light of the prior art considered alone or in combination the feature of:
"the wide-angle camera and the magnified camera to be matched by mapping a gaze estimation position of magnified camera within a matching area to a
foreground image of wide-angle camera in a calibration"
Sztuk et al discloses the method of
se
l
ectin
g
a particular image sensor of camera based on the gaze vector within the FOV of a particular camera (see para [0048] of Sztuk et al). This is completely different to the calibration method and working of the present invention where the wide-angle camera and the magnified camera are matched by mapping a gaze estimation position of the magnified camera within a
matching
area to a foreground image of the wide-angle camera in a calibration.
As indicated in paragraph [0050] of the specification, the present invention is different from prior art systems which may only capture rough foreground images which are not suitable for tracking the subtle gaze movements such as writing a Traditional Chinese character in a 1.5*1.5 cm2 lattice. Advantageously, the additional foreground camera may be used to capture fine foreground images.
Sztuk et al discloses a gaze direction of an eye or eyes of a user of a head mounted device is determined. One or more images are captured with a camera of a head mounted
device. Gaze-guided images are generated from the one or more images based on the gaze direction of the user.
Bastani et al discloses a method for providing imagery to a user on a display that includes receiving eye tracking data. The method also includes determining a gaze location on the display and at least one of a confidence factor of the gaze location, or a speed of the change of the gaze location using the eye tracking data. The method also includes establishing multiple tiles using the gaze location and at least one of the confidence factor or the speed of the change of the gaze location. The method also includes providing a foveated rendered image using the multiple tiles.
The present claims are directed to the inaccuracy of the prior art eye tracking systems which may vary much, sometimes over 3 degrees, depending on the calibration and applications (see page 5 line 38 to page 6 line 1). The inventors devised an eye tracking system that may be categorized into head-mounted and desktop-based on different applications, and the system accuracy of prior art eye trackers may range from 0.5 degrees to 1 degree (page 5 lines 33 to 36).
Sztuk et al does not disclose any device for eye tracking but rather disclose a system to capture or generate images based on the gaze direction of an eye of a user that has been determined by an eye-tracking system (see paragraph 21). As such, the foreground cameras (193A, 193B, 193C, 193D) and the cameras (108A-B) of the Sztuk et al could not achieve or direct a person skilled in the art to solve the problem of the present invention. Sztuk et al has taken for granted that the eye-tracking system is accurate in order to capture or generate images based on gaze direction. On the contrary, the present invention is provided to determine the unknown eye-gaze positions in high accuracy (0.5 degrees to 1 degree error) with the input of the foreground camera 102 and the eye-gaze camera 106. Therefore, Sztuk et al does not disclose the present invention defined in Claim 1 and Claim 11. Claim 1 and Claim 11 are novel when compared to Sztuk et al.
As Sztuk et al does not direct or teach a person skilled in the art to determine the gazing direction or improve the accuracy of the gazing direction, it could not teach or direct a person skilled in the art to the feature that the foreground cameras and the eye-gaze
cameras are adapted to capture images of both the target view of a user as well as the images of the eyes of the user simultaneously to determine eye-gaze positions of the user. Therefore, Claim 1 and Claim 11 are not obvious when compared to Sztuk et al.
The function of the FOV mentioned in Sztuk et al is to capture a panoramic image viewable by the user (see figure2C; paragraph 48); (See FOV's297A-D. The FOVs can be adjusted to be narrow or wide). The function of the wide-angle camera in Claim 2 and Claim 12 is to estimate a wider view of the user and that of a magnified camera having a second FOV smaller than the first FOV to estimate a visual focus of the user. Since Sztuk et al does not disclose a wide-angle camera and a magnified camera together, it is impossible for Sztuk et al to direct a person skilled in the art to feature that a wide-angle camera having a first FOV to estimate a wider view of the user and a magnified camera having a second FOV smaller than the first FOV to estimate a visual focus of the user. Therefore, Claim 2 and Claim 12 are not obvious when compared to Sztuk et al.
Claim 9 and Claim 19 further define the feature of using a combined gaze estimation method based on convolutional neural network and a geometric model to get an accurate gaze position estimation. (See Page 8 lines 31 to 33). Neither Sztuk et al nor Bastani et al disclose to nor direct a person skilled in the art that an eye-tracking system or method has an eye-gaze estimation module that is calibrated with respect to different target view areas of a user associated with different FOVs from a wider view of the user and a magnified camera (as defined in Claim 2 and Claim 12), and using a combined gaze estimation method based on convolutional neural network and a geometric model to get the accurate gaze position estimation. Therefore, Claim 9 and Claim 19 are novel and not obvious when compared to Sztuk et al and Bastani et al either alone or in combination.
It is submitted that the invention defined in the proposed amendment claims is novel and not obvious to a person skilled in the art when compared to Sztuk et al and Bastani et al. It is therefore submitted that all claims as amended are novel and inventive. Reconsideration of allowance of the application is requested respectfully.
If any further amendment to this application is believed necessary to advance prosecution and place this case in allowable form, the Examiner is courteously solicited to contact the undersigned representative of the Applicant to discuss the same.
Conclusion


CITED REFERENCE SUMMARIES: Ref US20230119935: Claims:
1. A head mounted device comprising:
an eye-tracking system including one or more sensors configured to determine a gaze direction of an eye in an eyebox region of the head mounted device; a first image sensor configured to capture first images of an external environment of the head mounted device; a second image sensor configured to capture second images of the external environment of the head mounted device, wherein the first image sensor has a first field of view (FOV) that is different from a second FOV of the second image sensor; and processing logic configured to:
receive the gaze direction from the eye-tracking system; and
select a selected image sensor between the first image sensor and the second image sensor to capture one or more gaze-guided images, wherein the first image sensor or the second image sensor is selected to capture the one or more gaze-guided images based on the gaze direction with respect to the first FOV and the second FOV.
2. The head mounted device of claim 1, wherein the selected image sensor is selected based in part on a gaze vector representative of the gaze direction being closest to a middle of a selected FOV of the selected image sensor.
3. The head mounted device of claim 1, wherein the processing logic is further configured to:
receive a subsequent-gaze direction from the eye-tracking system; and select a subsequent-selected image sensor that is different from the selected image sensor when a subsequent-gaze vector representative of the subsequent-gaze direction becomes closer to a subsequent-selected FOV of the subsequent-selected image sensor that is different from the selected image sensor.
4. The head mounted device of claim 3, wherein the first image sensor is the selected image sensor and the second image sensor is the subsequent-selected image sensor.
5. The head mounted device of claim 1 further comprising:
a memory, wherein the gaze-guided images are saved to the memory as a gaze-guided video file.
6. The head mounted device of claim 1, wherein the first FOV of the first image sensor does not overlap with the second FOV of the second image sensor.
7. The head mounted device of claim 1, wherein the first FOV of the first image sensor overlaps the second FOV of the second image sensor.
8. A head mounted device comprising:
an eye-tracking system including one or more sensors configured to determine a gaze direction of an eye in an eyebox region of the head mounted device; at least one camera configured to capture images of an external environment of the head mounted device; and processing logic configured to:
receive the gaze direction from the eye-tracking system; and
generate one or more gaze-guided images from the images based on the gaze direction.
9. The head mounted device of claim 8, wherein generating the one or more gaze-guided images includes:
cropping the one or more images to generate the gaze-guided images, wherein the one or more images are cropped in response to the gaze direction with respect to a field of view (FOV) of the at least one camera.
10. The head mounted device of claim 8, wherein the at least one camera includes a lens assembly configured to focus image light onto an image sensor of the camera, and wherein generating the one or more gaze-guided images includes:
driving an optical zoom of the lens assembly in response to the gaze direction.
11. The head mounted device of claim 8, wherein the at least one camera includes a lens assembly configured to focus image light onto an image sensor of the camera, and wherein generating the one or more gaze-guided images includes:
adjusting an auto-focus of the lens assembly in response to the gaze direction.
12. The head mounted device of claim 11, wherein adjusting the auto-focus of the lens assembly in response to the gaze direction includes:
identifying a subject in the images that corresponds to the gaze direction; determining an approximate focus distance to the subject in the images; and adjusting the auto-focus of the lens assembly to the focus distance to image the subject.
13. The head mounted device of claim 8, wherein the at least one camera includes a lens assembly configured to focus image light onto an image sensor of the camera, and wherein generating the one or more gaze-guided images includes:
rotating the image sensor and the lens assembly in response to the gaze direction.
14. A method of operating a head mounted device, the method comprising:
determining a gaze direction of an eye of a user of the head mounted device, wherein an eye-tracking system of the head mounted device determines the gaze direction of the user; capturing one or more images with at least one camera of the head mounted device, wherein the at least one camera is configured to image an external environment of the head mounted device; and generating one or more gaze-guided images from the one or more images based on the gaze direction of the user.
15. The method of claim 14, wherein the at least one camera is included in a plurality of cameras of the head mounted device, and wherein generating the one or more gaze-guided images includes:
selecting a selected camera among the plurality of cameras of the head mounted device, wherein the selected camera is selected to capture the one or more gaze-guided images based on the gaze direction.
16. The method of claim 14, wherein generating the one or more gaze-guided images includes:
cropping the one or more images to generate the gaze-guided images, wherein the one or more images are cropped in response to the gaze direction with respect to a field of view (FOV) of the at least one camera.
17. The method of claim 14, wherein generating the one or more gaze-guided images includes:
transmitting the gaze direction from the head mounted device to a mobile device; transmitting the one or more images from the head mounted device to the mobile device, wherein processing logic of the mobile device generates the gaze-guided images from the one or more images based on the gaze direction of the user.
18. The method of claim 14, wherein the at least one camera includes a lens assembly configured to focus image light onto an image sensor of the camera, and wherein generating the one or more gaze-guided images includes:
adjusting an auto-focus of the lens assembly in response to the gaze direction.
19. The method of claim 18, wherein adjusting the auto-focus of the lens assembly in response to the gaze direction includes:
identifying a subject in the images that corresponds to the gaze direction; determining an approximate focus distance to the subject in the images; and adjusting the auto-focus of the lens assembly to the focus distance to image the subject.
20. The method of claim 14, wherein generating the one or more gaze-guided images includes:
identifying a focus distance that corresponds with the gaze direction of the user; and applying blur effects to the one or more images to blur at least one of a foreground or a background in the one or more images, wherein the background has a background depth that is greater than the focus distance and the foreground has a foreground depth that is less than the focus distance.
Ref US9185352: Claims:
1. An eye tracking system that provides input to a computer, the system comprising:
a computing device;
a monitor coupled to the computing device, the monitor having an identifying region that is recognizable by an indication of the identifying region, wherein the computing device is programmed to recognize the indication of the identifying region;
a first camera adapted to collect information related to a location of the identifying region of the monitor based on a location of the indication of the identifying region, the first camera being adapted to be mounted to a user such that a lens of the first camera faces away from the user; and
a second camera having a lens that is adapted to be disposed in close proximity to and facing an eye of the user and being adapted to collect information from the eye of the user through the lens to provide an indication of a direction of gaze of the eye of the user, the information being provided as an input to the computing device.
2. The eye tracking system of claim 1, wherein the indication of the identifying region of the monitor includes at least one signal emission.
3. The eye tracking system of claim 1, further comprising at least one additional monitor coupled to the computing device and having a second identifying region that is recognizable by a second indication.
4. The eye tracking system of claim 1, further comprising at least a second monitor coupled to at least a second computing device, the second monitor having a second identifying region that is recognizable by a second indication, information collected from the second camera which provides the indication of the direction of the gaze of the eye being provided as input to the second computing device.
5. The eye tracking system of claim 1, further comprising a third camera having a lens that is adapted to be disposed in close proximity to and facing a second eye of the user, the third camera being adapted to collect supplemental information from the second eye which provides an indication of a direction of a gaze of the second eye.
6. The eye tracking system of claim 1, wherein the computing device is programmed to determine from where on the monitor the direction of gaze of the eye intersects information regarding an indication of the direction of the gaze of the eye the identifying region of the monitor.
7. A method for providing computer input by tracking a direction of a gaze of an eye, the method comprising:
providing a computing device;
providing a monitor of the computing device, the monitor having an indication marking an identifying region of the monitor;
mounting a first camera to a user and positioning a lens of the first camera to face generally in a direction of the field of view of the user to collect scene information;
mounting a second camera to the user and positioning a lens of the second camera to face toward an eye of the user to collect information relating to a direction of a gaze of an eye;
using the scene information collected from the first camera to determine a location of the identifying region of the monitor;
using the information collected from the second camera to determine the direction of the gaze of the eye; and
providing input to the computing device based on the direction of the gaze of the eye.
8. The method of claim 7, wherein using the scene information collected from the first camera to determine a location of the identifying region comprises detecting light emission from the indication.
9. The method of claim 7, further comprising providing at least one second monitor having a second indication marking a second identifying region of the at least one second monitor.
10. The method of claim 9, further comprising providing input to a second computing device when the direction of the gaze of the eye falls within a boundary of the second identifying region of the at least one second monitor.
11. The method of claim 7, further comprising correcting for parallax error by mounting a third camera to the user and positioning a lens of the third camera to face toward a second eye of the user to collect supplemental information relating to a direction of a gaze of the second eye.
12. The method of claim 7, wherein providing input to the computing device occurs when the direction of the gaze of the eye falls within the identifying region.
13. The method of claim 7, further comprising utilizing an algorithm that employs a perspective correction algorithm and a parallax correction algorithm to correct for error of the direction of the gaze of the eye when the direction of the gaze of the eye falls within the identifying region of the monitor.
14. At least one non-transitory computer storage medium comprising computer-executable instructions that, when executed by at least one processor, perform a method for calculating a location of a center of a pupil of an eye, the method comprising:
a) generating an image of a pupil of an eye comprising a distribution of pixels, each pixel having an intensity value and a geometric position on the image;
b) selecting pixels having intensity values that fall within a defined range of intensity values;
c) determining an average geometric position of the pixels having intensity values that fall within the defined range, the average geometric position being a first starting point for determining a location of a center of the pupil of the eye on the image;
d) making measurements of a value of pixel intensity along lines extending in a plurality of directions away from the starting point;
e) for each of the lines extending away from the starting point, determining a stop point, each stop point being determined based on a pixel intensity change exceeding a first threshold;
f) for each stop point, making measurements of a value of pixel intensity along lines extending in a plurality of directions from the stop point back toward the starting point and at a plurality of points adjacent to the starting point;
g) for each line from the stop point back to the starting point and a plurality of points adjacent to the starting point, defining a feature point having a geometric position, each feature point being determined based on a pixel intensity change exceeding second threshold; and
h) averaging the geometric positions of the feature points to calculate a second starting point for determining the location of the center of the pupil of the eye on the image.
15. The non-transitory computer storage medium of claim 14, further comprising repeating steps d), e), f), g) and h) for at least one iteration until a difference between a geometric position of the first starting point and the second starting point is less than about 1% in distance.
16. The non-transitory computer storage medium of claim 14, wherein averaging the geometric positions of the plurality of feature points comprises averaging the geometric positions of the plurality of feature points that are accumulated for every iteration of the at least one iteration.
17. The non-transitory computer storage medium of claim 14, wherein averaging the geometric positions of the plurality of feature points comprises averaging the geometric positions of the plurality of feature points that are accumulated for only a current iteration of the at least one iteration.
18. A method for estimating a location of a center of a pupil of an eye from an image of the eye having pixels, the method comprising:
generating an image of a pupil of an eye comprising a distribution of pixels, each pixel having an intensity value and a geometric position on the image;
selecting pixels having intensity values that fall within a defined range of intensity values;
determining the geometric positions of the pixels having intensity values that fall within the defined range of intensity values;
determining an average geometric position of the pixels having intensity values that fall within the defined range, the average geometric position being a first starting point for determining a location of a center of the pupil of the eye on the image;
making measurements of a value of pixel intensity along lines extending in a plurality of directions away from the starting point;
for each of the lines extending away from the starting point, determining a stop point, each stop point being determined based on a pixel intensity change exceeding a first threshold;
for each stop point, making measurements of a value of pixel intensity along lines extending in a plurality of directions from the stop point back toward the starting point and at a plurality of points adjacent to the starting point;
for each line from the stop point back to the starting point and a plurality of points adjacent to the starting point, defining a feature point having a geometric position, each feature point being determined based on a pixel intensity change exceeding a second threshold; and
averaging the geometric positions of the feature points to calculate a second starting point for determining the location of the center of the pupil of the eye on the image.
19. A mobile eye tracking system, the system comprising:
an identifying region of a field of view, the identifying region being recognizable by an indication;
a computing device programmed to recognize the indication of the field of view;
a first camera adapted to collect information related to a location of the identifying region of the field of view based on the indication of the identifying region, the first camera being adapted to be mounted to a user such that a lens of the first camera faces away from the user; and
a second camera adapted to collect information through a lens from the eyes from which a direction of a gaze of an eye of the user may be determined, the lens being adapted to be disposed in close proximity to and facing an eye of the user, the information collected from the second camera being provided as an input to the computing device.
20. A method for providing computer input based upon a direction of a gaze of an eye of a user, the method comprising:
providing an indication marking an identifying region of a field of view;
mounting a first camera to the user and positioning a lens of the first camera to face away from the user to collect scene information;
mounting a second camera to the user and positioning a lens of the second camera to face the eye of the user to collect information from the eye from which a direction of gaze of the eye may be determined;
determining a location of the identifying region of the field of view from the indication based upon the scene information collected from the first camera;
determining the direction of the gaze of the eye using the information collected from the second camera; and
providing an input to the computing device based on a location of an intersection of the direction of the gaze of the eye and the identifying region of the field of view.