Office Action 1: Non-Final Rejection (2023-08-17)
Rejection types: 102 - Lack of Novelty, 103 - Obviousness

CLAIMS:
CLAIMS

1\. An eye-gaze tracking apparatus, comprising:
\- a plurality of foreground cameras each arranged to
capture at least a portion of a target view area of a user; and
\- an eye-gaze estimation module having at least one eye-gaze camera each arranged to capture an image an eye of the user, so as to estimate a position of a
pupil of the eye based on the image of the eye;
wherein each of the plurality of foreground cameras includes a field-of-view (FOV) being different from each other.

2\. The eye-gaze tracking apparatus in accordance with claim 1, wherein te plurality of foreground cameras include a wide-angle camera having a first FOV and a magnified camera having a second FOV smaller than the first FOV.

3\. The eye-gaze tracking apparatus in accordance with claim 2, wherein the second FOV equals to 60° diagonal.

4\. The eye-gaze tracking apparatus in accordance with claim 1, wherein each of the at least one eye-gaze camera includes an infrared camera arranged to capture an infrared image of the eye of the user.

5\. The eye-gaze tracking apparatus in accordance with claim 4, further comprising at least one infrared LED arranged to illuminate the eye of the user.

6\. The eye-gaze tracking apparatus in accordance with claim 1, further comprising a calibration module arranged to facilitate a calibration of the eye-gaze estimation module according to characteristics of the eye of the user.
7\. The eye-gaze tracking apparatus in accordance with claim 6, wherein the calibration module is arranged to provide a plurality of displayed markers such that the eye-gaze estimation module is arranged to map a plurality of calibration positions of the pupil of the eye to the plurality of displayed markers overlaid at different positions in the target view area of a user.

8\. The eye-gaze tracking apparatus in accordance with claim 6, wherein the calibration module includes a plurality of physical markers such that the eye-gaze estimation module is arranged to map a plurality of calibration positions of the pupil of the eye to the plurality of physical markers disposed at different positions in the target view area of the user.

9\. The eye-gaze tracking apparatus in accordance with claim 6, wherein the calibration module is arranged to facilitate calibration of the eye-gaze estimation module with respect to different target view areas of a user associated with different FOVs of the plurality of foreground cameras.

10\. The eye-gaze tracking apparatus in accordance with claim 1, wherein the eye-gaze estimation module includes a neural network arranged to process the image of the eye captured by the at least one eye- gaze camera so as to estimate an eye-gaze angle of the user associated with the position of the pupil of the eye. ll.A method of eye-gaze tracking, comprising the steps of:
\- capturing at least a portion of a target view area of a user using a plurality of foreground cameras;
and - capturing an image an eye of the user using an eye- gaze camera, so as to estimate a position of a pupil of the eye based on the image of the eye;
wherein each of the plurality of foreground cameras
includes a field-of-view (FOV) being different from each other.

12\. The method of claim 11, wherein the plurality of foreground cameras include a wide-angle camera having a first FOV and a magnified camera having a second FOV smaller than the first FOV.

13\. The meLhod of claim 12, wherein the second FOV equals to 600 diagonal.

14\. The method of claim 11, wherein each of the at least one eye-gaze camera includes an infrared camera arranged to capture an infrared image of the eye of the user.

15\. The method of claim 14, further comprising the step of illuminating the eye of the user using at least one infrared LED.

16\. The method of claim 11, further comprising the step of calibrating an eye-gaze estimation module according to characteristics of the eye of the user, wherein the eye-gaze estimation module is arranged to estimate the position of the pupil of the eye.

17\. The method of claim 16, wherein the step of calibrating the eye-gaze estimation module comprises the steps of:
\- providing a plurality of displayed markers overlaid at different positions in the target view area of a
user; and
\- mapping a plurality of calibration positions of the pupil of the eye to the plurality of displayed markers.
18\. The method of claim 16, wherein the step of calibrating the eye-gaze estimation module comprises the steps of:
\- disposing a plurality of physica\_ markers at different positions in the target view area of the user; and
\- mapping a plurality of calibration positions of the pupil of the eye to the plurality of physical markers.

19\. The method of claim 16, wherein the eye-gaze estimation module is calibrated with respect to different target view areas of a user associated with different FOVs of the plurality of foreground cameras.

20\. The method of claim 11, further comprising the step of processing the image of the eye captured by the at least one eye-gaze camera using a neural network, so as to estimate an eye-gaze angle of the user associated with the position of the pupil of the eye. 22 23 24 25 5 5 10 15 20 25 30 35

OFFICE ACTION:
DETAILED ACTION
Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 2, 4-6, 9, 11, 12,14-16, and 19 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Sztuk et al (Publication number: US 2023/0119935).
Consider Claim 1, Sztuk et al shows an eye-gaze tracking apparatus (see figure 1), comprising:
(a) A plurality of foreground cameras (read as cameras 193A-B) each arranged to capture at least a portion of a target view area of a user (See figures 1 and 2A; paragraphs 34, 35 and 36); (Camera 193D is configured to image the external environment to the left of head mounted device 100. Camera 193B is disposed in the upper-right corner of frame 102 and configured to image the forward-right external environment of head mounted device 100. Camera 193C is disposed in the upper-left corner of frame 102 and configured to image the forward-left external environment of head mounted device 100. The field of view (FOV) of camera 193B may overlap a FOV of camera 193C).
(b) An eye-gaze estimation module (read as cameras 108A-B) having at least one eye-gaze camera each arranged to capture an image an eye of the user, so as to estimate a position of a pupil of the eye based on the image of the eye (figures 1 and 2A; paragraph 25); (Cameras 108A and 108B may be configured to image an eyebox region to image the eye of the user to capture eye data of the user. Cameras 108A and 108B may be included in an eye-tracking system that is configured to determine a gaze direction of an eye (or eyes) of a user of the head mounted device. Cameras 108A and 108B may image the eyebox region directly or indirectly).
(c) Wherein each of the plurality of foreground cameras includes a field-of-view (FOV) being different from each other (see figure 2C; paragraph 48); (See FOV’s 297A-D. The FOV’s can be adjusted to be narrow or wide).
Consider Claim 11, Sztuk et al shows a method of eye-gaze tracking (see figure 5), comprising the steps of:
(a) Capturing at least a portion of a target view area of a user using a plurality of foreground cameras (See figures 1 and 2A; paragraphs 34, 35 and 36); (Camera 193D is configured to image the external environment to the left of head mounted device 100. Camera 193B is disposed in the upper-right corner of frame 102 and configured to image the forward-right external environment of head mounted device 100. Camera 193C is disposed in the upper-left corner of frame 102 and configured to image the forward-left external environment of head mounted device 100. The field of view (FOV) of camera 193B may overlap a FOV of camera 193C).
(b) Capturing an image an eye of the user using an eye- gaze camera, so as to estimate a position of a pupil of the eye based on the image of the eye (figures 1 and 2A; paragraph 25); (Cameras 108A and 108B may be configured to image an eyebox region to image the eye of the user to capture eye data of the user. Cameras 108A and 108B may be included in an eye-tracking system that is configured to determine a gaze direction of an eye (or eyes) of a user of the head mounted device. Cameras 108A and 108B may image the eyebox region directly or indirectly).
(c) Wherein each of the plurality of foreground cameras includes a field-of-view (FOV) being different from each other (see figure 2C; paragraph 48); (See FOV’s 297A-D. The FOV’s can be adjusted to be narrow or wide).
Consider Claims 2 and 12, Sztuk et al shows that the plurality of foreground cameras includes a wide-angle camera having a first FOV and a magnified camera having a second FOV smaller than the first FOV (see figure 2C; paragraph 48); (See FOV’s 297A-D. The FOV’s can be adjusted to be narrow or wide).
Consider Claims 4 and 14, Sztuk et al shows that each of the at least one eye-gaze camera includes an infrared camera arranged to capture an infrared image of the eye of the user (see paragraphs 51 and 52); (A camera of eye-tracking system 460 may image eye 403 by receiving reflected infrared illumination light from an optical combiner layer included in optical element 410. The optical combiner layer may be configured to receive reflected infrared illumination light and redirect the reflected infrared illumination light to the camera of eye-tracking system 460. The camera would be oriented to receive the reflected infrared illumination light from the optical combiner layer of optical element 410).
Consider Claims 5 and 15, Sztuk et al shows at least one infrared LED arranged to illuminate the eye of the user (see paragraph 52).
Consider Claims 6 and 16, Sztuk et al shows a calibration module arranged to facilitate a calibration of the eye-gaze estimation module according to characteristics of the eye of the user (see paragraph 64); (Identifying a subject in an image that corresponds to a gaze direction of the user may utilize gaze calibration data that is recorded during a calibration procedure in an unboxing process of the head mounted device).
Consider Claims 9 and 19, Sztuk et al shows that the calibration module is arranged to facilitate calibration of the eye-gaze estimation module with respect to different target view areas of a user associated with different FOVs of the plurality of foreground cameras (see paragraph 64).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 7-8 and 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Sztuk et al (Publication number: US 2023/0119935) in view of Bastani et al (Publication number: US 2021/0166341).
Consider Claims 7 and 17, Sztuk et al does not specifically show that the calibration module is arranged to provide a plurality of displayed markers such that the eye-gaze estimation module is arranged to map a plurality of calibration positions of the pupil of the eye to the plurality of displayed markers overlaid at different positions in the target view area of a user.
In the same field of endeavor, Bastani et al shows the calibration module is arranged to provide a plurality of displayed markers such that the eye-gaze estimation module is arranged to map a plurality of calibration positions of the pupil of the eye to the plurality of displayed markers overlaid at different positions in the target view area of a user (See figures 6 and 7; paragraphs 53, 55,); (Display 164 can perform foveated rendering based on the calibrated eye tracking operation, which can indicate a gaze point corresponding to the gaze direction generated by the eye tracking operation).
Therefore, it would have been obvious to a person of ordinary skill in the art at the effective filing date of the application to incorporate the teaching of Bastani et al into the teaching of Sztuk et al in order to perform foveated rendering (see Bastani et al; paragraphs 54 and 55).
Consider Claims 8 and 18, Sztuk et al does not specifically show that the calibration module includes a plurality of physical markers such that the eye-gaze estimation module is arranged to map a plurality of calibration positions of the pupil of the eye to the plurality of physical markers disposed at different positions in the target view area of the user.
In the same field of endeavor, Bastani shows that the calibration module includes a plurality of physical markers such that the eye-gaze estimation module is arranged to map a plurality of calibration positions of the pupil of the eye to the plurality of physical markers disposed at different positions in the target view area of the user (See figures 6 and 7; paragraphs 53, 55,); (Display 164 can perform foveated rendering based on the calibrated eye tracking operation, which can indicate a gaze point corresponding to the gaze direction generated by the eye tracking operation).
Therefore, it would have been obvious to a person of ordinary skill in the art at the effective filing date of the application to incorporate the teaching of Bastani et al into the teaching of Sztuk et al in order to perform foveated rendering (see Bastani et al; paragraphs 54 and 55).
Claims 3 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Sztuk et al (Publication number: US 2023/0119935) in view of Trail et al (Patent number: US 10,928635).
Consider Claims 3 and 13, Sztuk et al does not specifically show that the second FOV equals to 60° diagonal.
In the same field of endeavor, Trail et al shows that the second FOV equals to 60° diagonal (see column 12, lines 15-30); (The expanded image light 340 provided to the user's eye 220 with a diagonal FOV of at least 60 degrees).
Therefore, it would have been obvious to a person of ordinary skill in the art at the effective filing date of the application to incorporate the teaching of Trail et al into the teaching of Sztuk et al in order to output expanded image light to the user’s eye (see column 12, lines 15-30).
Claims 10 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Sztuk et al (Publication number: US 2023/0119935) in view of Bathiche et al (Publication number: US 2022/0221932).
Consider Claims 10 and 20, Sztuk et al does not specifically show that the eye-gaze estimation module includes a neural network arranged to process the image of the eye captured by the at least one eye- gaze camera so as to estimate an eye-gaze angle of the user associated with the position of the pupil of the eye.
In related art, Bathiche et al shows that the eye-gaze estimation module includes a neural network arranged to process the image of the eye captured by the at least one eye- gaze camera so as to estimate an eye-gaze angle of the user associated with the position of the pupil of the eye (paragraphs 62 and 63).
Therefore, it would have been obvious to a person of ordinary skill in the art at the effective filing date of the application to incorporate the teaching of Bathiche et al into the teaching of Sztuk et al in order to generate gaze information (see Bathiche et al; paragraphs 62 and 63).
Conclusion

APPLICANT REBUTTAL: REMARKS
This amendment is in response to the Office Action dated August 17, 2023. To further the prosecution of this Application, Applicants submit the following remarks and amended claims. The claims as now presented are believed to be in allowable condition.
Drawings/Priority Papers/Information Disclosure Statement
The Applicant thanks the Examiner for indicating that the drawings currently on file are acceptable, acknowledging receipt and entry of the priority papers and acknowledging substantive consideration of the filed Information Disclosure Statement(s), concerning the above identified application, by returning a signed copy thereof with the most recent Office Action.
In the Claims:
Please amend claims 1, 2, 9, 11, 12, and 19 in which the claim additions are shown by underlining and/or the claim deletions are shown by either strikeout or brackets. Please enter the amended claims into the record of this case.
Prior art publication date
The applicant observed that the priority date of the current application is September 13, 2022. The publication date of the citation Sztuk et al. '953 is April 20, 2023. The applicant respectfully disagrees with the citation Sztuk et al. '953 qualifying as a printed publication before the effective filing date of the claimed Invention.
35 U.S.C. J 102 - Anticipation
Claims 1, 2, 4-6, 9, 11, 12,14-16, and 19 are rejected, under 35 U.S.C. §102 (a)(1), as being anticipated in view of Sztuk et al. '953 (U.S. Publication No. 2023/0119935). The Applicant acknowledges and respectfully traverses the raised anticipatory rejection in view of the following remarks.
35 U.S.C. § 103 - Obviousness
Claims 7-8 and 17-18 are rejected, under 35 U.S.C. § 103, as being unpatentable in view of Sztuk et al. '953 in view of Bastani et al. '341 (U.S. Publication No. 2021/0166341). The Applicant acknowledges and respectfully traverses the raised obviousness rejection in view of the above amendments and the following remarks.
In order to emphasize the noted distinctions between the presently claimed invention and the applied art, the independent claims of this application now recite the features of "wherein the foreground cameras and the eye-gaze cameras are adapted to capture images of both the target view of a user as well as the images of the eyes of the user simultaneously to determine eye-gaze positions of the user." Such features are believed to clearly and patentably distinguish the presently claimed invention from all of the art of record, including the applied art. The amended Claim 1 and Claim 11 are based on at least page 6 lines 19 to 29 of the description of the present application.
Additionally, the Applicant proposes to amend Claim 2 and Claim 12 to include the feature of "a wide-angle camera having a first FOV to estimate a wider view of the user and a magnified camera having a second FOV smaller than the first FOV to estimate a visual focus of the user." The amended Claim 2 and Claim 11 are based at least on Page 9 lines 26 to 39 of the description of the present application.
The Applicant proposes to amend Claim 9 and Claim 19 to include the feature of "using a combined gaze estimation method based on convolutional neural network and a geometric model to get the accurate gaze position estimation." The amended Claim 9 and Claim 19 are based on at least Page 8 lines 31 to 33 of the description of the present application.
Now turning to the cited art, Sztuk et al. '953 discloses a gaze direction of an eye or eyes of a user of a head mounted device is determined. One or more images are captured with a camera of a head mounted device. Gaze-guided images are generated from the one or more images based on the gaze direction of the user.
Bastani et al. '341 discloses a method for providing imagery to a user on a display that includes receiving eye tracking data. The method also includes determining a gaze location on the display and at least one of a confidence factor of the gaze location, or a speed of the change of the gaze location using the eye tracking data. The method also includes establishing multiple tiles using the gaze location and at least one of the confidence factors or the speed of the change of the gaze location. The method also includes providing a foveated rendered image using the multiple tiles.
The present invention is direct to the inaccuracy of the prior art eye tracking system which may vary much, sometimes over 3 degrees, depending on the calibration and applications (See, at least page 5 line 38 to page 6 line 1 of the present application). The inventors devised an eye tracking system that may be categorized into head-mounted and desktop-based on different applications, and the system accuracy of prior art eye trackers may range from 0.5 degrees to 1 degree (page 5 lines 33 to 36).
Sztuk et al. '953 does not disclose any device for eye tracking but rather discloses a system to capture or generate images based on the gaze direction of an eye of a user that has been determined by an eye-tracking system (see paragraph 21). As such, the foreground cameras (193A, 193B, 193C, 193D) and the cameras (108A-B) of the Sztuk et al. '953 could not achieve or direct a person skilled in the art to solve the problem of the present invention. Sztuk et al. '953 has taken for granted that the eye-tracking system is accurate in order to capture or generate images based on gaze direction. On the contrary, the present invention is provided to determine the unknown eye-gaze positions in high accuracy (0.5 degrees to 1 degree error) with the input of the foreground camera 102 and the eye-gaze camera 106. Therefore, Sztuk et al. '953 does not disclose the present invention defined in Claim 1 and Claim 11. Claim 1 and Claim 11 are novel when compared to Sztuk et al. '953.
As Sztuk et al. '953 does not direct or teach a person skilled in the art to determine the gazing direction or improve the accuracy of the gazing direction, it could not teach or direct a person skilled in the art to the feature that the foreground cameras and the eye-gaze cameras are adapted to capture images of both the target view of a user as well as the
images of the eyes of the user simultaneously to determine eye-gaze positions of the user. Therefore, Claim 1 and Claim 11 are not obvious when compared to Sztuk et al. '953.
Claim 2 and Claim 12 further define that a wide-angle camera has a first FOV to estimate a wider view of the user and a magnified camera has a second FOV smaller than the first FOV to estimate a visual focus of the user. (See Page 9 lines 26 to 39). There is no disclosure of a wide-angle camera and a magnified camera in Sztuk et al. '953. Therefore, Sztuk et al. '953 does not disclose the present invention defined in Claim 2 and Claim 12. Claim 2 and Claim 12 are novel when compared to Sztuk et al. '953.
The function of the FOV mentioned in Sztuk et al. '953 is to capture a panoramic image viewable by the user (see figure2C; paragraph 48); (See FOV's297A-D. The FOVs can be adjusted to be narrow or wide). The function of the wide-angle camera in Claim 2 and Claim 12 is to estimate a wider view of the user and that of a magnified camera having a second FOV smaller than the first FOV to estimate a visual focus of the user. Since Sztuk et al. '953 does not disclose a wide-angle camera and a magnified camera together, it is impossible for Sztuk et al. '953 to direct a person skilled in the art to feature that a wide- angle camera having a first FOV to estimate a wider view of the user and a magnified camera having a second FOV smaller than the first FOV to estimate a visual focus of the user. Therefore, Claim 2 and Claim 12 are not obvious when compared to Sztuk et al. '953
Claim 9 and Claim 19 further define the feature of using a combined gaze estimation method based on convolutional neural network and a geometric model to get an accurate gaze position estimation. (See Page 8 lines 31 to 33). Neither Sztuk et al. '953 nor Bastani et al. '341 disclose nor direct a person skilled in the art that an eye-tracking system or method has an eye-gaze estimation module that is calibrated with respect to different target view areas of a user associated with different FOVs from a wider view of the user and a magnified camera (as defined in Claim 2 and Claim 12), and using a combined gaze estimation method based on convolutional neural network and a geometric model to get the accurate gaze position estimation. Therefore, Claim 9 and Claim 19 are novel and not obvious when compared to Sztuk et al. '953 and Bastani et al. '341 either alone or in combination.
If any further amendment to this application is believed necessary to advance prosecution and place this case in allowable form, the Examiner is courteously solicited to contact the undersigned representative of the Applicant to discuss the same.
Conclusion

NEXT OFFICE ACTION: DETAILED ACTION
Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in response to the amendment filed by Applicant on 10/31/2023. This action is made FINAL.
Response to Arguments
Applicant’s arguments with respect to claims 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 4-6, 9, 11, 12,14-16, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Sztuk et al (Publication number: US 2023/0119935) in view of Jacques (Patent number: US 9,185,352).
Consider Claim 1, Sztuk et al shows an eye-gaze tracking apparatus (see figure 1), comprising:
(a) A plurality of foreground cameras (read as cameras 193A-B) each arranged to capture at least a portion of a target view area of a user (See figures 1 and 2A; paragraphs 34, 35 and 36); (Camera 193D is configured to image the external environment to the left of head mounted device 100. Camera 193B is disposed in the upper-right corner of frame 102 and configured to image the forward-right external environment of head mounted device 100. Camera 193C is disposed in the upper-left corner of frame 102 and configured to image the forward-left external environment of head mounted device 100. The field of view (FOV) of camera 193B may overlap a FOV of camera 193C).
(b) An eye-gaze estimation module (read as cameras 108A-B) having at least one eye-gaze camera each arranged to capture an image an eye of the user, so as to estimate a position of a pupil of the eye based on the image of the eye, wherein the foreground cameras and the eye-gaze cameras are adapted to capture images of the target view of the user as well as the images of the eyes of the user (figures 1 and 2A; paragraph 25); (Cameras 108A and 108B may be configured to image an eyebox region to image the eye of the user to capture eye data of the user. Cameras 108A and 108B may be included in an eye-tracking system that is configured to determine a gaze direction of an eye (or eyes) of a user of the head mounted device. Cameras 108A and 108B may image the eyebox region directly or indirectly).
(c) Wherein each of the plurality of foreground cameras includes a field-of-view (FOV) being different from each other (see figure 2C; paragraph 48); (See FOV’s 297A-D. The FOV’s can be adjusted to be narrow or wide).
However, Sztuk et al does not specifically show that the foreground cameras and the eye-gaze cameras are adapted to capture images of both the target view of a user as well as the images of the eyes of the user simultaneously to determine eye-gaze positions of the user.
In the same field of endeavor, Jacques shows that the foreground cameras and the eye-gaze cameras are adapted to capture images of both the target view of a user as well as the images of the eyes of the user simultaneously to determine eye-gaze positions of the user (column 2, lines 25-50); (Jacques specifically states that “a  first camera is mounted to the user and the lens of the first camera is positioned in a manner that faces away from the user so that the camera collects scene information generally associated with the field of view of the user. A second camera is mounted to the user and the lens of the second camera is positioned in a manner so as to be in close proximity to and face toward an eye of the user so that the camera collects image information of the eye. The system is calibrated such that a determination is made as to the direction of the gaze of the eye based on the image information collected of the eye by the second camera. Using the scene information as input, the system determines a location of the identifying region of the field of view. When the direction of the gaze of the user's eye tracked by the second camera is within the identifying region, input may subsequently be provided by the user through movements of the eye to the computing device”).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the application to incorporate the teaching of Jacques into the teaching of Sztuk in order to map images of the pupil to points in space (see Jacques; column 1, lines 10-25).
Consider Claim 11, Sztuk et al shows a method of eye-gaze tracking (see figure 5), comprising the steps of:
(a) Capturing at least a portion of a target view area of a user using a plurality of foreground cameras (See figures 1 and 2A; paragraphs 34, 35 and 36); (Camera 193D is configured to image the external environment to the left of head mounted device 100. Camera 193B is disposed in the upper-right corner of frame 102 and configured to image the forward-right external environment of head mounted device 100. Camera 193C is disposed in the upper-left corner of frame 102 and configured to image the forward-left external environment of head mounted device 100. The field of view (FOV) of camera 193B may overlap a FOV of camera 193C).
(b) Capturing an image an eye of the user using an eye- gaze camera, so as to estimate a position of a pupil of the eye based on the image of the eye (figures 1 and 2A; paragraph 25); (Cameras 108A and 108B may be configured to image an eyebox region to image the eye of the user to capture eye data of the user. Cameras 108A and 108B may be included in an eye-tracking system that is configured to determine a gaze direction of an eye (or eyes) of a user of the head mounted device. Cameras 108A and 108B may image the eyebox region directly or indirectly).
(c) Wherein each of the plurality of foreground cameras includes a field-of-view (FOV) being different from each other (see figure 2C; paragraph 48); (See FOV’s 297A-D. The FOV’s can be adjusted to be narrow or wide).
However, Sztuk et al does not specifically show that the foreground cameras and the eye-gaze cameras are adapted to capture images of both the target view of a user as well as the images of the eyes of the user simultaneously to determine eye-gaze positions of the user.
In the same field of endeavor, Jacques shows that the foreground cameras and the eye-gaze cameras are adapted to capture images of both the target view of a user as well as the images of the eyes of the user simultaneously to determine eye-gaze positions of the user (column 2, lines 25-50); (Jacques specifically states that “a  first camera is mounted to the user and the lens of the first camera is positioned in a manner that faces away from the user so that the camera collects scene information generally associated with the field of view of the user. A second camera is mounted to the user and the lens of the second camera is positioned in a manner so as to be in close proximity to and face toward an eye of the user so that the camera collects image information of the eye. The system is calibrated such that a determination is made as to the direction of the gaze of the eye based on the image information collected of the eye by the second camera. Using the scene information as input, the system determines a location of the identifying region of the field of view. When the direction of the gaze of the user's eye tracked by the second camera is within the identifying region, input may subsequently be provided by the user through movements of the eye to the computing device”).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the application to incorporate the teaching of Jacques into the teaching of Sztuk in order to map images of the pupil to points in space (see Jacques; column 1, lines 10-25).
Consider Claims 2 and 12, Sztuk et al shows that the plurality of foreground cameras includes a wide-angle camera having a first FOV to estimate a wider view of the user and a magnified camera having a second FOV smaller than the first FOV to estimate a visual focus of the user (see figure 2C; paragraph 48); (See FOV’s 297A-D. The FOV’s can be adjusted to be narrow or wide).
Consider Claims 4 and 14, Sztuk et al shows that each of the at least one eye-gaze camera includes an infrared camera arranged to capture an infrared image of the eye of the user (see paragraphs 51 and 52); (A camera of eye-tracking system 460 may image eye 403 by receiving reflected infrared illumination light from an optical combiner layer included in optical element 410. The optical combiner layer may be configured to receive reflected infrared illumination light and redirect the reflected infrared illumination light to the camera of eye-tracking system 460. The camera would be oriented to receive the reflected infrared illumination light from the optical combiner layer of optical element 410).
Consider Claims 5 and 15, Sztuk et al shows at least one infrared LED arranged to illuminate the eye of the user (see paragraph 52).
Consider Claims 6 and 16, Sztuk et al shows a calibration module arranged to facilitate a calibration of the eye-gaze estimation module according to characteristics of the eye of the user (see paragraph 64); (Identifying a subject in an image that corresponds to a gaze direction of the user may utilize gaze calibration data that is recorded during a calibration procedure in an unboxing process of the head mounted device).
Claims 7-8 and 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Sztuk et al (Publication number: US 2023/0119935) in view of Jacques in view of Bastani et al (Publication number: US 2021/0166341).
Consider Claims 7 and 17, Sztuk et al in view of Jacques does not specifically show that the calibration module is arranged to provide a plurality of displayed markers such that the eye-gaze estimation module is arranged to map a plurality of calibration positions of the pupil of the eye to the plurality of displayed markers overlaid at different positions in the target view area of a user.
In the same field of endeavor, Bastani et al shows the calibration module is arranged to provide a plurality of displayed markers such that the eye-gaze estimation module is arranged to map a plurality of calibration positions of the pupil of the eye to the plurality of displayed markers overlaid at different positions in the target view area of a user (See figures 6 and 7; paragraphs 53, 55,); (Display 164 can perform foveated rendering based on the calibrated eye tracking operation, which can indicate a gaze point corresponding to the gaze direction generated by the eye tracking operation).
Therefore, it would have been obvious to a person of ordinary skill in the art at the effective filing date of the application to incorporate the teaching of Bastani et al into the teaching of Sztuk et al and Jacques in order to perform foveated rendering (see Bastani et al; paragraphs 54 and 55).
Consider Claims 8 and 18, Sztuk et al in view of Jacques does not specifically show that the calibration module includes a plurality of physical markers such that the eye-gaze estimation module is arranged to map a plurality of calibration positions of the pupil of the eye to the plurality of physical markers disposed at different positions in the target view area of the user.
In the same field of endeavor, Bastani shows that the calibration module includes a plurality of physical markers such that the eye-gaze estimation module is arranged to map a plurality of calibration positions of the pupil of the eye to the plurality of physical markers disposed at different positions in the target view area of the user (See figures 6 and 7; paragraphs 53, 55,); (Display 164 can perform foveated rendering based on the calibrated eye tracking operation, which can indicate a gaze point corresponding to the gaze direction generated by the eye tracking operation).
Therefore, it would have been obvious to a person of ordinary skill in the art at the effective filing date of the application to incorporate the teaching of Bastani et al into the teaching of Sztuk et al and Jacques in order to perform foveated rendering (see Bastani et al; paragraphs 54 and 55).
Claims 3 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Sztuk et al (Publication number: US 2023/0119935) in view of Jacques in view of Trail et al (Patent number: US 10,928635).
Consider Claims 3 and 13, Sztuk et al in view of Jacques does not specifically show that the second FOV equals to 60° diagonal.
In the same field of endeavor, Trail et al shows that the second FOV equals to 60° diagonal (see column 12, lines 15-30); (The expanded image light 340 provided to the user's eye 220 with a diagonal FOV of at least 60 degrees).
Therefore, it would have been obvious to a person of ordinary skill in the art at the effective filing date of the application to incorporate the teaching of Trail et al into the teaching of Sztuk et al and Jacques in order to output expanded image light to the user’s eye (see column 12, lines 15-30).
Claims 10 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Sztuk et al (Publication number: US 2023/0119935) in view of Jacques in view of Bathiche et al (Publication number: US 2022/0221932).
Consider Claims 10 and 20, Sztuk et al in view of Jacques does not specifically show that the eye-gaze estimation module includes a neural network arranged to process the image of the eye captured by the at least one eye- gaze camera so as to estimate an eye-gaze angle of the user associated with the position of the pupil of the eye.
In related art, Bathiche et al shows that the eye-gaze estimation module includes a neural network arranged to process the image of the eye captured by the at least one eye- gaze camera so as to estimate an eye-gaze angle of the user associated with the position of the pupil of the eye (paragraphs 62 and 63).
Therefore, it would have been obvious to a person of ordinary skill in the art at the effective filing date of the application to incorporate the teaching of Bathiche et al into the teaching of Sztuk et al and Jacques in order to generate gaze information (see Bathiche et al; paragraphs 62 and 63).
Claims 9 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Sztuk et al (Publication number: US 2023/0119935) in view of Jacques in view of YU et al (Publication number: US 2021/0319420).
Consider Claims 9 and 19, Sztuk et al in view of Jacques shows that the calibration module is arranged to facilitate calibration of the eye-gaze estimation module with respect to different target view areas of a user associated with different FOVs of the plurality of foreground cameras (see paragraph 64).
However, Sztuk et al in view of Jacques does not specifically show using a combined gaze estimation method based on convolutional neural network and a geometric model to get an accurate gaze position estimation.
In related art, YU et al shows using a combined gaze estimation method based on convolutional neural network and a geometric model to get an accurate gaze position estimation (see paragraphs 95 and 96).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the application to incorporate the teaching of YU et al into the teaching of Sztuk et al and Jacques in order to achieve robust visual object tracking (see YU et al; paragraphs 95 and 96).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action.  Accordingly, THIS ACTION IS MADE FINAL.  See MPEP § 706.07(a).  Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action.  In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action.  In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.

CITED REFERENCE SUMMARIES: Ref US20230119935A1: Claims:
A head mounted device comprising: an eye-tracking system including one or more sensors configured to determine a gaze direction of an eye in an eyebox region of the head mounted device; a first image sensor configured to capture first images of an external environment of the head mounted device; a second image sensor configured to capture second images of the external environment of the head mounted device, wherein the first image sensor has a first field of view (FOV) that is different from a second FOV of the second image sensor; and processing logic configured to: receive the gaze direction from the eye-tracking system; and select a selected image sensor between the first image sensor and the second image sensor to capture one or more gaze-guided images, wherein the first image sensor or the second image sensor is selected to capture the one or more gaze-guided images based on the gaze direction with respect to the first FOV and the second FOV.
an eye-tracking system including one or more sensors configured to determine a gaze direction of an eye in an eyebox region of the head mounted device;
a first image sensor configured to capture first images of an external environment of the head mounted device;
a second image sensor configured to capture second images of the external environment of the head mounted device, wherein the first image sensor has a first field of view (FOV) that is different from a second FOV of the second image sensor; and
processing logic configured to: receive the gaze direction from the eye-tracking system; and select a selected image sensor between the first image sensor and the second image sensor to capture one or more gaze-guided images, wherein the first image sensor or the second image sensor is selected to capture the one or more gaze-guided images based on the gaze direction with respect to the first FOV and the second FOV.
receive the gaze direction from the eye-tracking system; and
select a selected image sensor between the first image sensor and the second image sensor to capture one or more gaze-guided images, wherein the first image sensor or the second image sensor is selected to capture the one or more gaze-guided images based on the gaze direction with respect to the first FOV and the second FOV.
The head mounted device of claim 1 , wherein the selected image sensor is selected based in part on a gaze vector representative of the gaze direction being closest to a middle of a selected FOV of the selected image sensor.
The head mounted device of claim 1 , wherein the processing logic is further configured to: receive a subsequent-gaze direction from the eye-tracking system; and select a subsequent-selected image sensor that is different from the selected image sensor when a subsequent-gaze vector representative of the subsequent-gaze direction becomes closer to a subsequent-selected FOV of the subsequent-selected image sensor that is different from the selected image sensor.
receive a subsequent-gaze direction from the eye-tracking system; and
select a subsequent-selected image sensor that is different from the selected image sensor when a subsequent-gaze vector representative of the subsequent-gaze direction becomes closer to a subsequent-selected FOV of the subsequent-selected image sensor that is different from the selected image sensor.
The head mounted device of claim 3 , wherein the first image sensor is the selected image sensor and the second image sensor is the subsequent-selected image sensor.
The head mounted device of claim 1 further comprising: a memory, wherein the gaze-guided images are saved to the memory as a gaze-guided video file.
a memory, wherein the gaze-guided images are saved to the memory as a gaze-guided video file.
The head mounted device of claim 1 , wherein the first FOV of the first image sensor does not overlap with the second FOV of the second image sensor.
The head mounted device of claim 1 , wherein the first FOV of the first image sensor overlaps the second FOV of the second image sensor.
A head mounted device comprising: an eye-tracking system including one or more sensors configured to determine a gaze direction of an eye in an eyebox region of the head mounted device; at least one camera configured to capture images of an external environment of the head mounted device; and processing logic configured to: receive the gaze direction from the eye-tracking system; and generate one or more gaze-guided images from the images based on the gaze direction.
at least one camera configured to capture images of an external environment of the head mounted device; and
processing logic configured to: receive the gaze direction from the eye-tracking system; and generate one or more gaze-guided images from the images based on the gaze direction.
generate one or more gaze-guided images from the images based on the gaze direction.
The head mounted device of claim 8 , wherein generating the one or more gaze-guided images includes: cropping the one or more images to generate the gaze-guided images, wherein the one or more images are cropped in response to the gaze direction with respect to a field of view (FOV) of the at least one camera.
cropping the one or more images to generate the gaze-guided images, wherein the one or more images are cropped in response to the gaze direction with respect to a field of view (FOV) of the at least one camera.
The head mounted device of claim 8 , wherein the at least one camera includes a lens assembly configured to focus image light onto an image sensor of the camera, and wherein generating the one or more gaze-guided images includes: driving an optical zoom of the lens assembly in response to the gaze direction.
driving an optical zoom of the lens assembly in response to the gaze direction.
The head mounted device of claim 8 , wherein the at least one camera includes a lens assembly configured to focus image light onto an image sensor of the camera, and wherein generating the one or more gaze-guided images includes: adjusting an auto-focus of the lens assembly in response to the gaze direction.
adjusting an auto-focus of the lens assembly in response to the gaze direction.
The head mounted device of claim 11 , wherein adjusting the auto-focus of the lens assembly in response to the gaze direction includes: identifying a subject in the images that corresponds to the gaze direction; determining an approximate focus distance to the subject in the images; and adjusting the auto-focus of the lens assembly to the focus distance to image the subject.
identifying a subject in the images that corresponds to the gaze direction;
determining an approximate focus distance to the subject in the images; and
adjusting the auto-focus of the lens assembly to the focus distance to image the subject.
The head mounted device of claim 8 , wherein the at least one camera includes a lens assembly configured to focus image light onto an image sensor of the camera, and wherein generating the one or more gaze-guided images includes: rotating the image sensor and the lens assembly in response to the gaze direction.
rotating the image sensor and the lens assembly in response to the gaze direction.
A method of operating a head mounted device, the method comprising: determining a gaze direction of an eye of a user of the head mounted device, wherein an eye-tracking system of the head mounted device determines the gaze direction of the user; capturing one or more images with at least one camera of the head mounted device, wherein the at least one camera is configured to image an external environment of the head mounted device; and generating one or more gaze-guided images from the one or more images based on the gaze direction of the user.
determining a gaze direction of an eye of a user of the head mounted device, wherein an eye-tracking system of the head mounted device determines the gaze direction of the user;
capturing one or more images with at least one camera of the head mounted device, wherein the at least one camera is configured to image an external environment of the head mounted device; and
generating one or more gaze-guided images from the one or more images based on the gaze direction of the user.
The method of claim 14 , wherein the at least one camera is included in a plurality of cameras of the head mounted device, and wherein generating the one or more gaze-guided images includes: selecting a selected camera among the plurality of cameras of the head mounted device, wherein the selected camera is selected to capture the one or more gaze-guided images based on the gaze direction.
selecting a selected camera among the plurality of cameras of the head mounted device, wherein the selected camera is selected to capture the one or more gaze-guided images based on the gaze direction.
The method of claim 14 , wherein generating the one or more gaze-guided images includes: cropping the one or more images to generate the gaze-guided images, wherein the one or more images are cropped in response to the gaze direction with respect to a field of view (FOV) of the at least one camera.
The method of claim 14 , wherein generating the one or more gaze-guided images includes: transmitting the gaze direction from the head mounted device to a mobile device; transmitting the one or more images from the head mounted device to the mobile device, wherein processing logic of the mobile device generates the gaze-guided images from the one or more images based on the gaze direction of the user.
transmitting the gaze direction from the head mounted device to a mobile device;
transmitting the one or more images from the head mounted device to the mobile device, wherein processing logic of the mobile device generates the gaze-guided images from the one or more images based on the gaze direction of the user.
The method of claim 14 , wherein the at least one camera includes a lens assembly configured to focus image light onto an image sensor of the camera, and wherein generating the one or more gaze-guided images includes: adjusting an auto-focus of the lens assembly in response to the gaze direction.
The method of claim 18 , wherein adjusting the auto-focus of the lens assembly in response to the gaze direction includes: identifying a subject in the images that corresponds to the gaze direction; determining an approximate focus distance to the subject in the images; and adjusting the auto-focus of the lens assembly to the focus distance to image the subject.
The method of claim 14 , wherein generating the one or more gaze-guided images includes: identifying a focus distance that corresponds with the gaze direction of the user; and applying blur effects to the one or more images to blur at least one of a foreground or a background in the one or more images, wherein the background has a background depth that is greater than the focus distance and the foreground has a foreground depth that is less than the focus distance.
identifying a focus distance that corresponds with the gaze direction of the user; and
applying blur effects to the one or more images to blur at least one of a foreground or a background in the one or more images, wherein the background has a background depth that is greater than the focus distance and the foreground has a foreground depth that is less than the focus distance.
Ref US20210166341A1: Claims:
A method for providing imagery to a user on a display, the method comprising: receiving eye tracking data; determining a gaze location on the display and at least one of a confidence factor of the gaze location, or a speed of the change of the gaze location using the eye tracking data; establishing a plurality of tiles using both the gaze location and at least one of the confidence factor or the speed of the change of the gaze location; and providing a foveated rendered image using the plurality of tiles.
receiving eye tracking data;
determining a gaze location on the display and at least one of a confidence factor of the gaze location, or a speed of the change of the gaze location using the eye tracking data;
establishing a plurality of tiles using both the gaze location and at least one of the confidence factor or the speed of the change of the gaze location; and
providing a foveated rendered image using the plurality of tiles.
The method of claim 1 , wherein establishing the plurality of tiles comprises defining or adjusting a size of one or more of the plurality of tiles, defining or adjusting a fall-off of a fovea region of the plurality of tiles, or defining or adjusting a radius of the fovea region, using at least one of: the gaze location on the display, the confidence factor of the gaze location, or the speed of the change of the gaze location.
the gaze location on the display, the confidence factor of the gaze location, or the speed of the change of the gaze location.
The method of claim 2 , wherein the fall-off is utilized as a rate of change of image quality of the plurality of tiles with respect to increased distance from the gaze location.
The method of claim 1 , wherein one or more tiles of the plurality of tiles that are proximate an edge of the display or in a user's peripheral vision have a larger size than at least one tile of the plurality of tiles that are proximate to a centerpoint of the display.
The method of claim 1 , wherein one or more tiles of the plurality of tiles that are proximate or at the gaze location have a smaller size than one or more tiles of the plurality of tiles that are distal from the gaze location.
The method of claim 1 , further comprising increasing a size of one or more tiles of the plurality of tiles proximate the gaze location in response to a decrease in the confidence factor of the gaze location.
The method of claim 1 , further comprising increasing the fall-off of the fovea region in response to an increase in the confidence factor of the gaze location.
The method of claim 1 , further comprising increasing a size of one or more tiles of the plurality of tiles that define the fovea region in response to an increase in the speed of the change of gaze location.
A head mounted display, comprising: a combiner configured to provide foveated imagery to a user; an imaging device configured to obtain eye tracking data; and processing circuitry configured to: receive the eye tracking data from the imaging device; determine a gaze location and at least one of a confidence factor of the gaze location or a time rate of change of the gaze location using the eye tracking data; define or adjust a fall-off of a fovea region defined by a plurality of tiles using the gaze location and at least one of the confidence factor or the time rate of change of the gaze location, the fall-off defining a change in image quality between adjacent tiles of the plurality of tiles; and provide a foveated rendered image to the user using the plurality of tiles and the combiner.
a combiner configured to provide foveated imagery to a user;
an imaging device configured to obtain eye tracking data; and
processing circuitry configured to: receive the eye tracking data from the imaging device; determine a gaze location and at least one of a confidence factor of the gaze location or a time rate of change of the gaze location using the eye tracking data; define or adjust a fall-off of a fovea region defined by a plurality of tiles using the gaze location and at least one of the confidence factor or the time rate of change of the gaze location, the fall-off defining a change in image quality between adjacent tiles of the plurality of tiles; and provide a foveated rendered image to the user using the plurality of tiles and the combiner.
receive the eye tracking data from the imaging device;
determine a gaze location and at least one of a confidence factor of the gaze location or a time rate of change of the gaze location using the eye tracking data;
define or adjust a fall-off of a fovea region defined by a plurality of tiles using the gaze location and at least one of the confidence factor or the time rate of change of the gaze location, the fall-off defining a change in image quality between adjacent tiles of the plurality of tiles; and
provide a foveated rendered image to the user using the plurality of tiles and the combiner.
The head mounted display of claim 9 , wherein one or more tiles of the plurality of tiles that are proximate an edge of the combiner or are in the user's peripheral vision are defined or adjusted to have a larger size than at least one tile of the plurality of tiles that are closer to a centerpoint of the combiner.
The head mounted display of claim 9 , wherein one or more tiles of the plurality of tiles that are proximate or at the gaze location have a smaller size than one or more tiles of the plurality of tiles that are distal from the gaze location.
The head mounted display of claim 9 , wherein the processing circuitry is configured to increase a size of one or more tiles of the plurality of tiles that are proximate the gaze location in response to a decrease in the confidence factor of the gaze location.
The head mounted display of claim 9 , wherein the processing circuitry is configured to increase the fall-off of the fovea region in response to an increase in the confidence factor of the gaze location.
The head mounted display of claim 9 , wherein the processing circuitry is configured to increase a size of one or more tiles of the plurality of tiles that define the fovea region in response to an increase in the speed of the change of the gaze location.
The head mounted display of claim 9 , wherein one or more tiles of the plurality of tiles that are proximate the gaze location are used to provide a higher quality image to the user, and one or more tiles of the plurality of tiles that are distal from the gaze location are used to provide a lower quality image to the user.
The head mounted display of claim 15 , wherein the fall-off defines the change of image quality between adjacent tiles of the plurality of tiles with different image quality with respect to increased distance from the gaze location.
A display system for providing foveated imagery to a user, the display system comprising: a combiner configured to provide imagery to the user; an imaging device configured to obtain eye tracking data; and processing circuitry configured to: receive the eye tracking data from the imaging device; determine a current gaze location, a confidence factor of the current gaze location, and a speed of the change of the current gaze location; define a first tile centered at the current gaze location and a plurality of tiles that surround the first tile using at least one of the current gaze location, the confidence factor of the gaze location, or the speed of the change of the current gaze location; and provide foveated imagery to the user using the first tile and the plurality of tiles.
a combiner configured to provide imagery to the user;
processing circuitry configured to: receive the eye tracking data from the imaging device; determine a current gaze location, a confidence factor of the current gaze location, and a speed of the change of the current gaze location; define a first tile centered at the current gaze location and a plurality of tiles that surround the first tile using at least one of the current gaze location, the confidence factor of the gaze location, or the speed of the change of the current gaze location; and provide foveated imagery to the user using the first tile and the plurality of tiles.
determine a current gaze location, a confidence factor of the current gaze location, and a speed of the change of the current gaze location;
define a first tile centered at the current gaze location and a plurality of tiles that surround the first tile using at least one of the current gaze location, the confidence factor of the gaze location, or the speed of the change of the curren
[Truncated to input limit]