Office Action 2: Final Rejection (2024-05-16)
Rejection types: 103 - Obviousness

CLAIMS:
AMENDMENTS TO THE CLAIMS

This listing of claims will replace all prior versions, and listings of claims in the Application.

Listing of the Claims:

1\. (CURRENTLY AMENDED) An eye-gaze tracking apparatus, comprising:- a plurality of foreground cameras each arranged to capture at least a portion of a target view area of a user; and- an eye-gaze estimation module having at least one eye-gaze camera each arranged to capture an image an eye of the user, so as to estimate a position of a pupil of the eye based on the image of the eye;
wherein each of the plurality of foreground cameras includes a field-of-view (FOV) being different from each other; and
wherein the foreground cameras and the eye-gaze cameras are adapted to capture images of both the target view of a user as well as the images of the eyes of the user simultaneously to determine eye-gaze positions of the user.
2\. (CURRENTLY AMENDED) The eye-gaze tracking apparatus in accordance with claim 1, wherein the plurality of foreground cameras include a wide-angle camera having a first FOV to estimate a wider view of the user and a magnified camera having a second FOV smaller than the first FOV to estimate a visual focus of the user.

3\. (ORIGINAL) The eye-gaze tracking apparatus in accordance with claim 2, wherein the second FOV equals to 600 diagonal.

4\. (ORIGINAL) The eye-gaze tracking apparatus in accordance with claim 1, wherein each of the at least one eye-gaze camera includes an infrared camera arranged to capture an infrared image of the eye of the user.

5\. (ORIGINAL) The eye-gaze tracking apparatus in accordance with claim 4, further comprising at least one infrared LED arranged to illuminate the eye of the user.

6\. (ORIGINAL) The eye-gaze tracking apparatus in accordance with claim 1, further comprising a calibration module arranged to facilitate a calibration of the eye-gaze estimation module according to characteristics of the eye of the user.

7\. (ORIGINAL) The eye-gaze tracking apparatus in accordance with claim 6, wherein the calibration module is arranged to provide a plurality of displayed markers such that the eye-gaze estimation module is arranged to map a plurality of calibration positions of the pupil of the eye to the plurality of displayed markers overlaid at different positions in the target view area of a user.

8\. (ORIGINAL) The eye-gaze tracking apparatus in accordance with claim 6, wherein the calibration module includes a plurality of physical markers such that the eye- gaze estimation module is arranged to map a plurality of calibration positions of the pupil of the eye to the plurality of physical markers disposed at different positions in the target view area of the user.
9\. (CURRENTLY AMENDED) The eye-gaze tracking apparatus in accordance with claim 6, wherein the calibration module is arranged to facilitate calibration of the eye-gaze estimation module with respect to different target view areas of a user associated with different FOVs of the plurality of foreground cameras, and using a combined gaze estimation method based on convolutional neural network and a geometric model to get an accurate gaze position estimation.

10\. (ORIGINAL) The eye-gaze tracking apparatus in accordance with claim 1, wherein the eye-gaze estimation module includes a neural network arranged to process the image of the eye captured by the at least one eye-gaze camera so as to estimate an eye-gaze angle of the user associated with the position of the pupil of the eye.

11\. (CURRENTLY AMENDED) A method of eye-gaze tracking, comprising the steps of:- capturing at least a portion of a target view area of a user using a plurality of foreground cameras; and- capturing an image an eye of the user using an eye-gaze camera, so as to estimate a position of a pupil of the eye based on the image of the eye;

wherein each of the plurality of foreground cameras includes a field-of-view (FOV) being different from each other; and
wherein the foreground cameras and the eye-gaze cameras are adapted to capture images of both the target view of a user as well as the images of the eyes of the user simultaneously to determine eye-gaze positions of the user.
12\. (CURRENTLY AMENDED) The method of claim 11, wherein the plurality of foreground cameras include a wide-angle camera having a first FOVto estimate a wider view of the user and a magnified camera having a second FOV smaller than the first FOV to estimate a visual focus of the user.

13\. (ORIGINAL) The method of claim 12, wherein the second FOV equals to 600 diagonal.

14\. (ORIGINAL) The method of claim 11, wherein each of the at least one eye- gaze camera includes an infrared camera arranged to capture an infrared image of the eye of the user.

15\. (ORIGINAL) The method of claim 14, further comprising the step of illuminating the eye of the user using at least one infrared LED.

16\. (ORIGINAL) The method of claim 11, further comprising the step of calibrating an eye-gaze estimation module according to characteristics of the eye of the user, wherein the eye-gaze estimation module is arranged to estimate the position of the pupil of the eye.

17\. (ORIGINAL) The method of claim 16, wherein the step of calibrating the eye-gaze estimation module comprises the steps of:- providing a plurality of displayed markers overlaid at different positions in the target view area of a user; and- mapping a plurality of calibration positions of the pupil of the eye to the plurality of displayed markers.

18\. (ORIGINAL) The method of claim 16, wherein the step of calibrating the eye-gaze estimation module comprises the steps of:- disposing a plurality of physical markers at different positions in the target view area of the user; and- mapping a plurality of calibration positions of the pupil of the eye to the plurality of physical markers.

19\. (CURRENTLY AMENDED) The method of claim 16, wherein the eye- gaze estimation module is calibrated with respect to different target view areas of a user associated with different FOVs of the plurality of foreground cameras, and using a combined gaze estimation method based on convolutional neural network and a geometric model to get an accurate gaze position estimation.

20\. (ORIGINAL) The method of claim 11, further comprising the step of processing the image of the eye captured by the at least one eye-gaze camera using a neural network, so as to estimate an eye-gaze angle of the user associated with the position of the pupil of the eye.

OFFICE ACTION:
DETAILED ACTION
Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in response to the amendment filed by Applicant on 10/31/2023. This action is made FINAL.
Response to Arguments
Applicant’s arguments with respect to claims 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 4-6, 9, 11, 12,14-16, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Sztuk et al (Publication number: US 2023/0119935) in view of Jacques (Patent number: US 9,185,352).
Consider Claim 1, Sztuk et al shows an eye-gaze tracking apparatus (see figure 1), comprising:
(a) A plurality of foreground cameras (read as cameras 193A-B) each arranged to capture at least a portion of a target view area of a user (See figures 1 and 2A; paragraphs 34, 35 and 36); (Camera 193D is configured to image the external environment to the left of head mounted device 100. Camera 193B is disposed in the upper-right corner of frame 102 and configured to image the forward-right external environment of head mounted device 100. Camera 193C is disposed in the upper-left corner of frame 102 and configured to image the forward-left external environment of head mounted device 100. The field of view (FOV) of camera 193B may overlap a FOV of camera 193C).
(b) An eye-gaze estimation module (read as cameras 108A-B) having at least one eye-gaze camera each arranged to capture an image an eye of the user, so as to estimate a position of a pupil of the eye based on the image of the eye, wherein the foreground cameras and the eye-gaze cameras are adapted to capture images of the target view of the user as well as the images of the eyes of the user (figures 1 and 2A; paragraph 25); (Cameras 108A and 108B may be configured to image an eyebox region to image the eye of the user to capture eye data of the user. Cameras 108A and 108B may be included in an eye-tracking system that is configured to determine a gaze direction of an eye (or eyes) of a user of the head mounted device. Cameras 108A and 108B may image the eyebox region directly or indirectly).
(c) Wherein each of the plurality of foreground cameras includes a field-of-view (FOV) being different from each other (see figure 2C; paragraph 48); (See FOV’s 297A-D. The FOV’s can be adjusted to be narrow or wide).
However, Sztuk et al does not specifically show that the foreground cameras and the eye-gaze cameras are adapted to capture images of both the target view of a user as well as the images of the eyes of the user simultaneously to determine eye-gaze positions of the user.
In the same field of endeavor, Jacques shows that the foreground cameras and the eye-gaze cameras are adapted to capture images of both the target view of a user as well as the images of the eyes of the user simultaneously to determine eye-gaze positions of the user (column 2, lines 25-50); (Jacques specifically states that “a  first camera is mounted to the user and the lens of the first camera is positioned in a manner that faces away from the user so that the camera collects scene information generally associated with the field of view of the user. A second camera is mounted to the user and the lens of the second camera is positioned in a manner so as to be in close proximity to and face toward an eye of the user so that the camera collects image information of the eye. The system is calibrated such that a determination is made as to the direction of the gaze of the eye based on the image information collected of the eye by the second camera. Using the scene information as input, the system determines a location of the identifying region of the field of view. When the direction of the gaze of the user's eye tracked by the second camera is within the identifying region, input may subsequently be provided by the user through movements of the eye to the computing device”).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the application to incorporate the teaching of Jacques into the teaching of Sztuk in order to map images of the pupil to points in space (see Jacques; column 1, lines 10-25).
Consider Claim 11, Sztuk et al shows a method of eye-gaze tracking (see figure 5), comprising the steps of:
(a) Capturing at least a portion of a target view area of a user using a plurality of foreground cameras (See figures 1 and 2A; paragraphs 34, 35 and 36); (Camera 193D is configured to image the external environment to the left of head mounted device 100. Camera 193B is disposed in the upper-right corner of frame 102 and configured to image the forward-right external environment of head mounted device 100. Camera 193C is disposed in the upper-left corner of frame 102 and configured to image the forward-left external environment of head mounted device 100. The field of view (FOV) of camera 193B may overlap a FOV of camera 193C).
(b) Capturing an image an eye of the user using an eye- gaze camera, so as to estimate a position of a pupil of the eye based on the image of the eye (figures 1 and 2A; paragraph 25); (Cameras 108A and 108B may be configured to image an eyebox region to image the eye of the user to capture eye data of the user. Cameras 108A and 108B may be included in an eye-tracking system that is configured to determine a gaze direction of an eye (or eyes) of a user of the head mounted device. Cameras 108A and 108B may image the eyebox region directly or indirectly).
(c) Wherein each of the plurality of foreground cameras includes a field-of-view (FOV) being different from each other (see figure 2C; paragraph 48); (See FOV’s 297A-D. The FOV’s can be adjusted to be narrow or wide).
However, Sztuk et al does not specifically show that the foreground cameras and the eye-gaze cameras are adapted to capture images of both the target view of a user as well as the images of the eyes of the user simultaneously to determine eye-gaze positions of the user.
In the same field of endeavor, Jacques shows that the foreground cameras and the eye-gaze cameras are adapted to capture images of both the target view of a user as well as the images of the eyes of the user simultaneously to determine eye-gaze positions of the user (column 2, lines 25-50); (Jacques specifically states that “a  first camera is mounted to the user and the lens of the first camera is positioned in a manner that faces away from the user so that the camera collects scene information generally associated with the field of view of the user. A second camera is mounted to the user and the lens of the second camera is positioned in a manner so as to be in close proximity to and face toward an eye of the user so that the camera collects image information of the eye. The system is calibrated such that a determination is made as to the direction of the gaze of the eye based on the image information collected of the eye by the second camera. Using the scene information as input, the system determines a location of the identifying region of the field of view. When the direction of the gaze of the user's eye tracked by the second camera is within the identifying region, input may subsequently be provided by the user through movements of the eye to the computing device”).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the application to incorporate the teaching of Jacques into the teaching of Sztuk in order to map images of the pupil to points in space (see Jacques; column 1, lines 10-25).
Consider Claims 2 and 12, Sztuk et al shows that the plurality of foreground cameras includes a wide-angle camera having a first FOV to estimate a wider view of the user and a magnified camera having a second FOV smaller than the first FOV to estimate a visual focus of the user (see figure 2C; paragraph 48); (See FOV’s 297A-D. The FOV’s can be adjusted to be narrow or wide).
Consider Claims 4 and 14, Sztuk et al shows that each of the at least one eye-gaze camera includes an infrared camera arranged to capture an infrared image of the eye of the user (see paragraphs 51 and 52); (A camera of eye-tracking system 460 may image eye 403 by receiving reflected infrared illumination light from an optical combiner layer included in optical element 410. The optical combiner layer may be configured to receive reflected infrared illumination light and redirect the reflected infrared illumination light to the camera of eye-tracking system 460. The camera would be oriented to receive the reflected infrared illumination light from the optical combiner layer of optical element 410).
Consider Claims 5 and 15, Sztuk et al shows at least one infrared LED arranged to illuminate the eye of the user (see paragraph 52).
Consider Claims 6 and 16, Sztuk et al shows a calibration module arranged to facilitate a calibration of the eye-gaze estimation module according to characteristics of the eye of the user (see paragraph 64); (Identifying a subject in an image that corresponds to a gaze direction of the user may utilize gaze calibration data that is recorded during a calibration procedure in an unboxing process of the head mounted device).
Claims 7-8 and 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Sztuk et al (Publication number: US 2023/0119935) in view of Jacques in view of Bastani et al (Publication number: US 2021/0166341).
Consider Claims 7 and 17, Sztuk et al in view of Jacques does not specifically show that the calibration module is arranged to provide a plurality of displayed markers such that the eye-gaze estimation module is arranged to map a plurality of calibration positions of the pupil of the eye to the plurality of displayed markers overlaid at different positions in the target view area of a user.
In the same field of endeavor, Bastani et al shows the calibration module is arranged to provide a plurality of displayed markers such that the eye-gaze estimation module is arranged to map a plurality of calibration positions of the pupil of the eye to the plurality of displayed markers overlaid at different positions in the target view area of a user (See figures 6 and 7; paragraphs 53, 55,); (Display 164 can perform foveated rendering based on the calibrated eye tracking operation, which can indicate a gaze point corresponding to the gaze direction generated by the eye tracking operation).
Therefore, it would have been obvious to a person of ordinary skill in the art at the effective filing date of the application to incorporate the teaching of Bastani et al into the teaching of Sztuk et al and Jacques in order to perform foveated rendering (see Bastani et al; paragraphs 54 and 55).
Consider Claims 8 and 18, Sztuk et al in view of Jacques does not specifically show that the calibration module includes a plurality of physical markers such that the eye-gaze estimation module is arranged to map a plurality of calibration positions of the pupil of the eye to the plurality of physical markers disposed at different positions in the target view area of the user.
In the same field of endeavor, Bastani shows that the calibration module includes a plurality of physical markers such that the eye-gaze estimation module is arranged to map a plurality of calibration positions of the pupil of the eye to the plurality of physical markers disposed at different positions in the target view area of the user (See figures 6 and 7; paragraphs 53, 55,); (Display 164 can perform foveated rendering based on the calibrated eye tracking operation, which can indicate a gaze point corresponding to the gaze direction generated by the eye tracking operation).
Therefore, it would have been obvious to a person of ordinary skill in the art at the effective filing date of the application to incorporate the teaching of Bastani et al into the teaching of Sztuk et al and Jacques in order to perform foveated rendering (see Bastani et al; paragraphs 54 and 55).
Claims 3 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Sztuk et al (Publication number: US 2023/0119935) in view of Jacques in view of Trail et al (Patent number: US 10,928635).
Consider Claims 3 and 13, Sztuk et al in view of Jacques does not specifically show that the second FOV equals to 60° diagonal.
In the same field of endeavor, Trail et al shows that the second FOV equals to 60° diagonal (see column 12, lines 15-30); (The expanded image light 340 provided to the user's eye 220 with a diagonal FOV of at least 60 degrees).
Therefore, it would have been obvious to a person of ordinary skill in the art at the effective filing date of the application to incorporate the teaching of Trail et al into the teaching of Sztuk et al and Jacques in order to output expanded image light to the user’s eye (see column 12, lines 15-30).
Claims 10 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Sztuk et al (Publication number: US 2023/0119935) in view of Jacques in view of Bathiche et al (Publication number: US 2022/0221932).
Consider Claims 10 and 20, Sztuk et al in view of Jacques does not specifically show that the eye-gaze estimation module includes a neural network arranged to process the image of the eye captured by the at least one eye- gaze camera so as to estimate an eye-gaze angle of the user associated with the position of the pupil of the eye.
In related art, Bathiche et al shows that the eye-gaze estimation module includes a neural network arranged to process the image of the eye captured by the at least one eye- gaze camera so as to estimate an eye-gaze angle of the user associated with the position of the pupil of the eye (paragraphs 62 and 63).
Therefore, it would have been obvious to a person of ordinary skill in the art at the effective filing date of the application to incorporate the teaching of Bathiche et al into the teaching of Sztuk et al and Jacques in order to generate gaze information (see Bathiche et al; paragraphs 62 and 63).
Claims 9 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Sztuk et al (Publication number: US 2023/0119935) in view of Jacques in view of YU et al (Publication number: US 2021/0319420).
Consider Claims 9 and 19, Sztuk et al in view of Jacques shows that the calibration module is arranged to facilitate calibration of the eye-gaze estimation module with respect to different target view areas of a user associated with different FOVs of the plurality of foreground cameras (see paragraph 64).
However, Sztuk et al in view of Jacques does not specifically show using a combined gaze estimation method based on convolutional neural network and a geometric model to get an accurate gaze position estimation.
In related art, YU et al shows using a combined gaze estimation method based on convolutional neural network and a geometric model to get an accurate gaze position estimation (see paragraphs 95 and 96).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the application to incorporate the teaching of YU et al into the teaching of Sztuk et al and Jacques in order to achieve robust visual object tracking (see YU et al; paragraphs 95 and 96).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action.  Accordingly, THIS ACTION IS MADE FINAL.  See MPEP § 706.07(a).  Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action.  In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action.  In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.

APPLICANT REBUTTAL: REMARKS
This amendment is in response to the Final Office Action dated May 16, 2024. To further the prosecution of this Application, Applicants submit the following remarks and amended claims. Additionally, Applicant submits a Request for Continued Examination. The claims as now presented are believed to be in allowable condition.
In the Claims:
Please amend claims 1, 3, 4, 6, 11, 13 and 14 in which the claim additions are shown by underlining and/or the claim deletions are shown by either strikeout or brackets. Please cancel claims 2 and 12. Please enter the amended claims into the record of this case.
Prior art publication date
The Applicant observed that the priority date of the current application is September 13, 2022. The publication date of the citation Sztuk et al. '953 is April 20, 2023. The applicant respectfully disagrees with the citation Sztuk et al. '953 qualifying as a printed publication before the effective filing date of the claimed Invention. Further, the Applicant objects to the combination of Jacques with the unpublished Sztuk et al for the rejections under 35 U.S.C. §103 as the combination would not be obvious before the effective filing date of the present application.
35 U.S.C. § 103 - Obviousness
The Office has rejected Claims 1, 2, 4-6, 9, 11, 12,14-16, and 19 under 35 U.S.C. 103 as being unpatentable over Sztuk et al (Publication number: US 2023/0119935) in view of Jacques (Patent number: US 9,185,352); Claims 7-8 and 17-18 under 35 U.S.C. 103 as being unpatentable over Sztuk et al in view of Jacques in view of Bastani et al (Publication number: US 2021/0166341); Claims 3 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Sztuk et al in view of Jacques in view of Trail et al (Patent number: US 10,928635); Claims 10 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Sztuk et al in view of Jacques in view of Bathiche et al (Publication number: US
2022/0221932); and Claims 9 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Sztuk et al in view of Jacques in view of YU et al (Publication number: US 2021/0319420). The Applicant acknowledges and respectfully traverses the raised obviousness rejections in view of the above amendments and the following remarks.
Assuming, arguendo, that Sztuk et al is a proper refence, the Applicant proposes to amend Claim 1 and Claim 11 to include the feature:
"wherein the plurality of foreground cameras comprise a wide-angle camera having a 100° diagonal FOV covering an entire target area to estimate a wider view of the user and a magnified camera having a 600 diagonal FOV associated with a "focus"
view or within a specific range or portion in the target area to obtain finer
foreground images of the specific area;
wherein the foreground cameras are adapted to capture images with a same
sampling frequency and a same maximum resolution of the captured images".
Support for these amendments to Claim 1 and Claim 11 are based on paragraphs [0047] and [0048] of the description.
Further, Claim 1 and Claim 11 include the feature:
"wherein the calibration module is adapted to carry out the steps of:
arranging four markers to be set in fixed positions of the foreground and be
distributed in four corners of a matching area,
obtaining a rectangular area consisted of the four markers both in images obtained
from the wide-angle camera and the magnified camera which are matching areas
obtaining a gaze estimation position in images firstly by the magnified camera and then obtaining the matching gaze estimation position in images of the wide-angle camera according to a relative position of the gaze estimation position in rectangle
formed by the four markers;
wherein the wide-angle camera and the magnified camera are matched by mapping a gaze estimation position of the magnified camera within a matching area to a foreground image of the wide-angle camera in a calibration".
This feature of the amended Claim 1 and Claim 11 is based on paragraphs [0060] and [0064] of the description.
The Applicant further proposes to amend Claim 3 and Claim 13 to include the feature of "the plurality of foreground cameras are adapted to capture a video stream of the users FOV at 30Hz with a highest image resolution of 1920* 1080 pixel, and the two near- eye infrared viewing cameras are adapted to capture user's eye images at 200*200 pixels at 200Hz". The amendments to Claim 3 and Claim 13 are based on paragraphs [0051] and [0052] of the description.
The Applicant proposes to amend Claim 4 and Claim 14 to include the feature of "near-eye infrared viewing cameras has a package size of 10*45*7 mm and are adapted to capture images at 860 nm wavelength". The amended Claim 4 and Claim 14 are based on paragraphs [0052] of the description.
Now turning to the cited art, Jacques discloses a device with a first camera mounted to the user and the lens of the first camera is positioned in a manner that faces away from the user so that the camera collects scene information generally associated with the field of view of the user. A second camera is mounted to the user and the lens of the second camera is positioned in a manner so as to be in close proximity to and face toward an eve of the user so that the camera collects image information of the eve. The system is calibrated such that a determination is made as to the direction of the gaze of the eve based on the image information collected of the eve by the second camera. Using the scene information as input, the system determines a location of the identifying region of the field of view. When the direction of the gaze of the user's eve tracked by the second camera is within the identifying region, input may subsequently be provided by the user through movements of the eve to the computing device".
However, the Applicant cannot find anywhere in Jacques that teaches, suggests or even motivates a person skilled in the art that the foreground cameras and the eye-gaze cameras are adapted to capture images of both the target view of a user as well as the images of the eyes of the user
simultaneously
to determine eye-gaze positions of the user.
The Applicant direct the Occie to paragraph [0048] of Sztuk et al which states that
"the FOVs can be rearranged to lap or overlap by moving the orientation of a camera or widening or narrowing the FOV by adjusting a lens assembly of the
camera"
A plurality of adjustable cameras is different from having a wide-angle camera and a magnified camera.
Sztuk et al does not teach or direct a person skilled in the art of a combination of a wide-angle camera and a magnified camera. As such, it is impossible for Sztuk et al to teach or direct the person skilled in the art of at least the feature:
diagonal FOV covering an entire target area to estimate a wider view of the user and a magnified camera having a 60° diagonal FOV associated with a "focus" view or within a specific range or portion in the target area to obtain finer foreground
images of the specific area;
wherein the foreground cameras are adapted to capture images with a same
sampling frequency and a same maximum resolution of the captured images"
Similarly, Jacques also does not disclose or direct a person skilled in the art the above feature.
Therefore, it is impossible for Sztuk et al, Jacques or any combination of prior art to further teach or direct a person skilled in the art to the calibration method with a plurality of foreground cameras. In particular, the calibration method comprising the step of
"arranging four markers to be set in fixed positions of the foreground and be
distributed in four corners of a matching area,
obtaining a rectangular area consisted of the four markers both in images obtained
from wide-angle camera and the magnified camera which are matching areas
obtaining a gaze estimation position in images firstly by the magnified camera and then obtaining the matching gaze estimation position in images of the wide-angle camera according to a relative position of the gaze estimation position in rectangle
formed by the four markers"
It is also impossible for a person skilled in the art to derive in light of the prior art considered alone or in combination the feature of:
"the wide-angle camera and the magnified camera to be matched by mapping a gaze estimation position of magnified camera within a matching area to a
foreground image of wide-angle camera in a calibration"
Sztuk et al discloses the method of
se
l
ectin
g
a particular image sensor of camera based on the gaze vector within the FOV of a particular camera (see para [0048] of Sztuk et al). This is completely different to the calibration method and working of the present invention where the wide-angle camera and the magnified camera are matched by mapping a gaze estimation position of the magnified camera within a
matching
area to a foreground image of the wide-angle camera in a calibration.
As indicated in paragraph [0050] of the specification, the present invention is different from prior art systems which may only capture rough foreground images which are not suitable for tracking the subtle gaze movements such as writing a Traditional Chinese character in a 1.5*1.5 cm2 lattice. Advantageously, the additional foreground camera may be used to capture fine foreground images.
Sztuk et al discloses a gaze direction of an eye or eyes of a user of a head mounted device is determined. One or more images are captured with a camera of a head mounted
device. Gaze-guided images are generated from the one or more images based on the gaze direction of the user.
Bastani et al discloses a method for providing imagery to a user on a display that includes receiving eye tracking data. The method also includes determining a gaze location on the display and at least one of a confidence factor of the gaze location, or a speed of the change of the gaze location using the eye tracking data. The method also includes establishing multiple tiles using the gaze location and at least one of the confidence factor or the speed of the change of the gaze location. The method also includes providing a foveated rendered image using the multiple tiles.
The present claims are directed to the inaccuracy of the prior art eye tracking systems which may vary much, sometimes over 3 degrees, depending on the calibration and applications (see page 5 line 38 to page 6 line 1). The inventors devised an eye tracking system that may be categorized into head-mounted and desktop-based on different applications, and the system accuracy of prior art eye trackers may range from 0.5 degrees to 1 degree (page 5 lines 33 to 36).
Sztuk et al does not disclose any device for eye tracking but rather disclose a system to capture or generate images based on the gaze direction of an eye of a user that has been determined by an eye-tracking system (see paragraph 21). As such, the foreground cameras (193A, 193B, 193C, 193D) and the cameras (108A-B) of the Sztuk et al could not achieve or direct a person skilled in the art to solve the problem of the present invention. Sztuk et al has taken for granted that the eye-tracking system is accurate in order to capture or generate images based on gaze direction. On the contrary, the present invention is provided to determine the unknown eye-gaze positions in high accuracy (0.5 degrees to 1 degree error) with the input of the foreground camera 102 and the eye-gaze camera 106. Therefore, Sztuk et al does not disclose the present invention defined in Claim 1 and Claim 11. Claim 1 and Claim 11 are novel when compared to Sztuk et al.
As Sztuk et al does not direct or teach a person skilled in the art to determine the gazing direction or improve the accuracy of the gazing direction, it could not teach or direct a person skilled in the art to the feature that the foreground cameras and the eye-gaze
cameras are adapted to capture images of both the target view of a user as well as the images of the eyes of the user simultaneously to determine eye-gaze positions of the user. Therefore, Claim 1 and Claim 11 are not obvious when compared to Sztuk et al.
The function of the FOV mentioned in Sztuk et al is to capture a panoramic image viewable by the user (see figure2C; paragraph 48); (See FOV's297A-D. The FOVs can be adjusted to be narrow or wide). The function of the wide-angle camera in Claim 2 and Claim 12 is to estimate a wider view of the user and that of a magnified camera having a second FOV smaller than the first FOV to estimate a visual focus of the user. Since Sztuk et al does not disclose a wide-angle camera and a magnified camera together, it is impossible for Sztuk et al to direct a person skilled in the art to feature that a wide-angle camera having a first FOV to estimate a wider view of the user and a magnified camera having a second FOV smaller than the first FOV to estimate a visual focus of the user. Therefore, Claim 2 and Claim 12 are not obvious when compared to Sztuk et al.
Claim 9 and Claim 19 further define the feature of using a combined gaze estimation method based on convolutional neural network and a geometric model to get an accurate gaze position estimation. (See Page 8 lines 31 to 33). Neither Sztuk et al nor Bastani et al disclose to nor direct a person skilled in the art that an eye-tracking system or method has an eye-gaze estimation module that is calibrated with respect to different target view areas of a user associated with different FOVs from a wider view of the user and a magnified camera (as defined in Claim 2 and Claim 12), and using a combined gaze estimation method based on convolutional neural network and a geometric model to get the accurate gaze position estimation. Therefore, Claim 9 and Claim 19 are novel and not obvious when compared to Sztuk et al and Bastani et al either alone or in combination.
It is submitted that the invention defined in the proposed amendment claims is novel and not obvious to a person skilled in the art when compared to Sztuk et al and Bastani et al. It is therefore submitted that all claims as amended are novel and inventive. Reconsideration of allowance of the application is requested respectfully.
If any further amendment to this application is believed necessary to advance prosecution and place this case in allowable form, the Examiner is courteously solicited to contact the undersigned representative of the Applicant to discuss the same.
Conclusion

NEXT OFFICE ACTION: DETAILED ACTION
Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection.  Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114.  Applicant's submission filed on 08/15/2024 has been entered.
EXAMINER’S AMENDMENT
An examiner’s amendment to the record appears below. Should the changes and/or additions be unacceptable to applicant, an amendment may be filed as provided by 37 CFR 1.312. To ensure consideration of such an amendment, it MUST be submitted no later than the payment of the issue fee.
Authorization for this examiner’s amendment was given in an interview with Ms. Kimberly Peaslee (64,311) on 08/27/2024.
The application has been amended as follows:
(Currently amended) An eye-gaze tracking apparatus, comprising:
a plurality of foreground cameras each arranged to capture at least a portion of a target view area of a user; and
an eye-gaze estimation module comprising two near-eye infrared viewing cameras each arranged to capture an image of an eye of the user, each estimate a position of a pupil of the eye based on the image of the eye of the user; and
a calibration module;
wherein the plurality of foreground cameras comprises a wide-angle camera having a 100° diagonal FOV covering an entire target area to estimate a wider view of the user and a magnified camera having a 60° diagonal FOV to provide comprising a portion of the entire
wherein the plurality of foreground cameras provide captured images with a same sampling frequency and a same maximum resolution
wherein the calibration module is adapted to carry out the steps of:
arranging four markers a foreground such that they create ;
obtaining a rectangular area marked by the four markers the captured images obtained from the wide-angle camera and the magnified camera which are the matching area[[s]];
obtaining a gaze estimation position in images captured by the magnified camera and then obtaining a matching gaze estimation position in images captured by s in [[a]]the rectangle formed by the four markers;
wherein the wide-angle camera and the magnified camera are matched by mapping [[a]]the gaze estimation position of the magnified camera within the[[a]] matching area to a foreground image of the wide-angle camera in a calibration; and
wherein the foreground cameras and the eye-gaze cameras are adapted to capture images of both the target view of a user and images of the eyes of the user simultaneously to determine eye-gaze positions of the user.
(Cancelled).
(Currently Amended) The eye-gaze tracking apparatus in accordance with claim 1, wherein the plurality of foreground cameras are adapted to capture a video stream of the user’s FOV at 30Hz with a highest image resolution of 1920*1080 pixel, and the two near-eye infrared viewing cameras are adapted to capture user’s eye images at 200*200 pixels at 200Hz.
(Currently amended) The eye-gaze tracking apparatus in accordance with claim 3, wherein each of the near-eye infrared viewing cameras has a package size of 10*45*7 mm and each are adapted to capture images at 860 nm wavelength.
(Original) The eye-gaze tracking apparatus in accordance with claim 4, further comprising at least one infrared LED arranged to illuminate the eye of the user.
(Previously amended) The eye-gaze tracking apparatus in accordance with claim 5, further comprising the calibration module arranged to facilitate a calibration of the eye-gaze estimation module according to characteristics of the eye of the user.
(Currently Amended) The eye-gaze tracking apparatus in accordance with claim 6, wherein the calibration module is arranged to provide a plurality of displayed markers such that the eye-gaze estimation module is arranged to map a plurality of calibration positions of the pupil of the eye of the user to the plurality of displayed markers overlaid at different positions in the target view area of a user.
(Currently Amended) The eye-gaze tracking apparatus in accordance with claim 6, wherein the calibration module includes a plurality of physical markers such that the eye-gaze estimation module is arranged to map a plurality of calibration positions of the pupil of the eye of the user to the plurality of physical markers disposed at different positions in the target view area of the user.
(Previously amended) The eye-gaze tracking apparatus in accordance with claim 6, wherein the calibration module is arranged to facilitate calibration of the eye-gaze estimation module with respect to different target view areas of a user associated with different FOVs of the plurality of foreground cameras, and using a combined gaze estimation method based on convolutional neural network and a geometric model to get an accurate gaze position estimation.
(Currently Amended) The eye-gaze tracking apparatus in accordance with claim 1, wherein the eye-gaze estimation module includes a neural network arranged to process the image of the eye captured by the two eye-gaze cameras
(Currently amended) A method of eye-gaze tracking, comprising the steps of:
carrying out a calibration method by a calibration module;
capturing at least a portion of a target view area of a user using a plurality of foreground cameras; and
capturing an image of each eye of the user using a respective near-eye infrared viewing camera[[s]], of the user based on the image of the eye;
wherein the plurality of foreground cameras comprises a wide-angle camera having a 100° diagonal FOV covering an entire target area to estimate a wider view of the user and a magnified camera having a 60° diagonal FOV associated with a “focus view comprising a portion of [[in]] the entire target area
wherein the plurality of foreground cameras are adapted to capture images with a same sampling frequency and a same maximum resolution
wherein the calibration method comprises the steps of:
arranging four markers a foreground such that they create four corners of a matching area[[,]];
obtaining a rectangular area marked by the four markers the matching area[[s]];
obtaining a gaze estimation position in images captured by the magnified camera and then obtaining a[[the]] matching gaze estimation position in images captured by [[of]] the wide-angle camera according to [[a]] relative positions of the gaze estimation position in [[a]]the rectangle formed by the four markers;
wherein the wide-angle camera and the magnified camera are matched by mapping [[a]]the gaze estimation position of the magnified camera within a matching area to a foreground image of the wide-angle camera in a calibration; and
wherein the foreground cameras and the eye-gaze cameras are adapted to capture images of both the target view of a user and the images of the eyes of the user simultaneously to determine eye-gaze positions of the user.
(Cancelled).
(Currently amended) The method of claim 11, wherein the plurality of foreground cameras are adapted to capture a video stream of the user’s FOV at 30Hz with a highest image resolution of 1920*1080 pixel, and the two near-eye infrared viewing cameras are adapted to capture user’s eye images at 200*200 pixels at 200Hz.
(Currently amended) The method of claim 13, wherein each of the two near-eye infrared viewing cameras has a package size of 10*45*7 mm and are adapted to capture images at 860 nm wavelength.
(Previously amended) The method of claim 14, further comprising the step of illuminating the eye of the user using at least one infrared LED.
(Original) The method of claim 11, further comprising the step of calibrating an eye-gaze estimation module according to characteristics of the eye of the user, wherein the eye-gaze estimation module is arranged to estimate the position of the pupil of the eye.
(Original) The method of claim 16, wherein the step of calibrating the eye-gaze estimation module comprises the steps of:
providing a plurality of displayed markers overlaid at different positions in the target view area of a user; and
mapping a plurality of calibration positions of the pupil of the eye to the plurality of displayed markers.
(Original) The method of claim 16, wherein the step of calibrating the eye-gaze estimation module comprises the steps of:
disposing a plurality of physical markers at different positions in the target view area of the user; and
mapping a plurality of calibration positions of the pupil of the eye to the plurality of physical markers.
(Previously amended) The method of claim 16, wherein the eye-gaze estimation module is calibrated with respect to different target view areas of a user associated with different FOVs of the plurality of foreground cameras and using a combined gaze estimation method based on convolutional neural network and a geometric model to get an accurate gaze position estimation.
(Currently Amended) The method of claim 11, further comprising the step of processing the image of the eye captured by the two eye-gaze cameras using a neural network,
Allowable Subject Matter
Claims 1, 3-11, and 13-20 are allowed.
The following is an examiner’s statement of reasons for allowance: The best prior art of record, i.e., Sztuk et al (Publication number: US 2023/0119935) in view of Jacques (Patent number; US 9,185,352) do not specifically show the limitations of “arranging four markers to be set in fixed positions of the a foreground such that they create and be distributed in four corners of a matching area, obtaining a rectangular area consisted of marked by the four markers both in the captured images obtained from the wide-angle camera and the magnified camera which are the matching area; obtaining a gaze estimation position in images firstly captured by the magnified camera and then obtaining the a matching gaze estimation position in images captured by of the wide-angle camera according to a relative position of the gaze estimation positions in the rectangle formed by the four markers; wherein the wide-angle camera and the magnified camera are matched by mapping the gaze estimation position of the magnified camera within the matching area to a foreground image of the wide-angle camera in a calibration; and wherein the foreground cameras and the eye-gaze cameras are adapted to capture images of both the target view of a user as well as the and images of the eyes of the user simultaneously to determine eye-gaze positions of the user.”
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee.  Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”
Conclusion

CITED REFERENCE SUMMARIES: Ref US9185352B1: Claims:
An eye tracking system that provides input to a computer, the system comprising: a computing device; a monitor coupled to the computing device, the monitor having an identifying region that is recognizable by an indication of the identifying region, wherein the computing device is programmed to recognize the indication of the identifying region; a first camera adapted to collect information related to a location of the identifying region of the monitor based on a location of the indication of the identifying region, the first camera being adapted to be mounted to a user such that a lens of the first camera faces away from the user; and a second camera having a lens that is adapted to be disposed in close proximity to and facing an eye of the user and being adapted to collect information from the eye of the user through the lens to provide an indication of a direction of gaze of the eye of the user, the information being provided as an input to the computing device.
a computing device;
a monitor coupled to the computing device, the monitor having an identifying region that is recognizable by an indication of the identifying region, wherein the computing device is programmed to recognize the indication of the identifying region;
a first camera adapted to collect information related to a location of the identifying region of the monitor based on a location of the indication of the identifying region, the first camera being adapted to be mounted to a user such that a lens of the first camera faces away from the user; and
a second camera having a lens that is adapted to be disposed in close proximity to and facing an eye of the user and being adapted to collect information from the eye of the user through the lens to provide an indication of a direction of gaze of the eye of the user, the information being provided as an input to the computing device.
The eye tracking system of claim 1 , wherein the indication of the identifying region of the monitor includes at least one signal emission.
The eye tracking system of claim 1 , further comprising at least one additional monitor coupled to the computing device and having a second identifying region that is recognizable by a second indication.
The eye tracking system of claim 1 , further comprising at least a second monitor coupled to at least a second computing device, the second monitor having a second identifying region that is recognizable by a second indication, information collected from the second camera which provides the indication of the direction of the gaze of the eye being provided as input to the second computing device.
The eye tracking system of claim 1 , further comprising a third camera having a lens that is adapted to be disposed in close proximity to and facing a second eye of the user, the third camera being adapted to collect supplemental information from the second eye which provides an indication of a direction of a gaze of the second eye.
The eye tracking system of claim 1 , wherein the computing device is programmed to determine from where on the monitor the direction of gaze of the eye intersects information regarding an indication of the direction of the gaze of the eye the identifying region of the monitor.
A method for providing computer input by tracking a direction of a gaze of an eye, the method comprising: providing a computing device; providing a monitor of the computing device, the monitor having an indication marking an identifying region of the monitor; mounting a first camera to a user and positioning a lens of the first camera to face generally in a direction of the field of view of the user to collect scene information; mounting a second camera to the user and positioning a lens of the second camera to face toward an eye of the user to collect information relating to a direction of a gaze of an eye; using the scene information collected from the first camera to determine a location of the identifying region of the monitor; using the information collected from the second camera to determine the direction of the gaze of the eye; and providing input to the computing device based on the direction of the gaze of the eye.
providing a computing device;
providing a monitor of the computing device, the monitor having an indication marking an identifying region of the monitor;
mounting a first camera to a user and positioning a lens of the first camera to face generally in a direction of the field of view of the user to collect scene information;
mounting a second camera to the user and positioning a lens of the second camera to face toward an eye of the user to collect information relating to a direction of a gaze of an eye;
using the scene information collected from the first camera to determine a location of the identifying region of the monitor;
using the information collected from the second camera to determine the direction of the gaze of the eye; and
providing input to the computing device based on the direction of the gaze of the eye.
The method of claim 7 , wherein using the scene information collected from the first camera to determine a location of the identifying region comprises detecting light emission from the indication.
The method of claim 7 , further comprising providing at least one second monitor having a second indication marking a second identifying region of the at least one second monitor.
The method of claim 9 , further comprising providing input to a second computing device when the direction of the gaze of the eye falls within a boundary of the second identifying region of the at least one second monitor.
The method of claim 7 , further comprising correcting for parallax error by mounting a third camera to the user and positioning a lens of the third camera to face toward a second eye of the user to collect supplemental information relating to a direction of a gaze of the second eye.
The method of claim 7 , wherein providing input to the computing device occurs when the direction of the gaze of the eye falls within the identifying region.
The method of claim 7 , further comprising utilizing an algorithm that employs a perspective correction algorithm and a parallax correction algorithm to correct for error of the direction of the gaze of the eye when the direction of the gaze of the eye falls within the identifying region of the monitor.
At least one non-transitory computer storage medium comprising computer-executable instructions that, when executed by at least one processor, perform a method for calculating a location of a center of a pupil of an eye, the method comprising: a) generating an image of a pupil of an eye comprising a distribution of pixels, each pixel having an intensity value and a geometric position on the image; b) selecting pixels having intensity values that fall within a defined range of intensity values; c) determining an average geometric position of the pixels having intensity values that fall within the defined range, the average geometric position being a first starting point for determining a location of a center of the pupil of the eye on the image; d) making measurements of a value of pixel intensity along lines extending in a plurality of directions away from the starting point; e) for each of the lines extending away from the starting point, determining a stop point, each stop point being determined based on a pixel intensity change exceeding a first threshold; f) for each stop point, making measurements of a value of pixel intensity along lines extending in a plurality of directions from the stop point back toward the starting point and at a plurality of points adjacent to the starting point; g) for each line from the stop point back to the starting point and a plurality of points adjacent to the starting point, defining a feature point having a geometric position, each feature point being determined based on a pixel intensity change exceeding second threshold; and h) averaging the geometric positions of the feature points to calculate a second starting point for determining the location of the center of the pupil of the eye on the image.
a) generating an image of a pupil of an eye comprising a distribution of pixels, each pixel having an intensity value and a geometric position on the image;
b) selecting pixels having intensity values that fall within a defined range of intensity values;
c) determining an average geometric position of the pixels having intensity values that fall within the defined range, the average geometric position being a first starting point for determining a location of a center of the pupil of the eye on the image;
d) making measurements of a value of pixel intensity along lines extending in a plurality of directions away from the starting point;
e) for each of the lines extending away from the starting point, determining a stop point, each stop point being determined based on a pixel intensity change exceeding a first threshold;
f) for each stop point, making measurements of a value of pixel intensity along lines extending in a plurality of directions from the stop point back toward the starting point and at a plurality of points adjacent to the starting point;
g) for each line from the stop point back to the starting point and a plurality of points adjacent to the starting point, defining a feature point having a geometric position, each feature point being determined based on a pixel intensity change exceeding second threshold; and
h) averaging the geometric positions of the feature points to calculate a second starting point for determining the location of the center of the pupil of the eye on the image.
The non-transitory computer storage medium of claim 14 , further comprising repeating steps d), e), f), g) and h) for at least one iteration until a difference between a geometric position of the first starting point and the second starting point is less than about 1% in distance.
The non-transitory computer storage medium of claim 14 , wherein averaging the geometric positions of the plurality of feature points comprises averaging the geometric positions of the plurality of feature points that are accumulated for every iteration of the at least one iteration.
The non-transitory computer storage medium of claim 14 , wherein averaging the geometric positions of the plurality of feature points comprises averaging the geometric positions of the plurality of feature points that are accumulated for only a current iteration of the at least one iteration.
A method for estimating a location of a center of a pupil of an eye from an image of the eye having pixels, the method comprising: generating an image of a pupil of an eye comprising a distribution of pixels, each pixel having an intensity value and a geometric position on the image; selecting pixels having intensity values that fall within a defined range of intensity values; determining the geometric positions of the pixels having intensity values that fall within the defined range of intensity values; determining an average geometric position of the pixels having intensity values that fall within the defined range, the average geometric position being a first starting point for determining a location of a center of the pupil of the eye on the image; making measurements of a value of pixel intensity along lines extending in a plurality of directions away from the starting point; for each of the lines extending away from the starting point, determining a stop point, each stop point being determined based on a pixel intensity change exceeding a first threshold; for each stop point, making measurements of a value of pixel intensity along lines extending in a plurality of directions from the stop point back toward the starting point and at a plurality of points adjacent to the starting point; for each line from the stop point back to the starting point and a plurality of points adjacent to the starting point, defining a feature point having a geometric position, each feature point being determined based on a pixel intensity change exceeding a second threshold; and averaging the geometric positions of the feature points to calculate a second starting point for determining the location of the center of the pupil of the eye on the image.
generating an image of a pupil of an eye comprising a distribution of pixels, each pixel having an intensity value and a geometric position on the image;
selecting pixels having intensity values that fall within a defined range of intensity values;
determining the geometric positions of the pixels having intensity values that fall within the defined range of intensity values;
determining an average geometric position of the pixels having intensity values that fall within the defined range, the average geometric position being a first starting point for determining a location of a center of the pupil of the eye on the image;
making measurements of a value of pixel intensity along lines extending in a plurality of directions away from the starting point;
for each of the lines extending away from the starting point, determining a stop point, each stop point being determined based on a pixel intensity change exceeding a first threshold;
for each stop point, making measurements of a value of pixel intensity along lines extending in a plurality of directions from the stop point back toward the starting point and at a plurality of points adjacent to the starting point;
for each line from the stop point back to the starting point and a plurality of points adjacent to the starting point, defining a feature point having a geometric position, each feature point being determined based on a pixel intensity change exceeding a second threshold; and
averaging the geometric positions of the feature points to calculate a second starting point for determining the location of the center of the pupil of the eye on the image.
A mobile eye tracking system, the system comprising: an identifying region of a field of view, the identifying region being recognizable by an indication; a computing device programmed to recognize the indication of the field of view; a first camera adapted to collect information related to a location of the identifying region of the field of view based on the indication of the identifying region, the first camera being adapted to be mounted to a user such that a lens of the first camera faces away from the user; and a second camera adapted to collect information through a lens from the eyes from which a direction of a gaze of an eye of the user may be determined, the lens being adapted to be disposed in close proximity to and facing an eye of the user, the information collected from the second camera being provided as an input to the computing device.
an identifying region of a field of view, the identifying region being recognizable by an indication;
a computing device programmed to recognize the indication of the field of view;
a first camera adapted to collect information related to a location of the identifying region of the field of view based on the indication of the identifying region, the first camera being adapted to be mounted to a user such that a lens of the first camera faces away from the user; and
a second camera adapted to collect information through a lens from the eyes from which a direction of a gaze of an eye of the user may be determined, the lens being adapted to be disposed in close proximity to and facing an eye of the user, the information collected from the second camera being provided as an input to the computing device.
A method for providing computer input based upon a direction of a gaze of an eye of a user, the method comprising: providing an indication marking an identifying region of a field of view; mounting a first camera to the user and positioning a lens of the first camera to face away from the user to collect scene information; mounting a second camera to the user and positioning a lens of the second camera to face the eye of the user to collect information from the eye from which a direction of gaze of the eye may be determined; determining a location of the identifying region of the field of view from the indication based upon the scene information collected from the first camera; determining the direction of the gaze of the eye using the information collected from the second camera; and providing an input to the computing device based on a location of an intersection of the direction of the gaze of the eye and the identifying region of the field of view.
providing an indication marking an identifying region of a field of view;
mounting a first camera to the user and positioning a lens of the first camera to face away from the user to collect scene information;
mounting a second camera to the user and positioning a lens of the second camera to face the eye of the user to collect information from the eye from which a direction of gaze of the eye may be determined;
determining a location of the identifying region of the field of view from the indication based upon the scene information collected from the first camera;
determining the direction of the gaze of the eye using the information collected from the second camera; and
providing an input to the computing device based on a location of an intersection of the direction of the gaze of the eye and the identifying region of the field of view.
Ref US20210319420A1: Claims:
A computer-implemented method for retail, comprising: tracking, based on a machine learning model with a deformable self-attention mechanism and a deformable cross-attention mechanism, a plurality of locations of a product in a plurality of images; determining a retail event type associated with the product based on the plurality of locations of the product in the plurality of images; and in response to the retail event type being irregular, generating a message for loss prevention.
tracking, based on a machine learning model with a deformable self-attention mechanism and a deformable cross-attention mechanism, a plurality of locations of a product in a plurality of images;
determining a retail event type associated with the product based on the plurality of locations of the product in the plurality of images; and
in response to the retail event type being irregular, generating a message for loss prevention.
The method of claim 1 , wherein the deformable self-attention mechanism comprises: generating, via spatial attention, self-spatial-attention search features to encode context information of search features of a search image of the product, and self-spatial-attention template features to encode context information of template features of a template; and generating, via channel attention, self-channel-attention search features to encode channel information of the search features, and self-channel-attention template features to encode channel information of the template features.
generating, via spatial attention, self-spatial-attention search features to encode context information of search features of a search image of the product, and self-spatial-attention template features to encode context information of template features of a template; and
generating, via channel attention, self-channel-attention search features to encode channel information of the search features, and self-channel-attention template features to encode channel information of the template features.
The method of claim 2 , wherein the deformable cross-attention mechanism further comprises: generating, via spatial attention, cross-spatial-attention search features and cross-spatial-attention template features to encode contextual interdependency information between the search features and the template features.
generating, via spatial attention, cross-spatial-attention search features and cross-spatial-attention template features to encode contextual interdependency information between the search features and the template features
[Truncated to input limit]