Office Action 1: Non-Final Rejection (2024-04-08)
Rejection types: 103 - Obviousness

CLAIMS:
AMENDMENTS TO THE CLAIMS

Applicant submits below a complete listing of the current claims, including marked-upclaim(s) with insertions indicated by underlining and deletions indicated by strikeouts and/or double bracketing. This listing of claims replaces all prior versions, and listings, of claims in the application.

Listing of the Claims:

1-20. (Canceled)219

21\. (New) An augmented reality system, comprising:at least one image sensor comprising:
a plurality of pixel cells, each pixel cell of the plurality of pixel cells comprising a light-sensitive component;
first circuitry coupled to light-sensitive components of pixel cells of the plurality of pixel cells and configurable to provide, as output signals from at least some pixel cells of the plurality of pixel cells, indications of light intensity; and
second circuitry coupled to light-sensitive components of pixel cells of the plurality of pixel cells and configurable to provide, as output signals from at least some pixel cells of the plurality of pixel cells, indications of change in a light property; and
at least one processor configured to:
for a first image data processing technique, receive, from the first circuitry, indications of light intensity; and
for a second image data processing technique, receive, from the second circuitry, indications of change in the light property.

22\. (New) The augmented reality system of claim 21, wherein the at least one processor is further configured to:perform the first image data processing technique, the first image data processing technique comprising generating or updating a world model using the indications of light intensity.

23\. (New) The augmented reality system of claim 21, wherein the at least one processor is further configured to:perform the first image data processing technique using indications of light intensity for a full image frame.

24\. (New) The augmented reality system of claim 21, wherein the at least one processor is further configured to:perform the first image data processing technique using indications of light intensity captured using a rolling shutter mode.

25\. (New) The augmented reality system of claim 21, wherein the at least one processor is further configured to:perform the first image data processing technique using indications of light intensity captured using a global shutter mode.

26\. (New) The augmented reality system of claim 21, wherein:the at least one image sensor comprises optical components disposed in light paths at least some pixel cells of the plurality of pixel cells;
the optical components are configured to convert an angle of incident light into light intensity; and
the at least one processor is further configured to perform the first image data processing technique, the first image data processing technique comprising generating passive depth information using the indications of light intensity.

27\. (New) The augmented reality system of claim 21, wherein the augmented reality system comprises a wearable headset.

28\. (New) Non-transitory computer-readable medium comprising computer-executable instructions, configured for execution on at least one processor of an augmented reality system comprising at least one image sensor comprising a plurality of pixel cells, each pixel cell of the plurality of pixel cells comprising a light-sensitive component; first circuitry coupled to light- sensitive components of pixel cells of the plurality of pixel cells and configurable to provide, as output signals from at least some pixel cells of the plurality of pixel cells, indications of light intensity; and second circuitry coupled to light-sensitive components of pixel cells of the plurality of pixel cells and configurable to provide, as output signals from at least some pixel cells of the plurality of pixel cells, indications of change in a light property, the computer-executable instructions configured to, when executed, control the at least one processor to:receive for processing, based on an image processing technique to be performed, indications of light intensity from the first circuitry or indications of change in the light property from the second circuitry, such that:
for a first image data processing technique, the at least one processor receives, from the first circuitry, indications of light intensity; and
for a second image data processing technique, the at least one processor receives,
from the second circuitry, indications of change in the light property

29\. (New) The non-transitory computer-readable medium of claim 28, wherein the computer- executable instructions further comprise computer-executable instructions configured to, when executed control the at least one processor to:perform the first image data processing technique, wherein the first image data processing technique comprises tracking an object or pose using the indications of change in the light property.

30\. (New) The non-transitory computer-readable medium of claim 28, wherein the computer- executable instructions further comprise computer-executable instructions configured to, when executed control the at least one processor to:

perform the second image data processing technique using the indications of change in the light property for an image patch.

31\. (New) The non-transitory computer-readable medium of claim 28, wherein the computer- executable instructions further comprise computer-executable instructions configured to, when executed control the at least one processor to:perform the first image data processing technique using indications of light intensity received at a first data rate; and
perform the second image data processing technique using indications of change in the light property received at a second data rate.

32\. (New) The non-transitory computer-readable medium of claim 28, wherein the computer- executable instructions further comprise computer-executable instructions configured to, when executed control the at least one processor to:perform the first image data processing technique using indications of light intensity in response to an indication to operate in a first power consumption mode; and
perform the second image data processing technique using indications of change in the light property in response to an indication to operate in a second power consumption mode.

33\. (New) A method of operating an augmented reality system, comprising at least one image sensor comprising a plurality of pixel cells, each pixel cell of the plurality of pixel cells comprising a light-sensitive component, first circuitry coupled to light-sensitive components of the plurality of pixel cells and configurable to provide, as output signals from at least some pixel cells of the plurality of pixel cells, indications of light intensity; and second circuitry coupled to light-sensitive components of the plurality of pixel cells and configurable to provide, as output signals from at least some pixel cells of the plurality of pixel cells, indications of change in a light property, the method comprising:using at least one processor to, for a first image data processing technique, receive from the first circuitry, indications of light intensity; and

using the at least one processor to for a second image data processing technique, receive, from the second circuitry, indications of change in the light property.

34\. (New) The method of claim 33, further comprising:using the at least one processor to perform the first image data processing technique using indications of light intensity for a full image frame.

35\. (New) The method of claim 33, further comprising:using the at least one processor to perform the first image data processing technique using indications of light intensity captured using a rolling shutter mode.

36\. (New) The method of claim 33, further comprising:using the at least one processor to perform the first image data processing technique using indications of light intensity captured using a global shutter mode.

37\. (New) The method of claim 33, further comprising:using the at least one processor to perform the second image data processing technique, the first image data processing technique comprising tracking an object or pose using the indications of change in the light property.

38\. (New) The method of claim 33, further comprising:using the at least one processor to perform the second image data processing technique using the indications of change in the light property for an image patch.

39\. (New) The method of claim 33, further comprising:using the at least one processor to perform the first image data processing technique using indications of light intensity received at a first data rate; and
using the at least one processor to perform the second image data processing technique using indications of change in the light property received at a second data rate.

40\. (New) The method of claim 33, further comprising:using the at least one processor to perform the first image data processing technique using indications of light intensity in response to an indication to operate in a first power consumption mode; and
using the at least one processor to perform the second image data processing technique using indications of change in the light property in response to an indication to operate in a second power consumption mode.

OFFICE ACTION:
DETAILED ACTION
Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA  35 U.S.C. 102 and 103 (or as subject to pre-AIA  35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA  to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary.  Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 21 – 25, 27, 28, 30, 33 – 36, and 38 is/are rejected under 35 U.S.C. 103 as being unpatentable over Sheffield et al US Patent No. 10,235,797 in view of Liu US Publication No. 2018/0376082.
Regarding claim 21 Sheffield et al discloses of, Fig. 1 – 9, of applicant’s an augmented reality system (column 10, line 3 augmented reality headset), comprising: at least one image sensor (column 17, line 23 – 30 camera megapixel senser) comprising: a plurality of pixel cells (column 6, line 61 – 67 red, green, and blue pixel values in an image are pixel cells); first circuitry coupled to light-sensitive components of pixel cells of the plurality of pixel cells and configurable to provide, as output signals from at least some pixel cells of the plurality of pixel cells, indications of light intensity (column 6, line 57 – 65 3D imaging system with a control unit in communication with a number of other components including a sensor array 206 and an interface device 208 where (column 9, line 23 – 39 sensor array 206 includes a camera device arranged in an array or matrix in rows and columns and where (column 11, 44 – 63) the computer system obtains information about an actual light source based on lighting conditions detected in a captured image so as to detect an object within the image that includes a known geometry and/or visual material property values. From the location and intensity of a highlight resulting from a reflection of the actual light source on the object, the computer system determines one or more properties of the light source. For example, the system may determine an approximate (or exact) location of the light source, an intensity of the light source, a wavelength (or color) or the light source, or any other suitable information related to the light source such that the 3D imaging system with a control unit is first circuitry coupled to pixel cells of the plurality of pixel cells in sensor array 206 that is configurable to provide, as output signals from at least some pixel cells of the plurality of pixel cells, indications of light intensity from a light source);
Sheffield et al further discloses of applicant’s and second circuitry coupled to pixel cells of the plurality of pixel cells and configurable to provide, as output signals from at least some pixel cells of the plurality of pixel cells, indications of change in a light property (column 6, line 57 – 65 3D imaging system with a control unit in communication with a number of other components including a sensor array 206 and an interface device 208 where (column 9, line 23 – 39 sensor array 206 includes a camera device arranged in an array or matrix in rows and columns and where (column 11, 18 – 63 the 3D imaging system detects a change in coloring or a change in reflectance values for the object and the computer system obtains information about an actual light source based on lighting conditions detected in a captured image so as to detect an object within the image that includes a known geometry and/or visual material property values. From the location and intensity of a highlight resulting from a reflection of the actual light source on the object, the computer system determines one or more properties of the light source. For example, the system may determine an approximate (or exact) location of the light source, an intensity of the light source, a wavelength (or color) or the light source, or any other suitable information related to the light source such that the 3D imaging system with a control unit is second circuitry coupled to pixel cells of the plurality of pixel cells in sensor array 206 that is configurable to provide, as output signals from at least some pixel cells of the plurality of pixel cells, indications of change in a light property of the pixels in sensor array 206 as a change in reflectance values for the object);
Sheffield et al further discloses of applicant’s and at least one processor (column 10, line 21 – 24 processor) configured to: for a first image data processing technique, receive, from the first circuitry, indications of light intensity (column 11, 18 – 63 receive, from the first circuitry of the 3D imaging system with a control unit, a first image data processing technique that indications of light intensity from a light source from the pixel cells in sensor array 206); and for a second image data processing technique, receive, from the second circuitry, indications of change in the light property (column 11, 18 – 63 receive, from the first circuitry of the 3D imaging system with a control unit, a second image data processing technique that indicates change in the light property from the pixels in sensor array 206 as a change in reflectance values for the object);
Sheffield et al further discloses an imaging method where an imager captured image data has both light intensity and light properties but does not expressively disclose a plurality of pixel cells, each pixel cell of the plurality of pixel cells comprising a light-sensitive component; first circuitry coupled to light-sensitive components of pixel cells and second circuitry coupled to light-sensitive components of pixel cells;
Liu teaches an imager with circuits coupled to a photodiode. Liu teaches of, Fig. 1 – 10, of applicant’s a plurality of pixel cells, each pixel cell of the plurality of pixel cells comprising a light-sensitive component (0076 pixel cell 600, part of a pixel array, includes a photodiode 602 a light-sensitive component); first circuitry coupled to light-sensitive components of pixel cells (paragraph 0076 pixel cell 600, part of a pixel array, includes a photodiode 602 a light-sensitive component have circuits in Fig. 6A where (paragraph 0096) the change of the analog voltage at analog output node 614 with respect to time when photodiode 602 is exposed to incident light with a first or second intensity level such that circuits in Fig. 6A is first and second circuitry coupled to photodiode 602 light-sensitive components of pixel cells). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to modify the circuitry of Sheffield et al in a manner similar to Liu. Doing so would result improving Sheffield et al invention in a similar way as Liu – namely the ability to provide an imager with circuits coupled to a photodiode, in Liu invention, to the imager that captured image data that has light both light intensity and light properties in Sheffield et al invention.
Regarding claim 22 of the combination of Sheffield et al in view of Liu, Sheffield et al further discloses of applicant’s wherein the at least one processor is further configured to: perform the first image data processing technique, the first image data processing technique comprising generating or updating a world model using the indications of light intensity (column 3, line 47 – 56 a three-dimensional (3D) imaging system configured to generate a 3D model of an object using material properties. The imaging system iterates through a number of sections of an object when generating a 3D model of the object where(column 11, line 15 – 22) the 3D imaging system that detects a change in coloring or a change in reflectance values for the object such that
the at least one processor is further configured to: perform the first image data processing technique, the first image data processing technique comprising generating or updating a world model of an object using material properties the indications of light intensity that detects a change in coloring or a change in reflectance values for the object).
Regarding claim 23 of the combination of Sheffield et al in view of Liu, Sheffield et al further discloses of applicant’s wherein the at least one processor is further configured to: perform the first image data processing technique using indications of light intensity for a full image frame (column 10, line 58 – 67 the computer system captures one or more images of an object (e.g., the object 104 where several properties of the object, the lighting condition and the diffuse/color texture of the object, are identified during the imaging process such that the at least one processor is further configured to: perform the first image data processing technique using indications of light intensity for a captured full image frame).
Regarding claim 24 the combination of Sheffield et al in view of Liu further discloses of applicant’s wherein the at least one processor is further configured to: perform the first image data processing technique using indications of light intensity captured using a rolling shutter mode (Sheffield et al in column 11, 44 – 63) the computer system obtains information about an actual light source based on lighting conditions detected in a captured image so as to detect an object within the image that includes a known geometry and/or visual material property values. From the location and intensity of a highlight resulting from a reflection of the actual light source on the object, the computer system determines one or more properties of the light source. For example, the system may determine an approximate (or exact) location of the light source, an intensity of the light source, a wavelength (or color) or the light source, or any other suitable information related to the light source. Liu in paragraph 0006 to generate an image, the pixel array is operated in a rolling shuttering fashion, in which each pixel row is exposed to incident lights to generate intensity data sequentially such that the first image data processing technique using indications of light intensity are captured using a rolling shutter mode is performed).
Regarding claim 25 the combination of Sheffield et al in view of Liu further discloses of applicant’s wherein the at least one processor is further configured to: perform the first image data processing technique using indications of light intensity captured using a global shutter mode (Sheffield et al in column 11, 44 – 63) the computer system obtains information about an actual light source based on lighting conditions detected in a captured image so as to detect an object within the image that includes a known geometry and/or visual material property values. From the location and intensity of a highlight resulting from a reflection of the actual light source on the object, the computer system determines one or more properties of the light source. For example, the system may determine an approximate (or exact) location of the light source, an intensity of the light source, a wavelength (or color) or the light source, or any other suitable information related to the light source. Liu in paragraph 0081 in pixel cell 600, each pixel cell 600 of a pixel array is exposed to incident lights and generate digital representations of the incident light intensity received at the pixel cell simultaneously in a global shutter operation such that the first image data processing technique using indications of light intensity captured using a global shutter mode is performed).
Regarding claim 27 of the combination of Sheffield et al in view of Liu, Sheffield et al further discloses of applicant’s wherein the augmented reality system comprises a wearable headset head (column 10, line 3 augmented reality headset is used in the augmented reality 3D system).
Regarding claim 28, claim 28 is rejected for being fully encompassed by the reasons found in rejected claim 1 above and where of the combination of Sheffield et al in view of Liu, Sheffield et al further teaches the additional claim limitation of applicant's non-transitory computer-readable medium comprising computer-executable instructions, configured for execution on at least one processor of an augmented reality system (column 10, line 13 – 28 non-transitory computer-readable storage medium comprising computer code executable instructions that are configured for execution on at least one processor of an augmented 3D reality system).
Regarding claim 30 of the combination of Sheffield et al in view of Liu, Sheffield et al further discloses of applicant’s wherein the computer-executable instructions further comprise computer-executable instructions configured to, when executed control the at least one processor to: perform the second image data processing technique using the indications of change in the light property for an image patch (column 11, line 4 – 26 the computer system identifies variances in properties of the object detected over its surface where the 3D imaging system detects a change in coloring or a change in reflectance values for the object. In this example, the area associated with a first set of material properties may be associated with a first section of the object and an area associated with a second set of material properties may be associated with a second section of the object. Column 11, 18 – 63 the 3D imaging system detects a change in coloring or a change in reflectance values for the object and the computer system obtains information about an actual light source based on lighting conditions detected in a captured image so as to detect an object within the image that includes a known geometry and/or visual material property values. From the location and intensity of a highlight resulting from a reflection of the actual light source on the object, the computer system determines one or more properties of the light source. For example, the system may determine an approximate (or exact) location of the light source, an intensity of the light source, a wavelength (or color) or the light source, or any other suitable information related to the light source such that the computer-executable instructions further comprise computer-executable instructions configured to, when executed control the at least one processor to: perform the second image data processing technique using the indications of change in the light property for an image area patch on the object with the pixels in sensor array 206 as a change in reflectance values for the object).
Regarding claim 33, claim 33 is rejected for being fully encompassed by the reasons found in rejected claim 1 above.
Regarding claim 34 of applicant’s further comprising: using the at least one processor to perform the first image data processing technique using indications of light intensity for a full image frame. Claim 34 is rejected for the reasons found in rejected claims 23 and 33 above.
Regarding claim 35 of applicant’s further comprising: using the at least one processor to perform the first image data processing technique using indications of light intensity captured using a rolling shutter mode. Claim 35 is rejected for the reasons found in rejected claims 24 and 33 above.
Regarding claim 36 of applicant’s further comprising: using the at least one processor to perform the first image data processing technique using indications of light intensity captured using a global shutter mode. Claim 36 is rejected for the reasons found in rejected claims 25 and 33 above.
Regarding claim 38 of applicant’s further comprising: using the at least one processor to perform the second image data processing technique using the indications of change in the light property for an image patch. Claim 38 is rejected for the reasons found in rejected claims 30 and 33 above.
Claim(s) 29 and 37 is/are rejected under 35 U.S.C. 103 as being unpatentable over Sheffield et al US Patent No. 10,235,797 in view of Liu US Publication No. 2018/0376082 as applied to claim 28 above, and further in view of Price et al US Publication No. 2018/0373348.
Regarding claim 29 of the combination of Sheffield et al in view of Liu, Sheffield et al further discloses of applicant’s wherein the computer-executable instructions further comprise computer-executable instructions configured to, when executed control the at least one processor to: perform the first image data processing technique, wherein the first image data processing technique comprises tracking an object or pose using the indications of change in the light property (column 11, 44 – 63 the computer system obtains information about an actual light source based on lighting conditions detected in a captured image so as to detect an object within the image that includes a known geometry and/or visual material property values. From the location and intensity of a highlight resulting from a reflection of the actual light source on the object, the computer system determines one or more properties of the light source. For example, the system may determine an approximate (or exact) location of the light source, an intensity of the light source, a wavelength (or color) or the light source, or any other suitable information related to the light source such that the computer-executable instructions further comprise computer-executable instructions configured to, when executed control the at least one processor to: perform the first image data processing technique, wherein the first image data processing technique determines intensity of the light source as indications of change in the light property);
The combination of Sheffield et al in view of Liu teaches an imaging method where an imager photodiode captures image data has both light intensity and light properties but do not expressively teach image data processing technique comprises tracking an object or pose using the indications of change in the light property;
Price et al teaches a method of teaching an object using reflected light illumination. Price et al teaches of Fig. 1 – 12, of applicant’s image data processing technique comprises tracking an object (paragraph 0025 an active illumination system is used to illuminate the tracking object on the handheld device. The active illumination system outputs a known illumination power. The tracking object may have a known reflectivity in the illumination wavelength range, and the brightness of the tracking object observed by a camera. Paragraph 0047 after capturing an IR-illuminated frame, the method 536 detects a tracking object in the frame at 542. Detecting the tracking object in the frame is based at least partially upon identifying a brightness, a size, a shape, or other predetermined property of the tracking object such that image data processing technique comprises tracking an object using the indications of the brightness change in the light property reflected off the tracked object) or pose using the indications of change in the light property. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to modify the circuitry of Sheffield et al in a manner similar to Price et al. Doing so would result improving Sheffield et al invention in a similar way as Price et al – namely the ability to provide a method of teaching an object using reflected light illumination, in Price et al invention, to the imager that captured image data that has light both light intensity and light properties in Sheffield et al invention.
Regarding claim 37 of applicant’s further comprising: using the at least one processor to perform the second image data processing technique, the first image data processing technique comprising tracking an object or pose using the indications of change in the light property. Claim 37 is rejected for the reasons found in rejected claims 29 and 33 above.
Claim(s) 31 and 39 is/are rejected under 35 U.S.C. 103 as being unpatentable over Sheffield et al US Patent No. 10,235,797 in view of Liu US Publication No. 2018/0376082 as applied to claim 28 above, and further in view of Aoyama et al US Publication No. 2018/0114566.
Regarding claim 31 of the combination of Sheffield et al in view of Liu, Sheffield et al further discloses of applicant’s wherein the computer-executable instructions further comprise computer-executable instructions configured to, when executed control the at least one processor to: perform the first image data processing technique using indications of light intensity received; and perform the second image data processing technique using indications of change in the light property received (column 11, 18 – 63 the 3D imaging system detects a change in coloring or a change in reflectance values for the object and the computer system obtains information about an actual light source based on lighting conditions detected in a captured image so as to detect an object within the image that includes a known geometry and/or visual material property values. From the location and intensity of a highlight resulting from a reflection of the actual light source on the object, the computer system determines one or more properties of the light source. For example, the system may determine an approximate (or exact) location of the light source, an intensity of the light source, a wavelength (or color) or the light source, or any other suitable information related to the light source such that the computer-executable instructions further comprise computer-executable instructions configured to, when executed control the at least one processor to: perform the first image data processing technique using indications of light intensity received from a light source and perform the second image data processing technique using indications of change in the light property received as a change in reflectance values for the object);
The combination of Sheffield et al in view of Liu teaches an imaging method where an imager photodiode captures image data has both light intensity and light properties but do not expressively teach image data processing technique comprises tracking an object or pose using the indications of change in the light property;
Aoyama et al teaches a method of setting an image first and second frame rate based on image luminance. Aoyama et al teaches of applicant’s light intensity received at a first data rate; and change in the light property received at a second data rate (paragraph 0594 the information method includes determining whether or not a length of the pattern of the plurality of bright lines included in the first bright line image is less than a predetermined length, the length being perpendicular to each of the plurality of bright lines; a frame rate changing step of changing a frame rate of the image sensor to a second frame rate lower than a first frame rate used when obtaining the first bright line image, in the case of determining that the length of the pattern is less than the predetermined length; a third bright line image obtainment step of obtaining a third bright line image which is an image including a plurality of bright lines, by capturing the first subject changing in luminance by the image sensor with the set first exposure time at the second frame rate such that by capturing the first subject changing in luminance by the image sensor with the set first exposure time the light intensity is received at a first data frame rate and a change in the light property received at a second data frame rate of the bright line image). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to modify the circuitry of Sheffield et al in a manner similar to Aoyama et al. Doing so would result improving Sheffield et al invention in a similar way as Aoyama et al – namely the ability to provide a method of setting an image first and second frame rate based on image luminance., in Aoyama et al invention, to the imager that captured image data that has light both light intensity and light properties in Sheffield et al invention.
Regarding claim 39 of applicant’s further comprising: using the at least one processor to perform the first image data processing technique using indications of light intensity received at a first data rate; and using the at least one processor to perform the second image data processing technique using indications of change in the light property received at a second data rate. Claim 39 is rejected for the reasons found in rejected claims 31 and 33 above.
Allowable Subject Matter
Claims 6, 32, and 40 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion


NEXT OFFICE ACTION: DETAILED ACTION
Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments, see page nine, first paragraph filed 07/03/2024, with respect to claims 21, 28, and 33 have been fully considered and are persuasive.  The rejection of claims 21, 28, and 33 has been withdrawn.
Allowable Subject Matter
Claims 21 – 40 are allowed.
The following is an examiner’s statement of reasons for allowance:
(Per amendments filed on 07/03/2024)
With respect to claim 21 the prior art discloses An augmented reality system, comprising:
at least one image sensor comprising:
a plurality of pixel cells, each pixel cell of the plurality of pixel cells comprising a light-sensitive component;
first circuitry coupled to light-sensitive components of pixel cells of the plurality of pixel cells and configurable to provide, as output signals from at least some pixel cells of the plurality of pixel cells, indications of light intensity;
and second circuitry coupled to light-sensitive components of pixel cells of the plurality of pixel cells and configurable to provide, as output signals from at least some pixel cells of the plurality of pixel cells.
However, the prior art does not teach or fairly suggest event-based indications of change in a light property based on the change in the light property exceeding a threshold;
and at least one processor configured to:
for a first image data processing technique, receive, from the first circuitry, indications of light intensity;
and for a second image data processing technique, receive, from the second circuitry, event-based indications of change in the light property.
With respect to claim 28 the prior art discloses Non-transitory computer-readable medium comprising computer-executable instructions, configured for execution on at least one processor of an augmented reality system comprising at least one image sensor comprising a plurality of pixel cells, each pixel cell of the plurality of pixel cells comprising a light-sensitive component;
first circuitry coupled to light-sensitive components of pixel cells of the plurality of pixel cells and configurable to provide, as output signals from at least some pixel cells of the plurality of pixel cells, indications of light intensity;
and second circuitry coupled to light-sensitive components of pixel cells of the plurality of pixel cells and configurable to provide, as output signals from at least some pixel cells of the plurality of pixel cells, indications of change in a light property, the computer-executable instructions configured to, when executed, control the at least one processor to.
However, the prior art does not teach or fairly suggest set an output mode of the at least one image sensor based on an image data processing technique to be performed, wherein:
the at least one image sensor is set to operate in a first mode in which the first circuitry provides, as output signals from at least some pixel cells of the plurality of pixel cells, indications of light intensity to perform a first image data processing technique;
the at least one image sensor is set to operate in a second mode in which the second circuitry provides, as output signals from at least some pixel cells of the plurality of pixel cells, indications of change in a light property to perform a second image data processing technique;
receive for processing, based on an image processing technique to be performed, indications of light intensity from the first circuitry or indications of change in the light property from the second circuitry, such that:
for the first image data processing technique, the at least one processor receives, from the first circuitry, indications of light intensity;
and for the second image data processing technique, the at least one processor receives, from the second circuitry, indications of change in the light property.
With respect to claim 33 the prior art discloses A method of operating an augmented reality system, comprising at least one image sensor comprising a plurality of pixel cells, each pixel cell of the plurality of pixel cells comprising a light-sensitive component, first circuitry coupled to light-sensitive components of the plurality of pixel cells and configurable to provide, as output signals from at least some pixel cells of the plurality of pixel cells, indications of light intensity;
and second circuitry coupled to light-sensitive components of the plurality of pixel cells and configurable to provide, as output signals from at least some pixel cells of the plurality of pixel cells, indications of change in a light property, the method comprising.
However, the prior art does not teach or fairly suggest controlling an output mode of the at least one image sensor such that the at least one image sensor operates in:
a first mode in which the first circuitry provides, as output signals from at least some pixel cells of the plurality of pixel cells, indications of light intensity;
or a second mode in which the second circuitry provides, as output signals from at least some pixel cells of the plurality of pixel cells, indications of change in a light property;
using at least one processor to, for a first image data processing technique, receive from the first circuitry, indications of light intensity;
and using the at least one processor to for a second image data processing technique, receive, from the second circuitry, indications of change in the light property.
Dependent claims 22 – 27, 29 – 32, and 34 - 40 are allowable for at least the reason that they depend on allowable independent claims 21, 28, and 33.
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee.  Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”
Conclusion

CITED REFERENCE SUMMARIES: Ref US10235797B1: [content unavailable]
Ref US20180376082A1: Claims:
A pixel cell comprising: a first semiconductor layer including a photodiode and one or more transistor devices configured to convert charges generated by the photodiode into an analog signal, the photodiode occupying a first region in the first semiconductor layer; and a second semiconductor layer including one or more transistor devices configured to convert the analog signal to one or more digital signals, the one or more transistor devices of the second semiconductor layer occupying a second region in the second semiconductor layer; wherein the first semiconductor layer forms a stack structure with the second semiconductor layer along an axis; and wherein the first region and the second region overlap at least partially along the axis.
a first semiconductor layer including a photodiode and one or more transistor devices configured to convert charges generated by the photodiode into an analog signal, the photodiode occupying a first region in the first semiconductor layer; and
a second semiconductor layer including one or more transistor devices configured to convert the analog signal to one or more digital signals, the one or more transistor devices of the second semiconductor layer occupying a second region in the second semiconductor layer;
wherein the first semiconductor layer forms a stack structure with the second semiconductor layer along an axis; and
wherein the first region and the second region overlap at least partially along the axis.
The pixel cell of claim 1 , wherein the first semiconductor layer includes a first surface and a second surface; wherein the first semiconductor layer includes one or more first metal interconnects disposed on the second surface; wherein the second semiconductor layer includes a third surface facing the second surface; wherein the second semiconductor layer includes one or more second metal interconnects disposed on the third surface; and wherein the pixel cell further includes one or more third metal interconnects to provide electrical connection between the one or more first metal interconnects and the one or more second metal interconnects.
wherein the first semiconductor layer includes a first surface and a second surface;
wherein the first semiconductor layer includes one or more first metal interconnects disposed on the second surface;
wherein the second semiconductor layer includes a third surface facing the second surface;
wherein the second semiconductor layer includes one or more second metal interconnects disposed on the third surface; and
wherein the pixel cell further includes one or more third metal interconnects to provide electrical connection between the one or more first metal interconnects and the one or more second metal interconnects.
The pixel cell of claim 2 , wherein the first surface is configured to receive light photons.
The pixel cell of claim 1 , wherein the first semiconductor layer includes a different doping profile from the second semiconductor layer.
The pixel cell of claim 4 , wherein the first semiconductor layer includes a doping gradient to introduce an electric field between a first surface and a second surface of the first semiconductor layer.
The pixel cell of claim 1 , wherein the first semiconductor layer has a different thickness from the second semiconductor layer.
The pixel cell of claim 6 , wherein a thickness of the first semiconductor is configured based on a target quantum efficiency for photons associated with a pre-determined frequency.
The pixel cell of claim 1 , wherein the one or more transistor devices of the first semiconductor layer comprises a first transistor with a source terminal coupled with the photodiode, a drain terminal configured as a capacitor, and a gate terminal being operable to control a flow of electrons generated by the photodiode to the capacitor; wherein the one or more transistor devices of the second semiconductor layer comprises a digital-to-analog converter configured to generate the one or more digital signals based on an analog voltage at the drain terminal of the first transistor.
wherein the one or more transistor devices of the first semiconductor layer comprises a first transistor with a source terminal coupled with the photodiode, a drain terminal configured as a capacitor, and a gate terminal being operable to control a flow of electrons generated by the photodiode to the capacitor;
wherein the one or more transistor devices of the second semiconductor layer comprises a digital-to-analog converter configured to generate the one or more digital signals based on an analog voltage at the drain terminal of the first transistor.
The pixel cell of claim 8 , wherein the digital-to-analog converter comprises a counter, a memory, and a voltage comparator; wherein the memory is configured to store one or more count values output by the counter; and wherein the voltage comparator is configured to control the storing of the one or more count values at the memory based on the analog voltage at the drain terminal of the first transistor to generate the one or more digital signals.
wherein the memory is configured to store one or more count values output by the counter; and
wherein the voltage comparator is configured to control the storing of the one or more count values at the memory based on the analog voltage at the drain terminal of the first transistor to generate the one or more digital signals.
The pixel cell of claim 8 , wherein the gate terminal is controlled to allow electrons to flow from the photodiode to the capacitor when a quantity of charges stored at the photodiode exceeds a threshold.
The pixel cell of claim 8 , wherein the one or more transistor devices of the first semiconductor layer comprises a second transistor configured to, during a reset mode, remove charges stored at the capacitor.
The pixel cell of claim 8 , wherein the one or more transistor devices of the first semiconductor layer comprises a second transistor configured to, during a reset mode, remove charges stored at the capacitor and at the photodiode.
A pixel cell comprising: a photodiode; a capacitor configured to store charges generated by the photodiode; and a processing circuit configured to perform at least one of a first mode of measurement and a second mode of measurement to generate data representing an intensity of light incident on the photodiode; wherein in the first mode of measurement, the processing circuit is configured to measure a quantity of charges stored at the capacitor when the capacitor is electrically coupled with the photodiode; and wherein in the second mode of measurement, the processing circuit is configured to measure a quantity of charges stored at the capacitor when the capacitor is electrically isolated from the photodiode.
a capacitor configured to store charges generated by the photodiode; and
a processing circuit configured to perform at least one of a first mode of measurement and a second mode of measurement to generate data representing an intensity of light incident on the photodiode;
wherein in the first mode of measurement, the processing circuit is configured to measure a quantity of charges stored at the capacitor when the capacitor is electrically coupled with the photodiode; and
wherein in the second mode of measurement, the processing circuit is configured to measure a quantity of charges stored at the capacitor when the capacitor is electrically isolated from the photodiode.
The pixel cell of claim 13 , wherein the processing circuit comprises a variable threshold generator, a comparator, and a counter.
The pixel cell of claim 14 , wherein, in the first mode of measurement: the variable threshold generator is configured to generate a fixed threshold voltage; the comparator is configured to compare the fixed threshold voltage against a voltage representing the quantity of charges stored at the capacitor to generate a decision output; the counter is configured to generate a set of count values based on the decision output; and one or more of the set of count values generated by the counter corresponding to a change in the decision output represents an intensity of light incident on the photodiode.
the variable threshold generator is configured to generate a fixed threshold voltage;
the comparator is configured to compare the fixed threshold voltage against a voltage representing the quantity of charges stored at the capacitor to generate a decision output;
the counter is configured to generate a set of count values based on the decision output; and
one or more of the set of count values generated by the counter corresponding to a change in the decision output represents an intensity of light incident on the photodiode.
The pixel cell of claim 15 , wherein the one or more of the set of count values decreases when the intensity of light incident on the photodiode increases.
The pixel cell of claim 15 , wherein the fixed threshold voltage corresponds to a saturation light intensity for the photodiode.
The pixel cell of claim 14 , wherein, in the second mode of measurement: the variable threshold generator is configured to generate a ramping voltage based on a set of count values generated by the counter; the comparator is configured to compare the ramping voltage against a voltage representing the quantity of charges stored at the capacitor to generate a decision output; the counter is configured to generate the set of count values based on the decision output; and one or more of the set of count values generated by the counter corresponding to a change in the decision output represents an intensity of light incident on the photodiode.
the variable threshold generator is configured to generate a ramping voltage based on a set of count values generated by the counter;
the comparator is configured to compare the ramping voltage against a voltage representing the quantity of charges stored at the capacitor to generate a decision output;
the counter is configured to generate the set of count values based on the decision output; and
The pixel cell of claim 18 , wherein the one or more of the set of count values increases when the intensity of light incident on the photodiode increases.
The pixel cell of claim 13 , wherein the processing circuit is configured to skip the second mode of measurement based on a determination, from the first mode of measurement, that a quantity of charges stored at the capacitor when the photodiode is electrically coupled with the photodiode exceeds a threshold.
Ref US20180373348A1: Claims:
An object tracking system comprising: an infrared illumination source mounted to a head mounted display; an infrared sensitive camera mounted to the head mounted display; and a handheld electronic device including a first tracking object with a first reflectivity in an infrared range.
an infrared illumination source mounted to a head mounted display;
an infrared sensitive camera mounted to the head mounted display; and
a handheld electronic device including a first tracking object with a first reflectivity in an infrared range.
The system of claim 1 , the tracking object including a retroreflective surface.
The system of claim 1 , the tracking object including a Lambertian surface.
The system of claim 1 , the tracking object being at least partially spherical.
The system of claim 1 , the head mounted display further comprising a processor configured to calculate X- and Y-position information of the first tracking object relative to a field of view of the head mounted display and calculate Z-position information of the first tracking object relative to the head mounted display based on a measured brightness of the first tracking object.
The system of claim 1 , the handheld electronic device including at least one sensor to monitor at least one of yaw, pitch, and roll of the handheld electronic device.
The system of claim 1 , further comprising a second tracking object having a second reflectivity in the infrared range, the second reflectivity being less than the first reflectivity.
A method of object tracking in a head-mounted display, the method comprising: illuminating a field of view with an infrared illumination source; capturing an infrared illuminated frame of the field of view with an infrared camera; detecting a tracking object in the field of view; calculating an x-position and a y-position of the tracking object in the field of view; measuring a maximum brightness of the tracking object; and calculating a position of the tracking object using the maximum brightness.
illuminating a field of view with an infrared illumination source;
capturing an infrared illuminated frame of the field of view with an infrared camera;
detecting a tracking object in the field of view;
calculating an x-position and a y-position of the tracking object in the field of view;
measuring a maximum brightness of the tracking object; and
calculating a position of the tracking object using the maximum brightness.
The method of claim 8 , measuring the maximum brightness including measuring a centroid of the tracking object.
The method of claim 8 , further comprising positioning the infrared illumination source and infrared camera equidistant from the tracking object.
The method of claim 8 , further comprising capturing an ambient frame of the field of view and subtracting the ambient frame from the illuminated frame.
The method of claim 8 , further comprising decreasing an illumination power of the infrared illumination source when the maximum brightness is greater than or equal to a saturation value.
The method of claim 8 , further comprising increasing an illumination power of the infrared illumination source when the maximum brightness is less than or equal to a threshold value.
The method of claim 8 , further comprising calculating rotation of the tracking object using one or more sensors connected to the tracking object.
The method of claim 8 , further comprising calculating an area of the tracking object.
The method of claim 15 , further comprising verifying a depth of the tracking object using the area of the tracking object.
A method of object tracking in a head-mounted display, the method comprising: illuminating a field of view with an infrared illumination source mounted to the head-mounted display; capturing an infrared illuminated frame of the field of view with an infrared camera mounted to the head-mounted display; detecting a first tracking object on a handheld device in the field of view; calculating a first x-position and a first y-position of the first tracking object in the field of view; measuring a first maximum brightness of the first tracking object; calculating a first depth of the first tracking object using the first maximum brightness; detecting a second tracking object on the handheld device in the field of view; calculating a second x-position and a second y-position of the second tracking object in the field of view; measuring a second maximum brightness of the second tracking object; and calculating a second depth of the second tracking object using the second maximum brightness.
illuminating a field of view with an infrared illumination source mounted to the head-mounted display;
capturing an infrared illuminated frame of the field of view with an infrared camera mounted to the head-mounted display;
detecting a first tracking object on a handheld device in the field of view;
calculating a first x-position and a first y-position of the first tracking object in the field of view;
measuring a first maximum brightness of the first tracking object;
calculating a first depth of the first tracking object using the first maximum brightness;
detecting a second tracking object on the handheld device in the field of view;
calculating a second x-position and a second y-position of the second tracking object in the field of view;
measuring a second maximum brightness of the second tracking object; and
calculating a second depth of the second tracking object using the second maximum brightness.
The method of claim 17 , further comprising calculating a pitch of the handheld device from the first tracking object and second tracking object.
The method of claim 17 , further comprising: detecting a third tracking object on the handheld device in the field of view; calculating a third x-position and a third y-position of the third tracking object in the field of view; measuring a third maximum brightness of the third tracking object; and calculating a third depth of the third tracking object using the third maximum brightness.
detecting a third tracking object on the handheld device in the field of view;
calculating a third x-position and a third y-position of the third tracking object in the field of view;
measuring a third maximum brightness of the third tracking object; and
calculating a third depth of the third tracking object using the third maximum brightness.
The method of claim 19 , further comprising calculating a yaw of the handheld device from the first tracking object, the second tracking object, and the third tracking object.
Ref US20180114566A1: Claims:
A display method for displaying an image on a display device, the display method comprising: obtaining a first image and a second image which includes light identification information by capturing a subject using an image sensor, the subject transmitting the light identification information by change in luminace; decoding the second image to obtain the light identification information; transmitting the light identification information to a server; a plurality of sets of (i) an augmented reality image, (ii) recognition information, and (iii) identification information being stored in the server, out of the plurality of sets, obtaining, from the server, the augmented reality image and the recognition information included in each of the sets including the identification information that corresponds to the light identification information; recognizing a target region in the first image using the obtained recognition information; and displaying the first display image with the obtained augmented reality image superimposed on the recognized target region.
obtaining a first image and a second image which includes light identification information by capturing a subject using an image sensor, the subject transmitting the light identification information by change in luminace;
decoding the second image to obtain the light identification information;
transmitting the light identification information to a server;
a plurality of sets of (i) an augmented reality image, (ii) recognition information, and (iii) identification information being stored in the server, out of the plurality of sets, obtaining, from the server, the augmented reality image and the recognition information included in each of the sets including the identification information that corresponds to the light identification information;
recognizing a target region in the first image using the obtained recognition information; and
displaying the first display image with the obtained augmented reality image superimposed on the recognized target region.
The display method according to claim 1 , wherein the recognition information includes: reference information for specifying a reference region in the firstimage; and target information indicating a relative position of the target region with respect to the reference region, and the recognizing of the target region includes: specifying the reference region from the first image by matching the reference information and the first image; and recognizing the target region in the first image at the relative position from the reference region using using the target information.
the recognition information includes:
reference information for specifying a reference region in the firstimage; and
target information indicating a relative position of 
[Truncated to input limit]