Office Action 1: Non-Final Rejection (2023-07-05)
Rejection types: 102 - Lack of Novelty, 103 - Obviousness

CLAIMS:
CLAIMS

1\. A hand-held controller comprising:
a main body;
a first extension extending from the main body;
a second extension extending from the main body;
a first light indicator located with respect to the main body, wherein the first light indicator is configured to emit light to be detected by a camera located at a bottom surface of a head-mounted display (HMD); and
a second light indicator located with respect to the main body, wherein the second light indicator is configured to emit light to be detected by the camera, wherein the lights emitted by the first and second light indicators facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD.

2\. The hand-held controller of claim 1, wherein the hand-held controller has an edge, wherein the first light indicator is located along the edge to be visible to the camera of the lower region of the HMD with the main body, wherein the second light indicator is located at a distance from the first light indicator to be visible to the camera.

3\. The hand-held controller of claim 1, wherein the main body has a touch surface for determining movements of fingers of a user, wherein the first light indicator includes a plurality of light sources, wherein the plurality of light sources are located at pre- determined locations with reference to the touch surface.

4\. The hand-held controller of claim 1, wherein the first light indicator is an infrared light emitter.

5\. The hand-held controller of claim 1, wherein the main body has a plurality of buttons, wherein the first light indicator is configured to face the camera to be in a field-of- view facing downward from the camera.

6\. The hand-held controller of claim 1, further comprising a touch surface, wherein the first light indicator is a string of light sources located along an edge of the main body and an edge of the touch surface.
7\. The hand-held controller of claim 1, further comprising a battery configured to provide power to the first light indicator and the second light indictor, wherein the second light indicator is a light emitting diode.

8\. The hand-held controller of claim 1, wherein the camera has a co-ordinate system having an x-axis, a y-axis, and a z-axis, wherein the orientation is determined with respect to the x, y, and z axes of the co-ordinate system.

9\. The hand-held controller of claim 1, further comprising a haptic feedback module, wherein the main body includes a joystick that is configured to be moved to generate input data for processing by a computing device, wherein the haptic feedback module is configured to provide tactile feedback to a user.

10\. The hand-held controller of claim 1, wherein the lights are detected by an additional camera located at the bottom surface of the HMD to determine the orientation of the main body, the first extension, and the second extension with reference to the additional camera.

11\. The hand-held controller of claim 1, wherein the main body has a touch surface, wherein the first light indicator is located below the touch surface and along an edge of the touch surface to be visible to the camera.

12\. A head-mounted display comprising:
a top surface;
a front surface located adjacent to the top surface;
a bottom surface located adjacent to the front surface;
a band coupled to the top surface; and
a camera located at the bottom surface, wherein the camera is configured to capture images of lights emitted from a first light indicator and a second light indictor of a hand-held controller to facilitate a determination of orientation of the hand-held controller with reference to the camera.

13\. The head-mounted display of claim 12, further including a haptic feedback device configured to vibrate the top, front, and bottom surfaces to provide tactile feedback to a head of a user.
14\. The head-mounted display of claim 12, further including an additional camera located at the bottom surface, wherein the additional camera is configured to capture images of the lights from the first and second light indicators to facilitate a determination of the orientation of the hand-held controller with reference to the additional camera.

15\. The head-mounted display of claim 12, further comprising:
an eye mask having the top surface, the front surface, the bottom surface, and a frame, wherein the frame is fitted to the bottom surface, wherein the band is a headband coupled to the frame and a portion of the top surface.

16\. The head-mounted display of claim 12, wherein the hand-held controller has a touch surface, wherein the first light indicator is located below the touch surface and along an edge of the touch surface to be visible to the camera.

17\. A system comprising:
a head-mounted display (HMD) including:
a top surface;
a front surface located adjacent to the top surface;
a bottom surface located adjacent to the front surface;
a band coupled to the top surface; and
a camera located at the bottom surface;
a hand-held controller including:
a main body;
a first extension extending from the main body;
a second extension extending from the main body;
a first light indicator located with respect to the main body, wherein the first light indicator is configured to emit light to be detected by the camera of the HMD;
and a second light indicator located with respect to the main body, wherein the second light indicator is configured to emit light to be detected by the camera of the HMD, wherein the lights emitted by the first and second light indicators facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD.
18\. The system of claim 17, wherein the hand-held controller has an edge, wherein the first light indicator is located along the edge to be visible to the camera of the lower region of the HMD with the main body, wherein the second light indicator is located at a distance from the first light indicator to be visible to the camera.

19\. The system of claim 17, wherein the HMD includes an additional camera located at the bottom surface, wherein the additional camera is configured to capture images of the lights from the first and second light indicators to facilitate a determination of the orientation of the hand-held controller with reference to the additional camera.

20\. The system of claim 17, wherein the main body has a touch surface, wherein the first light indicator is located below the touch surface and along an edge of the touch surface to be visible to the camera of the HMD. ASP/NVP SONYP360.C169PATENT APPLICATION SONYP360.C170PATENT APPLICATION SONYP360.C171PATENT APPLICATION SONYP360.C172PATENT APPLICATION

OFFICE ACTION:
DETAILED ACTION
Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
This application discloses and claims only subject matter disclosed in prior application No. 16/207, 209, filed November 30, 2018, and names the inventor or at least one joint inventor named in the prior application. Accordingly, this application constitutes a continuation. Applicant has claimed the benefit of the filing date of the prior application through to 35 U.S.C. 120, 37 CFR 1.78, and MPEP § 211 et seq.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on September 26, 2022 is in compliance with the provisions of 37 CFR 1.97.  Accordingly, the information disclosure statement is being considered by the examiner.
Claim Objections
Claims 7 and 12-16 are objected to because of the following informalities: Claims 7 and 12-16 include the recitation of “second light indictor” instead of “second light indicator” which appears to be a spelling error.  Appropriate correction is required. The Examiner interprets “indictor” as same as “indicator”, thus no rejection under 35 U.S.C. 112 is made.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-8, 10-12 and 14-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Samples et al., United States Patent Application Publication No. US 2018/0330521 A1.
Regarding claim 1, Samples discloses a hand-held controller (Fig. 1-4, handheld object, #106; Detailed Description, [0033-0035]) comprising:
a main body (Figs. 1-4, Detailed Description, [0035]);
a first extension extending from the main body (Fig. 4, 6DoF controller ,left hand, #406; Detailed Description, [0036-0050]);
a second extension extending from the main body (Fig. 4, 6DoF controller, right hand, #408; Detailed Description, [0036-0050]);
a first light indicator located with respect to the main body, wherein the first light indicator is configured to emit light to be detected by a camera located at a bottom surface of a head-mounted display (HMD) (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller”); and
a second light indicator located with respect to the main body, wherein the second light indicator is configured to emit light to be detected by the camera (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller…A handheld object may have any suitable number and arrangement of light sources.”),
wherein the lights emitted by the first and second light indicators facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD (Fig. 4, HMD, #402; Detailed Description, [0030-0050], “The optical pose of the handheld object with respect to the HMD (“HMD.fwdarw.LED Vision Poses” at 503) and the optical pose of the HMD relative to the world (“World.fwdarw.HMD Poses” at 505) are used to determine a pose of the handheld object with respect to the world”).
Regarding claim 2, Samples further discloses wherein the hand-held controller has an edge, wherein the first light indicator is located along the edge to be visible to the camera of the lower region of the HMD with the main body, wherein the second light indicator is located at a distance from the first light indicator to be visible to the camera (Fig. 3, Detailed Description, [0036-0041], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300”).
Regarding claim 3, Samples further discloses wherein the main body has a touch surface for determining movements of fingers of a user, wherein the first light indicator includes a plurality of light sources, wherein the plurality of light sources are located at pre-determined locations with reference to the touch surface (Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”).
Regarding claim 4, Samples further discloses wherein the first light indicator is an infrared light emitter (Detailed Description, [0035-0036], “Infrared light sources also may be used.”).
Regarding claim 5,  Samples further discloses wherein the main body has a plurality of buttons (Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”), wherein the first light indicator is configured to face the camera to be in a field-of-view facing downward from the camera (Fig. 2-3, the handheld device #202 is depicted as downward from the HMD which has the camera).
Regarding claim 6, Samples discloses a controller further comprising a touch surface (Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”), wherein the first light indicator is a string of light sources located along an edge of the main body and an edge of the touch surface (Fig. 3, Detailed Description, [0036-0041], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300”).
Regarding claim 7, Samples discloses a controller further comprising a battery configured to provide power to the first light indicator and the second light indicator (Detailed Description, [0061]), wherein the second light indicator is a light emitting diode (Detailed Description, [0036]).
Regarding claim 8, Samples discloses wherein the camera has a co-ordinate system having an x-axis, a y-axis, and a z-axis, wherein the orientation is determined with respect to the x, y, and z axes of the co-ordinate system (Detailed Description, [0052], “The computing device further predicts more possible uv-xyz correspondences, and refines these predictions. The predictions are thresholded on a reprojection error, the error between 2D detection points (u,v's) and the prediction of associated 3D points (x,y,z), assuming a certain pose”).
Regarding claim 10, Samples discloses wherein the lights are detected by an additional camera located at the bottom surface of the HMD to determine the orientation of the main body, the first extension, and the second extension with reference to the additional camera (Fig. 4, HMD, #402; Detailed Description, [0030-0050], “The optical pose of the handheld object with respect to the HMD (“HMD.fwdarw.LED Vision Poses” at 503) and the optical pose of the HMD relative to the world (“World.fwdarw.HMD Poses” at 505) are used to determine a pose of the handheld object with respect to the world”; See also Summary, [0003] for showing of multiple cameras in the stereo camera arrangement).
Regarding claim 11, Samples discloses wherein the main body has a touch surface (Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”), wherein the first light indicator is located below the touch surface and along an edge of the touch surface to be visible to the camera (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller…A handheld object may have any suitable number and arrangement of light sources.”).
Regarding claim 12, Samples discloses a head-mounted display (Fig. 1-4, #102, HMD)  comprising:
a top surface (See Fig. 2, top surface of HMD 200);
a front surface located adjacent to the top surface (Fig. 2, front surface is next to top surface);
a bottom surface located adjacent to the front surface (Fig. 2, #200);
a band coupled to the top surface (Fig. 1-2, back of HMD is a band for user); and
a camera located at the bottom surface, wherein the camera is configured to capture images of lights emitted from a first light indicator and a second light indictor of a hand-held controller to facilitate a determination of orientation of the hand-held controller with reference to the camera (Fig. 2, cameras, #204; second camera, #206; Detailed Description, [0030-0053], “One or more outward-facing cameras on the HMD 102 may acquire image data (e.g. visible light image data) of the surrounding environment and of a handheld object 106 held by the user 102. For example, FIG. 2 shows an example HMD 200 imaging a handheld object 202 using a stereo camera imaging system (indicated by first camera 204 and second camera 206)....The term “handheld object” is used herein for an object comprising trackable light sources 108 that may be held by hand as described herein, and signifies that the object may be handheld, whether or not the object is currently in the state of being held in a hand… For example, because the HMD may have more than one camera, a pose may be solved for each camera on the HMD. Such refinement is performed as postprocessing of a multitude of poses obtained from different cameras. In other examples, any other suitable pose estimation process may be used.”).
Regarding claim 14,  Samples discloses a head-mounted display further including an additional camera located at the bottom surface, wherein the additional camera is configured to capture images of the lights from the first and second light indicators to facilitate a determination of the orientation of the hand-held controller with reference to the additional camera (Fig. 2, cameras, #204; second camera, #206; Detailed Description, [0030-0053], “One or more outward-facing cameras on the HMD 102 may acquire image data (e.g. visible light image data) of the surrounding environment and of a handheld object 106 held by the user 102. For example, FIG. 2 shows an example HMD 200 imaging a handheld object 202 using a stereo camera imaging system (indicated by first camera 204 and second camera 206)....The term “handheld object” is used herein for an object comprising trackable light sources 108 that may be held by hand as described herein, and signifies that the object may be handheld, whether or not the object is currently in the state of being held in a hand… For example, because the HMD may have more than one camera, a pose may be solved for each camera on the HMD. Such refinement is performed as postprocessing of a multitude of poses obtained from different cameras. In other examples, any other suitable pose estimation process may be used.”).
Regarding claim 15, Samples discloses a head-mounted display further comprising: an eye mask having the top surface, the front surface, the bottom surface, and a frame, wherein the frame is fitted to the bottom surface, wherein the band is a headband coupled to the frame and a portion of the top surface (Fig. 1-2—generally; Examiner’ note—although eyemask and headband are not terms used, they are depicted as included in the HMD structure).
Regarding claim 16, Samples discloses wherein the hand-held controller has a touch surface (Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”), wherein the first light indicator is located below the touch surface and along an edge of the touch surface to be visible to the camera (Fig. 3, Detailed Description, [0036-0041], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300”).
Regarding claim 17, Samples disclose a system (Fig. 1-4, generally) comprising:
a head-mounted display (Fig. 1-4, #102, HMD)  including:
a top surface (See Fig. 2, top surface of HMD 200);
a front surface located adjacent to the top surface (Fig. 2, front surface is next to top surface);
a bottom surface located adjacent to the front surface (Fig. 2, #200);
a band coupled to the top surface (Fig. 1-2, back of HMD is a band for user); and
a camera located at the bottom surface (Fig. 2, cameras, #204; second camera, #206; Detailed Description, [0030-0053], “One or more outward-facing cameras on the HMD 102 may acquire image data (e.g. visible light image data) of the surrounding environment and of a handheld object 106 held by the user 102. For example, FIG. 2 shows an example HMD 200 imaging a handheld object 202 using a stereo camera imaging system (indicated by first camera 204 and second camera 206)....The term “handheld object” is used herein for an object comprising trackable light sources 108 that may be held by hand as described herein, and signifies that the object may be handheld, whether or not the object is currently in the state of being held in a hand… For example, because the HMD may have more than one camera, a pose may be solved for each camera on the HMD. Such refinement is performed as postprocessing of a multitude of poses obtained from different cameras. In other examples, any other suitable pose estimation process may be used.”);
a hand-held controller (Fig. 1-4, handheld object, #106; Detailed Description, [0033-0035]) including:
a main body (Figs. 1-4, Detailed Description, [0035]);
a first extension extending from the main body (Fig. 4, 6DoF controller ,left hand, #406; Detailed Description, [0036-0050]);
a second extension extending from the main body (Fig. 4, 6DoF controller, right hand, #408; Detailed Description, [0036-0050]);
a first light indicator located with respect to the main body, wherein the first light indicator is configured to emit light to be detected by the camera of the HMD (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller”); and
a second light indicator located with respect to the main body, wherein the second light indicator is configured to emit light to be detected by the camera of the HMD, wherein the lights emitted by the first and second light indicators facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD camera (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller…A handheld object may have any suitable number and arrangement of light sources.”; Fig. 4, HMD, #402; Detailed Description, [0030-0050], “The optical pose of the handheld object with respect to the HMD (“HMD.fwdarw.LED Vision Poses” at 503) and the optical pose of the HMD relative to the world (“World.fwdarw.HMD Poses” at 505) are used to determine a pose of the handheld object with respect to the world”).).
Regarding claim 18, Samples discloses wherein the hand-held controller has an edge, wherein the first light indicator is located along the edge to be visible to the camera of the lower region of the HMD with the main body, wherein the second light indicator is located at a distance from the first light indicator to be visible to the camera (Fig. 3, Detailed Description, [0036-0041], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300”).
Regarding claim 19, Samples discloses wherein the HMD includes an additional camera located at the bottom surface, wherein the additional camera is configured to capture images of the lights from the first and second light indicators to facilitate a determination of the orientation of the hand-held controller with reference to the additional camera  (Fig. 2, cameras, #204; second camera, #206; Detailed Description, [0030-0053], “One or more outward-facing cameras on the HMD 102 may acquire image data (e.g. visible light image data) of the surrounding environment and of a handheld object 106 held by the user 102. For example, FIG. 2 shows an example HMD 200 imaging a handheld object 202 using a stereo camera imaging system (indicated by first camera 204 and second camera 206)....The term “handheld object” is used herein for an object comprising trackable light sources 108 that may be held by hand as described herein, and signifies that the object may be handheld, whether or not the object is currently in the state of being held in a hand… For example, because the HMD may have more than one camera, a pose may be solved for each camera on the HMD. Such refinement is performed as postprocessing of a multitude of poses obtained from different cameras. In other examples, any other suitable pose estimation process may be used.”).
Regarding claim 20, Samples discloses wherein the main body has a touch surface (Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”)., wherein the first light indicator is located below the touch surface and along an edge of the touch surface to be visible to the camera of the HMD (Fig. 3, Detailed Description, [0036-0041], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300”).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 9 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Samples in view of Mallinson, United States Patent Application Publication No. US 2016/0025978 A1.
Regarding claim 9, Samples discloses every element of claim 1 and further discloses wherein
wherein the main body includes a joystick that is configured to be moved to generate input data for processing by a computing device (Detailed Description, [0035], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc. The handheld object 106 comprises an inertial measurement unit (IMU) (which may include an accelerometer, gyroscope, magnetometer, and/or other suitable sensors) that provides output related to changes in position and orientation of the handheld object 106”; See also Figs. 1-4, host computing device, #110).
Samples does not disclose the controller further comprising a haptic feedback module, wherein the haptic feedback module is configured to provide tactile feedback to a user.
Mallinson, in a similar field of endeavor, discloses a controller (Fig. 1) comprising a haptic feedback module, wherein the haptic feedback module is configured to provide tactile feedback to a user (Detailed Description, [0056], “Additionally, part of the processed media content may be transmitted to other devices, such as a game controller, that was used to provide input to the application. For example, video and audio streams of the updates may be provided to the HMD and the haptic feedback may be provided to the game controller.”).
It would have been obvious to one of ordinary skill in the art to have modified the controller of Samples to include the teachings of Mallison to add a haptic feedback module, wherein the haptic feedback module is configured to provide tactile feedback to a user. The motivation to combine these arts is to add to the immersion of the user within a AR/VR HMD system which enhances user experience (See Mallison, Summary, [0006-0009]). The fact the Mallison also discloses HMDs in use with game controllers, same as Samples,  makes this combination easily implemented.
Regarding claim 13, Samples discloses every element of claim 12 but does not explicitly disclose a HMD including a haptic feedback device configured to vibrate the top, front, and bottom surfaces to provide tactile feedback to a head of a user.
Mallinson, in a similar field of endeavor, discloses a head-mounted device (Fig. 1, HMD, #100) including a haptic feedback device configured to vibrate the top, front, and bottom surfaces to provide tactile feedback to a head of a user (Detailed Description, [0080-0085], “Other output modules include Light-Emitting Diodes (LED) 834, other visual markers or elements (which may also be used for visual tracking of the HMD), vibro-tactile feedback 850, speakers 830, and sound localization module 812, which performs sound localization for sounds to be delivered to speakers or headphones. Other output devices, such as headphones, can also connect to the HMD via the communication modules.”).
It would have been obvious to one of ordinary skill in the art to have modified the HMD of Samples to include the teachings of Mallison to add a haptic feedback device configured to vibrate the top, front, and bottom surfaces to provide tactile feedback to a head of a user. The motivation to combine these arts is to add to the immersion of the user within an AR/VR HMD system while wearing the HMD, which enhances user experience (See Mallison, Summary, [0006-0009][0081-0085]). The fact the Mallison also discloses HMDs in use with game controllers, same as Samples,  makes this combination easily implemented.
Conclusion


NEXT OFFICE ACTION: DETAILED ACTION
Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
As a result of the Amendment filed on October 4, 2023, claims 1-22 are pending. Claims 1, 3, 7 11 and 12 are amended. New claims 21-22 are added.
Response to Arguments
Applicant's arguments filed October 4, 2023 with respect to the art rejections have been fully considered but they are not persuasive. It is argued by Applicant that the Samples (US 2018/0330521 A1) reference does not teach the elements of claims 1-22, particularly that of “wherein the first light indicator is located along an edge of the touch surface to be visible to a camera located at a bottom surface of a head-mounted display (HMD)”. Applicant is submitting that Samples shows the light emitters distributed along a ring-like structure rather than an edge of the touch surface, and that in addition Samples does not teach a camera located at the bottom surface as recited in claims 12 and 17. (Remarks at pgs. 7-9).
The Office respectfully disagrees with the above conclusions. As cited before, Samples shows that the handheld object 106 can explicitly include a touch screen (Figs. 1-4, Detailed Description, [0035], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc.”). Furthermore, as an alternative interpretation, a portion of the ring-like structure shown in Fig. 2 of Samples can itself be interpreted as a touch surface, even without a touch screen, since the user is able to touch and manipulate the entirety of the handheld object 106. Next, the light sources are shown to have a plurality of arrangements and locations both within the exterior and interior of the structure of the controller (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller”). Thus, Samples teaches light sources alongside an edge of the touch surface since they can be located anywhere within or exterior of the ring-like structure of the handheld controller, which can include a touch screen. Also under the alternative interpretation, the light sources located on the edge of the ring-like structure, which under Broadest Reasonable Interpretation (BRI) in itself can be the touch surface. Finally, it is taught that the HMD comprises cameras and various camera arrangements (Detailed Description, [0151]), including at bottom surface of the HMD (Fig. 2, 204/206 are on lower portion of the HMD below the top band). Thus, Samples continues to teach all the claim imitations  of “wherein the first light indicator located along an edge of the touch surface to be visible to a camera located at a bottom surface of a head-mounted display (HMD)”. For the foregoing reasons, the claim rejections are maintained for all pending claims 1-22 in the manner set forth before. No new art is cited at this time.
Applicant’s arguments in light of the Amendments, see pg. 6 of the Remarks, filed October 4, 2023, with respect to the claim objections have been fully considered and are persuasive.  The claim objections of claims 7 and 12-16 due to typographical errors has been withdrawn.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-8, 10-12 and 14-22 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Samples et al., United States Patent Application Publication No. US 2018/0330521 A1.
Regarding claim 1, Samples discloses a hand-held controller (Fig. 1-4, handheld object, #106; Detailed Description, [0033-0035]) comprising:
a main body, wherein the main body has a touch surface (Figs. 1-4, Detailed Description, [0035], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc.”);
a first extension extending from the main body (Fig. 4, 6DoF controller ,left hand, #406; Detailed Description, [0036-0050]);
a second extension extending from the main body (Fig. 4, 6DoF controller, right hand, #408; Detailed Description, [0036-0050]);
a first light indicator located with respect to the main body, wherein the first light indicator located along an edge of the touch surface to be visible to a camera located at a bottom surface of a head-mounted display (HMD), wherein the first light indicator is configured to emit light to be detected by the camera (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller”); and
a second light indicator located with respect to the main body, wherein the second light indicator is configured to emit light to be detected by the camera (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller…A handheld object may have any suitable number and arrangement of light sources.”),
wherein the lights emitted by the first and second light indicators facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD (Fig. 4, HMD, #402; Detailed Description, [0030-0050], “The optical pose of the handheld object with respect to the HMD (“HMD.fwdarw.LED Vision Poses” at 503) and the optical pose of the HMD relative to the world (“World.fwdarw.HMD Poses” at 505) are used to determine a pose of the handheld object with respect to the world”).
Regarding claim 2, Samples further discloses wherein the hand-held controller has an edge, wherein the first light indicator is located along the edge to be visible to the camera of the lower region of the HMD with the main body, wherein the second light indicator is located at a distance from the first light indicator to be visible to the camera (Fig. 3, Detailed Description, [0036-0041], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300”).
Regarding claim 3, Samples further discloses wherein the a touch surface is configured to determine the fingers of a user, wherein the first light indicator includes a plurality of light sources, wherein the plurality of light sources are located at pre-determined locations with reference to the touch surface (Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”).
Regarding claim 4, Samples further discloses wherein the first light indicator is an infrared light emitter (Detailed Description, [0035-0036], “Infrared light sources also may be used.”).
Regarding claim 5,  Samples further discloses wherein the main body has a plurality of buttons (Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”), wherein the first light indicator is configured to face the camera to be in a field-of-view facing downward from the camera (Fig. 2-3, the handheld device #202 is depicted as downward from the HMD which has the camera).
Regarding claim 6, Samples discloses a controller further comprising a touch surface (Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”), wherein the first light indicator is a string of light sources located along an edge of the main body and an edge of the touch surface (Fig. 3, Detailed Description, [0036-0041], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300”).
Regarding claim 7, Samples discloses a controller further comprising a battery configured to provide power to the first light indicator and the second light indicator (Detailed Description, [0061]), wherein the second light indicator is a light emitting diode (Detailed Description, [0036]).
Regarding claim 8, Samples discloses wherein the camera has a co-ordinate system having an x-axis, a y-axis, and a z-axis, wherein the orientation is determined with respect to the x, y, and z axes of the co-ordinate system (Detailed Description, [0052], “The computing device further predicts more possible uv-xyz correspondences, and refines these predictions. The predictions are thresholded on a reprojection error, the error between 2D detection points (u,v's) and the prediction of associated 3D points (x,y,z), assuming a certain pose”).
Regarding claim 10, Samples discloses wherein the lights are detected by an additional camera located at the bottom surface of the HMD to determine the orientation of the main body, the first extension, and the second extension with reference to the additional camera (Fig. 4, HMD, #402; Detailed Description, [0030-0050], “The optical pose of the handheld object with respect to the HMD (“HMD.fwdarw.LED Vision Poses” at 503) and the optical pose of the HMD relative to the world (“World.fwdarw.HMD Poses” at 505) are used to determine a pose of the handheld object with respect to the world”; See also Summary, [0003] for showing of multiple cameras in the stereo camera arrangement).
Regarding claim 11, Samples discloses wherein the main body has a touch surface (Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”), wherein the first light indicator is located below the touch surface to be visible to the camera (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller…A handheld object may have any suitable number and arrangement of light sources.”).
Regarding claim 12, Samples discloses a head-mounted display (Fig. 1-4, #102, HMD)  comprising:
a top surface (See Fig. 2, top surface of HMD 200);
a front surface located adjacent to the top surface (Fig. 2, front surface is next to top surface);
a bottom surface located adjacent to the front surface (Fig. 2, #200);
a band coupled to the top surface (Fig. 1-2, back of HMD is a band for user); and
a camera located at the bottom surface, wherein the camera is configured to capture images of lights emitted from a first light indicator and a second light indicator of a hand-held controller to facilitate a determination of orientation of the hand-held controller with reference to the camera (Fig. 2, cameras, #204; second camera, #206; Detailed Description, [0030-0053], “One or more outward-facing cameras on the HMD 102 may acquire image data (e.g. visible light image data) of the surrounding environment and of a handheld object 106 held by the user 102. For example, FIG. 2 shows an example HMD 200 imaging a handheld object 202 using a stereo camera imaging system (indicated by first camera 204 and second camera 206)....The term “handheld object” is used herein for an object comprising trackable light sources 108 that may be held by hand as described herein, and signifies that the object may be handheld, whether or not the object is currently in the state of being held in a hand… For example, because the HMD may have more than one camera, a pose may be solved for each camera on the HMD. Such refinement is performed as postprocessing of a multitude of poses obtained from different cameras. In other examples, any other suitable pose estimation process may be used.”).
Regarding claim 14,  Samples discloses a head-mounted display further including an additional camera located at the bottom surface, wherein the additional camera is configured to capture images of the lights from the first and second light indicators to facilitate a determination of the orientation of the hand-held controller with reference to the additional camera (Fig. 2, cameras, #204; second camera, #206; Detailed Description, [0030-0053], “One or more outward-facing cameras on the HMD 102 may acquire image data (e.g. visible light image data) of the surrounding environment and of a handheld object 106 held by the user 102. For example, FIG. 2 shows an example HMD 200 imaging a handheld object 202 using a stereo camera imaging system (indicated by first camera 204 and second camera 206)....The term “handheld object” is used herein for an object comprising trackable light sources 108 that may be held by hand as described herein, and signifies that the object may be handheld, whether or not the object is currently in the state of being held in a hand… For example, because the HMD may have more than one camera, a pose may be solved for each camera on the HMD. Such refinement is performed as postprocessing of a multitude of poses obtained from different cameras. In other examples, any other suitable pose estimation process may be used.”).
Regarding claim 15, Samples discloses a head-mounted display further comprising: an eye mask having the top surface, the front surface, the bottom surface, and a frame, wherein the frame is fitted to the bottom surface, wherein the band is a headband coupled to the frame and a portion of the top surface (Fig. 1-2—generally; Examiner’ note—although eyemask and headband are not terms used, they are depicted as included in the HMD structure).
Regarding claim 16, Samples discloses wherein the hand-held controller has a touch surface (Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”), wherein the first light indicator is located below the touch surface and along an edge of the touch surface to be visible to the camera (Fig. 3, Detailed Description, [0036-0041], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300”).
Regarding claim 17, Samples disclose a system (Fig. 1-4, generally) comprising:
a head-mounted display (Fig. 1-4, #102, HMD)  including:
a top surface (See Fig. 2, top surface of HMD 200);
a front surface located adjacent to the top surface (Fig. 2, front surface is next to top surface);
a bottom surface located adjacent to the front surface (Fig. 2, #200);
a band coupled to the top surface (Fig. 1-2, back of HMD is a band for user); and
a camera located at the bottom surface (Fig. 2, cameras, #204; second camera, #206; Detailed Description, [0030-0053], “One or more outward-facing cameras on the HMD 102 may acquire image data (e.g. visible light image data) of the surrounding environment and of a handheld object 106 held by the user 102. For example, FIG. 2 shows an example HMD 200 imaging a handheld object 202 using a stereo camera imaging system (indicated by first camera 204 and second camera 206)....The term “handheld object” is used herein for an object comprising trackable light sources 108 that may be held by hand as described herein, and signifies that the object may be handheld, whether or not the object is currently in the state of being held in a hand… For example, because the HMD may have more than one camera, a pose may be solved for each camera on the HMD. Such refinement is performed as postprocessing of a multitude of poses obtained from different cameras. In other examples, any other suitable pose estimation process may be used.”);
a hand-held controller (Fig. 1-4, handheld object, #106; Detailed Description, [0033-0035]) including:
a main body (Figs. 1-4, Detailed Description, [0035]);
a first extension extending from the main body (Fig. 4, 6DoF controller ,left hand, #406; Detailed Description, [0036-0050]);
a second extension extending from the main body (Fig. 4, 6DoF controller, right hand, #408; Detailed Description, [0036-0050]);
a first light indicator located with respect to the main body, wherein the first light indicator is configured to emit light to be detected by the camera of the HMD (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller”); and
a second light indicator located with respect to the main body, wherein the second light indicator is configured to emit light to be detected by the camera of the HMD, wherein the lights emitted by the first and second light indicators facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD camera (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller…A handheld object may have any suitable number and arrangement of light sources.”; Fig. 4, HMD, #402; Detailed Description, [0030-0050], “The optical pose of the handheld object with respect to the HMD (“HMD.fwdarw.LED Vision Poses” at 503) and the optical pose of the HMD relative to the world (“World.fwdarw.HMD Poses” at 505) are used to determine a pose of the handheld object with respect to the world”).).
Regarding claim 18, Samples discloses wherein the hand-held controller has an edge, wherein the first light indicator is located along the edge to be visible to the camera of the lower region of the HMD with the main body, wherein the second light indicator is located at a distance from the first light indicator to be visible to the camera (Fig. 3, Detailed Description, [0036-0041], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300”).
Regarding claim 19, Samples discloses wherein the HMD includes an additional camera located at the bottom surface, wherein the additional camera is configured to capture images of the lights from the first and second light indicators to facilitate a determination of the orientation of the hand-held controller with reference to the additional camera  (Fig. 2, cameras, #204; second camera, #206; Detailed Description, [0030-0053], “One or more outward-facing cameras on the HMD 102 may acquire image data (e.g. visible light image data) of the surrounding environment and of a handheld object 106 held by the user 102. For example, FIG. 2 shows an example HMD 200 imaging a handheld object 202 using a stereo camera imaging system (indicated by first camera 204 and second camera 206)....The term “handheld object” is used herein for an object comprising trackable light sources 108 that may be held by hand as described herein, and signifies that the object may be handheld, whether or not the object is currently in the state of being held in a hand… For example, because the HMD may have more than one camera, a pose may be solved for each camera on the HMD. Such refinement is performed as postprocessing of a multitude of poses obtained from different cameras. In other examples, any other suitable pose estimation process may be used.”).
Regarding claim 20, Samples discloses wherein the main body has a touch surface (Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”)., wherein the first light indicator is located below the touch surface and along an edge of the touch surface to be visible to the camera of the HMD (Fig. 3, Detailed Description, [0036-0041], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300”).
Regarding claim 21, Samples further discloses wherein the first light indicator is a light bar located along the edge of the touch surface (Fig. 3, Detailed Description, [0036-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300… A handheld object may have any suitable number and arrangement of light sources.”).
Regarding claim 22, Samples further discloses the light bar is located below the touch surface (Fig. 3, Detailed Description, [0036-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300… A handheld object may have any suitable number and arrangement of light sources.”).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 9 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Samples in view of Mallinson, United States Patent Application Publication No. US 2016/0025978 A1.
Regarding claim 9, Samples discloses every element of claim 1 and further discloses wherein
wherein the main body includes a joystick that is configured to be moved to generate input data for processing by a computing device (Detailed Description, [0035], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc. The handheld object 106 comprises an inertial measurement unit (IMU) (which may include an accelerometer, gyroscope, magnetometer, and/or other suitable sensors) that provides output related to changes in position and orientation of the handheld object 106”; See also Figs. 1-4, host computing device, #110).
Samples does not disclose the controller further comprising a haptic feedback module, wherein the haptic feedback module is configured to provide tactile feedback to a user.
Mallinson, in a similar field of endeavor, discloses a controller (Fig. 1) comprising a haptic feedback module, wherein the haptic feedback module is configured to provide tactile feedback to a user (Detailed Description, [0056], “Additionally, part of the processed media content may be transmitted to other devices, such as a game controller, that was used to provide input to the application. For example, video and audio streams of the updates may be provided to the HMD and the haptic feedback may be provided to the game controller.”).
It would have been obvious to one of ordinary skill in the art to have modified the controller of Samples to include the teachings of Mallison to add a haptic feedback module, wherein the haptic feedback module is configured to provide tactile feedback to a user. The motivation to combine these arts is to add to the immersion of the user within a AR/VR HMD system which enhances user experience (See Mallison, Summary, [0006-0009]). The fact the Mallison also discloses HMDs in use with game controllers, same as Samples,  makes this combination easily implemented.
Regarding claim 13, Samples discloses every element of claim 12 but does not explicitly disclose a HMD including a haptic feedback device configured to vibrate the top, front, and bottom surfaces to provide tactile feedback to a head of a user.
Mallinson, in a similar field of endeavor, discloses a head-mounted device (Fig. 1, HMD, #100) including a haptic feedback device configured to vibrate the top, front, and bottom surfaces to provide tactile feedback to a head of a user (Detailed Description, [0080-0085], “Other output modules include Light-Emitting Diodes (LED) 834, other visual markers or elements (which may also be used for visual tracking of the HMD), vibro-tactile feedback 850, speakers 830, and sound localization module 812, which performs sound localization for sounds to be delivered to speakers or headphones. Other output devices, such as headphones, can also connect to the HMD via the communication modules.”).
It would have been obvious to one of ordinary skill in the art to have modified the HMD of Samples to include the teachings of Mallison to add a haptic feedback device configured to vibrate the top, front, and bottom surfaces to provide tactile feedback to a head of a user. The motivation to combine these arts is to add to the immersion of the user within an AR/VR HMD system while wearing the HMD, which enhances user experience (See Mallison, Summary, [0006-0009][0081-0085]). The fact the Mallison also discloses HMDs in use with game controllers, same as Samples,  makes this combination easily implemented.
Conclusion
THIS ACTION IS MADE FINAL.  Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action.  In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action.  In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.

CITED REFERENCE SUMMARIES: Ref US20180330521A1: Claims:
A head-mounted device, comprising: a stereo camera arrangement; a logic device configured to execute instructions; and a storage device comprising instructions executable by the logic device to for each camera in the stereo camera arrangement, receive image data of a field of view of the camera, detect a plurality of light sources of a handheld object in the image data, and based upon the plurality of light sources detected, determine a pose of the handheld object; and based upon the pose of the handheld object determined for each camera in the stereo camera arrangement, calibrate the stereo camera arrangement.
a stereo camera arrangement;
a logic device configured to execute instructions; and
a storage device comprising instructions executable by the logic device to for each camera in the stereo camera arrangement, receive image data of a field of view of the camera, detect a plurality of light sources of a handheld object in the image data, and based upon the plurality of light sources detected, determine a pose of the handheld object; and based upon the pose of the handheld object determined for each camera in the stereo camera arrangement, calibrate the stereo camera arrangement.
for each camera in the stereo camera arrangement, receive image data of a field of view of the camera, detect a plurality of light sources of a handheld object in the image data, and based upon the plurality of light sources detected, determine a pose of the handheld object; and
receive image data of a field of view of the camera,
detect a plurality of light sources of a handheld object in the image data, and
based upon the plurality of light sources detected, determine a pose of the handheld object; and
based upon the pose of the handheld object determined for each camera in the stereo camera arrangement, calibrate the stereo camera arrangement.
The head-mounted device of claim 1 , wherein the instructions are executable to calibrate the stereo camera arrangement during one or more of a motion of the head-mounted device and a motion of the handheld object.
The head-mounted device of claim 1 , wherein the instructions are executable to calibrate intrinsic characteristics for each camera of the stereo camera arrangement, the intrinsic characteristics comprising one or more of lens distortion, focal length, and principal point.
The head-mounted device of claim 1 , wherein the instructions are executable to calibrate extrinsic characteristics for each camera of the stereo camera arrangement, the extrinsic characteristics comprising one or more of a rotation and a translation.
The head-mounted device of claim 1 , wherein the instructions are executable to determine the pose by fitting an observed light pattern to a model light pattern via a rigid body transformation.
The head-mounted device of claim 1 , wherein the instructions are executable to determine the pose by, for each light source detected in the image data received from a first camera, searching along an epipolar line for a companion pixel in the image data received from a second camera, the companion pixel corresponding to the light source detected in the image data received from the first camera.
The head-mounted device of claim 1 , wherein the instructions are executable to calibrate the stereo camera arrangement utilizing one or more of a Kalman filter, a sliding window bundle adjustment, and a recursive Gauss-Newton filter.
The head-mounted device of claim 1 , wherein the instructions are executable to calibrate the stereo camera arrangement between 6 and 10 times per minute.
A method for determining a calibration for a handheld object via a head-mounted device comprising a stereo camera arrangement, the method comprising: receiving, from a first camera of the stereo camera arrangement, first image data from a perspective of the first camera; receiving, from a second camera of the stereo camera arrangement, second image data from a perspective of the second camera; detecting, in the first image data and the second image data, a plurality of light sources of the handheld object; determining a three-dimensional position of each light source of the plurality of light sources detected; and determining a calibration for the handheld object based upon each three-dimensional position determined.
receiving, from a first camera of the stereo camera arrangement, first image data from a perspective of the first camera;
receiving, from a second camera of the stereo camera arrangement, second image data from a perspective of the second camera;
detecting, in the first image data and the second image data, a plurality of light sources of the handheld object;
determining a three-dimensional position of each light source of the plurality of light sources detected; and
determining a calibration for the handheld object based upon each three-dimensional position determined.
The method of claim 9 , further comprising receiving from an inertial measurement unit (IMU) of the handheld object a sensed orientation of the handheld object, and determining a calibration for the IMU based upon the sensed orientation.
receiving from an inertial measurement unit (IMU) of the handheld object a sensed orientation of the handheld object, and
determining a calibration for the IMU based upon the sensed orientation.
The method of claim 10 , wherein determining the calibration for the IMU comprises determining a calibration for a position and a rotation of the IMU.
The method of claim 9 , wherein determining the calibration of the handheld object comprises determining calibration parameters via one or more of a Kalman filter, a sliding window bundle adjustment, and a recursive Gauss-Newton filter.
The method of claim 9 , wherein determining the three-dimensional position of each light source comprises fitting an observed light pattern to a model light pattern via a rigid body transformation.
A head-mounted device, comprising: a stereo camera arrangement; a logic device configured to execute instructions; and a storage device comprising instructions executable by the logic device to for each camera in the stereo camera arrangement, receive image data of a surrounding environment from a perspective of the camera, detect via the image data a plurality of feature points of a handheld object in a field of view of the camera, determine a location of each feature point detected, and determine a pose of the handheld object from the perspective of the camera based on the location of each feature point determined; update one or more calibration parameters for the stereo camera arrangement based on the pose of the handheld object determined for each camera; determine a three-dimensional position of each feature point based on the location of each feature point determined for each camera of the stereo camera arrangement; and update one or more calibration parameters for the handheld object based on the three-dimensional position of each feature point determined.
a storage device comprising instructions executable by the logic device to for each camera in the stereo camera arrangement, receive image data of a surrounding environment from a perspective of the camera, detect via the image data a plurality of feature points of a handheld object in a field of view of the camera, determine a location of each feature point detected, and determine a pose of the handheld object from the perspective of the camera based on the location of each feature point determined; update one or more calibration parameters for the stereo camera arrangement based on the pose of the handheld object determined for each camera; determine a three-dimensional position of each feature point based on the location of each feature point determined for each camera of the stereo camera arrangement; and update one or more calibration parameters for the handheld object based on the three-dimensional position of each feature point determined.
for each camera in the stereo camera arrangement, receive image data of a surrounding environment from a perspective of the camera, detect via the image data a plurality of feature points of a handheld object in a field of view of the camera, determine a location of each feature point detected, and determine a pose of the handheld object from the perspective of the camera based on the location of each feature point determined;
receive image data of a surrounding environment from a perspective of the camera,
detect via the image data a plurality of feature points of a handheld object in a field of view of the camera,
determine a location of each feature point detected, and
determine a pose of the handheld object from the perspective of the camera based on the location of each feature point determined;
update one or more calibration parameters for the stereo camera arrangement based on the pose of the handheld object determined for each camera;
determine a three-dimensional position of each feature point based on the location of each feature point determined for each camera of the stereo camera arrangement; and
update one or more calibration parameters for the handheld object based on the three-dimensional position of each feature point determined.
The head-mounted device of claim 14 , wherein the instructions are executable to receive the image data as a background process during use of the head-mounted device and the handheld object.
The head-mounted device of claim 14 , wherein the instructions are further executable to receive from an inertial measurement unit (IMU) of the handheld object a sensed orientation of the handheld object, and to determine a calibration for one or more of a translation and a rotation of the IMU based upon the sensed orientation.
The head-mounted device of claim 14 , wherein the one or more calibration parameters for the stereo camera arrangement comprise one or more of an intrinsic camera characteristic and an extrinsic camera characteristic.
The head-mounted device of claim 14 , wherein the instructions are executable to determine the pose of the handheld object by one or more of fitting an observed feature point pattern to a model feature point pattern via a rigid body transformation; and for each feature point detected in the image data received from a first camera of the stereo camera arrangement, searching along an epipolar line for a companion feature point in the image data received from a second camera of the stereo camera arrangement, the companion feature point corresponding to the feature point detected in the image data received from the first camera.
fitting an observed feature point pattern to a model feature point pattern via a rigid body transformation; and
for each feature point detected in the image data received from a first camera of the stereo camera arrangement, searching along an epipolar line for a companion feature point in the image data received from a second camera of the stereo camera arrangement, the companion feature point corresponding to the feature point detected in the image data received from the first camera.
The head-mounted device of claim 14 , wherein the instructions are executable to update the one or more calibration parameters for the stereo camera arrangement and update the one or more calibration parameters for the handheld object simultaneously.
The head-mounted device of claim 14 , wherein the instructions are executable to update one or more calibration parameters for one or more of the stereo camera arrangement and the handheld object via one or more of a Kalman filter, a sliding window bundle adjustment, and a recursive Gauss-Newton filter.
Ref US20160025978A1: Claims:
A device, comprising: a display screen having a front side and a back side, the display screen configured for rendering media content; first optics disposed adjacent to the front side of the display screen and configured to provide a focus for viewing the media content when rendered on the display screen; a shutter screen disposed adjacent to the backside of the display screen, wherein the shutter screen is switchable between an opaque mode and a transparent mode and wherein the opaque mode is active when the media content is viewable on the display screen; and second optics disposed behind the shutter screen such that the shutter screen is between the display screen and the second optics, the second optics providing an adjustment to the focus to allow viewing through the first optics, the display screen, the shutter screen and the second optics when the transparent mode is activated on the shutter screen.
a display screen having a front side and a back side, the display screen configured for rendering media content;
first optics disposed adjacent to the front side of the display screen and configured to provide a focus for viewing the media content when rendered on the display screen;
a shutter screen disposed adjacent to the backside of the display screen, wherein the shutter screen is switchable between an opaque mode and a transparent mode and wherein the opaque mode is active when the media content is viewable on the display screen; and
second optics disposed behind the shutter screen such that the shutter screen is between the display screen and the second optics, the second optics providing an adjustment to the focus to allow viewing through the first optics, the display screen, the shutter screen and the second optics when the transparent mode is activated on the shutter screen.
The device of claim 1 , wherein the adjustment provided in the second optics is to compensate for view distortion caused by the focus of the first optics.
The device of claim 1 , wherein the transparent mode is activated on a portion of the shutter screen and the opaque mode is activated for remaining portions of the shutter screen.
The device of claim 3 , wherein the display screen is rendered transparent.
The device of claim 3 , wherein at least a portion of the media content is rendered in portions of the display screen that correspond with the remaining portions of the shutter screen where the opaque mode is activated.
The device of claim 3 , wherein the activation of the transparent mode in the portion of the shutter screen causes a viewport to be defined through the portion of the shutter screen and through corresponding portions of the first optics, the display screen and the second optics.
The device of claim 3 , wherein the activation of the transparent mode in the portion of the shutter screen causes activation of a window in the portion of the display screen that corresponds to at least one of the remaining portions of the shutter screen that is in opaque mode, the window rendering a view of a web portal at a web site and providing options to interact with the web portal.
The device of claim 1 , further includes a switching circuit that is communicatively connected to the shutter screen to activate the transparent mode or the opaque mode.
The device of claim 8 , further includes a controller communicatively connected to the switching circuit to control the switching of the shutter screen to the transparent mode or the opaque mode.
The device of claim 8 , further includes an event detector circuit configured to generate a signal for selectively switching the shutter screen to transparent mode, in response to detection of an event occurring within the media content or within an external environment near the device.
The device of claim 10 , wherein the event detected is one of a change in an external environment condition in vicinity of the device, change in media content being rendered, change in environment condition within the media content, an audio signal detected in the vicinity of the device, audio signal generated in the media content, a visual cue detected in the vicinity of the user, or any combinations thereof.
The device of claim 10 , wherein when the event corresponds to movement of an object in an external environment in vicinity of the device, the event detector circuit is configured to track movement of the object and generate appropriate signals to cause selective switching of appropriate portions of the shutter screen corresponding to the movement of the object, to transparent mode to permit viewing of the moving object.
The device of claim 1 , further includes input device configured for user interaction, wherein the user interaction includes providing annotation on an object provided in the media content or on an object from an external environment, interaction with the media content or the object, blending of the object into the media content or any combinations thereof.
The device of claim 1 , wherein each of the first optic, the display screen, the shutter screen and the second optic is coated with an anti-reflective coating.
A pair of glasses, comprising, a view port provided with a multi-layer arrangement, the multi-layer arrangement includes, a first optic provided with a first focus setting; a display screen positioned behind the first optic, the display screen being transparent; a shutter screen positioned behind the display screen, the shutter screen being adjustable between a transparent mode and an opaque mode; and a second optic provided with a second focus setting, the second focus setting removing the first focus setting to provide a see through view through the first optic, the display screen, the shutter screen and the second optic, when the shutter screen is set to the transparent mode.
a view port provided with a multi-layer arrangement, the multi-layer arrangement includes, a first optic provided with a first focus setting; a display screen positioned behind the first optic, the display screen being transparent; a shutter screen positioned behind the display screen, the shutter screen being adjustable between a transparent mode and an opaque mode; and a second optic provided with a second focus setting, the second focus setting removing the first focus setting to provide a see through view through the first optic, the display screen, the shutter screen and the second optic, when the shutter screen is set to the transparent mode.
a first optic provided with a first focus setting;
a display screen positioned behind the first optic, the display screen being transparent;
a shutter screen positioned behind the display screen, the shutter screen being adjustable between a transparent mode and an opaque mode; and
a second optic provided with a second focus setting, the second focus setting removing the first focus setting to provide a see through view through the first optic, the display screen, the shutter screen and the second optic, when the shutter screen is set to the transparent mode.
The pair of glasses of claim 15 , further includes a switching circuit communicatively connected to the shutter screen and configured to activate the transparent mode or the opaque mode.
The pair of glasses of claim 16 , further includes a controller connected to the switching circuit to control switching of the shutter screen to the transparent mode or the opaque mode.
The pair of glasses of claim 16 , wherein the switching circuit is configured to activate the transparent mode in a select portion of the shutter screen.
The pair of glasses of claim 15 , wherein removal of the first focus setting is by configuring the second focus setting of the second optic to counterbalance the first focus setting so as to cancel out any distortion caused by the first focus setting when viewing through the pair of glasses.
The pair of glasses of claim 15 , wherein the first optic, the display screen, the shutter screen and the second optic are each coated with anti-reflective coating.
A method, comprising: receiving media content for rendering on a display screen of a pair of glasses, the pair of glasses including first optics in front of the display screen to provide a focus for viewing the media content rendering on the display screen; detecting an event t
[Truncated to input limit]