Office Action 2: Final Rejection (2023-12-08)
Rejection types: 102 - Lack of Novelty, 103 - Obviousness

CLAIMS:
IN THE CLAIMS

This listing of claims will replace all prior versions, and listings, of claims in the application:

Listing of Claims:

1\. (Currently amended) A hand-held controller comprising:a main body, wherein the main body has a touch surface;
a first extension extending from the main body;
a second extension extending from the main body;
a first light indicator located with respect to the main body, wherein the first light indicator is located along an edge of the touch surface to be visible to a camera located at a bottom surface of a head-mounted display (HMD), wherein the first light indicator is configured to emit light to be detected by the camera; and
a second light indicator located with respect to the main body, wherein the second light indicator is configured to emit light to be detected by the camera, wherein the lights emitted by the first and second light indicators facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD.
2\. (Original) The hand-held controller of claim 1, wherein the hand-held controller has an edge, wherein the first light indicator is located along the edge to be visible to the camera of the lower region of the HMD with the main body, wherein the second light indicator is located at a distance from the first light indicator to be visible to the camera.
3\. (Currently amended) The hand-held controller of claim 1, wherein the a touch surface is configured to determine movements of fingers of a user, wherein the first light indicator includes a plurality of light sources, wherein the plurality of light sources are located at pre-determined locations with reference to the touch surface.
4\. (Original) The hand-held controller of claim 1, wherein the first light indicator is an infrared light emitter.

5\. (Original) The hand-held controller of claim 1, wherein the main body has a plurality of buttons, wherein the first light indicator is configured to face the camera to be in a field-of- view facing downward from the camera.
6\. (Original) The hand-held controller of claim 1, further comprising a touch surface,
wherein the first light indicator is a string of light sources located along an edge of the main body and an edge of the touch surface.
7\. (Currently amended) The hand-held controller of claim 1, further comprising a battery configured to provide power to the first light indicator and the second light indicator, wherein the second light indicator is a light emitting diode.
8\. (Original) The hand-held controller of claim 1, wherein the camera has a co-ordinate system having an x-axis, a y-axis, and a z-axis, wherein the orientation is determined with respect to the x, y, and z axes of the co-ordinate system.
9\. (Original) The hand-held controller of claim 1, further comprising a haptic feedback module, wherein the main body includes a joystick that is configured to be moved to generate input data for processing by a computing device, wherein the haptic feedback module is configured to provide tactile feedback to a user.

10\. (Original) The hand-held controller of claim 1, wherein the lights are detected by an additional camera located at the bottom surface of the HMD to determine the orientation of the main body, the first extension, and the second extension with reference to the additional camera.

11\. (Currently amended) The hand-held controller of claim 1, wherein the main body has a touch surface, wherein the first light indicator is located below the touch surface to be visible to the camera.

12\. (Currently amended) A head-mounted display comprising:a top surface;
a front surface located adjacent to the top surface;
a bottom surface located adjacent to the front surface;

a band coupled to the top surface; and
a camera located at the bottom surface, wherein the camera is configured to capture images of lights emitted from a first light indicator and a second light indicator of a hand-held controller to facilitate a determination of orientation of the hand-held controller with reference to the camera.

13\. (Original) The head-mounted display of claim 12, further including a haptic feedback device configured to vibrate the top, front, and bottom surfaces to provide tactile feedback to a head of a user.

14\. (Original) The head-mounted display of claim 12, further including an additional camera located at the bottom surface, wherein the additional camera is configured to capture images of the lights from the first and second light indicators to facilitate a determination of the orientation of the hand-held controller with reference to the additional camera.

15\. (Original) The head-mounted display of claim 12, further comprising:an eye mask having the top surface, the front surface, the bottom surface, and a frame, wherein the frame is fitted to the bottom surface, wherein the band is a headband coupled to the frame and a portion of the top surface.

16\. (Original) The head-mounted display of claim 12, wherein the hand-held controller has a touch surface, wherein the first light indicator is located below the touch surface and along an edge of the touch surface to be visible to the camera.

17\. (Original) A system comprising:a head-mounted display (HMD) including:
a top surface;
a front surface located adjacent to the top surface;
a bottom surface located adjacent to the front surface;
a band coupled to the top surface; and a camera located at the bottom surface;
a hand-held controller including:
a main body;
a first extension extending from the main body;

a second extension extending from the main body;
a first light indicator located with respect to the main body, wherein the first light indicator is configured to emit light to be detected by the camera of the HMD;
and a second light indicator located with respect to the main body, wherein the second light indicator is configured to emit light to be detected by the camera of the HMD, wherein the lights emitted by the first and second light indicators facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD.

18\. (Original) The system of claim 17, wherein the hand-held controller has an edge, wherein the first light indicator is located along the edge to be visible to the camera of the lower region of the HMD with the main body, wherein the second light indicator is located at a distance from the first light indicator to be visible to the camera.

19\. (Original) The system of claim 17, wherein the HMD includes an additional camera located at the bottom surface, wherein the additional camera is configured to capture images of the lights from the first and second light indicators to facilitate a determination of the orientation of the hand-held controller with reference to the additional camera.

20\. (Original) The system of claim 17, wherein the main body has a touch surface, wherein the first light indicator is located below the touch surface and along an edge of the touch surface to be visible to the camera of the HMD.

21\. (New) The hand-held controller of claim 1, wherein the first light indicator is a light bar located along the edge of the touch surface.

22\. (New) The hand-held controller of claim 1, wherein the light bar is located below the touch surface.

OFFICE ACTION:
DETAILED ACTION
Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
As a result of the Amendment filed on October 4, 2023, claims 1-22 are pending. Claims 1, 3, 7 11 and 12 are amended. New claims 21-22 are added.
Response to Arguments
Applicant's arguments filed October 4, 2023 with respect to the art rejections have been fully considered but they are not persuasive. It is argued by Applicant that the Samples (US 2018/0330521 A1) reference does not teach the elements of claims 1-22, particularly that of “wherein the first light indicator is located along an edge of the touch surface to be visible to a camera located at a bottom surface of a head-mounted display (HMD)”. Applicant is submitting that Samples shows the light emitters distributed along a ring-like structure rather than an edge of the touch surface, and that in addition Samples does not teach a camera located at the bottom surface as recited in claims 12 and 17. (Remarks at pgs. 7-9).
The Office respectfully disagrees with the above conclusions. As cited before, Samples shows that the handheld object 106 can explicitly include a touch screen (Figs. 1-4, Detailed Description, [0035], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc.”). Furthermore, as an alternative interpretation, a portion of the ring-like structure shown in Fig. 2 of Samples can itself be interpreted as a touch surface, even without a touch screen, since the user is able to touch and manipulate the entirety of the handheld object 106. Next, the light sources are shown to have a plurality of arrangements and locations both within the exterior and interior of the structure of the controller (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller”). Thus, Samples teaches light sources alongside an edge of the touch surface since they can be located anywhere within or exterior of the ring-like structure of the handheld controller, which can include a touch screen. Also under the alternative interpretation, the light sources located on the edge of the ring-like structure, which under Broadest Reasonable Interpretation (BRI) in itself can be the touch surface. Finally, it is taught that the HMD comprises cameras and various camera arrangements (Detailed Description, [0151]), including at bottom surface of the HMD (Fig. 2, 204/206 are on lower portion of the HMD below the top band). Thus, Samples continues to teach all the claim imitations  of “wherein the first light indicator located along an edge of the touch surface to be visible to a camera located at a bottom surface of a head-mounted display (HMD)”. For the foregoing reasons, the claim rejections are maintained for all pending claims 1-22 in the manner set forth before. No new art is cited at this time.
Applicant’s arguments in light of the Amendments, see pg. 6 of the Remarks, filed October 4, 2023, with respect to the claim objections have been fully considered and are persuasive.  The claim objections of claims 7 and 12-16 due to typographical errors has been withdrawn.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-8, 10-12 and 14-22 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Samples et al., United States Patent Application Publication No. US 2018/0330521 A1.
Regarding claim 1, Samples discloses a hand-held controller (Fig. 1-4, handheld object, #106; Detailed Description, [0033-0035]) comprising:
a main body, wherein the main body has a touch surface (Figs. 1-4, Detailed Description, [0035], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc.”);
a first extension extending from the main body (Fig. 4, 6DoF controller ,left hand, #406; Detailed Description, [0036-0050]);
a second extension extending from the main body (Fig. 4, 6DoF controller, right hand, #408; Detailed Description, [0036-0050]);
a first light indicator located with respect to the main body, wherein the first light indicator located along an edge of the touch surface to be visible to a camera located at a bottom surface of a head-mounted display (HMD), wherein the first light indicator is configured to emit light to be detected by the camera (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller”); and
a second light indicator located with respect to the main body, wherein the second light indicator is configured to emit light to be detected by the camera (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller…A handheld object may have any suitable number and arrangement of light sources.”),
wherein the lights emitted by the first and second light indicators facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD (Fig. 4, HMD, #402; Detailed Description, [0030-0050], “The optical pose of the handheld object with respect to the HMD (“HMD.fwdarw.LED Vision Poses” at 503) and the optical pose of the HMD relative to the world (“World.fwdarw.HMD Poses” at 505) are used to determine a pose of the handheld object with respect to the world”).
Regarding claim 2, Samples further discloses wherein the hand-held controller has an edge, wherein the first light indicator is located along the edge to be visible to the camera of the lower region of the HMD with the main body, wherein the second light indicator is located at a distance from the first light indicator to be visible to the camera (Fig. 3, Detailed Description, [0036-0041], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300”).
Regarding claim 3, Samples further discloses wherein the a touch surface is configured to determine the fingers of a user, wherein the first light indicator includes a plurality of light sources, wherein the plurality of light sources are located at pre-determined locations with reference to the touch surface (Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”).
Regarding claim 4, Samples further discloses wherein the first light indicator is an infrared light emitter (Detailed Description, [0035-0036], “Infrared light sources also may be used.”).
Regarding claim 5,  Samples further discloses wherein the main body has a plurality of buttons (Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”), wherein the first light indicator is configured to face the camera to be in a field-of-view facing downward from the camera (Fig. 2-3, the handheld device #202 is depicted as downward from the HMD which has the camera).
Regarding claim 6, Samples discloses a controller further comprising a touch surface (Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”), wherein the first light indicator is a string of light sources located along an edge of the main body and an edge of the touch surface (Fig. 3, Detailed Description, [0036-0041], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300”).
Regarding claim 7, Samples discloses a controller further comprising a battery configured to provide power to the first light indicator and the second light indicator (Detailed Description, [0061]), wherein the second light indicator is a light emitting diode (Detailed Description, [0036]).
Regarding claim 8, Samples discloses wherein the camera has a co-ordinate system having an x-axis, a y-axis, and a z-axis, wherein the orientation is determined with respect to the x, y, and z axes of the co-ordinate system (Detailed Description, [0052], “The computing device further predicts more possible uv-xyz correspondences, and refines these predictions. The predictions are thresholded on a reprojection error, the error between 2D detection points (u,v's) and the prediction of associated 3D points (x,y,z), assuming a certain pose”).
Regarding claim 10, Samples discloses wherein the lights are detected by an additional camera located at the bottom surface of the HMD to determine the orientation of the main body, the first extension, and the second extension with reference to the additional camera (Fig. 4, HMD, #402; Detailed Description, [0030-0050], “The optical pose of the handheld object with respect to the HMD (“HMD.fwdarw.LED Vision Poses” at 503) and the optical pose of the HMD relative to the world (“World.fwdarw.HMD Poses” at 505) are used to determine a pose of the handheld object with respect to the world”; See also Summary, [0003] for showing of multiple cameras in the stereo camera arrangement).
Regarding claim 11, Samples discloses wherein the main body has a touch surface (Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”), wherein the first light indicator is located below the touch surface to be visible to the camera (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller…A handheld object may have any suitable number and arrangement of light sources.”).
Regarding claim 12, Samples discloses a head-mounted display (Fig. 1-4, #102, HMD)  comprising:
a top surface (See Fig. 2, top surface of HMD 200);
a front surface located adjacent to the top surface (Fig. 2, front surface is next to top surface);
a bottom surface located adjacent to the front surface (Fig. 2, #200);
a band coupled to the top surface (Fig. 1-2, back of HMD is a band for user); and
a camera located at the bottom surface, wherein the camera is configured to capture images of lights emitted from a first light indicator and a second light indicator of a hand-held controller to facilitate a determination of orientation of the hand-held controller with reference to the camera (Fig. 2, cameras, #204; second camera, #206; Detailed Description, [0030-0053], “One or more outward-facing cameras on the HMD 102 may acquire image data (e.g. visible light image data) of the surrounding environment and of a handheld object 106 held by the user 102. For example, FIG. 2 shows an example HMD 200 imaging a handheld object 202 using a stereo camera imaging system (indicated by first camera 204 and second camera 206)....The term “handheld object” is used herein for an object comprising trackable light sources 108 that may be held by hand as described herein, and signifies that the object may be handheld, whether or not the object is currently in the state of being held in a hand… For example, because the HMD may have more than one camera, a pose may be solved for each camera on the HMD. Such refinement is performed as postprocessing of a multitude of poses obtained from different cameras. In other examples, any other suitable pose estimation process may be used.”).
Regarding claim 14,  Samples discloses a head-mounted display further including an additional camera located at the bottom surface, wherein the additional camera is configured to capture images of the lights from the first and second light indicators to facilitate a determination of the orientation of the hand-held controller with reference to the additional camera (Fig. 2, cameras, #204; second camera, #206; Detailed Description, [0030-0053], “One or more outward-facing cameras on the HMD 102 may acquire image data (e.g. visible light image data) of the surrounding environment and of a handheld object 106 held by the user 102. For example, FIG. 2 shows an example HMD 200 imaging a handheld object 202 using a stereo camera imaging system (indicated by first camera 204 and second camera 206)....The term “handheld object” is used herein for an object comprising trackable light sources 108 that may be held by hand as described herein, and signifies that the object may be handheld, whether or not the object is currently in the state of being held in a hand… For example, because the HMD may have more than one camera, a pose may be solved for each camera on the HMD. Such refinement is performed as postprocessing of a multitude of poses obtained from different cameras. In other examples, any other suitable pose estimation process may be used.”).
Regarding claim 15, Samples discloses a head-mounted display further comprising: an eye mask having the top surface, the front surface, the bottom surface, and a frame, wherein the frame is fitted to the bottom surface, wherein the band is a headband coupled to the frame and a portion of the top surface (Fig. 1-2—generally; Examiner’ note—although eyemask and headband are not terms used, they are depicted as included in the HMD structure).
Regarding claim 16, Samples discloses wherein the hand-held controller has a touch surface (Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”), wherein the first light indicator is located below the touch surface and along an edge of the touch surface to be visible to the camera (Fig. 3, Detailed Description, [0036-0041], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300”).
Regarding claim 17, Samples disclose a system (Fig. 1-4, generally) comprising:
a head-mounted display (Fig. 1-4, #102, HMD)  including:
a top surface (See Fig. 2, top surface of HMD 200);
a front surface located adjacent to the top surface (Fig. 2, front surface is next to top surface);
a bottom surface located adjacent to the front surface (Fig. 2, #200);
a band coupled to the top surface (Fig. 1-2, back of HMD is a band for user); and
a camera located at the bottom surface (Fig. 2, cameras, #204; second camera, #206; Detailed Description, [0030-0053], “One or more outward-facing cameras on the HMD 102 may acquire image data (e.g. visible light image data) of the surrounding environment and of a handheld object 106 held by the user 102. For example, FIG. 2 shows an example HMD 200 imaging a handheld object 202 using a stereo camera imaging system (indicated by first camera 204 and second camera 206)....The term “handheld object” is used herein for an object comprising trackable light sources 108 that may be held by hand as described herein, and signifies that the object may be handheld, whether or not the object is currently in the state of being held in a hand… For example, because the HMD may have more than one camera, a pose may be solved for each camera on the HMD. Such refinement is performed as postprocessing of a multitude of poses obtained from different cameras. In other examples, any other suitable pose estimation process may be used.”);
a hand-held controller (Fig. 1-4, handheld object, #106; Detailed Description, [0033-0035]) including:
a main body (Figs. 1-4, Detailed Description, [0035]);
a first extension extending from the main body (Fig. 4, 6DoF controller ,left hand, #406; Detailed Description, [0036-0050]);
a second extension extending from the main body (Fig. 4, 6DoF controller, right hand, #408; Detailed Description, [0036-0050]);
a first light indicator located with respect to the main body, wherein the first light indicator is configured to emit light to be detected by the camera of the HMD (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller”); and
a second light indicator located with respect to the main body, wherein the second light indicator is configured to emit light to be detected by the camera of the HMD, wherein the lights emitted by the first and second light indicators facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD camera (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller…A handheld object may have any suitable number and arrangement of light sources.”; Fig. 4, HMD, #402; Detailed Description, [0030-0050], “The optical pose of the handheld object with respect to the HMD (“HMD.fwdarw.LED Vision Poses” at 503) and the optical pose of the HMD relative to the world (“World.fwdarw.HMD Poses” at 505) are used to determine a pose of the handheld object with respect to the world”).).
Regarding claim 18, Samples discloses wherein the hand-held controller has an edge, wherein the first light indicator is located along the edge to be visible to the camera of the lower region of the HMD with the main body, wherein the second light indicator is located at a distance from the first light indicator to be visible to the camera (Fig. 3, Detailed Description, [0036-0041], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300”).
Regarding claim 19, Samples discloses wherein the HMD includes an additional camera located at the bottom surface, wherein the additional camera is configured to capture images of the lights from the first and second light indicators to facilitate a determination of the orientation of the hand-held controller with reference to the additional camera  (Fig. 2, cameras, #204; second camera, #206; Detailed Description, [0030-0053], “One or more outward-facing cameras on the HMD 102 may acquire image data (e.g. visible light image data) of the surrounding environment and of a handheld object 106 held by the user 102. For example, FIG. 2 shows an example HMD 200 imaging a handheld object 202 using a stereo camera imaging system (indicated by first camera 204 and second camera 206)....The term “handheld object” is used herein for an object comprising trackable light sources 108 that may be held by hand as described herein, and signifies that the object may be handheld, whether or not the object is currently in the state of being held in a hand… For example, because the HMD may have more than one camera, a pose may be solved for each camera on the HMD. Such refinement is performed as postprocessing of a multitude of poses obtained from different cameras. In other examples, any other suitable pose estimation process may be used.”).
Regarding claim 20, Samples discloses wherein the main body has a touch surface (Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”)., wherein the first light indicator is located below the touch surface and along an edge of the touch surface to be visible to the camera of the HMD (Fig. 3, Detailed Description, [0036-0041], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300”).
Regarding claim 21, Samples further discloses wherein the first light indicator is a light bar located along the edge of the touch surface (Fig. 3, Detailed Description, [0036-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300… A handheld object may have any suitable number and arrangement of light sources.”).
Regarding claim 22, Samples further discloses the light bar is located below the touch surface (Fig. 3, Detailed Description, [0036-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300… A handheld object may have any suitable number and arrangement of light sources.”).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 9 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Samples in view of Mallinson, United States Patent Application Publication No. US 2016/0025978 A1.
Regarding claim 9, Samples discloses every element of claim 1 and further discloses wherein
wherein the main body includes a joystick that is configured to be moved to generate input data for processing by a computing device (Detailed Description, [0035], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc. The handheld object 106 comprises an inertial measurement unit (IMU) (which may include an accelerometer, gyroscope, magnetometer, and/or other suitable sensors) that provides output related to changes in position and orientation of the handheld object 106”; See also Figs. 1-4, host computing device, #110).
Samples does not disclose the controller further comprising a haptic feedback module, wherein the haptic feedback module is configured to provide tactile feedback to a user.
Mallinson, in a similar field of endeavor, discloses a controller (Fig. 1) comprising a haptic feedback module, wherein the haptic feedback module is configured to provide tactile feedback to a user (Detailed Description, [0056], “Additionally, part of the processed media content may be transmitted to other devices, such as a game controller, that was used to provide input to the application. For example, video and audio streams of the updates may be provided to the HMD and the haptic feedback may be provided to the game controller.”).
It would have been obvious to one of ordinary skill in the art to have modified the controller of Samples to include the teachings of Mallison to add a haptic feedback module, wherein the haptic feedback module is configured to provide tactile feedback to a user. The motivation to combine these arts is to add to the immersion of the user within a AR/VR HMD system which enhances user experience (See Mallison, Summary, [0006-0009]). The fact the Mallison also discloses HMDs in use with game controllers, same as Samples,  makes this combination easily implemented.
Regarding claim 13, Samples discloses every element of claim 12 but does not explicitly disclose a HMD including a haptic feedback device configured to vibrate the top, front, and bottom surfaces to provide tactile feedback to a head of a user.
Mallinson, in a similar field of endeavor, discloses a head-mounted device (Fig. 1, HMD, #100) including a haptic feedback device configured to vibrate the top, front, and bottom surfaces to provide tactile feedback to a head of a user (Detailed Description, [0080-0085], “Other output modules include Light-Emitting Diodes (LED) 834, other visual markers or elements (which may also be used for visual tracking of the HMD), vibro-tactile feedback 850, speakers 830, and sound localization module 812, which performs sound localization for sounds to be delivered to speakers or headphones. Other output devices, such as headphones, can also connect to the HMD via the communication modules.”).
It would have been obvious to one of ordinary skill in the art to have modified the HMD of Samples to include the teachings of Mallison to add a haptic feedback device configured to vibrate the top, front, and bottom surfaces to provide tactile feedback to a head of a user. The motivation to combine these arts is to add to the immersion of the user within an AR/VR HMD system while wearing the HMD, which enhances user experience (See Mallison, Summary, [0006-0009][0081-0085]). The fact the Mallison also discloses HMDs in use with game controllers, same as Samples,  makes this combination easily implemented.
Conclusion
THIS ACTION IS MADE FINAL.  Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action.  In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action.  In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.

APPLICANT REBUTTAL: REMARKS
Claims 1-20 and 23-24 are pending. Claims 1-22 are rejected. Claims 1-7, 9, 11, 12, 14, and 16-20 are amended. Claims 23-24 are newly added. Claims 21 and 22 are canceled. No new matter has been added.
Claim Rejections - 35 USC § 102
Claims 1-8, 10-12 and 14-22 are rejected under 35 U.S.C. §102(a)(1) as being
anticipated by Samples et al., United States Patent Application No. US 2018/0330521 Al ("Samples"). This rejection is respectfully traversed.
It is respectfully submitted that Samples does not disclose the recitations of claim 1. For example, Samples fails to disclose:
a light bar located with respect to the main body, wherein
the light bar is located along an edge of the touch surface to be visible to a camera located at a bottom surface of a head- mounted display (HMD), wherein the light bar is configured to
emit light to be detected by the camera; and
... wherein the lights emitted by the light bar and the light
indicator facilitate a determination of an orientation of the main
body, the first extension, and the second extension with
reference to the HMD
as is recited in claim 1. Rather, Samples describes a plurality of light sources that form patterns of light ("constellations") in image data (see paragraph 36, "FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300"). Figure 3 of Samples is reproduced below to show the light sources 302.
FIG. 3
However, the constellation pattern in samples is not "a light bar", recited in claim 1. Moreover, Samples describes that the handheld controller 300 may have "any suitable number and arrangement of light sources" (see paragraph 36). Regardless, none of the "suitable number and arrangement of light sources" in Samples describes "a light bar" of claim 1.
Also, because of the constellation light pattern, processing of image data having the constellation pattern to determine an orientation of the handheld controller 300 increases compared to processing of image data indicating the "light bar" and the "light indicator" of claim 1. For example, as described in Samples, with reference to Figure 6, reproduced below:
At 604, for each light source detected, a pixel location in the
image for that light source is determined, represented by pixel coordinates u,v. Next, at 606, each two-dimensional u,v pixel
may be mapped to a corresponding candidate three-dimensional data point x,y,z based on a three-dimensional coordinate frame system in space. At 608, the computing device may generate a hypothesis for the pose of the handheld object based on these
light source correspondences. At 610, the computing device may determine four of the closest detections to the u,v pixel
coordinates in pixel space and four of the closest model points to x,y,z in 3D model space. For unique combinations of 2D-3D
correspondences of size 4 (referring to the four closest
detections and model points), which includes the initial u,v and
x,y,z coordinates (at 612), the computing device may permute
through all possible orderings of the u,v's to fine
correspondence with x,y,z's, at 614.
(see paragraph 52).
-
G. 6
As such, the greater the number of light sources, the greater the number of pixel coordinates
u,v to be determined in Samples. The greater the number of pixel coordinates, the greater the
number of closest detections to the u,v pixel coordinates to be determined in Samples. For
each u,v location, there are four locations to be determined. As such, processing power and
time taken to determine a pose of the handheld object 300 based on the light sources
distributed on the handheld object 300 is greater compared to "determination of an orientation
of the main body, the first extension, and the second extension with reference to the HMD"
based on the lights emitted by the bar and the light indicator of claim 1. As such, for the
reasons set forth above, Samples fails to disclose:
a light bar located with respect to the main body, wherein the light bar is located along an edge of the touch surface to be visible to a camera located at a bottom surface of a head- mounted display (HMD), wherein the light bar is configured to
emit light to be detected by the camera; and
... wherein the lights emitted by the bar and the light indicator facilitate a determination of an orientation of the main
body, the first extension, and the second extension with
reference to the HMD
as is recited in claim 1. Hence, claim 1 is submitted to be novel in view of Samples.
Also, Samples does not disclose the recitations of claim 12. For example, for the same reasons set forth above, Samples fails to disclose that "the camera is configured to capture images of lights emitted from a light bar and a light indicator of a hand-held controller to facilitate a determination of an orientation of the hand-held controller with reference to the camera", as is recited in claim 12. Thus, for the reasons set forth above, claim 12 is submitted to be novel in view of Samples.
Moreover, Samples does not disclose the recitations of claim 17. For example, for the same reasons set forth above, Samples fails to disclose: a light bar located with respect to the main body, wherein the light bar is configured to emit light to be detected
by the camera of the HMD; and
a light indicator located with respect to the main body, wherein the light indicator is configured to emit light to be
detected by the camera of the HMD, wherein the lights emitted by the light bar and the light indicator facilitate a determination
of an orientation of the main body, the first extension, and the
second extension with reference to the HMD
as is recited in claim 17. Therefore, for the reasons set forth above, claim 17 is submitted to be novel in view of Samples.
Claims 21 and 22 are canceled. Claims 2-8 and 10-11 depend from independent claim
1. Also, claims 14-16 depend from independent claim 12 and claims 18-20 depend from independent claim 17. Hence, for the reasons set forth above, claims 2-8, 10-11, 14-16, and 18-20 are submitted to be novel over Samples.
As such, it is respectfully requested that the section 102 rejection of claims 1-8, 10-12 and 14-22 be withdrawn.
Claim Rejections - 35 USC § 103
Claims 9 and 13 are rejected under 35 U.S.C. §103 as being unpatentable over Samples in view of Mallinson, United States Patent Application No. US 2016/0025978 A1. This rejection is respectfully traversed.
Claim 9 depends from independent claim 1 and claim 13 depends from independent claim 12. The combination of Samples and Mallinson does not teach the recitations of claims
1 and 12. For example, the combination fails to teach:
a light bar located with respect to the main body, wherein
the light bar is located along an edge of the touch surface to be visible to a camera located at a bottom surface of a head- mounted display (HMD), wherein the light bar is configured to
emit light to be detected by the camera; and
... wherein the lights emitted by the light bar and the light
indicator facilitate a determination of an orientation of the main
body, the first extension, and the second extension with
reference to the HMD
as is recited in claim 1. As another example, the combination does not disclose or suggest the that "the camera is configured to capture images of lights emitted from a light bar and a light indicator of a hand-held controller to facilitate a determination of an orientation of the hand- held controller with reference to the camera", as is recited in claim 12. Samples is described above. Moreover, Mallinson is not cited to cure the deficiencies in Samples. Rather, Mallinson is cited to disclose "a controller ... comprising a haptic feedback module" and "a head-mounted device . .. including a haptic feedback device" (see office action, pages 15-16). Thus, for the reasons set forth above, independent claims 1 and 12 and the respective dependent claims 9 and 13 are submitted to be nonobvious in view of the combination of Samples and Mallinson.
As such, it is respectfully requested that the section 103 rejection of claims 9 and 13 be withdrawn.
New Claims
Claims 23-24 depend from independent claim 1, and are patentable for the same reasons the independent claim 1 is patentable.
Conclusion

NEXT OFFICE ACTION: DETAILED ACTION
Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection.  Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114.  Applicant's submission filed on February 26, 2024 has been entered.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on February 28, 2024 was filed after the mailing date of the Final Rejection on December 8, 2023 and before first action on RCE.  The submission is in compliance with the provisions of 37 CFR 1.97.  Accordingly, the information disclosure statement is being considered by the examiner.
Response to Amendment
As a result of the Amendment filed on RCE, claims 1-20 and 23-24 are pending.  Claims 1-7, 9, 11, 12,14 and 16-20 are amended. New claims 23-24 are added. Claims 21-22 are canceled.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-20 and 23-24 have been considered but are moot because the new ground of rejection does not rely on the same combination of references applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-8, 10-12, 14-20 and 23-24 are rejected under 35 U.S.C. 103 as being unpatentable over Samples et al., United States Patent Application Publication No. US 2018/0330521 A1 in view of Ohishima et al., United States Patent Application Publication No. US 2002/0084974 A1.
Regarding claim 1, Samples discloses a hand-held controller (Fig. 1-4, handheld object, #106; Detailed Description, [0033-0035]) comprising:
a main body, wherein the main body has a touch surface (Figs. 1-4, Detailed Description, [0035], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc.”);
a first extension extending from the main body (Fig. 4, 6DoF controller, left hand, #406; Detailed Description, [0036-0050]);
a second extension extending from the main body (Fig. 4, 6DoF controller, right hand, #408; Detailed Description, [0036-0050]);
a first light indicator located with respect to the main body, wherein the first light indicator located along an edge of the touch surface to be visible to a camera fitted to a lower region of a head-mounted display (HMD) (Fig. 2, first and second camera, #204/206; Detailed Description, [0033]), wherein the first light indicator is configured to emit light to be detected by the camera (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller”); and
a second light indicator located with respect to the main body, wherein the second light indicator is configured to emit light to be detected by the camera (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller…A handheld object may have any suitable number and arrangement of light sources.”), wherein the lights emitted by the first and second light indicators facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD (Fig. 4, HMD, #402; Detailed Description, [0030-0050], “The optical pose of the handheld object with respect to the HMD (“HMD.fwdarw.LED Vision Poses” at 503) and the optical pose of the HMD relative to the world (“World.fwdarw.HMD Poses” at 505) are used to determine a pose of the handheld object with respect to the world”).
Samples does not disclose the first light indicator explicitly as a light bar.
Ohishima, in a similar field of endeavor, discloses a hand-held controller comprising a light bar located with respect to the main body, where the light bar is located along an edge of the touch surface to be visible to a camera, wherein the light bar is configured to emit light to be detected by the camera (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173], “FIG. 2 is a side view of the game apparatus portion of the system of this embodiment. In an MR air hockey game, two players 2000 and 3000 face each other while holding mallets (260L, 260R) with their hands. The two players 2000 and 3000 wear head mount displays (to be abbreviated as HMDs hereinafter) 210L and 210R on their heads. The mallet of this embodiment has an infrared ray generator at its distal end. As will be described later, in this embodiment, the mallet position is detected by image processing. If each mallet has a feature in its shape or color, the mallet position can also be detected by pattern recognition using such feature….Mallet 260 held by each player has an infrared ray generator at its distal end, and each mallet position in a two-dimensional plane on the table 1000 is detected by a CCD camera 230 that detects the infrared rays. Specifically, the camera 230 is provided so that it may detect mallet positions of the players, and the detected positions of the players will be used for advancing or progressing the game, for this embodiment.”; the depicted mallet is a bar shape and emits infrared light).
It would have been obvious to one of ordinary skill in the art to have modified the form of at least one of the light indicator within Samples to be specifically a light bar, as taught in Ohishima, and provide  wherein the light bar is located along an edge of the touch surface to be visible to a camera fitted to a lower region of a head-mounted display (HMD), wherein the first light bar is configured to emit light to be detected by the camera and wherein the lights emitted by the light bar and light indicator facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD. The motivation to combine these arts is to provide the user with a controller with a holdable form factor, based on the particular function of the HMD (See Ohishima, Detailed Description, [0157-0166], in this case an air hockey game). In addition, the changing the light indicator shape to a bar would have been obvious to one of ordinary skill and pursued without undue experimentation as a matter of design choice (MPEP 2144.04 Part IV).  The fact that Ohishima discloses similar systems of HMD and handheld controllers as Samples, including using infrared emitters and cameras, makes this combination more easily implemented since they are within same field.
Regarding claim 2, Samples further discloses wherein the hand-held controller has an edge, wherein the first light indicator is located along the edge of the hand-held controller to be visible to the camera fitted to lower region of the HMD with the main body, wherein the second light indicator is located at a distance from the first light indicator to be visible to the camera (Fig. 3, Detailed Description, [0036-0041], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300”).
Ohishima also teaches wherein the hand-held controller has an edge, wherein the light bar is located along an edge of the hand-held controller to be visible to the camera (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173], “FIG. 2 is a side view of the game apparatus portion of the system of this embodiment. In an MR air hockey game, two players 2000 and 3000 face each other while holding mallets (260L, 260R) with their hands. The two players 2000 and 3000 wear head mount displays (to be abbreviated as HMDs hereinafter) 210L and 210R on their heads. The mallet of this embodiment has an infrared ray generator at its distal end. As will be described later, in this embodiment, the mallet position is detected by image processing. If each mallet has a feature in its shape or color, the mallet position can also be detected by pattern recognition using such feature….Mallet 260 held by each player has an infrared ray generator at its distal end, and each mallet position in a two-dimensional plane on the table 1000 is detected by a CCD camera 230 that detects the infrared rays. Specifically, the camera 230 is provided so that it may detect mallet positions of the players, and the detected positions of the players will be used for advancing or progressing the game, for this embodiment.”)
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1.
Regarding claim 3, Samples further discloses wherein the touch surface is configured to determine movements of fingers of a user, wherein the first light indicator includes a plurality of light sources, wherein the plurality of light sources are located at pre-determined locations with reference to the touch surface (Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”).
Ohishima also teaches wherein the light bar includes a plurality of light sources, wherein the plurality of light sources are located at pre-determined locations with reference to the touch surface (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173], “Mallet 260 held by each player has an infrared ray generator at its distal end, and each mallet position in a two-dimensional plane on the table 1000 is detected by a CCD camera 230 that detects the infrared rays. Specifically, the camera 230 is provided so that it may detect mallet positions of the players, and the detected positions of the players will be used for advancing or progressing the game, for this embodiment.”).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1.
Regarding claim 4, Samples further discloses wherein the first light indicator is an infrared light emitter (Detailed Description, [0035-0036], “Infrared light sources also may be used.”).
Ohishima also teaches wherein the light bar is an infrared light emitter (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173],
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1.
Regarding claim 5, Samples further discloses wherein the main body has a plurality of buttons (Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”), wherein the first light indicator is configured to face the camera to be in a field-of-view facing downward from the camera (Fig. 2-3, the handheld device #202 is depicted as downward from the HMD which has the camera).
Ohishima also discloses wherein the light bar is configured to face the camera to be in a field-of-view facing downward from the camera (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173], camera, #230).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1.
Regarding claim 6, Samples further discloses a hand-held controller comprising a touch panel situated between the first extension and the second extension, wherein the touch panel has a curvature Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”), wherein the first light indicator is a string of light sources located along an edge of the main body and an edge of the touch surface is situated above the touch panel  (Fig. 3, Detailed Description, [0036-0041], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300”). (Fig. 3, Detailed Description, [0036-0041], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300”; light sources above the handheld object touch surface at bottom).
Ohishima, in a similar field of endeavor, discloses a hand-held controller comprising a light bar (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173])
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1 with the modification of the light indicator as a light bar.
Regarding claim 7, Samples discloses a controller further comprising a battery configured to provide power to the first light indicator and the second light indicator (Detailed Description, [0061]), wherein the second light indicator is a light emitting diode (Detailed Description, [0036]).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1 with the modification of the light indicator as a light bar.
Regarding claim 8, Samples discloses wherein the camera has a co-ordinate system having an x-axis, a y-axis, and a z-axis, wherein the orientation is determined with respect to the x, y, and z axes of the co-ordinate system (Detailed Description, [0052], “The computing device further predicts more possible uv-xyz correspondences, and refines these predictions. The predictions are thresholded on a reprojection error, the error between 2D detection points (u,v's) and the prediction of associated 3D points (x,y,z), assuming a certain pose”).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1 with the modification of the light indicator as a light bar.
Regarding claim 10, Samples discloses wherein the lights are detected by an additional camera located at the bottom surface of the HMD to determine the orientation of the main body, the first extension, and the second extension with reference to the additional camera (Fig. 4, HMD, #402; Detailed Description, [0030-0050], “The optical pose of the handheld object with respect to the HMD (“HMD.fwdarw.LED Vision Poses” at 503) and the optical pose of the HMD relative to the world (“World.fwdarw.HMD Poses” at 505) are used to determine a pose of the handheld object with respect to the world”; See also Summary, [0003] for showing of multiple cameras in the stereo camera arrangement).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1 with the modification of the light indicator as a light bar.
Regarding claim 11, Samples in combination with Ohishima discloses every element of claim 1, and Ohshima further discloses wherein the light bar is located below the touch surface to be visible to the camera (Figs. 1-3, Detailed Description, [0162], “The mallet of this embodiment has an infrared ray generator at its distal end”; distal end would be away from and below where user is touching the mallet).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1 with the modification of the light indicator as a light bar.
Regarding claim 12, Samples discloses a head-mounted display (Fig. 1-4, #102, HMD)  comprising:
a top surface (See Fig. 2, top surface of HMD 200);
a front surface located adjacent to the top surface (Fig. 2, front surface is next to top surface);
a bottom surface located adjacent to the front surface (Fig. 2, #200);
a band coupled to the top surface (Fig. 1-2, back of HMD is a band for user); and
a camera located at the bottom surface, wherein the camera is configured to capture images of lights emitted from a first light indicator and a second light indicator of a hand-held controller to facilitate a determination of orientation of the hand-held controller with reference to the camera (Fig. 2, cameras, #204; second camera, #206; Detailed Description, [0030-0053], “One or more outward-facing cameras on the HMD 102 may acquire image data (e.g. visible light image data) of the surrounding environment and of a handheld object 106 held by the user 102. For example, FIG. 2 shows an example HMD 200 imaging a handheld object 202 using a stereo camera imaging system (indicated by first camera 204 and second camera 206)....The term “handheld object” is used herein for an object comprising trackable light sources 108 that may be held by hand as described herein, and signifies that the object may be handheld, whether or not the object is currently in the state of being held in a hand… For example, because the HMD may have more than one camera, a pose may be solved for each camera on the HMD. Such refinement is performed as postprocessing of a multitude of poses obtained from different cameras. In other examples, any other suitable pose estimation process may be used.”).
Samples does not disclose the first light indicator explicitly as a light bar.
Ohishima, in a similar field of endeavor, discloses a head-mounted display comprising a camera configured to capture images of light emitted from a light bar of a hand-held controller to facilitate a determination of orientation of the hand-held controller with reference to the camera (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173], “FIG. 2 is a side view of the game apparatus portion of the system of this embodiment. In an MR air hockey game, two players 2000 and 3000 face each other while holding mallets (260L, 260R) with their hands. The two players 2000 and 3000 wear head mount displays (to be abbreviated as HMDs hereinafter) 210L and 210R on their heads. The mallet of this embodiment has an infrared ray generator at its distal end. As will be described later, in this embodiment, the mallet position is detected by image processing. If each mallet has a feature in its shape or color, the mallet position can also be detected by pattern recognition using such feature….Mallet 260 held by each player has an infrared ray generator at its distal end, and each mallet position in a two-dimensional plane on the table 1000 is detected by a CCD camera 230 that detects the infrared rays. Specifically, the camera 230 is provided so that it may detect mallet positions of the players, and the detected positions of the players will be used for advancing or progressing the game, for this embodiment.”; the depicted mallet is a bar shape and emits infrared light).
It would have been obvious to one of ordinary skill in the art to have modified the form of at least one of the light indicator within Samples to be specifically a light bar, as taught in Ohishima, and provide the camera configured to capture images of light emitted from a light bar of a hand-held controller to facilitate a determination of orientation of the hand-held controller with reference to the camera. The motivation to combine these arts is to provide the user with a controller with a holdable form factor, based on the particular function of the HMD (See Ohishima, Detailed Description, [0157-0166], in this case an air hockey game). In addition, the changing the light indicator shape to a bar would have been obvious to one of ordinary skill and pursued without undue experimentation as a matter of design choice (MPEP 2144.04 Part IV).  The fact that Ohishima discloses similar systems of HMD and handheld controllers as Samples, including using infrared emitters and cameras, makes this combination more easily implemented since they are within same field.
Regarding claim 14, Samples discloses a head-mounted display further including an additional camera located at the bottom surface, wherein the additional camera is configured to capture images of the lights from the first and second light indicators to facilitate a determination of the orientation of the hand-held controller with reference to the additional camera (Fig. 2, cameras, #204; second camera, #206; Detailed Description, [0030-0053], “One or more outward-facing cameras on the HMD 102 may acquire image data (e.g. visible light image data) of the surrounding environment and of a handheld object 106 held by the user 102. For example, FIG. 2 shows an example HMD 200 imaging a handheld object 202 using a stereo camera imaging system (indicated by first camera 204 and second camera 206)....The term “handheld object” is used herein for an object comprising trackable light sources 108 that may be held by hand as described herein, and signifies that the object may be handheld, whether or not the object is currently in the state of being held in a hand… For example, because the HMD may have more than one camera, a pose may be solved for each camera on the HMD. Such refinement is performed as postprocessing of a multitude of poses obtained from different cameras. In other examples, any other suitable pose estimation process may be used.”).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 12 with the modification of the light indicator as a light bar.
Regarding claim 15, Samples discloses a head-mounted display further comprising: an eye mask having the top surface, the front surface, the bottom surface, and a frame, wherein the frame is fitted to the bottom surface, wherein the band is a headband coupled to the frame and a portion of the top surface (Fig. 1-2—generally; Examiner’ note—although eyemask and headband are not terms used, they are depicted as included in the HMD structure).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 12 with the modification of the light indicator as a light bar.
Regarding claim 16, Samples discloses wherein the hand-held controller has a touch surface (Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”), wherein the first light indicator is located below the touch surface and along an edge of the touch surface to be visible to the camera (Fig. 3, Detailed Description, [0036-0041], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300”).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 12 with the modification of the light indicator as a light bar.
Regarding claim 17, Samples disclose a system (Fig. 1-4, generally) comprising:
a head-mounted display (Fig. 1-4, #102, HMD)  including:
a top surface (See Fig. 2, top surface of HMD 200);
a front surface located adjacent to the top surface (Fig. 2, front surface is next to top surface);
a bottom surface located adjacent to the front surface (Fig. 2, #200);
a band coupled to the top surface (Fig. 1-2, back of HMD is a band for user); and
a camera located at the bottom surface (Fig. 2, cameras, #204; second camera, #206; Detailed Description, [0030-0053], “One or more outward-facing cameras on the HMD 102 may acquire image data (e.g. visible light image data) of the surrounding environment and of a handheld object 106 held by the user 102. For example, FIG. 2 shows an example HMD 200 imaging a handheld object 202 using a stereo camera imaging system (indicated by first camera 204 and second camera 206)....The term “handheld object” is used herein for an object comprising trackable light sources 108 that may be held by hand as described herein, and signifies that the object may be handheld, whether or not the object is currently in the state of being held in a hand… For example, because the HMD may have more than one camera, a pose may be solved for each camera on the HMD. Such refinement is performed as postprocessing of a multitude of poses obtained from different cameras. In other examples, any other suitable pose estimation process may be used.”);
a hand-held controller (Fig. 1-4, handheld object, #106; Detailed Description, [0033-0035]) including:
a main body (Figs. 1-4, Detailed Description, [0035]);
a first extension extending from the main body (Fig. 4, 6DoF controller ,left hand, #406; Detailed Description, [0036-0050]);
a second extension extending from the main body (Fig. 4, 6DoF controller, right hand, #408; Detailed Description, [0036-0050]);
a first light indicator located with respect to the main body, wherein the first light indicator is configured to emit light to be detected by the camera of the HMD (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller”); and
a second light indicator located with respect to the main body, wherein the second light indicator is configured to emit light to be detected by the camera of the HMD, wherein the lights emitted by the first and second light indicators facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD camera (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller…A handheld object may have any suitable number and arrangement of light sources.”; Fig. 4, HMD, #402; Detailed Description, [0030-0050], “The optical pose of the handheld object with respect to the HMD (“HMD.fwdarw.LED Vision Poses” at 503) and the optical pose of the HMD relative to the world (“World.fwdarw.HMD Poses” at 505) are used to determine a pose of the handheld object with respect to the world”).).
Samples does not disclose the first light indicator explicitly as a light bar.
Ohishima, in a similar field of endeavor, discloses a hand-held controller comprising a light bar located with respect to the main body, wherein the light bar is configured to emit light to be detected by the camera of the HMD  (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173], “FIG. 2 is a side view of the game apparatus portion of the system of this embodiment. In an MR air hockey game, two players 2000 and 3000 face each other while holding mallets (260L, 260R) with their hands. The two players 2000 and 3000 wear head mount displays (to be abbreviated as HMDs hereinafter) 210L and 210R on their heads. The mallet of this embodiment has an infrared ray generator at its distal end. As will be described later, in this embodiment, the mallet position is detected by image processing. If each mallet has a feature in its shape or color, the mallet position can also be detected by pattern recognition using such feature….Mallet 260 held by each player has an infrared ray generator at its distal end, and each mallet position in a two-dimensional plane on the table 1000 is detected by a CCD camera 230 that detects the infrared rays. Specifically, the camera 230 is provided so that it may detect mallet positions of the players, and the detected positions of the players will be used for advancing or progressing the game, for this embodiment.”; the depicted mallet is a bar shape and emits infrared light).
It would have been obvious to one of ordinary skill in the art to have modified the form of at least one of the light indicator within Samples to be specifically a light bar, as taught in Ohishima, and provide a light bar located with respect to the main body, wherein the light bar is configured to emit light to be detected by the camera of the HMD, wherein the first light bar is configured to emit light to be detected by the camera and wherein the lights emitted by the light bar and light indicator facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD. The motivation to combine these arts is to provide the user with a controller with a holdable form factor, based on the particular function of the HMD (See Ohishima, Detailed Description, [0157-0166], in this case an air hockey game). In addition, the changing the light indicator shape to a bar would have been obvious to one of ordinary skill and pursued without undue experimentation as a matter of design choice (MPEP 2144.04 Part IV).  The fact that Ohishima discloses similar systems of HMD and handheld controllers as Samples, including using infrared emitters and cameras, makes this combination more easily implemented since they are within same field.
Regarding claim 18, this is met by the rejection to claim 2 with the combination of Samples and Ohishima.
Regarding claim 19, this is met by the rejection to claim 14 with the combination of Samples and Ohishima.
Regarding claim 20, this is met by the rejection of claim 11 with the combination of Samples and Ohishima.
Regarding claim 23, Samples in combination with Ohishima discloses every element of claim 1, and Ohishima further discloses wherein the light bar has a first edge point and a second edge point, wherein the first and second edge points define the orientation of the main body, the first extension, and the second extension with reference to the HMD (Figs. 1-5, mallet, #260; Detailed Description, [0156-0182]; See also Detailed Description, [0192-210] and Figs. 8-10).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1 with the modification of the first light indicator as a light bar.
Regarding claim 24, Samples in combination with Ohishima discloses every element of claim 23, and Ohishima further discloses wherein the orientation is a first orientation, wherein the first and second edge points have a first set of locations with respect to a reference level passing through the light indictor to define the first orientation and have a second set of locations with respect to the reference level to define a second orientation of the main body, the first extension, and the second extension with reference to the HMD (Ohishima, Figs. 1-5, mallet, #260; Detailed Description, [0156-0182]; See also Detailed Description, [0192-210] and Figs. 8-10), “ As will be described later, in this embodiment, the mallet position is detected by image processing. If each mallet has a feature in its shape or color, the mallet position can also be detected by pattern recognition using such feature.”).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1 with the modification of the first light indicator as a light bar.
Claim(s) 9 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Samples, in view of Ohishima, further in view of Mallinson, United States Patent Application Publication No. US 2016/0025978 A1.
Regarding claim 9, Samples in combination with Ohishima discloses every element of claim 1 and Samples further discloses a touch panel (Detailed Description, [0035]).
Samples and Ohishima do not disclose the combination of the hand-held controller comprising a haptic feedback module and a touch panel, wherein the main body includes a first joystick and a second joystick that are configured to be moved to generate input data for processing by a computing device, wherein the touch panel is located between the first and second joysticks, wherein the haptic feedback module is configured to provide tactile feedback to a user.
However, Samples does provide the suggestion of the main body including using both a joystick and touch panel together (Detailed Description, [0035]).
Mallinson, in a similar field of endeavor, discloses a controller (Fig. 1) comprising a haptic feedback module and a touch panel (Detailed Description, [0081]), wherein the main body includes a first joystick and a second joystick that are configured to be moved to generate input data for processing by a computing device, (Mallison, Detailed Description, [0091], “A member of Clients 1110 is optionally configured to receive more than one audio or video stream. Input devices of Clients 1110 may include, for example, a one-hand game controller, a two-hand game controller, a gesture recognition system, a gaze recognition system, a voice recognition system, a keyboard, a joystick, a pointing device, a force feedback device, a motion and/or location sensing device, a mouse, a touch screen, a neural interface, a camera, input devices yet to be developed, and/or the like.”) wherein the haptic feedback module is configured to provide tactile feedback to a user (Detailed Description, [0056], “Additionally, part of the processed media content may be transmitted to other devices, such as a game controller, that was used to provide input to the application. For example, video and audio streams of the updates may be provided to the HMD and the haptic feedback may be provided to the game controller.”).
The above references do not explicitly disclose wherein the touch panel is located between the first and second joysticks.
However, it would have been obvious to one of ordinary skill in the art to have modified the combination of Samples and Ohishima to utilize the teachings and suggestion of Mallinson to provide the combination of the hand-held controller comprising a haptic feedback module and a touch panel, wherein the main body includes a first joystick and a second joystick that are configured to be moved to generate input data for processing by a computing device, wherein the touch panel is located between the first and second joysticks, wherein the haptic feedback module is configured to provide tactile feedback to a user. The motivation to combine these arts is to add to the immersion of the user while using a game controller within a AR/VR HMD system which enhances user experience (See Mallison, Summary, [0006-0009]). Furthermore, the modification to the prior art as a whole to provide wherein the touch panel is located between the first and second joysticks is a matter of rearrangement of parts, which would have been pursued by one of ordinary skill without undue experimentation since it would not have modified the overall operation of the HMD/controller system (MPEP 2144.04 Part VI). The fact the Mallison also discloses HMDs in use with game controllers, same as Samples and Ohishima, makes this combination easily implemented.
Regarding claim 13, Samples in combination with Ohishima discloses every element of claim 12 but those above references does not explicitly disclose a HMD including a haptic feedback device configured to vibrate the top, front, and bottom surfaces to provide tactile feedback to a head of a user.
Mallinson, in a similar field of endeavor, discloses a head-mounted device (Fig. 1, HMD, #100) including a haptic feedback device configured to vibrate the top, front, and bottom surfaces to provide tactile feedback to a head of a user (Detailed Description, [0080-0085], “Other output modules include Light-Emitting Diodes (LED) 834, other visual markers or elements (which may also be used for visual tracking of the HMD), vibro-tactile feedback 850, speakers 830, and sound localization module 812, which performs sound localization for sounds to be delivered to speakers or headphones. Other output devices, such as headphones, can also connect to the HMD via the communication modules.”).
It would have been obvious to one of ordinary skill in the art to have modified the HMD of Samples-Ohishima to include the teachings of Mallison to add a haptic feedback device configured to vibrate the top, front, and bottom surfaces to provide tactile feedback to a head of a user. The motivation to combine these arts is to add to the immersion of the user within an AR/VR HMD system while wearing the HMD, which enhances user experience (See Mallison, Summary, [0006-0009][0081-0085]). The fact the Mallison also discloses HMDs in use with game controllers, same as Samples, makes this combination easily implemented.
Conclusion

CITED REFERENCE SUMMARIES: Ref US20180330521A1: Claims:
A head-mounted device, comprising: a stereo camera arrangement; a logic device configured to execute instructions; and a storage device comprising instructions executable by the logic device to for each camera in the stereo camera arrangement, receive image data of a field of view of the camera, detect a plurality of light sources of a handheld object in the image data, and based upon the plurality of light sources detected, determine a pose of the handheld object; and based upon the pose of the handheld object determined for each camera in the stereo camera arrangement, calibrate the stereo camera arrangement.
a stereo camera arrangement;
a logic device configured to execute instructions; and
a storage device comprising instructions executable by the logic device to for each camera in the stereo camera arrangement, receive image data of a field of view of the camera, detect a plurality of light sources of a handheld object in the image data, and based upon the plurality of light sources detected, determine a pose of the handheld object; and based upon the pose of the handheld object determined for each camera in the stereo camera arrangement, calibrate the stereo camera arrangement.
for each camera in the stereo camera arrangement, receive image data of a field of view of the camera, detect a plurality of light sources of a handheld object in the image data, and based upon the plurality of light sources detected, determine a pose of the handheld object; and
receive image data of a field of view of the camera,
detect a plurality of light sources of a handheld object in the image data, and
based upon the plurality of light sources detected, determine a pose of the handheld object; and
based upon the pose of the handheld object determined for each camera in the stereo camera arrangement, calibrate the stereo camera arrangement.
The head-mounted device of claim 1 , wherein the instructions are executable to calibrate the stereo camera arrangement during one or more of a motion of the head-mounted device and a motion of the handheld object.
The head-mounted device of claim 1 , wherein the instructions are executable to calibrate intrinsic characteristics for each camera of the stereo camera arrangement, the intrinsic characteristics comprising one or more of lens distortion, focal length, and principal point.
The head-mounted device of claim 1 , wherein the instructions are executable to calibrate extrinsic characteristics for each camera of the stereo camera arrangement, the extrinsic characteristics comprising one or more of a rotation and a translation.
The head-mounted device of claim 1 , wherein the instructions are executable to determine the pose by fitting an observed light pattern to a model light pattern via a rigid body transformation.
The head-mounted device of claim 1 , wherein the instructions are executable to determine the pose by, for each light source detected in the image data received from a first camera, searching along an epipolar line for a companion pixel in the image data received from a second camera, the companion pixel corresponding to the light source detected in the image data received from the first camera.
The head-mounted device of claim 1 , wherein the instructions are executable to calibrate the stereo camera arrangement utilizing one or more of a Kalman filter, a sliding window bundle adjustment, and a recursive Gauss-Newton filter.
The head-mounted device of claim 1 , wherein the instructions are executable to calibrate the stereo camera arrangement between 6 and 10 times per minute.
A method for determining a calibration for a handheld object via a head-mounted device comprising a stereo camera arrangement, the method comprising: receiving, from a first camera of the stereo camera arrangement, first image data from a perspective of the first camera; receiving, from a second camera of the stereo camera arrangement, second image data from a perspective of the second camera; detecting, in the first image data and the second image data, a plurality of light sources of the handheld object; determining a three-dimensional position of each light source of the plurality of light sources detected; and determining a calibration for the handheld object based upon each three-dimensional position determined.
receiving, from a first camera of the stereo camera arrangement, first image data from a perspective of the first camera;
receiving, from a second camera of the stereo camera arrangement, second image data from a perspective of the second camera;
detecting, in the first image data and the second image data, a plurality of light sources of the handheld object;
determining a three-dimensional position of each light source of the plurality of light sources detected; and
determining a calibration for the handheld object based upon each three-dimensional position determined.
The method of claim 9 , further comprising receiving from an inertial measurement unit (IMU) of the handheld object a sensed orientation of the handheld object, and determining a calibration for the IMU based upon the sensed orientation.
receiving from an inertial measurement unit (IMU) of the handheld object a sensed orientation of the handheld object, and
determining a calibration for the IMU based upon the sensed orientation.
The method of claim 10 , wherein determining the calibration for the IMU comprises determining a calibration for a position and a rotation of the IMU.
The method of claim 9 , wherein determining the calibration of the handheld object comprises determining calibration parameters via one or more of a Kalman filter, a sliding window bundle adjustment, and a recursive Gauss-Newton filter.
The method of claim 9 , wherein determining the three-dimensional position of each light source comprises fitting an observed light pattern to a model light pattern via a rigid body transformation.
A head-mounted device, comprising: a stereo camera arrangement; a logic device configured to execute instructions; and a storage device comprising instructions executable by the logic device to for each camera in the stereo camera arrangement, receive image data of a surrounding environment from a perspective of the camera, detect via the image data a plurality of feature points of a handheld object in a field of view of the camera, determine a location of each feature point detected, and determine a pose of the handheld object from the perspective of the camera based on the location of each feature point determined; update one or more calibration parameters for the stereo camera arrangement based on the pose of the handheld object determined for each camera; determine a three-dimensional position of each feature point based on the location of each feature point determined for each camera of the stereo camera arrangement; and update one or more calibration parameters for the handheld object based on the three-dimensional position of each feature point determined.
a storage device comprising instructions executable by the logic device to for each camera in the stereo camera arrangement, receive image data of a surrounding environment from a perspective of the camera, detect via the image data a plurality of feature points of a handheld object in a field of view of the camera, determine a location of each feature point detected, and determine a pose of the handheld object from the perspective of the camera based on the location of each feature point determined; update one or more calibration parameters for the stereo camera arrangement based on the pose of the handheld object determined for each camera; determine a three-dimensional position of each feature point based on the location of each feature point determined for each camera of the stereo camera arrangement; and update one or more calibration parameters for the handheld object based on the three-dimensional position of each feature point determined.
for each camera in the stereo camera arrangement, receive image data of a surrounding environment from a perspective of the camera, detect via the image data a plurality of feature points of a handheld object in a field of view of the camera, determine a location of each feature point detected, and determine a pose of the handheld object from the perspective of the camera based on the location of each feature point determined;
receive image data of a surrounding environment from a perspective of the camera,
detect via the image data a plurality of feature points of a handheld object in a field of view of the camera,
determine a location of each feature point detected, and
determine a pose of the handheld object from the perspective of the camera based on the location of each feature point determined;
update one or more calibration parameters for the stereo camera arrangement based on the pose of the handheld object determined for each camera;
determine a three-dimensional position of each feature point based on the location of each feature point determined for each camera of the stereo camera arrangement; and
update one or more calibration parameters for the handheld object based on the three-dimensional position of each feature point determined.
The head-mounted device of claim 14 , wherein the instructions are executable to receive the image data as a background process during use of the head-mounted device and the handheld object.
The head-mounted device of claim 14 , wherein the instructions are further executable to receive from an inertial measurement unit (IMU) of the handheld object a sensed orientation of the handheld object, and to determine a calibration for one or more of a translation and a rotation of the IMU based upon the sensed orientation.
The head-mounted device of claim 14 , wherein the one or more calibration parameters for the stereo camera arrangement comprise one or more of an intrinsic camera characteristic and an extrinsic camera characteristic.
The head-mounted device of claim 14 , wherein the instructions are executable to determine the pose of the handheld object by one or more of fitting an observed feature point pattern to a model feature point pattern via a rigid body transformation; and for each feature point detected in the image data received from a first camera of the stereo camera arrangement, searching along an epipolar line for a companion feature point in the image data received from a second camera of the stereo camera arrangement, the companion feature point corresponding to the feature point detected in the image data received from the first camera.
fitting an observed feature point pattern to a model feature point pattern via a rigid body transformation; and
for each feature point detected in the image data received from a first camera of the stereo camera arrangement, searching along an epipolar line for a companion feature point in the image data received from a second camera of the stereo camera arrangement, the companion feature point corresponding to the feature point detected in the image data received from the first camera.
The head-mounted device of claim 14 , wherein the instructions are executable to update the one or more calibration parameters for the stereo camera arrangement and update the one or more calibration parameters for the handheld object simultaneously.
The head-mounted device of claim 14 , wherein the instructions are executable to update one or more calibration parameters for one or more of the stereo camera arrangement and the handheld object via one or more of a Kalman filter, a sliding window bundle adjustment, and a recursive Gauss-Newton filter.
Ref US20160025978A1: Claims:
A device, comprising: a display screen having a front side and a back side, the display screen configured for rendering media content; first optics disposed adjacent to the front side of the display screen and configured to provide a focus for viewing the media content when rendered on the display screen; a shutter screen disposed adjacent to the backside of the display screen, wherein the shutter screen is switchable between an opaque mode and a transparent mode and wherein the opaque mode is active when the media content is viewable on the display screen; and second optics disposed behind the shutter screen such that the shutter screen is between the display screen and the second optics, the second optics providing an adjustment to the focus to allow viewing through the first optics, the display screen, the shutter screen and the second optics when the transparent mode is activated on the shutter screen.
a display screen having a front side and a back side, the display screen configured for rendering media content;
first optics disposed adjacent to the front side of the display screen and configured to provide a focus for viewing the media content when rendered on the display screen;
a shutter screen disposed adjacent to the backside of the display screen, wherein the shutter screen is switchable between an opaque mode and a transparent mode and wherein the opaque mode is active when the media content is viewable on the display screen; and
second optics disposed behind the shutter screen such that the shutter screen is between the display screen and the second optics, the second optics providing an adjustment to the focus to allow viewing through the first optics, the display screen, the shutter screen and the second optics when the transparent mode is activated on the shutter screen.
The device of claim 1 , wherein the adjustment provided in the second optics is to compensate for view distortion caused by the focus of the first optics.
The device of claim 1 , wherein the transparent mode is activated on a portion of the shutter screen and the opaque mode is activated for remaining portions of the shutter screen.
The device of claim 3 , wherein the display screen is rendered transparent.
The device of claim 3 , wherein at least a portion of the media content is rendered in portions of the display screen that correspond with the remaining portions of the shutter screen where the opaque mode is activated.
The device of claim 3 , wherein the activation of the transparent mode in the portion of the shutter screen causes a viewport to be defined through the portion of the shutter screen and through corresponding portions of the first optics, the display screen and the second optics.
The device of claim 3 , wherein the activation of the transparent mode in the portion of the shutter screen causes activation of a window in the portion of the display screen that corresponds to at least one of the remaining portions of the shutter screen that is in opaque mode, the window rendering a view of a web portal at a web site and providing options to interact with the web portal.
The device of claim 1 , further includes a switching circuit that is communicatively connected to the shutter screen to activate the transparent mode or the opaque mode.
The device of claim 8 , further includes a controller communicatively connected to the switching circuit to control the switching of the shutter screen to the transparent mode or the opaque mode.
The device of claim 8 , further includes an event detector circuit configured to generate a signal for selectively switching the shutter screen to transparent mode, in response to detection of an event occurring within the media content or within an external environment near the device.
The device of claim 10 , wherein the event detected is one of a change in an external environment condition in vicinity of the device, change in media content being rendered, change in environment condition within the media content, an audio signal detected in the vicinity of the device, audio signal generated in the media content, a visual cue detected in the vicinity of the user, or any combinations thereof.
The device of claim 10 , wherein when the event corresponds to movement of an object in an external environment in vicinity of the device, the event detector circuit is configured to track movement of the object and generate appropriate signals to cause selective switching of appropriate portions of the shutter screen corresponding to the movement of the object, to transparent mode to permit viewing of the moving object.
The device of claim 1 , further includes input device configured for user interaction, wherein the user interaction includes providing annotation on an object provided in the media content or on an object from an external environment, interaction with the media content or the object, blending of the object into the media content or any combinations thereof.
The device of claim 1 , wherein each of the first optic, the display screen, the shutter screen and the second optic is coated with an anti-reflective coating.
A pair of glasses, comprising, a view port provided with a multi-layer arrangement, the multi-layer arrangement includes, a first optic provided with a first focus setting; a display screen positioned behind the first optic, the display screen being transparent; a shutter screen positioned behind the display screen, the shutter screen being adjustable between a transparent mode and an opaque mode; and a second optic provided with a second focus setting, the second focus setting removing the first focus setting to provide a see through view through the first optic, the display screen, the shutter screen and the second optic, when the shutter screen is set to the transparent mode.
a view port provided with a multi-layer arrangement, the multi-layer arrangement includes, a first optic provided with a first focus setting; a display screen positioned behind the first optic, the display screen being transparent; a shutter screen positioned behind the display screen, the shutter screen being adjustable between a transparent mode and an opaque mode; and a second optic provided with a second focus setting, the second focus setting removing the first focus setting to provide a see through view through the first optic, the display screen, the shutter screen and the second optic, when the shutter screen is set to the transparent mode.
a first optic provided with a first focus setting;
a display screen positioned behind the first optic, the display screen being transparent;
a shutter screen positioned behind the display screen, the shutter screen being adjustable between a transparent mode and an opaque mode; and
a second optic provided with a second focus setting, the second focus setting removing the first focus setting to provide a see through view through the first optic, the display screen, the shutter screen and the second optic, when the shutter screen is set to the transparent mode.
The pair of glasses of claim 15 , further includes a switching circuit communicatively connected to the shutter screen and configured to activate the transparent mode or the opaque mode.
The pair of glasses of claim 16 , further includes a controller connected to the switching circuit to control switching of the shutter screen to the transparent mode or the opaque mode.
The pair of glasses of claim 16 , wherein the switching circuit is configured to activate the transparent mode in a select portion of the shutter screen.
The pair of glasses of claim 15 , wherein removal of the first focus setting is by configuring the second focus setting of the second optic to counterbalance the first focus setting so as to cancel out any distortion caused by the first focus setting when viewing through the pair of glasses.
The pair of glasses of claim 15 , wherein the first optic, the display screen, the shutter screen and the second optic are each coated with anti-reflective coating.
A method, comprising: receiving media content for rendering on a display screen of a pair of glasses, the pair of glasses including first optics in front of the display screen to provide a focus for viewing the media content rendering on the display screen; detecting an event t
[Truncated to input limit]