Office Action 4: Final Rejection (2024-09-11)
Rejection types: 103 - Obviousness

CLAIMS:
IN THE CLAIMS

Please amend the claims are shown below to include underlined sections and delete strikeout or double bracketed ("\[\[ \]\]") sections.

Listing of Claims:

1\. (Currently amended) A hand-held controller comprising:a main body, wherein the main body has a touch surface;
a first extension extending from the main body;
a second extension extending from the main body;
a first group of buttons;
a second group of directional buttons;
a light bar located with respect to the main body, wherein the light bar is located along an edge of the touch surface to be visible to a camera fitted to a lower region of a head-mounted display (HMD), wherein the light bar extends between the first group of buttons and the second group of directional buttons, wherein the light bar is configured to emit light to be detected by the camera; and
a light indicator located with respect to the main body, wherein the light indicator is configured to emit light to be detected by the camera, wherein the lights emitted by the light bar and the light indicator facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD.
2\. (Currently amended) The hand-held controller of claim 1, wherein the hand-held controller has an edge, wherein the light bar is located along the edge of the hand-held controller to be visible to the camera fitted to the lower region of the HMD, wherein the light indicator is located at a distance from the light bar to be visible to the camera.

3\. (Previously presented) The hand-held controller of claim 1, wherein the touch surface is configured to determine movements of fingers of a user, wherein the light bar includes a plurality of light sources, wherein the plurality of light sources are located at pre-determined locations with reference to the touch surface.

4\. (Previously presented) The hand-held controller of claim 1, wherein the light bar is an infrared light emitter.
5\. (Currently amended) The hand-held controller of claim 1, wherein the light bar is configured to face the camera to be in a field- of-view facing downward from the camera.
6\. (Previously presented) The hand-held controller of claim 1, further comprising a touch panel situated between the first extension and the second extension, wherein the touch panel has a curvature, wherein the light bar is a string of light sources located along an edge of the main body and is situated above the touch panel.
7\. (Previously presented) The hand-held controller of claim 1, further comprising a battery configured to provide power to the light bar and the light indicator, wherein the light indicator is a light emitting diode.
8\. (Original) The hand-held controller of claim 1, wherein the camera has a co-ordinate system having an x-axis, a y-axis, and a z-axis, wherein the orientation is determined with respect to the x, y, and z axes of the co-ordinate system.
9\. (Previously presented) The hand-held controller of claim 1, further comprising a haptic feedback module and a touch panel, wherein the main body includes a first joystick and a second joystick that are configured to be moved to generate input data for processing by a computing device, wherein the touch panel is located between the first and second joysticks, wherein the haptic feedback module is configured to provide tactile feedback to a user.

10\. (Original) The hand-held controller of claim 1, wherein the lights are detected by an additional camera located at the bottom surface of the HMD to determine the orientation of the main body, the first extension, and the second extension with reference to the additional camera.

11\. (Previously presented) The hand-held controller of claim 1, wherein the light bar is located below the touch surface to be visible to the camera.

12\. (Currently amended) A head-mounted display comprising:a top surface;
a front surface located adjacent to the top surface;
a bottom surface located adjacent to the front surface;
a band coupled to the top surface; and
a camera located at the bottom surface, wherein the camera is configured to capture images of lights emitted from a light bar and a light indicator of a hand-held controller to facilitate a determination of an orientation of the hand-held controller with reference to the camera, wherein the light bar extends between a first group of buttons of the hand-held controller and a second group of directional buttons of the hand-held controller.

13\. (Original) The head-mounted display of claim 12, further including a haptic feedback device configured to vibrate the top, front, and bottom surfaces to provide tactile feedback to a head of a user.

14\. (Previously presented) The head-mounted display of claim 12, further including an additional camera located at the bottom surface, wherein the additional camera is configured to capture images of the lights from the light bar and the light indicator to facilitate a determination of the orientation of the hand-held controller with reference to the additional camera.

15\. (Original) The head-mounted display of claim 12, further comprising:an eye mask having the top surface, the front surface, the bottom surface, and a frame, wherein the frame is fitted to the bottom surface, wherein the band is a headband coupled to the frame and a portion of the top surface.

16\. (Previously presented) The head-mounted display of claim 12, wherein the hand-held controller has a touch surface, wherein the light bar is located below the touch surface and along an edge of the touch surface to be visible to the camera.

17\. (Currently amended) A system comprising:a head-mounted display (HMD) including:a top surface;
a front surface located adjacent to the top surface;
a bottom surface located adjacent to the front surface;

a band coupled to the top surface; and
a camera located at the bottom surface;
a hand-held controller including:
a main body;
a first extension extending from the main body;
a second extension extending from the main body;
a first group of buttons;
a second group of directional buttons;
a light bar located with respect to the main body, wherein the light bar extends between the first group of buttons and the second group of directional buttons,
wherein the light bar is configured to emit light to be detected by the camera of the HMD; anda light indicator located with respect to the main body, wherein the light indicator is configured to emit light to be detected by the camera of the HMD, wherein the lights emitted by the light bar and the light indicator facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD.

18\. (Previously presented) The system of claim 17, wherein the hand-held controller has an edge, wherein the light bar is located along the edge to be visible to the camera of the lower region of the HMD with the main body, wherein the light indicator is located at a distance from the light bar to be visible to the camera.

19\. (Previously presented) The system of claim 17, wherein the HMD includes an additional camera located at the bottom surface, wherein the additional camera is configured to capture images of the lights from the light bar and the light indicator to facilitate a determination of the orientation of the hand-held controller with reference to the additional camera.

20\. (Previously presented) The system of claim 17, wherein the main body has a touch surface, wherein the light bar is located below the touch surface and along an edge of the touch surface to be visible to the camera of the HMD.

21\. (Canceled)

22\. (Canceled)

23\. (Previously presented) The hand-held controller of claim 1, wherein the light bar has a first edge point and a second edge point, wherein the first and second edge points define the orientation of the main body, the first extension, and the second extension with reference to the HMD.

24\. (Previously presented) The hand-held controller of claim 23, wherein the orientation is a first orientation, wherein the first and second edge points have a first set of locations with respect to a reference level passing through the light indictor to define the first orientation and have a second set of locations with respect to the reference level to define a second orientation of the main body, the first extension, and the second extension with reference to the HMD.

25\. (New) The hand-held controller of claim 1, wherein the light bar has a plurality of edges, wherein the orientation is determined based on whether the plurality of edges are above or below a horizontal level that passes through the light indicator.

26\. (New) The hand-held controller of claim 25, wherein the plurality of edges include a first edge and a second edge, wherein the orientation is a first orientation of the first and second extensions when the plurality of edges are below the horizontal level, wherein the first and second extensions have a second orientation with reference to the HMD when the first edge is above the horizontal level and the second edge is below the horizontal level.

OFFICE ACTION:
DETAILED ACTION
Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
As a result of the Amendment foiled on July 12, 2024, claims 1-20 and 23-26 are pending. Claims 1, 2, 5, 12 and 17 are amended. Claims 21-22 were previously canceled. New claims 25-26 are added.
Response to Arguments
Applicant's arguments filed July 12, 2024 with respect the art rejections of claims 1-20 and 23-25 have been fully considered but they are not persuasive. Applicant is contending that the prior art does not show the amended features of a “first group of buttons”, “a second group of directional buttons”, and “wherein the first light bar extends between the first group of buttons and the second group of directional buttons”. (Applicant Remarks at pgs. 9-11). Furthermore, Applicant also argues the feature “wherein the lights emitted by the light indicator and light bar facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD” is not taught by Ohishima because Ohishima’s xy coordinate position is not an orientation (Applicant Remarks at pgs. 11-12).
The Office respectfully disagrees with the above conclusions. To start, Ohishima was used purely for its form-factor of a “light bar” and the structural modification to Samples merely changes the form-factor of the handheld object (Samples, Figs. 1-4, handheld object, #106). The rationale and reasoning for the modification has been set forth in the 103 rejection as a prima facie case of obviousness. (See rejection section for details). Otherwise, Samples teaches the remaining structural and functional aspects of the independent claims. Samples teaches a first group of buttons and a second group of directional buttons (Detailed Description, [0034-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc.”) because the cited section shows both buttons and directional pads can be input controls on the handheld object.  Next, Samples teaches “wherein the first light indicator extends between the first group of buttons and the second group of directional buttons” (See Figs. 2-3, light sources, #302; light sources, Detailed Description, [0034-0036][0130-0140], since light sources are all around the surface, it includes the areas in between the first group of buttons and second group of button) because Samples shows the light sources distributed along the ring, including in vector space in between the joystick and buttons on the handle.  It is noted that it was not affirmatively claimed that the light bar extends between the first group of buttons and second group of buttons within the same plane or within the same layer. Next, Samples continues to teaches  “wherein the lights emitted by the first light indicator and second light indicator facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD” (Fig. 4, HMD, #402; Detailed Description, [0030-0050], “The optical pose of the handheld object with respect to the HMD (“HMD.fwdarw.LED Vision Poses” at 503) and the optical pose of the HMD relative to the world (“World.fwdarw.HMD Poses” at 505) are used to determine a pose of the handheld object with respect to the world”; Detailed Description, [0130-140], “As the HMD makes a sequence of observations, the HMD may determine that alternative parameters may make the observations more likely. For example, after a handheld object is dropped and experiences a deformation in light source positions on the handheld object, the HMD may observe a light pattern that differs from the model light pattern, and may recompute the constellation positions and update one or more calibration parameters for the handheld object. The tracking system on the HMD may use new estimates of light source positions, normal, illumination fields, and IMU parameters to refine future tracking and pose estimates.”).
Since Ohishima modifies Sample’s first light indicator to be a rectangular shaped light bar, the combination and rationale for independent claims 1, 12 and 17 remain as set forth in the previous Office Action.  For the foregoing reasons, the rejection grounds are maintained for claims 1-20 and 23-25.
Applicant’s arguments in light of the new claims added with respect specifically to claim 26 have been fully considered and are persuasive.  Claim 26 is objected to as being dependent off of rejected base claim but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims (including claim 25).
An additional reference is cited on the PTO-892 is relevant to the Field of the Invention but is not necessarily used in the rejection grounds.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-8, 10-12, 14-20 and 23-25 are rejected under 35 U.S.C. 103 as being unpatentable over Samples et al., United States Patent Application Publication No. US 2018/0330521 A1 in view of Ohishima et al., United States Patent Application Publication No. US 2002/0084974 A1.
Regarding claim 1, Samples discloses a hand-held controller (Fig. 1-4, handheld object, #106; Detailed Description, [0033-0035]) comprising:
a main body, wherein the main body has a touch surface (Figs. 1-4, Detailed Description, [0035], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc.”);
a first extension extending from the main body (Fig. 4, 6DoF controller, left hand, #406; Detailed Description, [0036-0050]);
a second extension extending from the main body (Fig. 4, 6DoF controller, right hand, #408; Detailed Description, [0036-0050]);
a first group of buttons (Detailed Description, [0034-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc.”);
a second group of directional buttons (Detailed Description, [0034-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc.”)
a first light indicator located with respect to the main body, wherein the first light indicator extends between the first group of buttons and the second group of directional buttons (See Figs. 2-3, light sources, #302; light sources, Detailed Description, [0034-0036][0130-0140], since light sources are all around the surface, it includes the areas in between the first group of buttons and second group of button) wherein the first light indicator located along an edge of the touch surface to be visible to a camera fitted to a lower region of a head-mounted display (HMD) (Fig. 2, first and second camera, #204/206; Detailed Description, [0033]), wherein the first light indicator is configured to emit light to be detected by the camera (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller”); and
a second light indicator located with respect to the main body, wherein the second light indicator is configured to emit light to be detected by the camera (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller…A handheld object may have any suitable number and arrangement of light sources.”), wherein the lights emitted by the first and second light indicators facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD (Fig. 4, HMD, #402; Detailed Description, [0030-0050], “The optical pose of the handheld object with respect to the HMD (“HMD.fwdarw.LED Vision Poses” at 503) and the optical pose of the HMD relative to the world (“World.fwdarw.HMD Poses” at 505) are used to determine a pose of the handheld object with respect to the world”).
Samples does not disclose the first light indicator explicitly as a light bar.
Ohishima, in a similar field of endeavor, discloses a hand-held controller comprising a light bar located with respect to the main body, where the light bar is located along an edge of the touch surface to be visible to a camera, wherein the light bar is configured to emit light to be detected by the camera (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173], “FIG. 2 is a side view of the game apparatus portion of the system of this embodiment. In an MR air hockey game, two players 2000 and 3000 face each other while holding mallets (260L, 260R) with their hands. The two players 2000 and 3000 wear head mount displays (to be abbreviated as HMDs hereinafter) 210L and 210R on their heads. The mallet of this embodiment has an infrared ray generator at its distal end. As will be described later, in this embodiment, the mallet position is detected by image processing. If each mallet has a feature in its shape or color, the mallet position can also be detected by pattern recognition using such feature….Mallet 260 held by each player has an infrared ray generator at its distal end, and each mallet position in a two-dimensional plane on the table 1000 is detected by a CCD camera 230 that detects the infrared rays. Specifically, the camera 230 is provided so that it may detect mallet positions of the players, and the detected positions of the players will be used for advancing or progressing the game, for this embodiment.”; the depicted mallet is a bar shape and emits infrared light).
It would have been obvious to one of ordinary skill in the art to have modified the form of at least one of the light indicator within Samples to be specifically a light bar, as taught in Ohishima, and provide  wherein the light bar is located along an edge of the touch surface to be visible to a camera fitted to a lower region of a head-mounted display (HMD), wherein the first light bar is configured to emit light to be detected by the camera and wherein the lights emitted by the light bar and light indicator facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD. The motivation to combine these arts is to provide the user with a controller with a holdable form factor, based on the particular function of the HMD (See Ohishima, Detailed Description, [0157-0166], in this case an air hockey game). In addition, the changing the light indicator shape to a bar would have been obvious to one of ordinary skill and pursued without undue experimentation as a matter of design choice (MPEP 2144.04 Part IV).  The fact that Ohishima discloses similar systems of HMD and handheld controllers as Samples, including using infrared emitters and cameras, makes this combination more easily implemented since they are within same field.
Regarding claim 2, Samples further discloses wherein the hand-held controller has an edge, wherein the first light indicator is located along the edge of the hand-held controller to be visible to the camera fitted to lower region of the HMD with the main body, wherein the second light indicator is located at a distance from the first light indicator to be visible to the camera (Fig. 3, Detailed Description, [0036-0041], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300”).
Ohishima also teaches wherein the hand-held controller has an edge, wherein the light bar is located along an edge of the hand-held controller to be visible to the camera (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173], “FIG. 2 is a side view of the game apparatus portion of the system of this embodiment. In an MR air hockey game, two players 2000 and 3000 face each other while holding mallets (260L, 260R) with their hands. The two players 2000 and 3000 wear head mount displays (to be abbreviated as HMDs hereinafter) 210L and 210R on their heads. The mallet of this embodiment has an infrared ray generator at its distal end. As will be described later, in this embodiment, the mallet position is detected by image processing. If each mallet has a feature in its shape or color, the mallet position can also be detected by pattern recognition using such feature….Mallet 260 held by each player has an infrared ray generator at its distal end, and each mallet position in a two-dimensional plane on the table 1000 is detected by a CCD camera 230 that detects the infrared rays. Specifically, the camera 230 is provided so that it may detect mallet positions of the players, and the detected positions of the players will be used for advancing or progressing the game, for this embodiment.”)
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1.
Regarding claim 3, Samples further discloses wherein the touch surface is configured to determine movements of fingers of a user, wherein the first light indicator includes a plurality of light sources, wherein the plurality of light sources are located at pre-determined locations with reference to the touch surface (Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”).
Ohishima also teaches wherein the light bar includes a plurality of light sources, wherein the plurality of light sources are located at pre-determined locations with reference to the touch surface (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173], “Mallet 260 held by each player has an infrared ray generator at its distal end, and each mallet position in a two-dimensional plane on the table 1000 is detected by a CCD camera 230 that detects the infrared rays. Specifically, the camera 230 is provided so that it may detect mallet positions of the players, and the detected positions of the players will be used for advancing or progressing the game, for this embodiment.”).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1.
Regarding claim 4, Samples further discloses wherein the first light indicator is an infrared light emitter (Detailed Description, [0035-0036], “Infrared light sources also may be used.”).
Ohishima also teaches wherein the light bar is an infrared light emitter (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173],
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1.
Regarding claim 5, Samples further discloses wherein the first light indicator is configured to face the camera to be in a field-of-view facing downward from the camera (Fig. 2-3, the handheld device #202 is depicted as downward from the HMD which has the camera).
Ohishima also discloses wherein the light bar is configured to face the camera to be in a field-of-view facing downward from the camera (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173], camera, #230).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1.
Regarding claim 6, Samples further discloses a hand-held controller comprising a touch panel situated between the first extension and the second extension, wherein the touch panel has a curvature Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”), wherein the first light indicator is a string of light sources located along an edge of the main body and an edge of the touch surface is situated above the touch panel  (Fig. 3, Detailed Description, [0036-0041], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300”). (Fig. 3, Detailed Description, [0036-0041], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300”; light sources above the handheld object touch surface at bottom).
Ohishima, in a similar field of endeavor, discloses a hand-held controller comprising a light bar (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173])
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1 with the modification of the light indicator as a light bar.
Regarding claim 7, Samples discloses a controller further comprising a battery configured to provide power to the first light indicator and the second light indicator (Detailed Description, [0061]), wherein the second light indicator is a light emitting diode (Detailed Description, [0036]).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1 with the modification of the light indicator as a light bar.
Regarding claim 8, Samples discloses wherein the camera has a co-ordinate system having an x-axis, a y-axis, and a z-axis, wherein the orientation is determined with respect to the x, y, and z axes of the co-ordinate system (Detailed Description, [0052], “The computing device further predicts more possible uv-xyz correspondences, and refines these predictions. The predictions are thresholded on a reprojection error, the error between 2D detection points (u,v's) and the prediction of associated 3D points (x,y,z), assuming a certain pose”).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1 with the modification of the light indicator as a light bar.
Regarding claim 10, Samples discloses wherein the lights are detected by an additional camera located at the bottom surface of the HMD to determine the orientation of the main body, the first extension, and the second extension with reference to the additional camera (Fig. 4, HMD, #402; Detailed Description, [0030-0050], “The optical pose of the handheld object with respect to the HMD (“HMD.fwdarw.LED Vision Poses” at 503) and the optical pose of the HMD relative to the world (“World.fwdarw.HMD Poses” at 505) are used to determine a pose of the handheld object with respect to the world”; See also Summary, [0003] for showing of multiple cameras in the stereo camera arrangement).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1 with the modification of the light indicator as a light bar.
Regarding claim 11, Samples in combination with Ohishima discloses every element of claim 1, and Ohshima further discloses wherein the light bar is located below the touch surface to be visible to the camera (Figs. 1-3, Detailed Description, [0162], “The mallet of this embodiment has an infrared ray generator at its distal end”; distal end would be away from and below where user is touching the mallet).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1 with the modification of the light indicator as a light bar.
Regarding claim 12, Samples discloses a head-mounted display (Fig. 1-4, #102, HMD)  comprising:
a top surface (See Fig. 2, top surface of HMD 200);
a front surface located adjacent to the top surface (Fig. 2, front surface is next to top surface);
a bottom surface located adjacent to the front surface (Fig. 2, #200);
a band coupled to the top surface (Fig. 1-2, back of HMD is a band for user); and
a camera located at the bottom surface, wherein the camera is configured to capture images of lights emitted from a first light indicator and a second light indicator of a hand-held controller to facilitate a determination of orientation of the hand-held controller with reference to the camera (Fig. 2, cameras, #204; second camera, #206; Detailed Description, [0030-0053], “One or more outward-facing cameras on the HMD 102 may acquire image data (e.g. visible light image data) of the surrounding environment and of a handheld object 106 held by the user 102. For example, FIG. 2 shows an example HMD 200 imaging a handheld object 202 using a stereo camera imaging system (indicated by first camera 204 and second camera 206)....The term “handheld object” is used herein for an object comprising trackable light sources 108 that may be held by hand as described herein, and signifies that the object may be handheld, whether or not the object is currently in the state of being held in a hand… For example, because the HMD may have more than one camera, a pose may be solved for each camera on the HMD. Such refinement is performed as postprocessing of a multitude of poses obtained from different cameras. In other examples, any other suitable pose estimation process may be used.”)-, wherein the light indicator extends between a first group of buttons of the hand-held controller and a second group of directional buttons of the hand-held controller (See Figs. 2-3, light sources, #302; light sources, Detailed Description, [0034-0036][0130-0140], since light sources are all around the surface, it includes the areas in between the first group of buttons and second group of buttons),
Samples does not disclose the first light indicator explicitly as a light bar.
Ohishima, in a similar field of endeavor, discloses a head-mounted display comprising a camera configured to capture images of light emitted from a light bar of a hand-held controller to facilitate a determination of orientation of the hand-held controller with reference to the camera (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173], “FIG. 2 is a side view of the game apparatus portion of the system of this embodiment. In an MR air hockey game, two players 2000 and 3000 face each other while holding mallets (260L, 260R) with their hands. The two players 2000 and 3000 wear head mount displays (to be abbreviated as HMDs hereinafter) 210L and 210R on their heads. The mallet of this embodiment has an infrared ray generator at its distal end. As will be described later, in this embodiment, the mallet position is detected by image processing. If each mallet has a feature in its shape or color, the mallet position can also be detected by pattern recognition using such feature….Mallet 260 held by each player has an infrared ray generator at its distal end, and each mallet position in a two-dimensional plane on the table 1000 is detected by a CCD camera 230 that detects the infrared rays. Specifically, the camera 230 is provided so that it may detect mallet positions of the players, and the detected positions of the players will be used for advancing or progressing the game, for this embodiment.”; the depicted mallet is a bar shape and emits infrared light).
It would have been obvious to one of ordinary skill in the art to have modified the form of at least one of the light indicator within Samples to be specifically a light bar, as taught in Ohishima, and provide the camera configured to capture images of light emitted from a light bar of a hand-held controller to facilitate a determination of orientation of the hand-held controller with reference to the camera. The motivation to combine these arts is to provide the user with a controller with a holdable form factor, based on the particular function of the HMD (See Ohishima, Detailed Description, [0157-0166], in this case an air hockey game). In addition, the changing the light indicator shape to a bar would have been obvious to one of ordinary skill and pursued without undue experimentation as a matter of design choice (MPEP 2144.04 Part IV).  The fact that Ohishima discloses similar systems of HMD and handheld controllers as Samples, including using infrared emitters and cameras, makes this combination more easily implemented since they are within same field.
Regarding claim 14, Samples discloses a head-mounted display further including an additional camera located at the bottom surface, wherein the additional camera is configured to capture images of the lights from the first and second light indicators to facilitate a determination of the orientation of the hand-held controller with reference to the additional camera (Fig. 2, cameras, #204; second camera, #206; Detailed Description, [0030-0053], “One or more outward-facing cameras on the HMD 102 may acquire image data (e.g. visible light image data) of the surrounding environment and of a handheld object 106 held by the user 102. For example, FIG. 2 shows an example HMD 200 imaging a handheld object 202 using a stereo camera imaging system (indicated by first camera 204 and second camera 206)....The term “handheld object” is used herein for an object comprising trackable light sources 108 that may be held by hand as described herein, and signifies that the object may be handheld, whether or not the object is currently in the state of being held in a hand… For example, because the HMD may have more than one camera, a pose may be solved for each camera on the HMD. Such refinement is performed as postprocessing of a multitude of poses obtained from different cameras. In other examples, any other suitable pose estimation process may be used.”).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 12 with the modification of the light indicator as a light bar.
Regarding claim 15, Samples discloses a head-mounted display further comprising: an eye mask having the top surface, the front surface, the bottom surface, and a frame, wherein the frame is fitted to the bottom surface, wherein the band is a headband coupled to the frame and a portion of the top surface (Fig. 1-2—generally; Examiner’ note—although eyemask and headband are not terms used, they are depicted as included in the HMD structure).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 12 with the modification of the light indicator as a light bar.
Regarding claim 16, Samples discloses wherein the hand-held controller has a touch surface (Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”), wherein the first light indicator is located below the touch surface and along an edge of the touch surface to be visible to the camera (Fig. 3, Detailed Description, [0036-0041], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300”).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 12 with the modification of the light indicator as a light bar.
Regarding claim 17, Samples disclose a system (Fig. 1-4, generally) comprising:
a head-mounted display (Fig. 1-4, #102, HMD)  including:
a top surface (See Fig. 2, top surface of HMD 200);
a front surface located adjacent to the top surface (Fig. 2, front surface is next to top surface);
a bottom surface located adjacent to the front surface (Fig. 2, #200);
a band coupled to the top surface (Fig. 1-2, back of HMD is a band for user); and
a camera located at the bottom surface (Fig. 2, cameras, #204; second camera, #206; Detailed Description, [0030-0053], “One or more outward-facing cameras on the HMD 102 may acquire image data (e.g. visible light image data) of the surrounding environment and of a handheld object 106 held by the user 102. For example, FIG. 2 shows an example HMD 200 imaging a handheld object 202 using a stereo camera imaging system (indicated by first camera 204 and second camera 206)....The term “handheld object” is used herein for an object comprising trackable light sources 108 that may be held by hand as described herein, and signifies that the object may be handheld, whether or not the object is currently in the state of being held in a hand… For example, because the HMD may have more than one camera, a pose may be solved for each camera on the HMD. Such refinement is performed as postprocessing of a multitude of poses obtained from different cameras. In other examples, any other suitable pose estimation process may be used.”);
a hand-held controller (Fig. 1-4, handheld object, #106; Detailed Description, [0033-0035]) including:
a main body (Figs. 1-4, Detailed Description, [0035]);
a first extension extending from the main body (Fig. 4, 6DoF controller ,left hand, #406; Detailed Description, [0036-0050]);
a second extension extending from the main body (Fig. 4, 6DoF controller, right hand, #408; Detailed Description, [0036-0050]);
a first group of buttons (Detailed Description, [0034-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc.”);
a second group of directional buttons (Detailed Description, [0034-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc.”)
a first light indicator located with respect to the main body, wherein the first light indicator extends between the first group of buttons and the second group of directional buttons (See Figs. 2-3, light sources, #302; light sources, Detailed Description, [0034-0036][0130-0140], since light sources are all around the surface, it includes the areas in between the first group of buttons and second group of button), wherein the first light indicator is configured to emit light to be detected by the camera of the HMD (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller”); and
a second light indicator located with respect to the main body,wherein the second light indicator is configured to emit light to be detected by the camera of the HMD, wherein the lights emitted by the first and second light indicators facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD camera (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller…A handheld object may have any suitable number and arrangement of light sources.”; Fig. 4, HMD, #402; Detailed Description, [0030-0050], “The optical pose of the handheld object with respect to the HMD (“HMD.fwdarw.LED Vision Poses” at 503) and the optical pose of the HMD relative to the world (“World.fwdarw.HMD Poses” at 505) are used to determine a pose of the handheld object with respect to the world”).).
Samples does not disclose the first light indicator explicitly as a light bar.
Ohishima, in a similar field of endeavor, discloses a hand-held controller comprising a light bar located with respect to the main body, wherein the light bar is configured to emit light to be detected by the camera of the HMD  (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173], “FIG. 2 is a side view of the game apparatus portion of the system of this embodiment. In an MR air hockey game, two players 2000 and 3000 face each other while holding mallets (260L, 260R) with their hands. The two players 2000 and 3000 wear head mount displays (to be abbreviated as HMDs hereinafter) 210L and 210R on their heads. The mallet of this embodiment has an infrared ray generator at its distal end. As will be described later, in this embodiment, the mallet position is detected by image processing. If each mallet has a feature in its shape or color, the mallet position can also be detected by pattern recognition using such feature….Mallet 260 held by each player has an infrared ray generator at its distal end, and each mallet position in a two-dimensional plane on the table 1000 is detected by a CCD camera 230 that detects the infrared rays. Specifically, the camera 230 is provided so that it may detect mallet positions of the players, and the detected positions of the players will be used for advancing or progressing the game, for this embodiment.”; the depicted mallet is a bar shape and emits infrared light).
It would have been obvious to one of ordinary skill in the art to have modified the form of at least one of the light indicator within Samples to be specifically a light bar, as taught in Ohishima, and provide a light bar located with respect to the main body, wherein the light bar is configured to emit light to be detected by the camera of the HMD, wherein the first light bar is configured to emit light to be detected by the camera and wherein the lights emitted by the light bar and light indicator facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD. The motivation to combine these arts is to provide the user with a controller with a holdable form factor, based on the particular function of the HMD (See Ohishima, Detailed Description, [0157-0166], in this case an air hockey game). In addition, the changing the light indicator shape to a bar would have been obvious to one of ordinary skill and pursued without undue experimentation as a matter of design choice (MPEP 2144.04 Part IV).  The fact that Ohishima discloses similar systems of HMD and handheld controllers as Samples, including using infrared emitters and cameras, makes this combination more easily implemented since they are within same field.
Regarding claim 18, this is met by the rejection to claim 2 with the combination of Samples and Ohishima.
Regarding claim 19, this is met by the rejection to claim 14 with the combination of Samples and Ohishima.
Regarding claim 20, this is met by the rejection of claim 11 with the combination of Samples and Ohishima.
Regarding claim 23, Samples in combination with Ohishima discloses every element of claim 1, and Ohishima further discloses wherein the light bar has a first edge point and a second edge point, wherein the first and second edge points define the orientation of the main body, the first extension, and the second extension with reference to the HMD (Figs. 1-5, mallet, #260; Detailed Description, [0156-0182]; See also Detailed Description, [0192-210] and Figs. 8-10).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1 with the modification of the first light indicator as a light bar.
Regarding claim 24, Samples in combination with Ohishima discloses every element of claim 23, and Ohishima further discloses wherein the orientation is a first orientation, wherein the first and second edge points have a first set of locations with respect to a reference level passing through the light indictor to define the first orientation and have a second set of locations with respect to the reference level to define a second orientation of the main body, the first extension, and the second extension with reference to the HMD (Ohishima, Figs. 1-5, mallet, #260; Detailed Description, [0156-0182]; See also Detailed Description, [0192-210] and Figs. 8-10), “ As will be described later, in this embodiment, the mallet position is detected by image processing. If each mallet has a feature in its shape or color, the mallet position can also be detected by pattern recognition using such feature.”).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1 with the modification of the first light indicator as a light bar.
Regarding claim 25, Samples in combination with Ohishima discloses every element of claim 1, and Ohishima further discloses wherein the light bar has a plurality of edges, wherein the orientation is determined based on whether the plurality of edges are above or below a horizontal level that passes through the light indicator (Figs. 1-5, mallet, #260; Detailed Description, [0156-0182]; See also Detailed Description, [0192-210] and Figs. 8-10; it is inherent that the edges would be above or below an arbitrary horizontal level).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1 with the modification of the first light indicator as a light bar.
Claim(s) 9 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Samples, in view of Ohishima, further in view of Mallinson, United States Patent Application Publication No. US 2016/0025978 A1.
Regarding claim 9, Samples in combination with Ohishima discloses every element of claim 1 and Samples further discloses a touch panel (Detailed Description, [0035]).
Samples and Ohishima do not disclose the combination of the hand-held controller comprising a haptic feedback module and a touch panel, wherein the main body includes a first joystick and a second joystick that are configured to be moved to generate input data for processing by a computing device, wherein the touch panel is located between the first and second joysticks, wherein the haptic feedback module is configured to provide tactile feedback to a user.
However, Samples does provide the suggestion of the main body including using both a joystick and touch panel together (Detailed Description, [0035]).
Mallinson, in a similar field of endeavor, discloses a controller (Fig. 1) comprising a haptic feedback module and a touch panel (Detailed Description, [0081]), wherein the main body includes a first joystick and a second joystick that are configured to be moved to generate input data for processing by a computing device, (Mallison, Detailed Description, [0091], “A member of Clients 1110 is optionally configured to receive more than one audio or video stream. Input devices of Clients 1110 may include, for example, a one-hand game controller, a two-hand game controller, a gesture recognition system, a gaze recognition system, a voice recognition system, a keyboard, a joystick, a pointing device, a force feedback device, a motion and/or location sensing device, a mouse, a touch screen, a neural interface, a camera, input devices yet to be developed, and/or the like.”) wherein the haptic feedback module is configured to provide tactile feedback to a user (Detailed Description, [0056], “Additionally, part of the processed media content may be transmitted to other devices, such as a game controller, that was used to provide input to the application. For example, video and audio streams of the updates may be provided to the HMD and the haptic feedback may be provided to the game controller.”).
The above references do not explicitly disclose wherein the touch panel is located between the first and second joysticks.
However, it would have been obvious to one of ordinary skill in the art to have modified the combination of Samples and Ohishima to utilize the teachings and suggestion of Mallinson to provide the combination of the hand-held controller comprising a haptic feedback module and a touch panel, wherein the main body includes a first joystick and a second joystick that are configured to be moved to generate input data for processing by a computing device, wherein the touch panel is located between the first and second joysticks, wherein the haptic feedback module is configured to provide tactile feedback to a user. The motivation to combine these arts is to add to the immersion of the user while using a game controller within a AR/VR HMD system which enhances user experience (See Mallison, Summary, [0006-0009]). Furthermore, the modification to the prior art as a whole to provide wherein the touch panel is located between the first and second joysticks is a matter of rearrangement of parts, which would have been pursued by one of ordinary skill without undue experimentation since it would not have modified the overall operation of the HMD/controller system (MPEP 2144.04 Part VI). The fact the Mallison also discloses HMDs in use with game controllers, same as Samples and Ohishima, makes this combination easily implemented.
Regarding claim 13, Samples in combination with Ohishima discloses every element of claim 12 but those above references does not explicitly disclose a HMD including a haptic feedback device configured to vibrate the top, front, and bottom surfaces to provide tactile feedback to a head of a user.
Mallinson, in a similar field of endeavor, discloses a head-mounted device (Fig. 1, HMD, #100) including a haptic feedback device configured to vibrate the top, front, and bottom surfaces to provide tactile feedback to a head of a user (Detailed Description, [0080-0085], “Other output modules include Light-Emitting Diodes (LED) 834, other visual markers or elements (which may also be used for visual tracking of the HMD), vibro-tactile feedback 850, speakers 830, and sound localization module 812, which performs sound localization for sounds to be delivered to speakers or headphones. Other output devices, such as headphones, can also connect to the HMD via the communication modules.”).
It would have been obvious to one of ordinary skill in the art to have modified the HMD of Samples-Ohishima to include the teachings of Mallison to add a haptic feedback device configured to vibrate the top, front, and bottom surfaces to provide tactile feedback to a head of a user. The motivation to combine these arts is to add to the immersion of the user within an AR/VR HMD system while wearing the HMD, which enhances user experience (See Mallison, Summary, [0006-0009][0081-0085]). The fact the Mallison also discloses HMDs in use with game controllers, same as Samples, makes this combination easily implemented.
Allowable Subject Matter
Claim 26 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
The prior art of record does not teach nor suggest the additional structural features of hand-held controller of claim 26 wherein the plurality of edges include a first edge and a second edge, wherein the orientation is a first orientation of the first and second extensions when the plurality of edges are below the horizontal level, wherein the first and second extensions have a second orientation with reference to the HMD when the first edge is above the horizontal level and the second edge is below the horizontal level.
Since the Ohishima reference teaches tracking the position of the mallet in 2-D space (See Detailed Description, [0191-0230], discussing XY coordinates), it does not teach nor suggest orientations when the first edge is above the horizontal level and the second is below the horizontal level, and thereby would teach away from a modification to arrive at the features of claim 26.
Conclusion
THIS ACTION IS MADE FINAL.  Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action.  In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action.  In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.

APPLICANT REBUTTAL: REMARKS
Claims 1-20 are pending. Claims 1-20 and 23-25 are rejected. Claims 1, 8, 10, 12, 14, 17, and 19 are amended. Claim 26 is objected to. Claims 23-26 are canceled. No new matter has been added.
Claim Rejection - 35 USC § 103
Claims 1-8, 10-12, 14-20, and 23-25 are rejected under 35 U.S.C. §103 as being
unpatentable over Samples et al., (US 2018/0330521 Al) ("Samples") in view of Ohishima et al., ( US 2002/0084974 Al) ("Ohishima"). This rejection is respectfully traversed. However, to expedite the prosecution of the present patent application, claim 1 is amended to include the recitations similar to that of allowable claim 26. Also, claim 12 is amended to include the recitations similar to that of allowable claim 26 and claim 17 is amended to include the recitations similar to that of allowable claim 26. Hence, the independent claims 1, 12, and 17 are submitted to be in condition for allowance.
Claims 23-25 are canceled. Claims 2-8 and 10-11 depend from independent claim 1. Also, claims 14-16 depend from independent claim 12 and claims 18-20 depend from independent claim 17. Hence, for the reasons set forth above, the dependent claims 2-8, 10- 11, 14-16, and 18-20 are submitted to be in condition for allowance.
As such, it is respectfully requested that the section 103 rejection of claims 1-8, 10-12, 14-20, and 23-25 be withdrawn.
Claim Rejection - 35 USC § 103
Claims 9 and 13 are rejected under 35 U.S.C. §103 as being unpatentable over Samples, in view of Ohishima, further in view of Mallinson, United States Patent Application No. US 2016/0025978 Al. This rejection is respectfully traversed.
Claim 9 depends from independent claim 1 and claim 13 depends from independent claim 12. The independent claims 1 and 12 are submitted to be in condition for allowance.
Thus, for the reasons set forth above, the dependent claims 9 and 13 are submitted to be in condition for allowance.
As such, it is respectfully requested that the section 103 rejection of claims 9 and 13 be withdrawn.
Allowable Subject Matter
Claim 26 is objected to. Applicant thanks the Examiner for the indication of allowable subject matter. Claim 26 has been canceled.
Conclusion

NEXT OFFICE ACTION: DETAILED ACTION
Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
As a result of the After Final proposed amendment on October 31, 2024, claims 1-20 are pending. Claims 1, 8, 10, 12, 14, 17 and 19 are amended. Claims 21-26 are canceled or were previously canceled.  This amendment is entered as it puts the application in condition for allowance.
Allowable Subject Matter
Claims 1-20 are allowed.
The following is an examiner’s statement of reasons for allowance:
Regarding independent claim 1, 12, or 17, the prior art of record does not disclose a hand-held controller, a head-mounted display, or a system comprising:
wherein the light bar has a plurality of edges, wherein the plurality of orientations are determined based on whether each of the plurality of edges is above or below a horizontal level that passes through the light indicator, wherein the plurality of orientations include a first orientation and a second orientation,
wherein the plurality of edges include a first edge and a second edge, wherein the first orientation of the first and second extensions is achieved when the plurality of edges are below the horizontal level, wherein the second orientation of the first and second extensions is achieved when the first edge is above the horizontal level and the second edge is below the horizontal level.
The prior art of record does not teach nor suggest the additional structural features of apparatus claims 1, 12 17, in addition to the other structural aspects of the claims, comprising wherein the light bar has a plurality of edges, wherein the plurality of orientations are determined based on whether each of the plurality of edges is above or below a horizontal level that passes through the light indicator, wherein the plurality of orientations include a first orientation and a second orientation, wherein the plurality of edges include a first edge and a second edge, wherein the first orientation of the first and second extensions is achieved when the plurality of edges are below the horizontal level, wherein the second orientation of the first and second extensions is achieved when the first edge is above the horizontal level and the second edge is below the horizontal level.
Since the previously cited Ohishima reference teaches tracking the position of the mallet in 2-D space (See Detailed Description, [0191-0230], discussing XY coordinates), it does not teach nor suggest orientations when the first edge is above the horizontal level and the second is below the horizontal level, and thereby would teach away from a modification to arrive at the features of the independent claims. Thus, claims 1, 12 and 17 are free of and unobvious over the prior art and are allowed.
Additional prior art is cited on the PTO-892 as relevant to the Field of invention but do not teach nor suggest the features of claims 1-20. The remaining claims are dependent off of claims 1, 12 or 17 and are allowed as result of their dependencies.
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee.  Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”
Conclusion

CITED REFERENCE SUMMARIES: Ref US20190212808A1: Claims:
A method, comprising: projecting, by a processing system including a processor, first infrared light into an operational space associated with an accessory device; receiving, by the processing system, second infrared light from the operational space associated with the accessory device; generating, by the processing system, a hand image according to a first portion of the second infrared light that is received; determining, by the processing system, hand feature information from the hand image according to device location information associated with the accessory device; mapping, by the processing system, the hand feature information and the device location information to generate hand-device interaction information associated with the operational space; modifying, by the processing system, the hand-device interaction information according to input data associated with the accessory device to generate modified hand-device interaction information; and transmitting, by the processing system, the modified hand-device interaction information to a virtual reality operating system, wherein the modified hand-device interaction information is provided to a virtual reality application by the virtual reality operating system, wherein the virtual reality application superimposes a virtual hand and a virtual accessory device onto a virtual reality image according to the modified hand-device interaction information.
projecting, by a processing system including a processor, first infrared light into an operational space associated with an accessory device;
receiving, by the processing system, second infrared light from the operational space associated with the accessory device;
generating, by the processing system, a hand image according to a first portion of the second infrared light that is received;
determining, by the processing system, hand feature information from the hand image according to device location information associated with the accessory device;
mapping, by the processing system, the hand feature information and the device location information to generate hand-device interaction information associated with the operational space;
modifying, by the processing system, the hand-device interaction information according to input data associated with the accessory device to generate modified hand-device interaction information; and
transmitting, by the processing system, the modified hand-device interaction information to a virtual reality operating system, wherein the modified hand-device interaction information is provided to a virtual reality application by the virtual reality operating system, wherein the virtual reality application superimposes a virtual hand and a virtual accessory device onto a virtual reality image according to the modified hand-device interaction information.
The method of claim 1 , wherein the second infrared light that is received includes a first portion of the first infrared light that is projected into the operational space and is reflected by a hand operating the accessory device.
The method of claim 1 , wherein the second infrared light that is received includes third infrared light that is emitted from a hand operating the accessory device.
The method of claim 1 , wherein the first infrared light is generated by a light source attached to the accessory device, and wherein the second infrared light is received via a camera attached to the accessory device.
The method of claim 1 , further comprising: receiving, by the processing system, a second portion of the second infrared light from the operational space associated with the accessory device; and determining, by the processing system, the device location information associated with the accessory device according to the second portion of the second infrared light that is received.
receiving, by the processing system, a second portion of the second infrared light from the operational space associated with the accessory device; and
determining, by the processing system, the device location information associated with the accessory device according to the second portion of the second infrared light that is received.
The method of claim 5 , wherein the second portion of the second infrared light that is received is reflected or emitted from the accessory device.
The method of claim 5 , wherein the second portion of the second infrared light that is received is associated with an alignment marking of the accessory device, and wherein the device location information is further determined according to a position of the alignment marking that is extracted from the second portion of the second infrared light.
The method of claim 1 , wherein the device location information associated with the accessory device includes longitudinal, lateral, and rotational information associated with the accessory device.
The method of claim 1 , wherein the device location information associated with the accessory device includes depth of field information associated with the accessory device.
The method of claim 1 , wherein the accessory device comprises a keyboard, a keypad, a mouse, a game controller, or any combinations thereof.
The method of claim 1 , further comprising: receiving, by the processing system, the input data associated with the accessory device; and comparing, by the processing system, the input data associated with the accessory device to the hand-device interaction information to generate the modified hand-device interaction information.
receiving, by the processing system, the input data associated with the accessory device; and
comparing, by the processing system, the input data associated with the accessory device to the hand-device interaction information to generate the modified hand-device interaction information.
The method of claim 11 , further comprising transmitting, by the processing system, the input data associated with the accessory device to the virtual reality operation system, wherein the input data is provided to a virtual reality application by the virtual reality operating system.
The method of claim 11 , further comprising: recording, by the processing system, an association between first stimulation data that can be generated by the accessory device and a first substitute accessory stimulation; receiving, by the processing system, stimulation data from the accessory device; detecting, by the processing system, a match between the stimulation data that is received from the accessory device and the association between the first stimulation data and the substitute accessory stimulation; retrieving, by the processing system, the substitute accessory stimulations responsive to detecting the match; and replacing, by the processing system, the stimulation data that is received from the accessory device with the substitute accessory stimulation to generate the input data associated with the accessory device prior to transmission of the input data to the virtual reality operating system.
recording, by the processing system, an association between first stimulation data that can be generated by the accessory device and a first substitute accessory stimulation;
receiving, by the processing system, stimulation data from the accessory device;
detecting, by the processing system, a match between the stimulation data that is received from the accessory device and the association between the first stimulation data and the substitute accessory stimulation;
retrieving, by the processing system, the substitute accessory stimulations responsive to detecting the match; and
replacing, by the processing system, the stimulation data that is received from the accessory device with the substitute accessory stimulation to generate the input data associated with the accessory device prior to transmission of the input data to the virtual reality operating system.
The method of claim 1 , further comprising translating, by the processing system, the modified hand-device interaction information into a format compatible with the virtual reality operating system prior to transmission of the hand-device interaction information to the virtual reality operating system, wherein the modified hand-device interaction information is converted to the format compatible with the virtual reality operating system by a conversion unit attached to the accessory device.
A machine-readable storage medium, comprising instructions, wherein responsive to executing the instructions, a processing system including a processor performs operations comprising: receiving image information from an operational space associated with an accessory device; generating a hand image from a first portion of the image information that is received; determining hand feature information from the hand image according to device location information associated with the accessory device; generating hand-device interaction information associated with the operational space according to the hand feature information and the device location information; and transmitting the hand-device interaction information to a virtual reality operating system, wherein the hand-device interaction information is provided to a virtual reality application by the virtual reality operating system, wherein the virtual reality application superimposes a virtual hand and a virtual accessory device onto a virtual reality image according to the hand-device interaction information.
receiving image information from an operational space associated with an accessory device;
generating a hand image from a first portion of the image information that is received;
determining hand feature information from the hand image according to device location information associated with the accessory device;
generating hand-device interaction information associated with the operational space according to the hand feature information and the device location information; and
transmitting the hand-device interaction information to a virtual reality operating system, wherein the hand-device interaction information is provided to a virtual reality application by the virtual reality operating system, wherein the virtual reality application superimposes a virtual hand and a virtual accessory device onto a virtual reality image according to the hand-device interaction information.
The machine-readable storage medium of claim 15 , wherein the operations further comprise updating the hand-device interaction information according to input data associated with the accessory device to generate modified hand-device interaction information, wherein the modified hand-device interaction information is transmitted to the virtual reality operating system.
The machine-readable storage medium of claim 15 , wherein the operations further comprise: receiving input data associated with the accessory device; detecting that the hand-device interaction information required modification responsive to comparing the input data associated with the accessory device to the hand-device interaction information; and responsive to the detecting, transmitting the input data associated with the accessory device to the virtual reality operation system to update the virtual hand and the virtual accessory device in the virtual reality image according to the input data.
receiving input data associated with the accessory device;
detecting that the hand-device interaction information required modification responsive to comparing the input data associated with the accessory device to the hand-device interaction information; and
responsive to the detecting, transmitting the input data associated with the accessory device to the virtual reality operation system to update the virtual hand and the virtual accessory device in the virtual reality image according to the input data.
The machine-readable storage medium of claim 15 , wherein the operations further comprise: receiving a second portion of the image information from the operational space associated with the accessory device; and determining the device location information associated with the accessory device according to the second portion of the image information that is received.
receiving a second portion of the image information from the operational space associated with the accessory device; and
determining the device location information associated with the accessory device according to the second portion of the image information that is received.
A device, comprising: a memory to store instructions; and a processing system including a processor coupled to the memory, wherein responsive to executing the instructions, the processor performs operations comprising: receiving a signal from an operational space associated with an accessory device; determining hand feature information according to the signal ; generating hand-device interaction information according to the hand feature information and device location information associated with the accessory device; and transmitting the hand-device interaction information to a virtual reality system, wherein the virtual reality system generates, according to the hand-device interaction information, a virtual hand and a virtual accessory device in a virtual reality image.
a memory to store instructions; and
a processing system including a processor coupled to the memory, wherein responsive to executing the instructions, the processor performs operations comprising:
receiving a signal from an operational space associated with an accessory device;
determining hand feature information according to the signal ;
generating hand-device interaction information according to the hand feature information and device location information associated with the accessory device; and
transmitting the hand-device interaction information to a virtual reality system, wherein the virtual reality system generates, according to the hand-device interaction information, a virtual hand and a virtual accessory device in a virtual reality image.
The device of claim 19 , wherein the operations further comprise: receiving input data associated with the accessory device; and transmitting the input data to the virtual reality system to update the virtual hand.
receiving input data associated with the accessory device; and
transmitting the input data to the virtual reality system to update the virtual hand.
Ref US20180330521A1: Claims:
A head-mounted device, comprising: a stereo camera arrangement; a logic device configured to execute instructions; and a storage device comprising instructions executable by the logic device to for each camera in the stereo camera arrangement, receive image data of a field of view of the camera, detect a plurality of light sources of a handheld object in the image data, and based upon the plurality of light sources detected, determine a pose of the handheld object; and based upon the pose of the handheld object determined for each camera in the stereo camera arrangement, calibrate the stereo camera arrangement.
a stereo camera arrangement;
a logic device configured to execute instructions; and
a storage device comprising instructions executable by the logic device to for each camera in the stereo camera arrangement, receive image data of a field of view of the camera, detect a plurality of light sources of a handheld object in the image data, and based upon the plurality of light sources detected, determine a pose of the handheld object; and based upon the pose of the handheld object determined for each camera in the stereo camera arrangement, calibrate the stereo camera arrangement.
for each camera in the stereo camera arrangement, receive image data of a field of view of the camera, detect a plurality of light sources of a handheld object in the image data, and based upon the plurality of light sources detected, determine a pose of the handheld object; and
receive image data of a field of view of the camera,
detect a plurality of light sources of a handheld object in the image data, and
based upon the plurality of light sources detected, determine a pose of the handheld object; and
based upon the pose of the handheld object determined for each camera in the stereo camera arrangement, calibrate the stereo camera arrangement.
The head-mounted device of claim 1 , wherein the instructions are executable to calibrate the stereo camera arrangement during one or more of a motion of the head-mounted device and a motion of the handheld object.
The head-mounted device of claim 1 , wherein the instructions are executable to calibrate intrinsic characteristics for each camera of the stereo camera arrangement, the intrinsic characteristics comprising one or more of lens distortion, focal length, and principal point.
The head-mounted device of claim 1 , wherein the instructions are executable to calibrate extrinsic characteristics for each camera of the stereo camera arrangement, the extrinsic characteristics comprising one or more of a rotation and a translation.
The head-mounted device of claim 1 , wherein the instructions are executable to determine the pose by fitting an observed light pattern to a model light pattern via a rigid body transformation.
The head-mounted device of claim 1 , wherein the instructions are executable to determine the pose by, for each light source detected in the image data received from a first camera, searching along an epipolar line for a companion pixel in the image data received from a second camera, the companion pixel corresponding to the light source detected in the image data received from the first camera.
The head-mounted device of claim 1 , wherein the instructions are executable to calibrate the stereo camera arrangement utilizing one or more of a Kalman filter, a sliding window bundle adjustment, and a recursive Gauss-Newton filter.
The head-mounted device of claim 1 , wherein the instructions are executable to calibrate the stereo camera arrangement between 6 and 10 times per minute.
A method for determining a calibration for a handheld object via a head-mounted device comprising a stereo camera arrangement, the method comprising: receiving, from a first camera of the stereo camera arrangement, first image data from a perspective of the first camera; receiving, from a second camera of the stereo camera arrangement, second image data from a perspective of the second camera; detecting, in the first image data and the second image data, a plurality of light sources of the handheld object; determining a three-dimensional position of each light source of the plurality of light sources detected; and determining a calibration for the handheld object based upon each three-dimensional position determined.
receiving, from a first camera of the stereo camera arrangement, first image data from a perspective of the first camera;
receiving, from a second camera of the stereo camera arrangement, second image data from a perspective of the second camera;
detecting, in the first image data and the second image data, a plurality of light sources of the handheld object;
determining a three-dimensional position of each light source of the plurality of light sources detected; and
determining a calibration for the handheld object based upon each three-dimensional position determined.
The method of claim 9 , further comprising receiving from an inertial measurement unit (IMU) of the handheld object a sensed orientation of the handheld object, and determining a calibration for the IMU based upon the sensed orientation.
receiving from an inertial measurement unit (IMU) of the handheld object a sensed orientation of the handheld object, and
determining a calibration for the IMU based upon the sensed orientation.
The method of claim 10 , wherein determining the calibration for the IMU comprises determining a calibration for a position and a rotation of the IMU.
The method of claim 9 , wherein determining the calibration of the handheld object comprises
[Truncated to input limit]