Office Action 3: Non-Final Rejection (2024-04-18)
Rejection types: 103 - Obviousness

CLAIMS:
IN THE CLAIMS

Please amend the claims are shown below to include underlined sections and delete strikeout or double bracketed ("\[\[ \]\]") sections.

Listing of Claims:

1\. (Currently amended) A hand-held controller comprising:a main body, wherein the main body has a touch surface;
a first extension extending from the main body;
a second extension extending from the main body;
a light bar located with respect to the main body, wherein the light bar is located along an edge of the touch surface to be visible to a camera fitted to a lower region of a head-mounted display (HMD), wherein the light bar is configured to emit light to be detected by the camera; and
a light indicator located with respect to the main body, wherein the light indicator is configured to emit light to be detected by the camera, wherein the lights emitted by the light bar and the light indicator facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD.
2\. (Currently amended) The hand-held controller of claim 1, wherein the hand-held controller has an edge, wherein the light bar is located along the edge of the hand-held controller to be visible to the camera \[\[of\]\] fitted to the lower region of the HMD , wherein the light indicator is located at a distance from the first light indicator to be visible to the camera.
3\. (Currently amended) The hand-held controller of claim 1, wherein the touch surface is configured to determine movements of fingers of a user, wherein the light bar includes a plurality of light sources, wherein the plurality of light sources are located at pre-determined locations with reference to the touch surface.
4\. (Currently amended) The hand-held controller of claim 1, wherein the light bar is an infrared light emitter.

5\. (Currently amended) The hand-held controller of claim 1, wherein the main body has a plurality of buttons, wherein the light bar is configured to face the camera to be in a field-of-view facing downward from the camera.
6\. (Currently amended) The hand-held controller of claim 1, further comprising a touch panel situated between the first extension and the second extension,wherein the touch panel has a curvature, wherein the light bar is a string of light sources located along an edge of the main body and is situated above the touch panel.
7\. (Currently amended) The hand-held controller of claim 1, further comprising a battery configured to provide power to the light bar and the light indicator, wherein the light indicator is a light emitting diode.
8\. (Original) The hand-held controller of claim 1, wherein the camera has a co-ordinate system having an x-axis, a y-axis, and a z-axis, wherein the orientation is determined with respect to the x, y, and z axes of the co-ordinate system.
9\. (Currently amended) The hand-held controller of claim 1, further comprising a haptic feedback module and a touch panel, wherein the main body includes a first joystick and a second joystick that \[\[is\]\] are configured to be moved to generate input data for processing by a computing device, wherein the touch panel is located between the first and second joysticks, wherein the haptic feedback module is configured to provide tactile feedback to a user.

10\. (Original) The hand-held controller of claim 1, wherein the lights are detected by an additional camera located at the bottom surface of the HMD to determine the orientation of the main body, the first extension, and the second extension with reference to the additional camera.

11\. (Currently amended) The hand-held controller of claim 1, wherein the light bar is located below the touch surface to be visible to the camera.

12\. (Currently amended) A head-mounted display comprising:a top surface;
a front surface located adjacent to the top surface;
a bottom surface located adjacent to the front surface;
a band coupled to the top surface; and
a camera located at the bottom surface, wherein the camera is configured to capture images of lights emitted from a light bar and a light indicator of a hand- held controller to facilitate a determination of an orientation of the hand-held controller with reference to the camera.

13\. (Original) The head-mounted display of claim 12, further including a haptic feedback device configured to vibrate the top, front, and bottom surfaces to provide tactile feedback to a head of a user.

14\. (Currently amended) The head-mounted display of claim 12, further including an additional camera located at the bottom surface, wherein the additional camera is configured to capture images of the lights from the light bar and the light indicator to facilitate a determination of the orientation of the hand-held controller with reference to the additional camera.

15\. (Original) The head-mounted display of claim 12, further comprising:an eye mask having the top surface, the front surface, the bottom surface, and a frame, wherein the frame is fitted to the bottom surface, wherein the band is a headband coupled to the frame and a portion of the top surface.

16\. (Currently amended) The head-mounted display of claim 12, wherein the hand-held controller has a touch surface, wherein the light bar is located below the touch surface and along an edge of the touch surface to be visible to the camera.

17\. (Currently amended) A system comprising:a head-mounted display (HMD) including:a top surface;
a front surface located adjacent to the top surface;
a bottom surface located adjacent to the front surface;

a band coupled to the top surface; and a camera located at the bottom surface;
a hand-held controller including:
a main body;
a first extension extending from the main body;
a second extension extending from the main body;
a light bar located with respect to the main body, wherein the light bar is configured to emit light to be detected by the camera of the HMD; and
a light indicator located with respect to the main body, wherein the light indicator is configured to emit light to be detected by the camera of the HMD, wherein the lights emitted by the light bar and the light indicator facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD.

18\. (Currently amended) The system of claim 17, wherein the hand-held controller has an edge, wherein the light bar is located along the edge to be visible to the camera of the lower region of the HMD with the main body, wherein the light indicator is located at a distance from the light bar to be visible to the camera.

19\. (Currently amended) The system of claim 17, wherein the HMD includes an additional camera located at the bottom surface, wherein the additional camera is configured to capture images of the lights from the light bar and the light indicator to facilitate a determination of the orientation of the hand-held controller with reference to the additional camera.

20\. (Currently amended) The system of claim 17, wherein the main body has a touch surface, wherein the light bar is located below the touch surface and along an edge of the touch surface to be visible to the camera of the HMD.

21\. (Canceled)

22\. (Canceled)

23\. (New) The hand-held controller of claim 1, wherein the light bar has a first edge point and a second edge point, wherein the first and second edge points define the orientation of the main body, the first extension, and the second extension with reference to the HMD.

24\. (New) The hand-held controller of claim 23, wherein the orientation is a first orientation, wherein the first and second edge points have a first set of locations with respect to a reference level passing through the light indictor to define the first orientation and have a second set of locations with respect to the reference level to define a second orientation of the main body, the first extension, and the second extension with reference to the HMD.

OFFICE ACTION:
DETAILED ACTION
Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection.  Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114.  Applicant's submission filed on February 26, 2024 has been entered.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on February 28, 2024 was filed after the mailing date of the Final Rejection on December 8, 2023 and before first action on RCE.  The submission is in compliance with the provisions of 37 CFR 1.97.  Accordingly, the information disclosure statement is being considered by the examiner.
Response to Amendment
As a result of the Amendment filed on RCE, claims 1-20 and 23-24 are pending.  Claims 1-7, 9, 11, 12,14 and 16-20 are amended. New claims 23-24 are added. Claims 21-22 are canceled.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-20 and 23-24 have been considered but are moot because the new ground of rejection does not rely on the same combination of references applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-8, 10-12, 14-20 and 23-24 are rejected under 35 U.S.C. 103 as being unpatentable over Samples et al., United States Patent Application Publication No. US 2018/0330521 A1 in view of Ohishima et al., United States Patent Application Publication No. US 2002/0084974 A1.
Regarding claim 1, Samples discloses a hand-held controller (Fig. 1-4, handheld object, #106; Detailed Description, [0033-0035]) comprising:
a main body, wherein the main body has a touch surface (Figs. 1-4, Detailed Description, [0035], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc.”);
a first extension extending from the main body (Fig. 4, 6DoF controller, left hand, #406; Detailed Description, [0036-0050]);
a second extension extending from the main body (Fig. 4, 6DoF controller, right hand, #408; Detailed Description, [0036-0050]);
a first light indicator located with respect to the main body, wherein the first light indicator located along an edge of the touch surface to be visible to a camera fitted to a lower region of a head-mounted display (HMD) (Fig. 2, first and second camera, #204/206; Detailed Description, [0033]), wherein the first light indicator is configured to emit light to be detected by the camera (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller”); and
a second light indicator located with respect to the main body, wherein the second light indicator is configured to emit light to be detected by the camera (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller…A handheld object may have any suitable number and arrangement of light sources.”), wherein the lights emitted by the first and second light indicators facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD (Fig. 4, HMD, #402; Detailed Description, [0030-0050], “The optical pose of the handheld object with respect to the HMD (“HMD.fwdarw.LED Vision Poses” at 503) and the optical pose of the HMD relative to the world (“World.fwdarw.HMD Poses” at 505) are used to determine a pose of the handheld object with respect to the world”).
Samples does not disclose the first light indicator explicitly as a light bar.
Ohishima, in a similar field of endeavor, discloses a hand-held controller comprising a light bar located with respect to the main body, where the light bar is located along an edge of the touch surface to be visible to a camera, wherein the light bar is configured to emit light to be detected by the camera (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173], “FIG. 2 is a side view of the game apparatus portion of the system of this embodiment. In an MR air hockey game, two players 2000 and 3000 face each other while holding mallets (260L, 260R) with their hands. The two players 2000 and 3000 wear head mount displays (to be abbreviated as HMDs hereinafter) 210L and 210R on their heads. The mallet of this embodiment has an infrared ray generator at its distal end. As will be described later, in this embodiment, the mallet position is detected by image processing. If each mallet has a feature in its shape or color, the mallet position can also be detected by pattern recognition using such feature….Mallet 260 held by each player has an infrared ray generator at its distal end, and each mallet position in a two-dimensional plane on the table 1000 is detected by a CCD camera 230 that detects the infrared rays. Specifically, the camera 230 is provided so that it may detect mallet positions of the players, and the detected positions of the players will be used for advancing or progressing the game, for this embodiment.”; the depicted mallet is a bar shape and emits infrared light).
It would have been obvious to one of ordinary skill in the art to have modified the form of at least one of the light indicator within Samples to be specifically a light bar, as taught in Ohishima, and provide  wherein the light bar is located along an edge of the touch surface to be visible to a camera fitted to a lower region of a head-mounted display (HMD), wherein the first light bar is configured to emit light to be detected by the camera and wherein the lights emitted by the light bar and light indicator facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD. The motivation to combine these arts is to provide the user with a controller with a holdable form factor, based on the particular function of the HMD (See Ohishima, Detailed Description, [0157-0166], in this case an air hockey game). In addition, the changing the light indicator shape to a bar would have been obvious to one of ordinary skill and pursued without undue experimentation as a matter of design choice (MPEP 2144.04 Part IV).  The fact that Ohishima discloses similar systems of HMD and handheld controllers as Samples, including using infrared emitters and cameras, makes this combination more easily implemented since they are within same field.
Regarding claim 2, Samples further discloses wherein the hand-held controller has an edge, wherein the first light indicator is located along the edge of the hand-held controller to be visible to the camera fitted to lower region of the HMD with the main body, wherein the second light indicator is located at a distance from the first light indicator to be visible to the camera (Fig. 3, Detailed Description, [0036-0041], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300”).
Ohishima also teaches wherein the hand-held controller has an edge, wherein the light bar is located along an edge of the hand-held controller to be visible to the camera (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173], “FIG. 2 is a side view of the game apparatus portion of the system of this embodiment. In an MR air hockey game, two players 2000 and 3000 face each other while holding mallets (260L, 260R) with their hands. The two players 2000 and 3000 wear head mount displays (to be abbreviated as HMDs hereinafter) 210L and 210R on their heads. The mallet of this embodiment has an infrared ray generator at its distal end. As will be described later, in this embodiment, the mallet position is detected by image processing. If each mallet has a feature in its shape or color, the mallet position can also be detected by pattern recognition using such feature….Mallet 260 held by each player has an infrared ray generator at its distal end, and each mallet position in a two-dimensional plane on the table 1000 is detected by a CCD camera 230 that detects the infrared rays. Specifically, the camera 230 is provided so that it may detect mallet positions of the players, and the detected positions of the players will be used for advancing or progressing the game, for this embodiment.”)
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1.
Regarding claim 3, Samples further discloses wherein the touch surface is configured to determine movements of fingers of a user, wherein the first light indicator includes a plurality of light sources, wherein the plurality of light sources are located at pre-determined locations with reference to the touch surface (Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”).
Ohishima also teaches wherein the light bar includes a plurality of light sources, wherein the plurality of light sources are located at pre-determined locations with reference to the touch surface (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173], “Mallet 260 held by each player has an infrared ray generator at its distal end, and each mallet position in a two-dimensional plane on the table 1000 is detected by a CCD camera 230 that detects the infrared rays. Specifically, the camera 230 is provided so that it may detect mallet positions of the players, and the detected positions of the players will be used for advancing or progressing the game, for this embodiment.”).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1.
Regarding claim 4, Samples further discloses wherein the first light indicator is an infrared light emitter (Detailed Description, [0035-0036], “Infrared light sources also may be used.”).
Ohishima also teaches wherein the light bar is an infrared light emitter (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173],
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1.
Regarding claim 5, Samples further discloses wherein the main body has a plurality of buttons (Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”), wherein the first light indicator is configured to face the camera to be in a field-of-view facing downward from the camera (Fig. 2-3, the handheld device #202 is depicted as downward from the HMD which has the camera).
Ohishima also discloses wherein the light bar is configured to face the camera to be in a field-of-view facing downward from the camera (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173], camera, #230).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1.
Regarding claim 6, Samples further discloses a hand-held controller comprising a touch panel situated between the first extension and the second extension, wherein the touch panel has a curvature Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”), wherein the first light indicator is a string of light sources located along an edge of the main body and an edge of the touch surface is situated above the touch panel  (Fig. 3, Detailed Description, [0036-0041], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300”). (Fig. 3, Detailed Description, [0036-0041], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300”; light sources above the handheld object touch surface at bottom).
Ohishima, in a similar field of endeavor, discloses a hand-held controller comprising a light bar (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173])
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1 with the modification of the light indicator as a light bar.
Regarding claim 7, Samples discloses a controller further comprising a battery configured to provide power to the first light indicator and the second light indicator (Detailed Description, [0061]), wherein the second light indicator is a light emitting diode (Detailed Description, [0036]).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1 with the modification of the light indicator as a light bar.
Regarding claim 8, Samples discloses wherein the camera has a co-ordinate system having an x-axis, a y-axis, and a z-axis, wherein the orientation is determined with respect to the x, y, and z axes of the co-ordinate system (Detailed Description, [0052], “The computing device further predicts more possible uv-xyz correspondences, and refines these predictions. The predictions are thresholded on a reprojection error, the error between 2D detection points (u,v's) and the prediction of associated 3D points (x,y,z), assuming a certain pose”).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1 with the modification of the light indicator as a light bar.
Regarding claim 10, Samples discloses wherein the lights are detected by an additional camera located at the bottom surface of the HMD to determine the orientation of the main body, the first extension, and the second extension with reference to the additional camera (Fig. 4, HMD, #402; Detailed Description, [0030-0050], “The optical pose of the handheld object with respect to the HMD (“HMD.fwdarw.LED Vision Poses” at 503) and the optical pose of the HMD relative to the world (“World.fwdarw.HMD Poses” at 505) are used to determine a pose of the handheld object with respect to the world”; See also Summary, [0003] for showing of multiple cameras in the stereo camera arrangement).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1 with the modification of the light indicator as a light bar.
Regarding claim 11, Samples in combination with Ohishima discloses every element of claim 1, and Ohshima further discloses wherein the light bar is located below the touch surface to be visible to the camera (Figs. 1-3, Detailed Description, [0162], “The mallet of this embodiment has an infrared ray generator at its distal end”; distal end would be away from and below where user is touching the mallet).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1 with the modification of the light indicator as a light bar.
Regarding claim 12, Samples discloses a head-mounted display (Fig. 1-4, #102, HMD)  comprising:
a top surface (See Fig. 2, top surface of HMD 200);
a front surface located adjacent to the top surface (Fig. 2, front surface is next to top surface);
a bottom surface located adjacent to the front surface (Fig. 2, #200);
a band coupled to the top surface (Fig. 1-2, back of HMD is a band for user); and
a camera located at the bottom surface, wherein the camera is configured to capture images of lights emitted from a first light indicator and a second light indicator of a hand-held controller to facilitate a determination of orientation of the hand-held controller with reference to the camera (Fig. 2, cameras, #204; second camera, #206; Detailed Description, [0030-0053], “One or more outward-facing cameras on the HMD 102 may acquire image data (e.g. visible light image data) of the surrounding environment and of a handheld object 106 held by the user 102. For example, FIG. 2 shows an example HMD 200 imaging a handheld object 202 using a stereo camera imaging system (indicated by first camera 204 and second camera 206)....The term “handheld object” is used herein for an object comprising trackable light sources 108 that may be held by hand as described herein, and signifies that the object may be handheld, whether or not the object is currently in the state of being held in a hand… For example, because the HMD may have more than one camera, a pose may be solved for each camera on the HMD. Such refinement is performed as postprocessing of a multitude of poses obtained from different cameras. In other examples, any other suitable pose estimation process may be used.”).
Samples does not disclose the first light indicator explicitly as a light bar.
Ohishima, in a similar field of endeavor, discloses a head-mounted display comprising a camera configured to capture images of light emitted from a light bar of a hand-held controller to facilitate a determination of orientation of the hand-held controller with reference to the camera (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173], “FIG. 2 is a side view of the game apparatus portion of the system of this embodiment. In an MR air hockey game, two players 2000 and 3000 face each other while holding mallets (260L, 260R) with their hands. The two players 2000 and 3000 wear head mount displays (to be abbreviated as HMDs hereinafter) 210L and 210R on their heads. The mallet of this embodiment has an infrared ray generator at its distal end. As will be described later, in this embodiment, the mallet position is detected by image processing. If each mallet has a feature in its shape or color, the mallet position can also be detected by pattern recognition using such feature….Mallet 260 held by each player has an infrared ray generator at its distal end, and each mallet position in a two-dimensional plane on the table 1000 is detected by a CCD camera 230 that detects the infrared rays. Specifically, the camera 230 is provided so that it may detect mallet positions of the players, and the detected positions of the players will be used for advancing or progressing the game, for this embodiment.”; the depicted mallet is a bar shape and emits infrared light).
It would have been obvious to one of ordinary skill in the art to have modified the form of at least one of the light indicator within Samples to be specifically a light bar, as taught in Ohishima, and provide the camera configured to capture images of light emitted from a light bar of a hand-held controller to facilitate a determination of orientation of the hand-held controller with reference to the camera. The motivation to combine these arts is to provide the user with a controller with a holdable form factor, based on the particular function of the HMD (See Ohishima, Detailed Description, [0157-0166], in this case an air hockey game). In addition, the changing the light indicator shape to a bar would have been obvious to one of ordinary skill and pursued without undue experimentation as a matter of design choice (MPEP 2144.04 Part IV).  The fact that Ohishima discloses similar systems of HMD and handheld controllers as Samples, including using infrared emitters and cameras, makes this combination more easily implemented since they are within same field.
Regarding claim 14, Samples discloses a head-mounted display further including an additional camera located at the bottom surface, wherein the additional camera is configured to capture images of the lights from the first and second light indicators to facilitate a determination of the orientation of the hand-held controller with reference to the additional camera (Fig. 2, cameras, #204; second camera, #206; Detailed Description, [0030-0053], “One or more outward-facing cameras on the HMD 102 may acquire image data (e.g. visible light image data) of the surrounding environment and of a handheld object 106 held by the user 102. For example, FIG. 2 shows an example HMD 200 imaging a handheld object 202 using a stereo camera imaging system (indicated by first camera 204 and second camera 206)....The term “handheld object” is used herein for an object comprising trackable light sources 108 that may be held by hand as described herein, and signifies that the object may be handheld, whether or not the object is currently in the state of being held in a hand… For example, because the HMD may have more than one camera, a pose may be solved for each camera on the HMD. Such refinement is performed as postprocessing of a multitude of poses obtained from different cameras. In other examples, any other suitable pose estimation process may be used.”).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 12 with the modification of the light indicator as a light bar.
Regarding claim 15, Samples discloses a head-mounted display further comprising: an eye mask having the top surface, the front surface, the bottom surface, and a frame, wherein the frame is fitted to the bottom surface, wherein the band is a headband coupled to the frame and a portion of the top surface (Fig. 1-2—generally; Examiner’ note—although eyemask and headband are not terms used, they are depicted as included in the HMD structure).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 12 with the modification of the light indicator as a light bar.
Regarding claim 16, Samples discloses wherein the hand-held controller has a touch surface (Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”), wherein the first light indicator is located below the touch surface and along an edge of the touch surface to be visible to the camera (Fig. 3, Detailed Description, [0036-0041], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300”).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 12 with the modification of the light indicator as a light bar.
Regarding claim 17, Samples disclose a system (Fig. 1-4, generally) comprising:
a head-mounted display (Fig. 1-4, #102, HMD)  including:
a top surface (See Fig. 2, top surface of HMD 200);
a front surface located adjacent to the top surface (Fig. 2, front surface is next to top surface);
a bottom surface located adjacent to the front surface (Fig. 2, #200);
a band coupled to the top surface (Fig. 1-2, back of HMD is a band for user); and
a camera located at the bottom surface (Fig. 2, cameras, #204; second camera, #206; Detailed Description, [0030-0053], “One or more outward-facing cameras on the HMD 102 may acquire image data (e.g. visible light image data) of the surrounding environment and of a handheld object 106 held by the user 102. For example, FIG. 2 shows an example HMD 200 imaging a handheld object 202 using a stereo camera imaging system (indicated by first camera 204 and second camera 206)....The term “handheld object” is used herein for an object comprising trackable light sources 108 that may be held by hand as described herein, and signifies that the object may be handheld, whether or not the object is currently in the state of being held in a hand… For example, because the HMD may have more than one camera, a pose may be solved for each camera on the HMD. Such refinement is performed as postprocessing of a multitude of poses obtained from different cameras. In other examples, any other suitable pose estimation process may be used.”);
a hand-held controller (Fig. 1-4, handheld object, #106; Detailed Description, [0033-0035]) including:
a main body (Figs. 1-4, Detailed Description, [0035]);
a first extension extending from the main body (Fig. 4, 6DoF controller ,left hand, #406; Detailed Description, [0036-0050]);
a second extension extending from the main body (Fig. 4, 6DoF controller, right hand, #408; Detailed Description, [0036-0050]);
a first light indicator located with respect to the main body, wherein the first light indicator is configured to emit light to be detected by the camera of the HMD (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller”); and
a second light indicator located with respect to the main body, wherein the second light indicator is configured to emit light to be detected by the camera of the HMD, wherein the lights emitted by the first and second light indicators facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD camera (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller…A handheld object may have any suitable number and arrangement of light sources.”; Fig. 4, HMD, #402; Detailed Description, [0030-0050], “The optical pose of the handheld object with respect to the HMD (“HMD.fwdarw.LED Vision Poses” at 503) and the optical pose of the HMD relative to the world (“World.fwdarw.HMD Poses” at 505) are used to determine a pose of the handheld object with respect to the world”).).
Samples does not disclose the first light indicator explicitly as a light bar.
Ohishima, in a similar field of endeavor, discloses a hand-held controller comprising a light bar located with respect to the main body, wherein the light bar is configured to emit light to be detected by the camera of the HMD  (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173], “FIG. 2 is a side view of the game apparatus portion of the system of this embodiment. In an MR air hockey game, two players 2000 and 3000 face each other while holding mallets (260L, 260R) with their hands. The two players 2000 and 3000 wear head mount displays (to be abbreviated as HMDs hereinafter) 210L and 210R on their heads. The mallet of this embodiment has an infrared ray generator at its distal end. As will be described later, in this embodiment, the mallet position is detected by image processing. If each mallet has a feature in its shape or color, the mallet position can also be detected by pattern recognition using such feature….Mallet 260 held by each player has an infrared ray generator at its distal end, and each mallet position in a two-dimensional plane on the table 1000 is detected by a CCD camera 230 that detects the infrared rays. Specifically, the camera 230 is provided so that it may detect mallet positions of the players, and the detected positions of the players will be used for advancing or progressing the game, for this embodiment.”; the depicted mallet is a bar shape and emits infrared light).
It would have been obvious to one of ordinary skill in the art to have modified the form of at least one of the light indicator within Samples to be specifically a light bar, as taught in Ohishima, and provide a light bar located with respect to the main body, wherein the light bar is configured to emit light to be detected by the camera of the HMD, wherein the first light bar is configured to emit light to be detected by the camera and wherein the lights emitted by the light bar and light indicator facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD. The motivation to combine these arts is to provide the user with a controller with a holdable form factor, based on the particular function of the HMD (See Ohishima, Detailed Description, [0157-0166], in this case an air hockey game). In addition, the changing the light indicator shape to a bar would have been obvious to one of ordinary skill and pursued without undue experimentation as a matter of design choice (MPEP 2144.04 Part IV).  The fact that Ohishima discloses similar systems of HMD and handheld controllers as Samples, including using infrared emitters and cameras, makes this combination more easily implemented since they are within same field.
Regarding claim 18, this is met by the rejection to claim 2 with the combination of Samples and Ohishima.
Regarding claim 19, this is met by the rejection to claim 14 with the combination of Samples and Ohishima.
Regarding claim 20, this is met by the rejection of claim 11 with the combination of Samples and Ohishima.
Regarding claim 23, Samples in combination with Ohishima discloses every element of claim 1, and Ohishima further discloses wherein the light bar has a first edge point and a second edge point, wherein the first and second edge points define the orientation of the main body, the first extension, and the second extension with reference to the HMD (Figs. 1-5, mallet, #260; Detailed Description, [0156-0182]; See also Detailed Description, [0192-210] and Figs. 8-10).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1 with the modification of the first light indicator as a light bar.
Regarding claim 24, Samples in combination with Ohishima discloses every element of claim 23, and Ohishima further discloses wherein the orientation is a first orientation, wherein the first and second edge points have a first set of locations with respect to a reference level passing through the light indictor to define the first orientation and have a second set of locations with respect to the reference level to define a second orientation of the main body, the first extension, and the second extension with reference to the HMD (Ohishima, Figs. 1-5, mallet, #260; Detailed Description, [0156-0182]; See also Detailed Description, [0192-210] and Figs. 8-10), “ As will be described later, in this embodiment, the mallet position is detected by image processing. If each mallet has a feature in its shape or color, the mallet position can also be detected by pattern recognition using such feature.”).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1 with the modification of the first light indicator as a light bar.
Claim(s) 9 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Samples, in view of Ohishima, further in view of Mallinson, United States Patent Application Publication No. US 2016/0025978 A1.
Regarding claim 9, Samples in combination with Ohishima discloses every element of claim 1 and Samples further discloses a touch panel (Detailed Description, [0035]).
Samples and Ohishima do not disclose the combination of the hand-held controller comprising a haptic feedback module and a touch panel, wherein the main body includes a first joystick and a second joystick that are configured to be moved to generate input data for processing by a computing device, wherein the touch panel is located between the first and second joysticks, wherein the haptic feedback module is configured to provide tactile feedback to a user.
However, Samples does provide the suggestion of the main body including using both a joystick and touch panel together (Detailed Description, [0035]).
Mallinson, in a similar field of endeavor, discloses a controller (Fig. 1) comprising a haptic feedback module and a touch panel (Detailed Description, [0081]), wherein the main body includes a first joystick and a second joystick that are configured to be moved to generate input data for processing by a computing device, (Mallison, Detailed Description, [0091], “A member of Clients 1110 is optionally configured to receive more than one audio or video stream. Input devices of Clients 1110 may include, for example, a one-hand game controller, a two-hand game controller, a gesture recognition system, a gaze recognition system, a voice recognition system, a keyboard, a joystick, a pointing device, a force feedback device, a motion and/or location sensing device, a mouse, a touch screen, a neural interface, a camera, input devices yet to be developed, and/or the like.”) wherein the haptic feedback module is configured to provide tactile feedback to a user (Detailed Description, [0056], “Additionally, part of the processed media content may be transmitted to other devices, such as a game controller, that was used to provide input to the application. For example, video and audio streams of the updates may be provided to the HMD and the haptic feedback may be provided to the game controller.”).
The above references do not explicitly disclose wherein the touch panel is located between the first and second joysticks.
However, it would have been obvious to one of ordinary skill in the art to have modified the combination of Samples and Ohishima to utilize the teachings and suggestion of Mallinson to provide the combination of the hand-held controller comprising a haptic feedback module and a touch panel, wherein the main body includes a first joystick and a second joystick that are configured to be moved to generate input data for processing by a computing device, wherein the touch panel is located between the first and second joysticks, wherein the haptic feedback module is configured to provide tactile feedback to a user. The motivation to combine these arts is to add to the immersion of the user while using a game controller within a AR/VR HMD system which enhances user experience (See Mallison, Summary, [0006-0009]). Furthermore, the modification to the prior art as a whole to provide wherein the touch panel is located between the first and second joysticks is a matter of rearrangement of parts, which would have been pursued by one of ordinary skill without undue experimentation since it would not have modified the overall operation of the HMD/controller system (MPEP 2144.04 Part VI). The fact the Mallison also discloses HMDs in use with game controllers, same as Samples and Ohishima, makes this combination easily implemented.
Regarding claim 13, Samples in combination with Ohishima discloses every element of claim 12 but those above references does not explicitly disclose a HMD including a haptic feedback device configured to vibrate the top, front, and bottom surfaces to provide tactile feedback to a head of a user.
Mallinson, in a similar field of endeavor, discloses a head-mounted device (Fig. 1, HMD, #100) including a haptic feedback device configured to vibrate the top, front, and bottom surfaces to provide tactile feedback to a head of a user (Detailed Description, [0080-0085], “Other output modules include Light-Emitting Diodes (LED) 834, other visual markers or elements (which may also be used for visual tracking of the HMD), vibro-tactile feedback 850, speakers 830, and sound localization module 812, which performs sound localization for sounds to be delivered to speakers or headphones. Other output devices, such as headphones, can also connect to the HMD via the communication modules.”).
It would have been obvious to one of ordinary skill in the art to have modified the HMD of Samples-Ohishima to include the teachings of Mallison to add a haptic feedback device configured to vibrate the top, front, and bottom surfaces to provide tactile feedback to a head of a user. The motivation to combine these arts is to add to the immersion of the user within an AR/VR HMD system while wearing the HMD, which enhances user experience (See Mallison, Summary, [0006-0009][0081-0085]). The fact the Mallison also discloses HMDs in use with game controllers, same as Samples, makes this combination easily implemented.
Conclusion

APPLICANT REBUTTAL: REMARKS
Claims 1-20 and 23-26 are pending. Claims 1-20 and 23-24 are rejected. Claims 1, 2, 5, 12, and 17 are amended. Claims 25 and 26 are newly added. No new matter has been added.
Claim Rejection - 35 USC § 103
Claims 1-8, 10-12, 14-20, and 23-24 are rejected under 35 U.S.C. §103 as being
unpatentable over Samples et al., (US 2018/0330521 Al) ("Samples") in view of Ohishima et al., ( US 2002/0084974 Al) ("Ohishima"). This rejection is respectfully traversed.
It is respectfully submitted that the combination of Samples and Ohishima does not disclose the recitations of claim 1. For example, the combination fails to teach that "the light bar extends between the first group of buttons and the second group of directional buttons", as is recited in claim 1. The Examiner agrees that Samples does not "disclose the first light indicator explicitly as
a light bar"
(see Office Action, page 5)
(Emphasis in Original).
Ohishima describes a mallet having an infrared ray generator. Figure 2 of Ohishima is reproduced below.
It is described in paragraph 162 of Ohishima that:
FIG. 2 is a side view of the game apparatus portion of the system of this embodiment. In an MR air hockey game, two
players 2000 and 3000 face each other while holding mallets Attorney Docket No. SONYP360.C1
(260L, 260R) with their hands. The two players 2000 and 3000
wear head mount displays (to be abbreviated as HMDs
hereinafter) 210L and 210R on their heads. The mallet of this embodiment has an infrared ray generator at its distal end. As
will be described later, in this embodiment, the mallet position is detected by image processing. If each mallet has a feature in
its shape or color, the mallet position can also be detected by
pattern recognition using such feature.
There is no description or suggestion in Ohishima that the infrared ray generator of the mallet is a "light bar" that "extends between the first group of buttons and the second group of directional buttons", as is recited in claim 1. As such, the combination of Samples and Ohishima fails to teach that "the light bar extends between the first group of buttons and the second group of directional buttons", as is recited in claim 1. By extending between the "first group of buttons and the second group of directional buttons", the light bar facilitates determination "of an orientation of the main body, the first extension, and the second extension with reference to the HMD", as is recited in claim 1.
Moreover, the combination of Samples and Ohishima does not disclose or suggest
that "the lights emitted by the light bar and the light indicator facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD", as is recited in claim 1. As explained above, the Examiner agrees that Samples does not disclose or suggest the "light bar" of claim 1. Also, Ohishima describes that a position of the mallet is determined. The position of the mallet is not an orientation. Figure 8 of Ohishima is reproduced below.

eor  eaost  or  a o, c am   s su m e o t to oovounve o  t e com  na  e  stexenion  h
pl
", as is recited in claim 12. Thsfonteresose fort abve clama1siteraosstfrhaoecli
Atore
SONP30.C an1
o    o n o    m sa
a e o  coeo ugs
Moreover, the combination of Samples and Ohishima does not disclose or suggest the recitations of claim 17. For example, for the same reasons set forth above, the combination of Samples and Ohishima does not teach: the light bar extends between the first group of buttons and the second group of directional buttons... ; and
. .the lights emitted by the light bar and the light indicator facilitate a determination of an orientation of the main body, the first extension, and the second extension with
reference to the HMD
as is recited in claim 17. Therefore, for the reasons set forth above, claim 17 is submitted to be nonobvious in view of the combination of Samples and Ohishima.
Claims 2-8, 10-11, 23, and 24 depend from independent claim 1. Also, claims 14-16 depend from independent claim 12 and claims 18-20 depend from independent claim 17. Hence, for the reasons set forth above, claims 2-8, 10-11, 14-16, 18-20, 23, and 24 are submitted to be nonobvious in view of the combination of Samples and Ohishima.
As such, it is respectfully requested that the section 103 rejection of claims 1-8, 10- 12, 14-20, and 23-24 be withdrawn.
Claim Rejection - 35 USC s 103
Claims 9 and 13 are rejected under 35 U.S.C. §103 as being unpatentable over Samples, in view of Ohishima, further in view of Mallinson, United States Patent Application No. US 2016/0025978 AL. This rejection is respectfully traversed.
Claim 9 depends from independent claim 1 and claim 13 depends from independent claim 12. The combination of Samples, Ohishima, and Mallinson does not teach the recitations of claims 1 and 12. For example, the combination fails to teach:
the light bar extends between the first group of buttons and
the second group of directional buttons; and
... the lights emitted by the light bar and the light indicator
facilitate a determination of an orientation of the main body,
the first extension, and the second extension with reference to
the HMD
as is recited in claim 1. As another example, the combination does not disclose or suggest the that "the light bar extends between a first group of buttons of the hand-held controller and a Attorney Docket No. SONYP360.C1
second group of directional buttons of the hand-held controller", as is recited in claim 12. The combination of Samples and Ohishima is described above. Moreover, Mallinson is not cited to cure the deficiencies in the combination of Samples and Ohishima. Rather, Mallinson is cited to disclose "a controller . . . comprising a haptic feedback module" (see office action, page 20). Thus, for the reasons set forth above, independent claims 1 and 12 and the respective dependent claims 9 and 13 are submitted to be nonobvious in view of the combination of Samples, Ohishima, and Mallinson.
As such, it is respectfully requested that the section 103 rejection of claims 9 and 13 be withdrawn.
New Claim
Claims 25 and 26 depend from independent claim 1, and are patentable for the same reasons the independent claim 1 is patentable.

Conclusion

NEXT OFFICE ACTION: DETAILED ACTION
Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
As a result of the Amendment foiled on July 12, 2024, claims 1-20 and 23-26 are pending. Claims 1, 2, 5, 12 and 17 are amended. Claims 21-22 were previously canceled. New claims 25-26 are added.
Response to Arguments
Applicant's arguments filed July 12, 2024 with respect the art rejections of claims 1-20 and 23-25 have been fully considered but they are not persuasive. Applicant is contending that the prior art does not show the amended features of a “first group of buttons”, “a second group of directional buttons”, and “wherein the first light bar extends between the first group of buttons and the second group of directional buttons”. (Applicant Remarks at pgs. 9-11). Furthermore, Applicant also argues the feature “wherein the lights emitted by the light indicator and light bar facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD” is not taught by Ohishima because Ohishima’s xy coordinate position is not an orientation (Applicant Remarks at pgs. 11-12).
The Office respectfully disagrees with the above conclusions. To start, Ohishima was used purely for its form-factor of a “light bar” and the structural modification to Samples merely changes the form-factor of the handheld object (Samples, Figs. 1-4, handheld object, #106). The rationale and reasoning for the modification has been set forth in the 103 rejection as a prima facie case of obviousness. (See rejection section for details). Otherwise, Samples teaches the remaining structural and functional aspects of the independent claims. Samples teaches a first group of buttons and a second group of directional buttons (Detailed Description, [0034-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc.”) because the cited section shows both buttons and directional pads can be input controls on the handheld object.  Next, Samples teaches “wherein the first light indicator extends between the first group of buttons and the second group of directional buttons” (See Figs. 2-3, light sources, #302; light sources, Detailed Description, [0034-0036][0130-0140], since light sources are all around the surface, it includes the areas in between the first group of buttons and second group of button) because Samples shows the light sources distributed along the ring, including in vector space in between the joystick and buttons on the handle.  It is noted that it was not affirmatively claimed that the light bar extends between the first group of buttons and second group of buttons within the same plane or within the same layer. Next, Samples continues to teaches  “wherein the lights emitted by the first light indicator and second light indicator facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD” (Fig. 4, HMD, #402; Detailed Description, [0030-0050], “The optical pose of the handheld object with respect to the HMD (“HMD.fwdarw.LED Vision Poses” at 503) and the optical pose of the HMD relative to the world (“World.fwdarw.HMD Poses” at 505) are used to determine a pose of the handheld object with respect to the world”; Detailed Description, [0130-140], “As the HMD makes a sequence of observations, the HMD may determine that alternative parameters may make the observations more likely. For example, after a handheld object is dropped and experiences a deformation in light source positions on the handheld object, the HMD may observe a light pattern that differs from the model light pattern, and may recompute the constellation positions and update one or more calibration parameters for the handheld object. The tracking system on the HMD may use new estimates of light source positions, normal, illumination fields, and IMU parameters to refine future tracking and pose estimates.”).
Since Ohishima modifies Sample’s first light indicator to be a rectangular shaped light bar, the combination and rationale for independent claims 1, 12 and 17 remain as set forth in the previous Office Action.  For the foregoing reasons, the rejection grounds are maintained for claims 1-20 and 23-25.
Applicant’s arguments in light of the new claims added with respect specifically to claim 26 have been fully considered and are persuasive.  Claim 26 is objected to as being dependent off of rejected base claim but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims (including claim 25).
An additional reference is cited on the PTO-892 is relevant to the Field of the Invention but is not necessarily used in the rejection grounds.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-8, 10-12, 14-20 and 23-25 are rejected under 35 U.S.C. 103 as being unpatentable over Samples et al., United States Patent Application Publication No. US 2018/0330521 A1 in view of Ohishima et al., United States Patent Application Publication No. US 2002/0084974 A1.
Regarding claim 1, Samples discloses a hand-held controller (Fig. 1-4, handheld object, #106; Detailed Description, [0033-0035]) comprising:
a main body, wherein the main body has a touch surface (Figs. 1-4, Detailed Description, [0035], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc.”);
a first extension extending from the main body (Fig. 4, 6DoF controller, left hand, #406; Detailed Description, [0036-0050]);
a second extension extending from the main body (Fig. 4, 6DoF controller, right hand, #408; Detailed Description, [0036-0050]);
a first group of buttons (Detailed Description, [0034-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc.”);
a second group of directional buttons (Detailed Description, [0034-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc.”)
a first light indicator located with respect to the main body, wherein the first light indicator extends between the first group of buttons and the second group of directional buttons (See Figs. 2-3, light sources, #302; light sources, Detailed Description, [0034-0036][0130-0140], since light sources are all around the surface, it includes the areas in between the first group of buttons and second group of button) wherein the first light indicator located along an edge of the touch surface to be visible to a camera fitted to a lower region of a head-mounted display (HMD) (Fig. 2, first and second camera, #204/206; Detailed Description, [0033]), wherein the first light indicator is configured to emit light to be detected by the camera (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller”); and
a second light indicator located with respect to the main body, wherein the second light indicator is configured to emit light to be detected by the camera (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller…A handheld object may have any suitable number and arrangement of light sources.”), wherein the lights emitted by the first and second light indicators facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD (Fig. 4, HMD, #402; Detailed Description, [0030-0050], “The optical pose of the handheld object with respect to the HMD (“HMD.fwdarw.LED Vision Poses” at 503) and the optical pose of the HMD relative to the world (“World.fwdarw.HMD Poses” at 505) are used to determine a pose of the handheld object with respect to the world”).
Samples does not disclose the first light indicator explicitly as a light bar.
Ohishima, in a similar field of endeavor, discloses a hand-held controller comprising a light bar located with respect to the main body, where the light bar is located along an edge of the touch surface to be visible to a camera, wherein the light bar is configured to emit light to be detected by the camera (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173], “FIG. 2 is a side view of the game apparatus portion of the system of this embodiment. In an MR air hockey game, two players 2000 and 3000 face each other while holding mallets (260L, 260R) with their hands. The two players 2000 and 3000 wear head mount displays (to be abbreviated as HMDs hereinafter) 210L and 210R on their heads. The mallet of this embodiment has an infrared ray generator at its distal end. As will be described later, in this embodiment, the mallet position is detected by image processing. If each mallet has a feature in its shape or color, the mallet position can also be detected by pattern recognition using such feature….Mallet 260 held by each player has an infrared ray generator at its distal end, and each mallet position in a two-dimensional plane on the table 1000 is detected by a CCD camera 230 that detects the infrared rays. Specifically, the camera 230 is provided so that it may detect mallet positions of the players, and the detected positions of the players will be used for advancing or progressing the game, for this embodiment.”; the depicted mallet is a bar shape and emits infrared light).
It would have been obvious to one of ordinary skill in the art to have modified the form of at least one of the light indicator within Samples to be specifically a light bar, as taught in Ohishima, and provide  wherein the light bar is located along an edge of the touch surface to be visible to a camera fitted to a lower region of a head-mounted display (HMD), wherein the first light bar is configured to emit light to be detected by the camera and wherein the lights emitted by the light bar and light indicator facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD. The motivation to combine these arts is to provide the user with a controller with a holdable form factor, based on the particular function of the HMD (See Ohishima, Detailed Description, [0157-0166], in this case an air hockey game). In addition, the changing the light indicator shape to a bar would have been obvious to one of ordinary skill and pursued without undue experimentation as a matter of design choice (MPEP 2144.04 Part IV).  The fact that Ohishima discloses similar systems of HMD and handheld controllers as Samples, including using infrared emitters and cameras, makes this combination more easily implemented since they are within same field.
Regarding claim 2, Samples further discloses wherein the hand-held controller has an edge, wherein the first light indicator is located along the edge of the hand-held controller to be visible to the camera fitted to lower region of the HMD with the main body, wherein the second light indicator is located at a distance from the first light indicator to be visible to the camera (Fig. 3, Detailed Description, [0036-0041], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300”).
Ohishima also teaches wherein the hand-held controller has an edge, wherein the light bar is located along an edge of the hand-held controller to be visible to the camera (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173], “FIG. 2 is a side view of the game apparatus portion of the system of this embodiment. In an MR air hockey game, two players 2000 and 3000 face each other while holding mallets (260L, 260R) with their hands. The two players 2000 and 3000 wear head mount displays (to be abbreviated as HMDs hereinafter) 210L and 210R on their heads. The mallet of this embodiment has an infrared ray generator at its distal end. As will be described later, in this embodiment, the mallet position is detected by image processing. If each mallet has a feature in its shape or color, the mallet position can also be detected by pattern recognition using such feature….Mallet 260 held by each player has an infrared ray generator at its distal end, and each mallet position in a two-dimensional plane on the table 1000 is detected by a CCD camera 230 that detects the infrared rays. Specifically, the camera 230 is provided so that it may detect mallet positions of the players, and the detected positions of the players will be used for advancing or progressing the game, for this embodiment.”)
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1.
Regarding claim 3, Samples further discloses wherein the touch surface is configured to determine movements of fingers of a user, wherein the first light indicator includes a plurality of light sources, wherein the plurality of light sources are located at pre-determined locations with reference to the touch surface (Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”).
Ohishima also teaches wherein the light bar includes a plurality of light sources, wherein the plurality of light sources are located at pre-determined locations with reference to the touch surface (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173], “Mallet 260 held by each player has an infrared ray generator at its distal end, and each mallet position in a two-dimensional plane on the table 1000 is detected by a CCD camera 230 that detects the infrared rays. Specifically, the camera 230 is provided so that it may detect mallet positions of the players, and the detected positions of the players will be used for advancing or progressing the game, for this embodiment.”).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1.
Regarding claim 4, Samples further discloses wherein the first light indicator is an infrared light emitter (Detailed Description, [0035-0036], “Infrared light sources also may be used.”).
Ohishima also teaches wherein the light bar is an infrared light emitter (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173],
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1.
Regarding claim 5, Samples further discloses wherein the first light indicator is configured to face the camera to be in a field-of-view facing downward from the camera (Fig. 2-3, the handheld device #202 is depicted as downward from the HMD which has the camera).
Ohishima also discloses wherein the light bar is configured to face the camera to be in a field-of-view facing downward from the camera (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173], camera, #230).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1.
Regarding claim 6, Samples further discloses a hand-held controller comprising a touch panel situated between the first extension and the second extension, wherein the touch panel has a curvature Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”), wherein the first light indicator is a string of light sources located along an edge of the main body and an edge of the touch surface is situated above the touch panel  (Fig. 3, Detailed Description, [0036-0041], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300”). (Fig. 3, Detailed Description, [0036-0041], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300”; light sources above the handheld object touch surface at bottom).
Ohishima, in a similar field of endeavor, discloses a hand-held controller comprising a light bar (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173])
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1 with the modification of the light indicator as a light bar.
Regarding claim 7, Samples discloses a controller further comprising a battery configured to provide power to the first light indicator and the second light indicator (Detailed Description, [0061]), wherein the second light indicator is a light emitting diode (Detailed Description, [0036]).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1 with the modification of the light indicator as a light bar.
Regarding claim 8, Samples discloses wherein the camera has a co-ordinate system having an x-axis, a y-axis, and a z-axis, wherein the orientation is determined with respect to the x, y, and z axes of the co-ordinate system (Detailed Description, [0052], “The computing device further predicts more possible uv-xyz correspondences, and refines these predictions. The predictions are thresholded on a reprojection error, the error between 2D detection points (u,v's) and the prediction of associated 3D points (x,y,z), assuming a certain pose”).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1 with the modification of the light indicator as a light bar.
Regarding claim 10, Samples discloses wherein the lights are detected by an additional camera located at the bottom surface of the HMD to determine the orientation of the main body, the first extension, and the second extension with reference to the additional camera (Fig. 4, HMD, #402; Detailed Description, [0030-0050], “The optical pose of the handheld object with respect to the HMD (“HMD.fwdarw.LED Vision Poses” at 503) and the optical pose of the HMD relative to the world (“World.fwdarw.HMD Poses” at 505) are used to determine a pose of the handheld object with respect to the world”; See also Summary, [0003] for showing of multiple cameras in the stereo camera arrangement).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1 with the modification of the light indicator as a light bar.
Regarding claim 11, Samples in combination with Ohishima discloses every element of claim 1, and Ohshima further discloses wherein the light bar is located below the touch surface to be visible to the camera (Figs. 1-3, Detailed Description, [0162], “The mallet of this embodiment has an infrared ray generator at its distal end”; distal end would be away from and below where user is touching the mallet).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1 with the modification of the light indicator as a light bar.
Regarding claim 12, Samples discloses a head-mounted display (Fig. 1-4, #102, HMD)  comprising:
a top surface (See Fig. 2, top surface of HMD 200);
a front surface located adjacent to the top surface (Fig. 2, front surface is next to top surface);
a bottom surface located adjacent to the front surface (Fig. 2, #200);
a band coupled to the top surface (Fig. 1-2, back of HMD is a band for user); and
a camera located at the bottom surface, wherein the camera is configured to capture images of lights emitted from a first light indicator and a second light indicator of a hand-held controller to facilitate a determination of orientation of the hand-held controller with reference to the camera (Fig. 2, cameras, #204; second camera, #206; Detailed Description, [0030-0053], “One or more outward-facing cameras on the HMD 102 may acquire image data (e.g. visible light image data) of the surrounding environment and of a handheld object 106 held by the user 102. For example, FIG. 2 shows an example HMD 200 imaging a handheld object 202 using a stereo camera imaging system (indicated by first camera 204 and second camera 206)....The term “handheld object” is used herein for an object comprising trackable light sources 108 that may be held by hand as described herein, and signifies that the object may be handheld, whether or not the object is currently in the state of being held in a hand… For example, because the HMD may have more than one camera, a pose may be solved for each camera on the HMD. Such refinement is performed as postprocessing of a multitude of poses obtained from different cameras. In other examples, any other suitable pose estimation process may be used.”)-, wherein the light indicator extends between a first group of buttons of the hand-held controller and a second group of directional buttons of the hand-held controller (See Figs. 2-3, light sources, #302; light sources, Detailed Description, [0034-0036][0130-0140], since light sources are all around the surface, it includes the areas in between the first group of buttons and second group of buttons),
Samples does not disclose the first light indicator explicitly as a light bar.
Ohishima, in a similar field of endeavor, discloses a head-mounted display comprising a camera configured to capture images of light emitted from a light bar of a hand-held controller to facilitate a determination of orientation of the hand-held controller with reference to the camera (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173], “FIG. 2 is a side view of the game apparatus portion of the system of this embodiment. In an MR air hockey game, two players 2000 and 3000 face each other while holding mallets (260L, 260R) with their hands. The two players 2000 and 3000 wear head mount displays (to be abbreviated as HMDs hereinafter) 210L and 210R on their heads. The mallet of this embodiment has an infrared ray generator at its distal end. As will be described later, in this embodiment, the mallet position is detected by image processing. If each mallet has a feature in its shape or color, the mallet position can also be detected by pattern recognition using such feature….Mallet 260 held by each player has an infrared ray generator at its distal end, and each mallet position in a two-dimensional plane on the table 1000 is detected by a CCD camera 230 that detects the infrared rays. Specifically, the camera 230 is provided so that it may detect mallet positions of the players, and the detected positions of the players will be used for advancing or progressing the game, for this embodiment.”; the depicted mallet is a bar shape and emits infrared light).
It would have been obvious to one of ordinary skill in the art to have modified the form of at least one of the light indicator within Samples to be specifically a light bar, as taught in Ohishima, and provide the camera configured to capture images of light emitted from a light bar of a hand-held controller to facilitate a determination of orientation of the hand-held controller with reference to the camera. The motivation to combine these arts is to provide the user with a controller with a holdable form factor, based on the particular function of the HMD (See Ohishima, Detailed Description, [0157-0166], in this case an air hockey game). In addition, the changing the light indicator shape to a bar would have been obvious to one of ordinary skill and pursued without undue experimentation as a matter of design choice (MPEP 2144.04 Part IV).  The fact that Ohishima discloses similar systems of HMD and handheld controllers as Samples, including using infrared emitters and cameras, makes this combination more easily implemented since they are within same field.
Regarding claim 14, Samples discloses a head-mounted display further including an additional camera located at the bottom surface, wherein the additional camera is configured to capture images of the lights from the first and second light indicators to facilitate a determination of the orientation of the hand-held controller with reference to the additional camera (Fig. 2, cameras, #204; second camera, #206; Detailed Description, [0030-0053], “One or more outward-facing cameras on the HMD 102 may acquire image data (e.g. visible light image data) of the surrounding environment and of a handheld object 106 held by the user 102. For example, FIG. 2 shows an example HMD 200 imaging a handheld object 202 using a stereo camera imaging system (indicated by first camera 204 and second camera 206)....The term “handheld object” is used herein for an object comprising trackable light sources 108 that may be held by hand as described herein, and signifies that the object may be handheld, whether or not the object is currently in the state of being held in a hand… For example, because the HMD may have more than one camera, a pose may be solved for each camera on the HMD. Such refinement is performed as postprocessing of a multitude of poses obtained from different cameras. In other examples, any other suitable pose estimation process may be used.”).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 12 with the modification of the light indicator as a light bar.
Regarding claim 15, Samples discloses a head-mounted display further comprising: an eye mask having the top surface, the front surface, the bottom surface, and a frame, wherein the frame is fitted to the bottom surface, wherein the band is a headband coupled to the frame and a portion of the top surface (Fig. 1-2—generally; Examiner’ note—although eyemask and headband are not terms used, they are depicted as included in the HMD structure).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 12 with the modification of the light indicator as a light bar.
Regarding claim 16, Samples discloses wherein the hand-held controller has a touch surface (Detailed Description, [0035-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc”), wherein the first light indicator is located below the touch surface and along an edge of the touch surface to be visible to the camera (Fig. 3, Detailed Description, [0036-0041], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300”).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 12 with the modification of the light indicator as a light bar.
Regarding claim 17, Samples disclose a system (Fig. 1-4, generally) comprising:
a head-mounted display (Fig. 1-4, #102, HMD)  including:
a top surface (See Fig. 2, top surface of HMD 200);
a front surface located adjacent to the top surface (Fig. 2, front surface is next to top surface);
a bottom surface located adjacent to the front surface (Fig. 2, #200);
a band coupled to the top surface (Fig. 1-2, back of HMD is a band for user); and
a camera located at the bottom surface (Fig. 2, cameras, #204; second camera, #206; Detailed Description, [0030-0053], “One or more outward-facing cameras on the HMD 102 may acquire image data (e.g. visible light image data) of the surrounding environment and of a handheld object 106 held by the user 102. For example, FIG. 2 shows an example HMD 200 imaging a handheld object 202 using a stereo camera imaging system (indicated by first camera 204 and second camera 206)....The term “handheld object” is used herein for an object comprising trackable light sources 108 that may be held by hand as described herein, and signifies that the object may be handheld, whether or not the object is currently in the state of being held in a hand… For example, because the HMD may have more than one camera, a pose may be solved for each camera on the HMD. Such refinement is performed as postprocessing of a multitude of poses obtained from different cameras. In other examples, any other suitable pose estimation process may be used.”);
a hand-held controller (Fig. 1-4, handheld object, #106; Detailed Description, [0033-0035]) including:
a main body (Figs. 1-4, Detailed Description, [0035]);
a first extension extending from the main body (Fig. 4, 6DoF controller ,left hand, #406; Detailed Description, [0036-0050]);
a second extension extending from the main body (Fig. 4, 6DoF controller, right hand, #408; Detailed Description, [0036-0050]);
a first group of buttons (Detailed Description, [0034-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc.”);
a second group of directional buttons (Detailed Description, [0034-0036], “The handheld object 106 may also include one or more input controls, such as a button, trigger, joystick, directional pad, touch screen, etc.”)
a first light indicator located with respect to the main body, wherein the first light indicator extends between the first group of buttons and the second group of directional buttons (See Figs. 2-3, light sources, #302; light sources, Detailed Description, [0034-0036][0130-0140], since light sources are all around the surface, it includes the areas in between the first group of buttons and second group of button), wherein the first light indicator is configured to emit light to be detected by the camera of the HMD (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller”); and
a second light indicator located with respect to the main body,wherein the second light indicator is configured to emit light to be detected by the camera of the HMD, wherein the lights emitted by the first and second light indicators facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD camera (Detailed Description, [0030-0050], “The handheld object also includes a plurality of light sources distributed over a surface of the handheld object. FIG. 3 shows an example handheld object 300 having a plurality of light sources 302 distributed along both an exterior and interior of a ring-like structure of the handheld controller 300. The light sources are configured to form patterns of light (“constellations”) in image data acquired by the camera or cameras of the HMD, such that a pose of the handheld object may be determined from an image capturing the controller…A handheld object may have any suitable number and arrangement of light sources.”; Fig. 4, HMD, #402; Detailed Description, [0030-0050], “The optical pose of the handheld object with respect to the HMD (“HMD.fwdarw.LED Vision Poses” at 503) and the optical pose of the HMD relative to the world (“World.fwdarw.HMD Poses” at 505) are used to determine a pose of the handheld object with respect to the world”).).
Samples does not disclose the first light indicator explicitly as a light bar.
Ohishima, in a similar field of endeavor, discloses a hand-held controller comprising a light bar located with respect to the main body, wherein the light bar is configured to emit light to be detected by the camera of the HMD  (Figs. 2-3, mallets, #260L/R, Detailed Description, [0162-0173], “FIG. 2 is a side view of the game apparatus portion of the system of this embodiment. In an MR air hockey game, two players 2000 and 3000 face each other while holding mallets (260L, 260R) with their hands. The two players 2000 and 3000 wear head mount displays (to be abbreviated as HMDs hereinafter) 210L and 210R on their heads. The mallet of this embodiment has an infrared ray generator at its distal end. As will be described later, in this embodiment, the mallet position is detected by image processing. If each mallet has a feature in its shape or color, the mallet position can also be detected by pattern recognition using such feature….Mallet 260 held by each player has an infrared ray generator at its distal end, and each mallet position in a two-dimensional plane on the table 1000 is detected by a CCD camera 230 that detects the infrared rays. Specifically, the camera 230 is provided so that it may detect mallet positions of the players, and the detected positions of the players will be used for advancing or progressing the game, for this embodiment.”; the depicted mallet is a bar shape and emits infrared light).
It would have been obvious to one of ordinary skill in the art to have modified the form of at least one of the light indicator within Samples to be specifically a light bar, as taught in Ohishima, and provide a light bar located with respect to the main body, wherein the light bar is configured to emit light to be detected by the camera of the HMD, wherein the first light bar is configured to emit light to be detected by the camera and wherein the lights emitted by the light bar and light indicator facilitate a determination of an orientation of the main body, the first extension, and the second extension with reference to the HMD. The motivation to combine these arts is to provide the user with a controller with a holdable form factor, based on the particular function of the HMD (See Ohishima, Detailed Description, [0157-0166], in this case an air hockey game). In addition, the changing the light indicator shape to a bar would have been obvious to one of ordinary skill and pursued without undue experimentation as a matter of design choice (MPEP 2144.04 Part IV).  The fact that Ohishima discloses similar systems of HMD and handheld controllers as Samples, including using infrared emitters and cameras, makes this combination more easily implemented since they are within same field.
Regarding claim 18, this is met by the rejection to claim 2 with the combination of Samples and Ohishima.
Regarding claim 19, this is met by the rejection to claim 14 with the combination of Samples and Ohishima.
Regarding claim 20, this is met by the rejection of claim 11 with the combination of Samples and Ohishima.
Regarding claim 23, Samples in combination with Ohishima discloses every element of claim 1, and Ohishima further discloses wherein the light bar has a first edge point and a second edge point, wherein the first and second edge points define the orientation of the main body, the first extension, and the second extension with reference to the HMD (Figs. 1-5, mallet, #260; Detailed Description, [0156-0182]; See also Detailed Description, [0192-210] and Figs. 8-10).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1 with the modification of the first light indicator as a light bar.
Regarding claim 24, Samples in combination with Ohishima discloses every element of claim 23, and Ohishima further discloses wherein the orientation is a first orientation, wherein the first and second edge points have a first set of locations with respect to a reference level passing through the light indictor to define the first orientation and have a second set of locations with respect to the reference level to define a second orientation of the main body, the first extension, and the second extension with reference to the HMD (Ohishima, Figs. 1-5, mallet, #260; Detailed Description, [0156-0182]; See also Detailed Description, [0192-210] and Figs. 8-10), “ As will be described later, in this embodiment, the mallet position is detected by image processing. If each mallet has a feature in its shape or color, the mallet position can also be detected by pattern recognition using such feature.”).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1 with the modification of the first light indicator as a light bar.
Regarding claim 25, Samples in combination with Ohishima discloses every element of claim 1, and Ohishima further discloses wherein the light bar has a plurality of edges, wherein the orientation is determined based on whether the plurality of edges are above or below a horizontal level that passes through the light indicator (Figs. 1-5, mallet, #260; Detailed Description, [0156-0182]; See also Detailed Description, [0192-210] and Figs. 8-10; it is inherent that the edges would be above or below an arbitrary horizontal level).
Thus, it would have remained obvious to have combined Samples and Ohishima in the manner described in claim 1 with the modification of the first light indicator as a light bar.
Claim(s) 9 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Samples, in view of Ohishima, further in view of Mallinson, United States Patent Application Publication No. US 2016/0025978 A1.
Regarding claim 9, Samples in combination with Ohishima discloses every element of claim 1 and Samples further discloses a touch panel (Detailed Description, [0035]).
Samples and Ohishima do not disclose the combination of the hand-held controller comprising a haptic feedback module and a touch panel, wherein the main body includes a first joystick and a second joystick that are configured to be moved to generate input data for processing by a computing device, wherein the touch panel is located between the first and second joysticks, wherein the haptic feedback module is configured to provide tactile feedback to a user.
However, Samples does provide the suggestion of the main body including using both a joystick and touch panel together (Detailed Description, [0035]).
Mallinson, in a similar field of endeavor, discloses a controller (Fig. 1) comprising a haptic feedback module and a touch panel (Detailed Description, [0081]), wherein the main body includes a first joystick and a second joystick that are configured to be moved to generate input data for processing by a computing device, (Mallison, Detailed Description, [0091], “A member of Clients 1110 is optionally configured to receive more than one audio or video stream. Input devices of Clients 1110 may include, for example, a one-hand game controller, a two-hand game controller, a gesture recognition system, a gaze recognition system, a voice recognition system, a keyboard, a joystick, a pointing device, a force feedback device, a motion and/or location sensing device, a mouse, a touch screen, a neural interface, a camera, input devices yet to be developed, and/or the like.”) wherein the haptic feedback module is configured to provide tactile feedback to a user (Detailed Description, [0056], “Additionally, part of the processed media content may be transmitted to other devices, such as a game controller, that was used to provide input to the application. For example, video and audio streams of the updates may be provided to the HMD and the haptic feedback may be provided to the game controller.”).
The above references do not explicitly disclose wherein the touch panel is located between the first and second joysticks.
However, it would have been obvious to one of ordinary skill in the art to have modified the combination of Samples and Ohishima to utilize the teachings and suggestion of Mallinson to provide the combination of the hand-held controller comprising a haptic feedback module and a touch panel, wherein the main body includes a first joystick and a second joystick that are configured to be moved to generate input data for processing by a computing device, wherein the touch panel is located between the first and second joysticks, wherein the haptic feedback module is configured to provide tactile feedback to a user. The motivation to combine these arts is to add to the immersion of the user while using a game controller within a AR/VR HMD system which enhances user experience (See Mallison, Summary, [0006-0009]). Furthermore, the modification to the prior art as a whole to provide wherein the touch panel is located between the first and second joysticks is a matter of rearrangement of parts, which would have been pursued by one of ordinary skill without undue experimentation since it would not have modified the overall operation of the HMD/controller system (MPEP 2144.04 Part VI). The fact the Mallison also discloses HMDs in use with game controllers, same as Samples and Ohishima, makes this combination easily implemented.
Regarding claim 13, Samples in combination with Ohishima discloses every element of claim 12 but those above references does not explicitly disclose a HMD including a haptic feedback device configured to vibrate the top, front, and bottom surfaces to provide tactile feedback to a head of a user.
Mallinson, in a similar field of endeavor, discloses a head-mounted device (Fig. 1, HMD, #100) including a haptic feedback device configured to vibrate the top, front, and bottom surfaces to provide tactile feedback to a head of a user (Detailed Description, [0080-0085], “Other output modules include Light-Emitting Diodes (LED) 834, other visual markers or elements (which may also be used for visual tracking of the HMD), vibro-tactile feedback 850, speakers 830, and sound localization module 812, which performs sound localization for sounds to be delivered to speakers or headphones. Other output devices, such as headphones, can also connect to the HMD via the communication modules.”).
It would have been obvious to one of ordinary skill in the art to have modified the HMD of Samples-Ohishima to include the teachings of Mallison to add a haptic feedback device configured to vibrate the top, front, and bottom surfaces to provide tactile feedback to a head of a user. The motivation to combine these arts is to add to the immersion of the user within an AR/VR HMD system while wearing the HMD, which enhances user experience (See Mallison, Summary, [0006-0009][0081-0085]). The fact the Mallison also discloses HMDs in use with game controllers, same as Samples, makes this combination easily implemented.
Allowable Subject Matter
Claim 26 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
The prior art of record does not teach nor suggest the additional structural features of hand-held controller of claim 26 wherein the plurality of edges include a first edge and a second edge, wherein the orientation is a first orientation of the first and second extensions when the plurality of edges are below the horizontal level, wherein the first and second extensions have a second orientation with reference to the HMD when the first edge is above the horizontal level and the second edge is below the horizontal level.
Since the Ohishima reference teaches tracking the position of the mallet in 2-D space (See Detailed Description, [0191-0230], discussing XY coordinates), it does not teach nor suggest orientations when the first edge is above the horizontal level and the second is below the horizontal level, and thereby would teach away from a modification to arrive at the features of claim 26.
Conclusion
THIS ACTION IS MADE FINAL.  Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action.  In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action.  In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.

CITED REFERENCE SUMMARIES: Ref US20020084974A1: Claims:
A mixed reality presentation apparatus which generates a three-dimensional virtual image associated with a collaborative operation to be done by a plurality of operators in a predetermined mixed reality environment, and displays the generated virtual image on see-through display devices respectively attached to the plurality of operators, comprising: first sensor means for detecting a position of each of actuators which are operated by the plurality of operators and move as the collaborative operation progresses; second sensor means for detecting a view point position of each of the plurality of operators in an environment of the collaborative operation; and generation means for generating three-dimensional images for the see-through display devices of the individual operators, said generation means generating a three-dimensional virtual image representing an operation result of the collaborative operation that has progressed according to a change in position of each of the plurality of actuators detected by said first sensor means when viewed from the view point position of each operator detected by said second sensor means, and outputting the generated three-dimensional virtual image to each see-through display device.
first sensor means for detecting a position of each of actuators which are operated by the plurality of operators and move as the collaborative operation progresses;
second sensor means for detecting a view point position of each of the plurality of operators in an environment of the collaborative operation; and
generation means for generating three-dimensional images for the see-through display devices of the individual operators, said generation means generating a three-dimensional virtual image representing an operation result of the collaborative operation that has progressed according to a change in position of each of the plurality of actuators detected by said first sensor means when viewed from the view point position of each operator detected by said second sensor means, and outputting the generated three-dimensional virtual image to each see-through display device.
The apparatus according to claim 1 , wherein said first sensor means comprises: an image sensing camera which includes a maximum range of the actuator within a field of view thereof, the position of said actuator moving upon operation of the operator; and image processing means for performing image-processing to detect a position of each actuator in an image obtained by said camera.
an image sensing camera which includes a maximum range of the actuator within a field of view thereof, the position of said actuator moving upon operation of the operator; and
image processing means for performing image-processing to detect a position of each actuator in an image obtained by said camera.
The apparatus according to claim 1 , wherein the actuator includes an illuminator emitting light having a predetermined wavelength, and said first sensor means comprises a camera which is sensitive to the light having the predetermined wavelength.
The apparatus according to claim 1 , wherein the actuator is a mallet operated by a hand of the operator.
The apparatus according to claim 1 , wherein the see-through display device comprises an optic al transmission type display device.
The apparatus according to claim 1 , wherein said second sensor means detects a head position and posture of each operator, and calculates a view point position in accordance with the detected head position and posture.
The apparatus according to claim 5 , wherein said second sensor means comprises: a generator for generating an AC magnetic field; and a magnetic sensor attached to the head portion of each operator.
a generator for generating an AC magnetic field; and
a magnetic sensor attached to the head portion of each operator.
The apparatus according to claim 1 , wherein said generation means comprises: storage means for storing a rule of the collaborative operation; means for generating a virtual image representing a progress result of the collaborative operation in accordance with the rule stored in said storage means in correspondence with detected changes in position of the plurality of actuators; and means for generating a three-dimensional virtual image for each view point position by transferring a coordinate position for each view point position of each operator detected by said second sensor means.
storage means for storing a rule of the collaborative operation;
means for generating a virtual image representing a progress result of the collaborative operation in accordance with the rule stored in said storage means in correspondence with detected changes in position of the plurality of actuators; and
means for generating a three-dimensional virtual image for each view point position by transferring a coordinate position for each view point position of each operator detected by said second sensor means.
A mixed reality presentation apparatus which generates a three-dimensional virtual image associated with a collaborative operation to be done by a plurality of operators in a predetermined mixed reality environment, and displays the generated virtual image on see-through display devices respectively attached to the plurality of operators, comprising: a camera arranged so as to include a plurality of actuators operated by the plurality of operators in the collaborative operation within a field of view thereof; actuator position detection means for outputting information relating to positions of the actuators associated with a coordinate system of that environment on the basis of an image sensed by said camera; sensor means for detecting and outputting a view point position of each of the plurality of operators in the environment of the collaborative operation; and image generation means for generating a three-dimensional virtual image of a progress result viewed from the view point position of each operator detected by said sensor means to each see-through display device so as to present the progress result of the collaborative operation that has progressed according to detected changes in position of the actuator to each operator.
a camera arranged so as to include a plurality of actuators operated by the plurality of operators in the collaborative operation within a field of view thereof;
actuator position detection means for outputting information relating to positions of the actuators associated with a coordinate system of that environment on the basis of an image sensed by said camera;
sensor means for detecting and outputting a view point position of each of the plurality of operators in the environment of the collaborative operation; and
image generation means for generating a three-dimensional virtual image of a progress result viewed from the view point position of each operator detected by said sensor means to each see-through display device so as to present the progress result of the collaborative operation that has progressed according to detected changes in position of the actuator to each operator.
A mixed reality presentation apparatus which generates a three-dimensional virtual image associated with a collaborative operation to be done by a plurality of operators in a predetermined mixed reality environment, and displays the generated virtual image on see-through display devices respectively attached to the plurality of operators, comprising: a first camera which substantially includes the plurality of operators within a field of view thereof; a first processor arranged so as to calculate operation positions of the plurality of operators on the basis of an image obtained by said first camera; a detection device detecting a view point position of each operator using a plurality of sensors attached to the plurality of operators; a plurality of second cameras sensing front fields of the individual operators, at least one second camera being attached to each of the plurality of operators; a second processor calculating information associated with a line of sight of each operator on the basis of each of images from said plurality of second cameras; a third processor correcting the view point position of each operator detected by the sensor using the line of sight information from said second processor and outputting the corrected view point position as a position on a coordinate system of the mixed reality environment; a first image processing device making the collaborative operation virtually progress on the basis of the operation position of each operator calculated by said first processor, and generating three-dimensional virtual images representing results that have changed along with the progress of the collaborative operation for the plurality of operators; and a second image processing device transferring coordinate positions of the three-dimensional virtual images for the individual operators generated by said first image processing device in accordance with the individual corrected view point positions calculated by said third processor, and outputting the coordinate-transferred images to the see-through display devices.
a first camera which substantially includes the plurality of operators within a field of view thereof;
a first processor arranged so as to calculate operation positions of the plurality of operators on the basis of an image obtained by said first camera;
a detection device detecting a view point position of each operator using a plurality of sensors attached to the plurality of operators;
a plurality of second cameras sensing front fields of the individual operators, at least one second camera being attached to each of the plurality of operators;
a second processor calculating information associated with a line of sight of each operator on the basis of each of images from said plurality of second cameras;
a third processor correcting the view point position of each operator detected by the sensor using the line of sight information from said second processor and outputting the corrected view point position as a position on a coordinate system of the mixed reality environment;
a first image processing device making the collaborative operation virtually progress on the basis of the operation position of each operator calculated by said first processor, and generating three-dimensional virtual images representing results that have changed along with the progress of the collaborative operation for the plurality of operators; and
a second image processing device transferring coordinate positions of the three-dimensional virtual images for the individual operators generated by said first image processing device in accordance with the individual corrected view point positions calculated by said third processor, and outputting the coordinate-transferred images to the see-through display devices.
A method of generating a three-dimensional virtual image associated with a collaborative operation to be done within a predetermined mixed reality environment so as to display the image on see-through display devices attached to a plurality of operators in the mixed reality environment, comprising: the image sensing step of sensing a plurality of actuators operated by the plurality of operators by a camera that includes the plurality of operators within a field of view thereof; the actuator position acquisition step of calculating information relating to positions of the actuators associated with a coordinate system of the environment on the basis of the image sensed by the camera; the view point position detection step of detecting a view point position of each of the plurality of operators in the environment of the collaborative operation on the coordinate system of the environment; the progress step of making the collaborative operation virtually progress in accordance with changes in position of the plurality of actuators calculated in the actuator position acquisition step; and the image generation step of outputting a three-dimensional virtual image of a progress result in the progress step viewed from the view point position of each operator detected in the view point position detection step to each see-through display device so as to present the progress result in the progress step to each operator.
the image sensing step of sensing a plurality of actuators operated by the plurality of operators by a camera that includes the plurality of operators within a field of view thereof;
the actuator position acquisition step of calculating information relating to positions of the actuators associated with a coordinate system of the environment on the basis of the image sensed by the camera;
the view point position detection step of detecting a view point position of each of the plurality of operators in the environment of the collaborative operation on the coordinate system of the environment;
the progress step of making the collaborative operation virtually progress in accordance with changes in position of the plurality of actuators calculated in the actuator position acquisition step; and
the image generation step of outputting a three-dimensional virtual image of a progress result in the progress step viewed from the view point position of each operator detected in the view point position detection step to each see-through display device so as to present the progress result in the progress step to each operator.
A mixed reality presentation method for generating a three-dimensional virtual image associated with a collaborative operation to be done by a plurality of operators in a predetermined mixed reality environment, and displaying the generated virtual image on see-through display devices respectively attached to the plurality of operators, comprising: the first image sensing step of capturing an image using a first camera which substantially includes the plurality of operators within a field of view thereof; the first detection step of detecting operation positions of the plurality of operators on the basis of the image sensed by the first camera; the second detection step of detecting a view point position of each operator using a plurality of sensors respectively attached to the plurality of operators; the second image sensing step of sensing a front field of each operator using each of a plurality of second cameras, at least one second camera being attached to each of the plurality of operators; the line of sight calculation step of calculating information associated with a line of sight of each operator on the basis of each of images obtained from the plurality of second cameras; the correction step of correcting the view point position of each operator detected by the sensor on the basis of the line of sight information calculated in the line of sight calculation step, and obtaining the corrected view point position as a position on a coordinate system of the mixed reality environment; the generation step of making the collaborative operation virtually progress on the basis of the operation positions of the individual operators detected in the first detection step, and generating three-dimensional virtual images that represent results of the collaborative operation and are viewed from the view point positions of the plurality of operators; and the step of transferring coordinate positions of the three-dimensional virtual images for the individual operators generated in the generation step in accordance with the individual corrected view point positions obtained in the correction step, and outputting the coordinate-transferred images to the see-through display devices.
the first image sensing step of capturing an image using a first camera which substantially includes the plurality of operators within a field of view thereof;
the first detection step of detecting operation positions of the plurality of operators on the basis of the image sensed by the first camera;
the second detection step of detecting a view point position of each operator using a plurality of sensors respectively attached to the plurality of operators;
the second image sensing step of sensing a front field of each operator using each of a plurality of second cameras, at least one second camera being attached to each of the plurality of operators;
the line of sight calculation step of calculating information associated with a line of sight of each operator on the basis of each of images obtained from the plurality of second cameras;
the correction step of correcting the view point position of each operator detected by the sensor on the basis of the line of sight information calculated in the line of sight calculation step, and obtaining the corrected view point position as a position on a coordinate system of the mixed reality environment;
the generation step of making the collaborative operation virtually progress on the basis of the operation positions of the individual operators detected in the first detection step, and generating three-dimensional virtual images that represent results of the collaborative operation and are viewed from the view point positions of the plurality of operators; and
the step of transferring coordinate positions of the three-dimensional virtual images for the individual operators generated in the generation step in accordance with the individual corrected view point positions obtained in the correction step, and outputting the coordinate-transferred images to the see-through display devices.
A storage medium which stores a program that implements a method of claim 11 .
A storage medium which stores a program that implements a method of claim 12 .
A game apparatus which incorporates a storage medium of claim 11 .
A game apparatus having a mixed reality presentation apparatus of claim 1 .
A position/posture detection apparatus for detecting an operation position of an operator so as to generate a three-dimensional virtual image that represents an operation done by the operator in a predetermined mixed reality environment, comprising: a position/posture sensor for measuring a three-dimensional position and posture of the operator to output an operator's position and posture signal; a camera sensing images of a first plurality of markers arranged at known positions in the environment; detection means for processing an image signal from said camera, tracking a marker of the first plurality of markers, and detecting a coordinate value of the tracked marker in a coordinate system; and calculation means for calculating a portion-position and -posture representing a position and posture of the operating portion, on the basis of the coordinate value of the tracked marker detected by said detection means and the operator's position and posture signal outputted from the position/posture sensor.
a position/posture sensor for measuring a three-dimensional position and posture of the operator to output an operator's position and posture signal;
a camera sensing images of a first plurality of markers arranged at known positions in the environment;
detection means for processing an image signal from said camera, tracking a marker of the first plurality of markers, and detecting a coordinate value of the tracked marker in a coordinate system; and
calculation means for calculating a portion-position and -posture representing a position and posture of the operating portion, on the basis of the coordinate value of the tracked marker detected by said detection means and the operator's position and posture signal outputted from the position/posture sensor.
The apparatus according to claim 17 , wherein a distance between one marker and another marker of the plurality of markers in a direction crossing in front of the operator is set to be larger as the markers are farther from the operator.
The apparatus according to claim 17 , wherein where a plurality of operators perform a collaborative operation, markers for one operato
[Truncated to input limit]