Patent Description:
A virtual reality (VR) system generates an immersive virtual environment for a user. For example, the immersive environment can be three-dimensional (3D) and can include multiple virtual objects with which the user may interact. The user can experience the immersive virtual environment via various display devices such as, for example, a helmet or other head mounted device including a display, glasses, or goggles that a user looks through when viewing a display device.

Once immersed in the 3D virtual environment, the user can move through the virtual environment and move to other areas of the virtual environment, through physical movement and/or manipulation of an electronic device to interact with the virtual environment and personalize interaction with the virtual environment. For example, the VR system can include sensors to track the user's head or body movement. Additionally, the VR system can include a handheld device that includes sensors, and other such electronic components. The user can use this handheld device to navigate in and interact with the virtual environment.

<NPL> discloses a method to align components in a VR environment.

<NPL> discloses a further method to align components in a VR environment.

The proposed solution relates to a method as stated in claim <NUM> of the accompanying claim set, a computer program product as stated in claim <NUM> and an electronic apparatus as stated in claim <NUM>. Some aspects perform a method that includes detecting a first input from a handheld controller of a virtual reality system, responsive to detecting the first input, instructing a user to orient a handheld controller in a designated direction, detecting a second input from the handheld controller; and responsive to detecting the second input, storing alignment data representative of an alignment of the handheld controller.

Some aspects include a computer program product including a nontransitory storage medium, the computer program product including code that, when executed by processing circuitry of a virtual reality system configured to produce a virtual reality environment, causes the processing circuitry to perform a method that includes detecting a first input from a handheld controller of the virtual reality system, responsive to detecting the first input: communicatively coupling the handheld controller to a head-mounted display of the virtual reality system and instructing a user to orient a handheld controller in a designated direction, detecting a second input from the handheld controller, and responsive to detecting the second input, storing alignment data representative of an alignment of the handheld controller.

Some aspects include an electronic apparatus configured to produce a virtual reality environment, the electronic apparatus including memory and controlling circuitry coupled to the memory, the controlling circuitry being configured to: detect a first input from a handheld controller of the virtual reality system, responsive to detecting the first input: communicatively couple the handheld controller to a head-mounted display of the virtual reality system via a wireless signal, and instruct a user to orient a handheld controller in a designated direction, detect a second input from the handheld controller, and responsive to detecting the second input: determine a controller orientation of the handheld controller, determine a display orientation of a head-mounted display; and store alignment data representative of an alignment of the handheld controller.

Some aspects perform a method that includes detecting an input from a handheld controller of a virtual reality system, communicatively coupling the handheld controller to a head-mounted display of the virtual reality system in response to the input, displaying in a virtual reality scene a first symbol associated with the handheld controller, and a second symbol associated with the head-mounted display in response to communicatively coupling the handheld controller to the head-mounted display, and storing data representative of alignment of the handheld controller to the head-mounted display when the first symbol at least partially overlaps with the second symbol in the virtual reality scene.

Some aspects perform a method that includes detecting an input from a handheld controller of a virtual reality system, communicatively coupling the handheld controller to a head-mounted display of the virtual reality system in response to the input, instructing a user to orient the head-mounted display in a designated direction while the input is detected, displaying in a virtual reality scene an indicator representing the orientation of the head-mounted display, and storing data representative of an alignment of the head-mounted display when the input is no longer detected.

According to one aspect a computer-readable medium has recorded and embodied thereon instructions that, when executed by a processor of a computer system, cause the computer system to perform any of the methods and processes disclosed herein.

Reference will now be made in detail to non-limiting examples of this disclosure, examples of which are illustrated in the accompanying drawings. The examples are described below by referring to the drawings, wherein like reference numerals refer to like elements. When like reference numerals are shown, corresponding description(s) are not repeated and the interested reader is referred to the previously discussed figure(s) for a description of the like element(s).

Turning to <FIG>, a block diagram of an example virtual reality (VR) system <NUM> for creating and interacting with a three-dimensional (3D) VR environment in accordance with the teachings of this disclosure is shown. In general, the system <NUM> provides the 3D VR environment and VR content that enable a user to access, view, use and/or interact with the examples described herein. The system <NUM> can provide the user with options for accessing the content, applications, virtual objects, and VR controls using, for example, eye gaze and/or movements within the VR environment. The example VR system <NUM> of <FIG> includes a head-mounted display (HMD) <NUM> and a handheld controller <NUM>. Also shown is a user <NUM> who is wearing the HMD <NUM> and holding the handheld controller <NUM>.

Using the examples disclosed herein, the user <NUM> can align the HMD <NUM> and the controller <NUM> with each other. For example, the alignment can be performed during an initialization process or to correct for drifting apart between the HMD <NUM> and the controller <NUM> over time. By aligning the HMD <NUM> and the controller <NUM> with each other, the location at which a representation of the controller <NUM> in the VR environment within the HMD <NUM> is aligned with the position of the actual controller <NUM> relative to the HMD <NUM>. Additionally, the alignment of one or both of the HMD <NUM> and the controller <NUM> can drift relative to a VR scene. Embodiments disclosed herein can be used to re-align one or both of the HMD <NUM> and the controller <NUM> to a VR scene.

In some embodiments, the HMD <NUM> and the controller <NUM> each measure movement and/or rotational changes independently (e.g., using separate inertial motion units). Using these measurement, the HMD <NUM> and controller <NUM> each independently track their own orientations and/or poses. In some embodiments, the pose includes a position and an orientation. For example, position can be represented by three degrees of freedom (3DOF) including forward/backward (X coordinate), right/left (Y coordinate), and up/down (Z coordinate) and orientation can include three rotational components such as pitch (i.e., rotation about the Y-axis), yaw (i.e., rotation about the Z-axis), and roll (i.e., rotation about the X-axis). Accordingly, in some embodiments, pose is represented by six degrees of freedom (6DOF), including forward/backward (X coordinate), right/left (Y coordinate), up/down (Z coordinate), pitch (i.e., rotation about the Y-axis), yaw (i.e., rotation about the Z-axis), and roll (i.e., rotation about the X axis). Although many of the example herein are described in terms of alignment of poses of the HMD <NUM> and the controller <NUM>, other embodiments are possible as well. For example, some embodiments align one or the other of position or orientation of the HMD <NUM> and the controller <NUM>. Embodiments of the technology described herein align the controller <NUM> and/or the HMD <NUM> with each other and/or with a VR scene.

Over time, as the HMD <NUM> and the controller <NUM> independently track their poses, numerical discrepancies can accumulate causing the alignment of the HMD <NUM> and the controller <NUM> to drift. For example, after time, the pose of the controller <NUM> relative to the HMD <NUM> in real space may be different than the pose of the virtual representation of the controller <NUM> relative to the virtual representation of the HMD <NUM> in the VR environment.

Thus, the user <NUM> may think they are pointing the controller <NUM> in a particular direction relative to looking in that direction (e.g., looking straight ahead with their eyes, and/or by pointing their HMD <NUM> straight ahead). However, in actuality, the user <NUM> may be looking and pointing in slightly different directions. This effect is referred to as drift, and may cause the user <NUM> to have some difficulty interacting with virtual objects in a VR environment. Accordingly, examples are disclosed herein that enable a user to easily align the HMD <NUM> to the controller <NUM> during, for example, VR application startup and/or later during operation. An alignment procedure adjusts the pose of the controller <NUM> in the HMD <NUM> to align with the pose of the HMD <NUM>. In some examples, the alignment takes place together with the communicative coupling of the HMD <NUM> and controller <NUM> via, for example, a short-range wireless signal, such as Bluetooth®, Wi-Fi®, etc. In this way, a single actuation of an input device (e.g., pressing a button into an activated state) can be used to trigger communicative coupling and alignment of the HMD <NUM> and the controller <NUM>. In some embodiments, the alignment procedure is also performed after the communicative coupling. For example, in some embodiments, the alignment procedure is performed multiple times during a VR session to perform an initial alignment and/or then, later, for re-alignment. While for simplicity, references are made herein to buttons and button presses, any other input devices may be used to trigger communicative coupling and alignment according to this disclosure. For example, a touch or gesture on a trackpad, etc. or any other detectable input.

An example alignment process includes detecting pressing of a button of a first component (e.g., the handheld controller <NUM>), communicatively coupling the controller <NUM> to a second component (e.g., the HMD <NUM>) in response to the button pressing, displaying in a VR scene a first symbol (e.g., a target <NUM>) associated with the HMD <NUM>, and a second symbol (e.g., a cursor <NUM>) associated with the controller <NUM> in response to the communicative coupling. In some examples, the cursor <NUM> can be represented as a laser pointer or a ray associated with (or emitted from) the handheld controller. The user is instructed to orient the HMD and/or their eyes toward the first symbol, and the point the handheld controller <NUM> at the first symbol. And storing data representative of alignment (also can be referred to as alignment data) of the controller <NUM> to the HMD <NUM> when the target <NUM> and cursor <NUM> overlap in the VR scene. In some examples, the VR scene changes while the target <NUM> and the cursor <NUM> are displayed in the changing VR scene.

In some examples, the alignment data represents a correction in at least one component of pose of the controller <NUM> relative to the HMD <NUM>. For example, when a user attempts to point both the controller <NUM> and the HMD <NUM> in a designated direction (e.g., straight forward, to the right, etc.); the controller <NUM> may be pointed in a different direction than the HMD <NUM>. Accordingly, example alignment data represents an offset to be applied to one or more components of a 3DoF or a 6DoF reference model of the controller <NUM> that is maintained by the HMD <NUM> or another component of the system <NUM>. Alignment data may be applied by adding the values of the alignment data to, for example, 3DoF or 6DoF coordinates such that a measured pose is changed to a corrected pose. For example, alignment data could represent a difference between the pose of an HMD and a handheld controller. Alignment data may similarly be applied between HMD pose and an origin or designated alignment point of a VR scene (see <FIG>).

In some examples, the button may need to be pressed and held. In some instances, the roles of the controller <NUM> and the HMD <NUM> are reversed, e.g., a button is pressed on the HMD <NUM> rather than on the controller <NUM>. Or, the buttons need to be pressed and held on both the handheld controller <NUM> and the HMD <NUM>.

The pose (e.g., in 6DoF, or in 3DoF) of the HMD <NUM> may be determined using emitters or images (one of which is designated at reference numeral <NUM>) that can be used to determine the pose of the HMD <NUM>, and/or using sensors or cameras (one of which is designated at reference numeral <NUM>) that can be used to determine the orientation of the HMD <NUM>. The HMD <NUM> can include a camera <NUM> (see <FIG>) that can be used to sense the emitters/images <NUM>, and/or an emitter <NUM> (see <FIG>) for sensing by the sensors/cameras <NUM>. Any number and/or type(s) of emitters, images, sensors and/or cameras, and any method of using the same to determine the pose of the HMD <NUM> may be used. The pose of the handheld controller <NUM> can, likewise, be determined using the sensor/camera <NUM>, and an emitter <NUM> (see <FIG>) of the handheld controller <NUM>. Determination of pose, including positions and/or orientation, may be performed by the HMD <NUM> or another device (e.g., any of devices <NUM>-<NUM> discussed below) of the VR system <NUM>, and/or may be implemented by the example computing devices P00 and P50 of <FIG>.

As shown in <FIG>, the example VR system <NUM> includes any number of computing and/or electronic devices that can exchange data over a network <NUM>. The devices may represent clients or servers, and can communicate via the network <NUM> or any other additional and/or alternative network(s). Example client devices include, but are not limited to, a mobile device <NUM> (e.g., a smartphone, a personal digital assistant, a portable media player, etc.), an electronic tablet (not shown), a laptop or netbook <NUM>, a camera (not shown), the HMD <NUM>, a desktop computer <NUM>, the VR handheld controller <NUM>, a gaming device (not shown), and any other electronic or computing devices that can communicate using the network <NUM> or other network(s) with other computing or electronic devices or systems, or that may be used to access VR content or operate within a VR environment. The devices <NUM>, <NUM>, and <NUM>-<NUM> may represent client or server devices. The devices <NUM>, <NUM>, and <NUM>-<NUM> can execute a client operating system, and one or more client applications that can access, render, provide, or display VR content on a display device included in or in conjunction with each respective device <NUM>, <NUM>, and <NUM>-<NUM>.

In some examples, the mobile device <NUM> can be placed, located, or otherwise implemented or operated in conjunction within the HMD <NUM> to provide a display that can be used as the screen for the HMD <NUM>. The mobile device <NUM> can additionally or alternatively include hardware and/or software for executing VR applications.

<FIG> is a schematic diagram of an example HMD <NUM> that may be used, among other things, to implement the example HMD <NUM> of <FIG>. The example HMD <NUM> of <FIG> includes an optics module <NUM>, a processor module <NUM>, and a sensing module <NUM>.

The optics module <NUM> includes, but is not limited to, lenses, mirrors, coatings, apertures, etc. that enable a wearer of the HMD <NUM> to view a display <NUM>. The example display <NUM> displays side-by-side images and/or videos for respective eyes of a wearer of the HMD <NUM>. In other examples, two (or more) of the display <NUM> are used to collectively display, respectively, the side-by-side images and/or videos.

To control the HMD <NUM>, the example processor module <NUM> of <FIG> includes a processor <NUM> in the form of a microcontroller, a central processing unit (CPU) and/or a graphics processing unit (GPU) programmed or configured to execute machine-readable instructions stored in a memory <NUM>. In some examples, the instructions, when executed, cause the processor <NUM> to, among other things, implement VR applications, determine the pose of the HMD <NUM> and/or the controller <NUM>, and/or perform an alignment procedure to align the HMD <NUM> with the controller <NUM>.

In some examples, alignment data and/or parameters are stored in the memory <NUM>. In some embodiments, the alignment data represents a difference or offset between a first direction in which the HMD <NUM> is oriented, and a second direction in which a handheld controller is oriented during an alignment procedure in which it is assumed that the handheld controller is being held with the intention of being aligned with the HMD <NUM>. For example, the alignment data can represent an offset between a longitudinal axis of the controller relative to a direction a front surface of the HMD <NUM> (e.g., a vector normal to the surface of the HMD <NUM>). Thus, example alignment data could indicate that a user unintentionally points the controller <NUM> to the right by <NUM> degree and upward by <NUM> degrees when pointing the controller and looking at the same point in a VR scene. The HMD <NUM> or another device <NUM>-<NUM> executing a VR application can use the alignment data to correct the measured orientation of the handheld controller so that it is pointing leftward by <NUM> degree and downward by <NUM> degrees so the handheld controller is now more accurately pointing at what the user is looking at. In some examples, what a wearer is looking at is determined by the position of the HMD. In other examples, what a wearer is looking at is determined using eye tracking. The alignment data and/or parameters can also compensate for any drifting apart of a target of the HMD and a target of a handheld controller due to, for example, accumulating numerical discrepancies, mechanical changes over time, user fatigue, etc. In some embodiments, the alignment data includes data that can be used to align the poses of the controller and the HMD (e.g., offsets for both orientation and position).

In some embodiments, the alignment data represents a reference controller coordinate system for the controller and/or a reference HMD coordinate system. For example, the reference controller coordinate system may store an X-axis that is aligned with the longitudinal axis of the controller user is aiming the controller forward as instructed during the alignment procedure, while the reference HMD coordinate system may store an X-axis that is aligned with the direction in which the front surface of the HMD is facing when the user is facing forward as instructed during the alignment procedure.

To communicatively couple the example HMD <NUM> to a handheld controller, such as the controller <NUM> of <FIG>, the example HMD <NUM> of <FIG> includes a communication module <NUM>. An example communication module <NUM> is a short-range wireless communication module in accordance with the Bluetooth standard. However, other communication signals and/or protocols, such as Wi-Fi or universal serial bus (USB) may be used.

To help in the determination of the orientation of the HMD <NUM>, the example sensing module <NUM> of <FIG> can include a forward-facing sensor or camera <NUM> that can be used to determine the orientation of the HMD <NUM> using the example emitters/images <NUM> of <FIG>. Additionally or alternatively, the example sensing module <NUM> can include an emitter <NUM> that other devices of a VR system, such as the example devices <NUM>-<NUM>, can use to determine the orientation of the HMD <NUM>. Some embodiments include an inertial measurement unit (IMU) <NUM> that can be used to determine the orientation and/or detect motion of the HMD <NUM>. In various embodiments, the IMU <NUM> includes various types of sensors such as, for example, an accelerometer, a gyroscope, a magnetometer, and other such sensors. A position and orientation of the HMD <NUM> may be detected and tracked based on data provided by the sensors included in the IMU <NUM>. The detected positon and orientation of the HMD <NUM> may allow the system to in turn, detect and track the user's head gaze direction and movement.

To determine the direction in which a wearer is gazing, the example processor module <NUM> includes an eye tracking module <NUM>. Using any number and/or type(s) of algorithms, methods and logic, the of <FIG> processes images of one or more of the wearer's eyes to determine the direction in which the wearer is gazing.

To enable a user to control or operate the HMD <NUM>, the example processor module <NUM> includes an input device <NUM>. Example of the input device <NUM> include buttons and other user-actuatable controls. The example input device <NUM> of the HMD <NUM> can be operated by a user to initiate communicative coupling of the HMD <NUM> to a handheld controller (not shown) via the communication modules <NUM> and <NUM> (see <FIG>), and to initiate alignment of the HMD <NUM> and the handheld controller.

The modules <NUM>, <NUM> and <NUM>, and the elements therein may be implemented using any number and/or type(s) of components and/or machine-readable instructions. In some examples, the processor module <NUM> is implemented using a mobile device such as a mobile phone or smartphone that can be communicatively and/or physically coupled to the HMD <NUM>. The sensing module <NUM> The sensing module <NUM> may additionally be implemented by the mobile device.

<FIG> illustrates an example handheld controller <NUM> that may be used to implement the example handheld controller <NUM> of <FIG>. To control the handheld controller <NUM> of <FIG>, the example handheld controller <NUM> includes a processor <NUM> in, for example, the form of a CPU or a microcontroller programmed or configured to execute machine-readable instructions stored in a memory <NUM>. In some examples, the instructions, when executed, cause the processor <NUM> to, among other things, interact with the HMDs <NUM>, <NUM>, and operate a communication module <NUM>.

To communicatively couple the example handheld controller <NUM> to the HMDs <NUM>, <NUM>, the example handheld controller <NUM> of <FIG> includes the communication module <NUM>. An example communication module <NUM> is a short-range wireless communication module in accordance with the Bluetooth standard. However, other communication signals and/or protocols, such as Wi-Fi, USB, etc. may be used.

To help in the determination of the location of the handheld controller <NUM>, the example handheld controller <NUM> of <FIG> includes an emitter <NUM> that other devices of a VR system, such as the example device <NUM>, <NUM> and <NUM>-<NUM>, can use to determine the orientation of the handheld controller <NUM>. In some examples, reflectors may be implemented to reflect light emitted by other devices to determine the orientation of the controller <NUM>. Additionally or alternatively, some embodiments of the handheld controller <NUM> include an inertial measurement unit (IMU) <NUM> to track the orientation and/or motion of the handheld controller <NUM>. The IMU <NUM> may be similar to the IMU <NUM> of the HMD <NUM>.

To enable a user to control or operate the handheld controller <NUM>, the example handheld controller <NUM> includes an input device <NUM> that can be actuated by a user. An example of the input device <NUM> is a button that the user can press to actuate. The example input device <NUM> of the handheld controller <NUM> can be operated by a user to initiate communicative coupling of the handheld controller <NUM> and an HMD via the communication modules <NUM> and <NUM>, and to initiate alignment of the handheld controller <NUM> and the HMD.

To provide illumination for the sensor/camera <NUM>, and/or others of the devices <NUM>-<NUM> that can determine the location of the handheld controller <NUM>, the example handheld controller <NUM> includes an emitter <NUM>. The emitter <NUM> may emit, for example, visible light and/or infrared light.

<FIG> is a flowchart of an example process <NUM> that may, for example, be implemented as machine-readable instructions carried out by one or more processors, such as the example processors of <FIG>, to implement the example HMDs disclosed herein. The example method <NUM> will be described with reference to the example HMD <NUM>, but may be used to implement other HMDs. The example method <NUM> of <FIG> includes the processor <NUM> starting a VR mode on the HMD <NUM> (block <NUM>), and starting the communication module <NUM> in, for example, a Bluetooth discovery mode (block <NUM>).

If a user actuates an input device such as the input device <NUM> (e.g., by pressing and holding a button) (block <NUM>), the communication module <NUM> scans for nearby handheld controllers (block <NUM>). If a recently connected handheld controller is found (block <NUM>), the communication module <NUM> attempts to connect with that handheld controller (block <NUM>). In some examples, the input device <NUM> does not need to be actuated and, instead, the HMD <NUM>, while in VR mode, remains in a listening mode actively listening for Bluetooth queries from handheld controllers.

If communication is successfully obtained (block <NUM>), the processor <NUM> initiates display of a target <NUM> (see <FIG>) in a portion <NUM> (see <FIG>) of a larger VR scene <NUM> (see <FIG>) on the display <NUM> (within an HMD), and indicates to the user to look at the target <NUM>, so that the HMD and the controller are pointed in the same direction, and point the connected handheld controller at the target <NUM> (block <NUM>). The VR portion <NUM> of the VR scene <NUM>, the target <NUM>, and the cursor <NUM> are displayed in the HMD. When a cursor <NUM> (see <FIG>) under control of a handheld controller aligns with (e.g., overlaps) with the target <NUM>, the input device <NUM> (if actuated) or the input device <NUM> (if actuated) is released to indicate that the target <NUM> and cursor <NUM> are aligned (block <NUM>), the position and orientation of the HMD and handheld controller are saved as alignment parameters in, for instance, the memory <NUM> (block <NUM>), and VR applications are started (block <NUM>). In some examples, the alignment data represents a correction in at least one orientation component of the controller <NUM> relative to the HMD <NUM>. For example, when the user indicates that both the controller and the HMD are pointed in the same direction, the controller <NUM> may actually be pointed in a different direction than the HMD <NUM>. Accordingly, example alignment data can include data representing an offset to be applied to one or more components of the pose of the controller <NUM> or HMD <NUM> (e.g., in any of the components of a 3DoF or a 6DoF reference coordinate system). Alignment parameters may be determined, for example, using the emitters <NUM> and <NUM>, the sensor/camera <NUM>, and/or the eye tracking module <NUM>. Control exits from the example method <NUM> until a new communicative coupling and/or alignment needs to be performed, which may be triggered by a user request, an expiration of a time period, an event occurring in the VR application, or any other type of event. Alignment data, such as alignment parameters, are applied subsequently when determining the position and/or orientation of the handheld controller, which may then be used to determine a target location (or object) at which the connected handheld controller is pointed. Due to application of the alignment data, the determined position and orientation can be more accurately determined. Thus, in some implementations, the target location (or object) is also determined more accurately based on application of the alignment data. In other words, the representation of the handheld controller in the VR environment can more accurately point at the user's intended target by using the alignment data to correct previous discrepancies or drifting in determining the orientation and position of the handheld controller.

Returning to block <NUM>, if the connection is not able to be established, control exits from the example method <NUM> until a new attempt to communicatively couple and/or align the HMD to a handheld controller is initiated.

Returning to block <NUM>, if a previously connected handheld controller is not found, the communication module <NUM> searches for nearby handheld controllers (block <NUM>). If a nearby handheld controller is not found (block <NUM>), control exits from the example method <NUM> until a new attempt to communicatively couple and/or align the HMD to a handheld controller is initiated. Otherwise, the communication module <NUM> attempts to connect with the closest handheld controller (block <NUM>), and control proceeds to block <NUM>.

<FIG> and, similarly, <FIG> are shown from the perspective of a <NUM>rd person viewing a VR environment from within that VR environment. The person depicted in these figures is in this VR environment with the <NUM>rd person, and is as seen by the <NUM>rd person.

<FIG> is a flowchart of an example process <NUM> that may, for example, be implemented as machine-readable instructions carried out by one or more processors, such as the example processors of <FIG>, to implement the example handheld controllers disclosed herein. The example method <NUM> will be described with reference to the example handheld controller <NUM>, but may be used to implement other handheld controllers. The example method <NUM> of <FIG> includes waiting for a user to actuate and hold the example input device <NUM> (block <NUM>). When the input device <NUM> is actuated and held (block <NUM>), the communication module <NUM> is activated in, for example, a Bluetooth discovery mode (block <NUM>), and the communication module <NUM> scans for nearby HMDs (block <NUM>). If a recently connected HMD is found (block <NUM>), the communication module <NUM> attempts to connect with that HMD (block <NUM>).

If a connection is established (block <NUM>), the processor <NUM> waits for release of the actuated input device <NUM> (block <NUM>). When determined that the input device <NUM> has been released (e.g., a button is released), the processor <NUM> notifies the HMD <NUM> or other VR application implementing the alignment (block <NUM>), control exits from the method <NUM> until a new attempt to communicatively couple and/or align the controller to an HMD is initiated.

Returning to block <NUM>, if the connection is not established (block <NUM>), control exits from the example method <NUM> until a new attempt to communicatively couple and/or align the controller to an HMD is initiated.

Returning to block <NUM>, if a previously connected HMD is not found, the communication module <NUM> searches for nearby HMDs (block <NUM>). If a nearby HMD is not found (block <NUM>), control exits from the example method <NUM> until a new attempt to communicatively couple and/or align the controller to an HMD is initiated. Otherwise, the controller attempts to connect with the closest HMD, and control proceeds to block <NUM>.

Turning to <FIG>, a VR scene <NUM> that includes the portion <NUM> of <FIG> is shown. As shown, the target <NUM> and the cursor <NUM> are overlaid on the VR scene <NUM> and, if a user moves their gaze or body, the target <NUM> and cursor <NUM> will be displayed on a different portion of the VR scene <NUM>.

<FIG> sequentially illustrate an example alignment of an HMD to a VR home screen (or other VR scene element, e.g., object). <FIG> illustrates an example VR scene <NUM> shown in an HMD, such as the HMD <NUM>. The example VR scene <NUM> includes a home screen <NUM> comprising a plurality of buttons, one of which is designated at reference numeral <NUM>. Because of drift, the centerline <NUM> of the home screen <NUM> is offset with respect to the centerline <NUM> of the VR scene <NUM>, which is also the centerline of the HMD. To align the HMD to the home screen <NUM>, a user may press and hold a button on the handheld controller. When instructed, the wearer of the HMD looks (e.g., with their eyes and/or by pointing their HMD) at the center <NUM> of the home screen <NUM> and releases the button. Based on the alignment data captured by the HMD, the home screen <NUM> is snapped back to the center of the VR scene (also HMD), as shown in <FIG>. In the example of <FIG>, only a side-to-side offset is depicted for clarify of explanation. Other offsets may be corrected including ones, for example, in any of one or more coordinates of a 3DoF or 6DoF reference module. Such offsets may result in rotations, twists, etc..

<FIG> is a flowchart of an example process <NUM> that may, for example, be implemented as machine-readable instructions carried out by one or more processors, such as the example processors of <FIG>, to implement the example HMDs disclosed herein. The example method <NUM> will be described with reference to the example HMD <NUM>, but may be used to implement other HMDs. The example method of method <NUM> may be used to carry out the example alignment of an HMD and a VR scene as discussed above in connection with <FIG>. Portions of the example method <NUM> are similar to the example method <NUM> of <FIG>. Thus, descriptions of the identical portions of <FIG> and <FIG> are not repeated here. Instead, interested readers are referred back to the descriptions of <FIG> for the identical portions.

At block <NUM>, the user, while wearing the HMD <NUM>, is instructed to look at the center of a VR element, such as a VR object, home screen, etc. and then stop actuating the input device <NUM> (e.g., by releasing a button). When it is determined that the input device <NUM> is no longer being actuated (block <NUM>), alignment data representing an offset between the direction the HMD <NUM> was pointed and the location of the centerline of the VR element is recorded and stored (block <NUM>). At block <NUM>, VR application(s) are started and their position(s) are corrected using the stored alignment data. For example, as shown in <FIG>, a home screen consisting of a set of buttons is shifted leftward to align with the center of the VR screen as defined by the alignment of the HMD <NUM> (block <NUM>). Control then exits from the example method <NUM>.

<FIG> is a flowchart of an example process <NUM> that may, for example, be implemented as machine-readable instructions carried out by one or more processors, such as the example processors of <FIG>, to implement the example HMDs disclosed herein. The example method <NUM> will be described with reference to the example HMD <NUM>, but may be used to implement other HMDs. The example method of method <NUM> may be used to align a handheld controller to an HMD. Portions of the example method <NUM> are similar to the example method <NUM> of <FIG>. Thus, descriptions of the identical portions of <FIG> and <FIG> are not repeated here. Instead, interested readers are referred back to the descriptions of <FIG> for the identical portions.

At block <NUM>, the user <NUM> (see <FIG>) is instructed to look in a designated direction (e.g., straight ahead, to the right, etc.) and simultaneously point the controller <NUM> in that direction at the same location <NUM> (see <FIG>), and then stop actuating the input device <NUM> (e.g., by releasing a depressed button). When it is determined that the input device <NUM> is no longer being actuated (block <NUM>), alignment data representing an offset between the pose of the HMD <NUM> and the pose of the handheld controller is stored (block <NUM>). In some embodiments, the alignment data represents an offset between the orientation of the HMD <NUM> and the orientation of the handheld controller. At block <NUM>, VR application(s) are started and the pose of the controller relative to the HMD <NUM> is changed based on the alignment data, so they are aligned with each other.

The examples of <FIG>, <FIG>, and <FIG> may be combined to simultaneously align an HMD to a VR element (e.g., home screen), and to align a handheld controller to the HMD. In such an example, the user would be instructed to look toward and point toward the center of a VR element.

In <FIG>, <FIG>, and <FIG>, the alignment parameters are shown as being saved after the button is released. However, in some examples, the alignment parameters can be stored, possibly temporarily, as the HMD is waiting for the button to be released.

One or more of the elements and interfaces disclosed herein may be duplicated, implemented in the parallel, implemented in the singular, combined, divided, rearranged, omitted, eliminated, and/or implemented in any other way. Further, any of the disclosed elements and interfaces may be implemented by the example processor platforms P00 and P50 of <FIG>, and/or one or more circuit(s), programmable processor(s), fuses, application-specific integrated circuit(s) (ASIC(s)), programmable logic device(s) (PLD(s)), field-programmable logic device(s) (FPLD(s)), and/or field-programmable gate array(s) (FPGA(s)), etc. Any of the elements and interfaces disclosed herein may, for example, be implemented as machine-readable instructions carried out by one or more processors. A processor, a controller and/or any other suitable processing device such as those shown in <FIG> may be used, configured, and/or programmed to execute and/or carry out the examples disclosed herein. For example, any of these interfaces and elements may be embodied in program code and/or machine-readable instructions stored on a tangible and/or non-transitory computer-readable medium accessible by a processor, a computer and/or other machine having a processor, such as that discussed below in connection with <FIG>. Machine-readable instructions comprise, for example, instructions that cause a processor, a computer and/or a machine having a processor to perform one or more particular processes. The order of execution of methods may be changed, and/or one or more of the blocks and/or interactions described may be changed, eliminated, sub-divided, or combined. Additionally, they may be carried out sequentially and/or carried out in parallel by, for example, separate processing threads, processors, devices, discrete logic, circuits, etc..

The example methods disclosed herein may, for example, be implemented as machine-readable instructions carried out by one or more processors. A processor, a controller and/or any other suitable processing device such as that shown in <FIG> may be used, configured and/or programmed to execute and/or carry out the example methods. For example, they may be embodied in program code and/or machine-readable instructions stored on a tangible and/or non-transitory computer-readable medium accessible by a processor, a computer and/or other machine having a processor, such as that discussed below in connection with <FIG>. Machine-readable instructions comprise, for example, instructions that cause a processor, a computer, and/or a machine having a processor to perform one or more particular processes. Many other methods of implementing the example methods may be employed. For example, the order of execution may be changed, and/or one or more of the blocks and/or interactions described may be changed, eliminated, sub-divided, or combined. Additionally, any or the entire example methods may be carried out sequentially and/or carried out in parallel by, for example, separate processing threads, processors, devices, discrete logic, circuits, etc..

As used herein, the terms "computer-readable medium" and "machine-readable medium" are expressly defined to include any type of readable medium and to expressly exclude propagating signals. Example computer-readable medium and machine-readable medium include, but are not limited to, one or any combination of a volatile and/or non-volatile memory, a volatile and/or non-volatile memory device, a compact disc (CD), a digital versatile disc (DVD), a read-only memory (ROM), a random-access memory (RAM), a programmable ROM (PROM), an electronically-programmable ROM (EPROM), an electronically-erasable PROM (EEPROM), an optical storage disk, an optical storage device, a magnetic storage disk, a magnetic storage device, a cache, and/or any other storage media in which information is stored for any duration (e.g., for extended time periods, permanently, brief instances, for temporarily buffering, and/or for caching of the information) and that can be accessed by a processor, a computer and/or other machine having a processor.

Returning to <FIG>, the VR system <NUM> may include any number of VR content systems <NUM> storing content and/or VR software modules (e.g., in the form of VR applications <NUM>) that can generate, modify, and/or execute VR scenes. In some examples, the devices <NUM>, <NUM>, and <NUM>-<NUM> and the VR content system <NUM> include one or more processors and one or more memory devices, which can execute a client operating system and one or more client applications. The HMD <NUM>, the controller <NUM>, the other devices <NUM>-<NUM> or the VR content system <NUM> may be implemented by the example computing devices P00 and P50 of <FIG>.

The VR applications <NUM> can be configured to execute on any or all of the devices <NUM>, <NUM>, and <NUM>-<NUM>. The HMD <NUM> can be connected to devices <NUM>-<NUM> to access VR content on VR content system <NUM>, for example. Device <NUM>-<NUM> can be connected (wired or wirelessly) to HMD <NUM>, which can provide VR content for display. A VR system can be HMD <NUM> alone, or a combination of device <NUM>-<NUM> and HMD <NUM>.

The HMD <NUM> may represent a VR HMD, glasses, an eyepiece, or any other wearable device capable of displaying VR content. In operation, the HMD <NUM> can execute a VR application <NUM> that can playback received, rendered and/or processed images for a user. In some instances, the VR application <NUM> can be hosted by one or more of the devices <NUM>-<NUM>.

In some implementations, one or more content servers (e.g., VR content system <NUM>) and one or more computer-readable storage devices can communicate with the computing devices <NUM> and <NUM>-<NUM> using the network <NUM> to provide VR content to the devices <NUM> and <NUM>-<NUM>.

In some implementations, the mobile device <NUM> can execute the VR application <NUM> and provide the content for the VR environment. In some implementations, the laptop computing device <NUM> can execute the VR application <NUM> and can provide content from one or more content servers (e.g., VR content server <NUM>). The one or more content servers and one or more computer-readable storage devices can communicate with the mobile device <NUM> and/or laptop computing device <NUM> using the network <NUM> to provide content for display in HMD <NUM>.

In the event that the HMD <NUM> is wirelessly coupled to device <NUM> or device <NUM>, the coupling may include use of any wireless communication protocol. A non-exhaustive list of wireless communication protocols that may be used individually or in combination includes, but is not limited to, the Institute of Electrical and Electronics Engineers (IEEE®) family of <NUM>. x standards a. Wi-Fi or wireless local area network (WLAN), Bluetooth, Transmission Control Protocol/Internet Protocol (TCP/IP), a satellite data network, a cellular data network, a Wi-Fi hotspot, the Internet, and a wireless wide area network (WWAN).

In the event that the HMD <NUM> is electrically coupled to device <NUM> or <NUM>, a cable with an appropriate connector on either end for plugging into device <NUM> or <NUM> can be used. A non-exhaustive list of wired communication protocols that may be used individually or in combination includes, but is not limited to, IEEE <NUM>. 3x (Ethernet), a powerline network, the Internet, a coaxial cable data network, a fiber optic data network, a broadband or a dialup modem over a telephone network, a private communications network (e.g., a private local area network (LAN), a leased line, etc.).

A cable can include a Universal Serial Bus (USB) connector on both ends. The USB connectors can be the same USB type connector or the USB connectors can each be a different type of USB connector. The various types of USB connectors can include, but are not limited to, USB A-type connectors, USB B-type connectors, micro-USB A connectors, micro-USB B connectors, micro-USB AB connectors, USB five pin Mini-b connectors, USB four pin Mini-b connectors, USB <NUM> A-type connectors, USB <NUM> B-type connectors, USB <NUM> Micro B connectors, and USB C-type connectors. Similarly, the electrical coupling can include a cable with an appropriate connector on either end for plugging into the HMD <NUM> and device <NUM> or device <NUM>. For example, the cable can include a USB connector on both ends. The USB connectors can be the same USB type connector or the USB connectors can each be a different type of USB connector. Either end of a cable used to couple device <NUM> or <NUM> to HMD <NUM> may be fixedly connected to device <NUM> or <NUM> and/or HMD <NUM>.

<FIG> shows an example of a generic computer device P00 and a generic mobile computer device P50, which may be used with the techniques described here. Computing device P00 is intended to represent various forms of digital computers, such as laptops, desktops, tablets, workstations, personal digital assistants, televisions, servers, blade servers, mainframes, and other appropriate computing devices. Computing device P50 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, and other similar computing devices.

Computing device P00 includes a processor P02, memory P04, a storage device P06, a high-speed interface P08 connecting to memory P04 and high-speed expansion ports P10, and a low speed interface P12 connecting to low speed bus P14 and storage device P06. The processor P02 can be a semiconductor-based processor. The memory P04 can be a semiconductor-based memory. Each of the components P02, P04, P06, P08, P10, and P12, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor P02 can process instructions for execution within the computing device P00, including instructions stored in the memory P04 or on the storage device P06 to display graphical information for a GUI on an external input/output device, such as display P16 coupled to high speed interface P08. Also, multiple computing devices P00 may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

The memory P04 stores information within the computing device P00. In one implementation, the memory P04 is a volatile memory unit or units. In another implementation, the memory P04 is a non-volatile memory unit or units. The memory P04 may also be another form of computer-readable medium, such as a magnetic or optical disk.

The storage device P06 is capable of providing mass storage for the computing device P00. In one implementation, the storage device P06 may be or contain a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. The information carrier is a computer- or machine-readable medium, such as the memory P04, the storage device P06, or memory on processor P02.

The high speed controller P08 manages bandwidth-intensive operations for the computing device P00, while the low speed controller P12 manages lower bandwidth-intensive operations. In one implementation, the high-speed controller P08 is coupled to memory P04, display P16 (e.g., through a graphics processor or accelerator), and to high-speed expansion ports P10, which may accept various expansion cards (not shown). In the implementation, low-speed controller P12 is coupled to storage device P06 and low-speed expansion port P14.

The computing device P00 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server P20, or multiple times in a group of such servers. It may also be implemented as part of a rack server system P24. In addition, it may be implemented in a personal computer such as a laptop computer P22. Alternatively, components from computing device P00 may be combined with other components in a mobile device (not shown), such as device P50. Each of such devices may contain one or more of computing device P00, P50, and an entire system may be made up of multiple computing devices P00, P50 communicating with each other.

Computing device P50 includes a processor P52, memory P64, an input/output device such as a display P54, a communication interface P66, and a transceiver P68, among other components. The device P50 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the components P50, P52, P64, P54, P66, and P68, are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other manners as appropriate.

The processor P52 can execute instructions within the computing device P50, including instructions stored in the memory P64. The processor may provide, for example, for coordination of the other components of the device P50, such as control of user interfaces, applications run by device P50, and wireless communication by device P50.

Processor P52 may communicate with a user through control interface P58 and display interface P56 coupled to a display P54. The display P54 may be, for example, a TFT LCD (Thin-Film-Transistor Liquid Crystal Display) or an OLED (Organic Light Emitting Diode) display, or other appropriate display technology. The display interface P56 may comprise appropriate circuitry for driving the display P54 to present graphical and other information to a user. The control interface P58 may receive commands from a user and convert them for submission to the processor P52. In addition, an external interface P62 may be provided in communication with processor P52, so as to enable near area communication of device P50 with other devices. External interface P62 may provide, for example, for wired communication in some implementations, or for wireless communication in other implementations, and multiple interfaces may also be used.

The memory P64 stores information within the computing device P50. The memory P64 can be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. Expansion memory P74 may also be provided and connected to device P50 through expansion interface P72, which may include, for example, a SIMM (Single In Line Memory Module) card interface. Such expansion memory P74 may provide extra storage space for device P50, or may also store applications or other information for device P50. Specifically, expansion memory P74 may include instructions to carry out or supplement the processes described above, and may include secure information also. Thus, for example, expansion memory P74 may be provide as a security module for device P50, and may be programmed with instructions that permit secure use of device P50.

The information carrier is a computer- or machine-readable medium, such as the memory P64, expansion memory P74, or memory on processor P52 that may be received, for example, over transceiver P68 or external interface P62.

Device P50 may communicate wirelessly through communication interface P66, which may include digital signal processing circuitry where necessary. Communication interface P66 may provide for communications under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication may occur, for example, through radio-frequency transceiver P68. In addition, short-range communication may occur, such as using a Bluetooth, Wi-Fi, or other such transceiver (not shown). In addition, GPS (Global Positioning System) receiver module P70 may provide additional navigation- and location-related wireless data to device P50, which may be used as appropriate by applications running on device P50.

Device P50 may also communicate audibly using audio codec P60, which may receive spoken information from a user and convert it to usable digital information. Audio codec P60 may likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of device P50. Such sound may include sound from voice telephone calls, may include recorded sound (e.g., voice messages, music files, etc.) and may also include sound generated by applications operating on device P50.

The computing device P50 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a cellular telephone P80. It may also be implemented as part of a smart phone P82, personal digital assistant, or other similar mobile device.

In some implementations, the computing devices depicted in <FIG> can include sensors that interface with a virtual reality (VR headset/HMD device P90) to generate an immersive environment. For example, one or more sensors included on a computing device P50 or other computing device depicted in <FIG>, can provide input to VR headset P90 or in general, provide input to a VR space. The sensors can include, but are not limited to, a touchscreen, accelerometers, gyroscopes, pressure sensors, biometric sensors, temperature sensors, humidity sensors, and ambient light sensors. The computing device P50 can use the sensors to determine an absolute position and/or a detected rotation of the computing device in the VR space that can then be used as input to the VR space (e.g., to perform alignments as described herein or to interact with the VR space after the alignment has been performed). For example, the computing device P50 may be incorporated into the VR space as a virtual object, such as a controller, a laser pointer, a keyboard, a weapon, etc. Positioning of the computing device/virtual object by the user when incorporated into the VR space can allow the user to position the computing device so as to view the virtual object in certain manners in the VR space. For example, if the virtual object represents a laser pointer, the user can manipulate the computing device as if it were an actual laser pointer. The user can move the computing device left and right, up and down, in a circle, etc., and use the device in a similar fashion to using a laser pointer.

In some implementations, one or more input devices included on, or connected to, the computing device P50 can be used as input to the VR space. The input devices can include, but are not limited to, a touchscreen, a keyboard, one or more buttons, a trackpad, a touchpad, a pointing device, a mouse, a trackball, a joystick, a camera, a microphone, earphones or buds with input functionality, a gaming controller, or other connectable input device. A user interacting with an input device included on the computing device P50 when the computing device is incorporated into the VR space can cause a particular action to occur in the VR space.

In some implementations, a touchscreen of the computing device P50 can be rendered as a touchpad in VR space. A user can interact with the touchscreen of the computing device P50. The interactions are rendered, in VR headset P90 for example, as movements on the rendered touchpad in the VR space. The rendered movements can control virtual objects in the VR space.

In some implementations, one or more output devices included on the computing device P50 can provide output and/or feedback to a user of the VR headset P90 in the VR space. The output and feedback can be visual, tactical, or audio. The output and/or feedback can include, but is not limited to, vibrations, turning on and off or blinking and/or flashing of one or more lights or strobes, sounding an alarm, playing a chime, playing a song, and playing of an audio file. The output devices can include, but are not limited to, vibration motors, vibration coils, piezoelectric devices, electrostatic devices, light emitting diodes (LEDs), strobes, and speakers.

In some implementations, the computing device P50 may appear as another object in a computer-generated, 3D environment. Interactions by the user with the computing device P50 (e.g., rotating, shaking, touching a touchscreen, swiping a finger across a touch screen) can be interpreted as interactions with the object in the VR space. In the example of the laser pointer in a VR space, the computing device P50 appears as a virtual laser pointer in the computer-generated, 3D environment. As the user manipulates the computing device P50, the user in the VR space sees movement of the laser pointer. The user receives feedback from interactions with the computing device P50 in the VR environment on the computing device P50 or on the VR headset P90. The alignment techniques described herein may allow these interactions to feel more natural and intuitive to a user because the position of the corresponding object being rendered in the VR headset P90 will be aligned with the actual object.

In some implementations, a computing device P50 may include a touchscreen. For example, a user can interact with the touchscreen in a particular manner that can mimic what happens on the touchscreen with what happens in the VR space. For example, a user may use a pinching-type motion to zoom content displayed on the touchscreen. This pinching-type motion on the touchscreen can cause information provided in the VR space to be zoomed. In another example, the computing device may be rendered as a virtual book in a computer-generated, 3D environment. In the VR space, the pages of the book can be displayed in the VR space and the swiping of a finger of the user across the touchscreen can be interpreted as turning/flipping a page of the virtual book. As each page is turned/flipped, in addition to seeing the page contents change, the user may be provided with audio feedback, such as the sound of the turning of a page in a book.

In some implementations, one or more input devices in addition to the computing device (e.g., a mouse, a keyboard) can be rendered in a computer-generated, 3D environment. The rendered input devices (e.g., the rendered mouse, the rendered keyboard) can be used as rendered in the VR space to control objects in the VR space.

Claim 1:
A method comprising:
instructing a user (<NUM>) to orient a handheld controller (<NUM>, <NUM>) in a designated direction by: displaying a target within a scene displayed by a head-mounted display (<NUM>, <NUM>); and
instructing the user (<NUM>) to look at the target (<NUM>), so that the head-mounted display and the handheld controller (<NUM>, <NUM>) are pointed in the same direction;
if a connection between the head-mounted display and the handheld controller (<NUM>, <NUM>) is established (<NUM>):
detecting an input from the handheld controller (<NUM>, <NUM>); and
responsive to detecting the input:
determining an orientation of the handheld controller (<NUM>, <NUM>);
determining an orientation associated with the head-mounted display (<NUM>, <NUM>); by the following steps;
indicating to point the connected handheld controller at the target (<NUM>);
when a cursor (<NUM>) under control of the handheld controller (<NUM>, <NUM>) aligns with the target (<NUM>), an input device of the handheld controller (<NUM>, <NUM>), if actuated, is released to indicate that the target (<NUM>) and cursor (<NUM>) are aligned (<NUM>); and
storing alignment data representative of an alignment of the handheld controller (<NUM>, <NUM>), the alignment data including data representative of an offset between the determined orientation of the handheld controller (<NUM>, <NUM>) and the determined orientation associated with the head-mounted display (<NUM>, <NUM>).