Patent Description:
Advances in technology, driven, for example, by improved processing power and speed, ever more efficient software rendering techniques, consumer convenience, and the like, have supported a growing movement in developing and designing simulated or three-dimensional (3D) virtual reality (VR) environments. VR systems which support such VR environments generally include hardware such as headsets, goggles, handheld or wearable devices, and the like. Operatively, such hardware continuously tracks user movements, updates user positions, orientations, and the like, and receives interactive input from users in a VR environment.

While certain VR systems can include complex and often expensive hardware and other equipment, such as an omni-directional treadmill, for tracking and translating real-world user movement into the VR environment, an average user may not have requisite capital (or physical space) to support such complex and often expensive equipment. Accordingly, certain challenges arise when designing and creating intuitive and interactive controls for users in a VR environment. Therefore, there is a need for improved interactive processes and techniques operable by simple VR equipment.

Previously proposed arrangements are disclosed in <CIT>, <CIT>, and <CIT>.

<CIT> discloses a game device, where an operation signal to operate a character displayed on a screen is input to proceed a game, is provided with a controller that has an acceleration sensor to detect acceleration in a predetermined direction, a GUI control unit that controls a graphic user interface for the display and operation of the game, and an object control unit that makes the coordinate position of a ball change in accordance with input data calculated on the basis of the operation signal input from the controller.

<CIT> discloses a game system that includes a game apparatus and a controller. The controller is furnished with an acceleration sensor for detecting accelerations in at least two axis directions. Game processing corresponding to the kind of an acceleration input by means of the controller is executed. For determining the kind, reference timing when acceleration in a first-axis direction is below a threshold value to take on a minimum value is detected. Then, it is determined whether or not an angle between acceleration change vectors before and after the reference timing is equal to or more than a predetermined angle. When the angle is not equal to or more than the predetermined angle, it is determined that the acceleration input is an acceleration input in any one of the two axis directions, and when the angle is equal to or more than the predetermined angle, it is determined that the acceleration input is an acceleration input in a direction including the two-axis directions as components.

<CIT> discloses a movable game controller for controlling aspects of a computer-controlled game display with apparatus for determining the linear and angular motion of that movable controller. The apparatus includes a plurality of self-contained inertial sensors for sensing the tri-axial linear and tri-axial angular motion of the moving controller. Each sensor is mounted at a fixed linear position and orientation with respect to the others. The linear and angular motion of the controller is computed from the correlated motion sensor readings of each of the plurality of self-contained inertial sensors.

This invention is defined by appended claims <NUM>, <NUM> and <NUM>. Preferred aspects and features of the invention are defined in the appended dependent claims.

The embodiments herein may be better understood by referring to the following description in conjunction with the accompanying drawings in which like reference numerals indicate identical or functionally similar elements.

Various embodiments of the disclosure are discussed in detail below. While specific implementations are discussed, it should be understood that this is done for illustration purposes only.

As discussed in herein, the subject disclosure relates to interactive controls particularly suitable for virtual reality (VR) environments. These interactive controls are employed by simple controllers without requiring complex and often expensive equipment. For example, the interactive controls may include an interactive control process/service that can be employed by simple controllers, head-sets, and/or VR consoles. Such interactive control process may, for example, detect movement of a controller associated with a virtual reality (VR) environment and includes steps to determine angles of rotation, magnitudes of force (e.g., acceleration), and the like, based on the movement. For example, the interactive control process may map or assign portions of the movement to vectors in a coordinate system and determine a path in the VR environment that corresponds to the vectors (e.g., based on averages, unions, differences, superposition, or other combinations of vectors/vector-elements). The path may be used as a guide to move the user in the 3D environment and/or as a selection tool to select or retrieve objects as discussed in greater detail below.

Referring now to the figures, <FIG> illustrates a schematic diagram of a Virtual Reality (VR) environment <NUM>. VR environment <NUM> includes various devices, components, sub-components, hardware/software, interfaces, and the like, which collectively operate form a VR system that provides an immersive and interactive simulated experience to a user <NUM>.

As shown, VR environment <NUM> includes equipment such as a console <NUM>, controller(s) <NUM>, and a headset <NUM>. Console <NUM> represents centralized hardware/software that communicates with controller <NUM> and/or headset <NUM>, as well as communicates with various other devices, servers, databases, and the like over a communication network <NUM> (e.g., the Internet), as is appreciated by those skilled in the art.

Communication network <NUM> represents a network of devices/nodes interconnected over network interfaces/links/segments/etc. and operable to exchange data such as a data packet <NUM> and transport data to/from end devices/nodes (e.g., console <NUM>, controller <NUM>, and/or headset <NUM>).

Data packets <NUM> include network traffic/messages which are exchanged between devices over communication network <NUM> using predefined network communication protocols such as certain known wired protocols, wireless protocols (e.g., IEEE Std. <NUM>, WiFi, Bluetooth®, etc.), PLC protocols, or other shared-media protocols where appropriate.

Controller <NUM> wirelessly communicates with console <NUM> over network <NUM>, or (in some embodiments) it may be coupled to console <NUM> over another network (not shown). Controller <NUM> facilitates user interaction with and within VR environment <NUM> and is operable to, for example, detect, track, or otherwise monitor movement and biometric information, communicate data signals with headset <NUM> and console <NUM>, and provide feedback (e.g., tactile, audible, etc.) to a user <NUM>. In this fashion, controller <NUM> can comprise any number of sensors, gyros, radios, processors, touch detectors, transmitters, receivers, feedback circuitry, and the like.

Headset <NUM>, similar to controller <NUM>, wirelessly communicates with console <NUM>. Headset <NUM> displays or projects simulated graphical elements that form simulated VR environments to user <NUM>, tracks eye movements, and measure biometric data from user <NUM>.

With respect to the devices discussed above, it is appreciated that certain devices may be adapted to include (or exclude) certain functionality and that the components shown are shown for purposes of discussion, not limitation. As discussed, console <NUM>, controller <NUM>, and headset <NUM> cooperate to provide an immersive and interactive VR environment to user <NUM>.

<FIG> illustrates a block diagram of an example device <NUM> that represents one or more devices shown in <FIG> (e.g., console <NUM>, controller <NUM>, headset <NUM>, etc.). Device <NUM> includes one or more network interfaces <NUM>, an input interface <NUM>, a processor <NUM>, and a memory <NUM> interconnected by a system bus <NUM>.

Network interface(s) <NUM> contain mechanical, electrical, and signaling circuitry for communicating data between devices over a network such as communication network <NUM>. Input interface <NUM> includes hardware/software that receives user commands, and detects movement or gestures, and may also be configured to provide user-feedback (e.g., tactile, visual, audio, etc.). For example, input interface <NUM> can include switches, buttons, accelerometers, sensors, processors, radios, display elements, and the like. Memory <NUM> comprises a plurality of storage locations that are addressable by processor <NUM> for storing software programs and data structures associated with the embodiments described herein.

Processor <NUM> may comprise necessary elements or logic adapted to execute the software programs and manipulate data structures <NUM>. An operating system <NUM>, portions of which are typically resident in memory <NUM> and executed by processor <NUM>, functionally organizes the device by, inter alia, invoking operations in support of software processes and/or services executing on the device. These software processes and/or services may comprise an illustrative "interactive control" process/service <NUM>. Note that while process/service <NUM> is shown in centralized memory <NUM>, it may be configured to operate in a distributed network.

It will be apparent to those skilled in the art that other processor and memory types, including various computer-readable media, may be used to store and execute program instructions pertaining to the techniques described herein. Also, while the description illustrates various processes, it is expressly contemplated that various processes may be embodied as modules configured to operate in accordance with the techniques herein (e.g., according to the functionality of a similar process). Further, while the processes have been shown separately, those skilled in the art will appreciate that processes may be routines or modules within other processes. For example, processor <NUM> can include one or more programmable processors, e.g., microprocessors or microcontrollers, or fixed-logic processors. In the case of a programmable processor, any associated memory, e.g., memory <NUM>, may be any type of tangible processor readable memory, e.g., random access, read-only, etc., that is encoded with or stores instructions that can implement program modules, e.g., a module having interactive control process <NUM> encoded thereon. Processor <NUM> can also include a fixed-logic processing device, such as an application specific integrated circuit (ASIC) or a digital signal processor that is configured with firmware comprised of instructions or logic that can cause the processor to perform the functions described herein. Thus, program modules may be encoded in one or more tangible computer readable storage media for execution, such as with fixed logic or programmable logic, e.g., software/computer instructions executed by a processor, and any processor may be a programmable processor, programmable digital logic, e.g., field programmable gate array, or an ASIC that comprises fixed digital logic, or a combination thereof. In general, any process logic may be embodied in a processor or computer readable medium that is encoded with instructions for execution by the processor that, when executed by the processor, are operable to cause the processor to perform the functions described herein.

<FIG> illustrates a block diagram of a VR environment <NUM>, particularly showing a simulate art gallery from the third person perspective view of user <NUM>. As mentioned, user <NUM> experiences and interacts with VR environment <NUM> using headset <NUM> and controller <NUM>. However, as discussed above, certain challenges arise when designing intuitive and interactive controls for VR environments, such as VR environment <NUM>, especially when designing controls operable by relatively simple and inexpensive controller components - here, controller <NUM> and/or headset <NUM>. Accordingly, the processes and techniques of this disclosure include improved interactive controls, particularly suitable for VR environments and for such relatively simple and inexpensive controller components.

<FIG> each illustrate a third person point view of a VR environment <NUM>, and show movement by controller <NUM> as well as a path resulting from the movement, respectively.

<FIG> particularly illustrates movement of controller <NUM> from a first position to a second position within a three-dimensional (3D) coordinate system. The movement of the controller, between the first position and the second position, includes a change in controller orientation or direction, indicated by an angle "α", as well as a change in acceleration, indicated by a magnitude of a corresponding vector. With respect to corresponding vectors, the first position and the second position of controller <NUM> are represented in vector form - i.e., vector <NUM>' and vector <NUM>" - which include a respective direction and magnitude, where the magnitude represents a force measured at the respective position. In this fashion, controller <NUM> monitors and measures changes in orientation, position, angular momentum, acceleration, and the like, and further performs or executes the interactive control processes/services (e.g., interactive control process <NUM>), as described herein.

<FIG> particularly shows a path <NUM> that results from the movement of controller <NUM> from the first position to the second position. Path <NUM> originates at a current location of user <NUM>, follows an arc (which is determined by a change in angle α over time), ends at a terminal location indicated by an anchor point <NUM> (a graphical display element ("X")), and is projected to user <NUM> by headset <NUM>. Path <NUM> can represent a travel path that guides movement of user <NUM> in the VR environment and/or it can represent a selection path that selects objects intersecting therewith.

In operation, controller <NUM> detects movement from the first position to the second position, determines an angle of rotation (α) based on the movement, and determines a magnitude of force (e.g., an acceleration force, etc.) associated with the movement and/or the respective positions. Controller <NUM> further determines a path in the VR environment - here, path <NUM> - that corresponds to the angle of rotation and the magnitude of force and headset <NUM> projects the path to user <NUM> in the VR environment. While the foregoing operations are described with respect to specific devices/controllers it is appreciated that any combination of devices may perform the same or substantially similar functionality.

<FIG> illustrates a first person point view of VR environment <NUM>, viewed from the perspective of user <NUM> wearing headset <NUM>. Here, VR environment <NUM> is illustrated with a grid pattern that represents a coordinate system overlaying a bounding plane or surface (e.g., a ground surface).

As discussed, the movement of controller <NUM> from the first position to the second position may be mapped into a 3D coordinate system where vectors represent respective direction and magnitude at each position. The 3D coordinate system may further include a real-world coordinate system and/or a VR environment coordinate system. With respect to the real-world coordinate system, additional processing may be needed to translate the movement and calculate the path resulting therefrom into the VR environment coordinate system.

As shown here, path <NUM> is represented by a graphical dash line that terminates in a 3D graphical "X" element, which represents anchor point <NUM>, and controller <NUM> is represented by a 3D graphical controller element. Collectively, these graphical components/elements show an exemplary a first person perspective view of user <NUM> within VR environment <NUM>. As mentioned, path <NUM> is calculated, in part, from changes in controller orientation as well as changes in force (e.g., acceleration). In one embodiment, path <NUM> represents a curved path or an arcing path and may be generated by a simulated casting motion similar to casting, for example, a fishing line. The distance and direction of the curved path are derived by the above mentioned changes in controller orientation/force, as is appreciated by those skilled in the art. Notably, in some embodiments, the casting motion may operate in conjunction with other input controls (e.g., button press/release/etc.) to indicate the user's intention to generate path <NUM> in environment <NUM>.

<FIG> illustrates a first person point of view of VR environment <NUM>, particularly from the perspective of user <NUM>. Here, user <NUM> moved controller <NUM> from the first position to the second position (as discussed above), and controller <NUM> (and/or other hardware components) determined a resultant a path - i.e., path <NUM> - which is displayed to user <NUM> by headset <NUM>. Path <NUM> represents a travel path in VR environment <NUM> that guides movement of user <NUM> from a current position to the anchor point <NUM>. Notably, in <FIG>, path <NUM> moves user <NUM> closer to an interactive object <NUM> - i.e., a painting. As is appreciated by those skilled in the art, user <NUM> may provide additional input to prompt subsequent movement along path <NUM> in VR environment <NUM>. Such subsequent movement along the path may be graphically projected/animated for user <NUM> by headset <NUM>, as is appreciated by those skilled in the art.

<FIG> illustrates the first person point of view of VR environment <NUM> in <FIG>, further showing path <NUM> intersecting interactive object <NUM>. In addition to calculating a path based on the movement of controller <NUM>, the interactive control process or techniques (e.g., interactive control process <NUM>) may incorporate object intersection processes that detect an intersection (or collision) between portions of the path and objects in the VR environment. Alternatively (or in addition), the intersection may include fuzzy logic to detects proximate intersection/collision as well as actual intersection. Here, the interactive control process determines portions of path <NUM> intersect (or are proximate to intersection) with interactive object <NUM>. Due to such intersection (or proximate intersection), path <NUM> may be modified to help user <NUM> select interactive object <NUM> and/or call up additional menu options (example options include "add", "adjust", "full screen", "zoom", "crop", and the like). In addition, VR environment <NUM> also provides a selection element or border <NUM> that indicates selection of interactive object <NUM>.

Here, path <NUM> represents a selection path that selects interactive object <NUM>. The selection path may guide movement between the user and the selected object in the VR environment and/or the selection path may retrieve the selected object to the user (e.g., move the object from its current location to a current location of the user). As is appreciated by those skilled in the art, user <NUM> may provide additional input to prompt subsequent movement of the user and/or the object in VR environment <NUM>.

<FIG> illustrates a third person point of view of another VR environment, showing movement of two controllers that each support user interaction with and within the VR environment;.

<FIG> each illustrate a third person point view of a VR environment <NUM>, where <FIG> shows movement by two controllers -- controller <NUM> and a controller <NUM> - in a three-dimensional (3D) coordinate system and <FIG> shows respective paths resulting from the movements. The movement of controller <NUM> in <FIG> is the same as the movement of controller <NUM> shown in <FIG>, discussed above. Accordingly, discussion with respect to <FIG> below focuses on a second controller - controller <NUM> - and its movement.

<FIG> specifically illustrates controller <NUM> movement from a first position to a second position and controller <NUM> movement from a respective first position to a second position. The respective movements of each controller include a change in controller orientation or direction, indicated by an angle "α" (for controller <NUM>) or angle "β" (for controller <NUM>), as well as respective changes in acceleration, indicated by corresponding vectors.

With respect to corresponding vectors, in addition to vectors <NUM>' and <NUM>" which correspond to controller <NUM>, <FIG> also provides vectors <NUM>' and <NUM>", which correspond to the first position and the second position of controller <NUM> and show directions and magnitudes. As discussed above, the magnitude represents a force measured at a respective position.

<FIG> particularly shows paths resulting from the respective movements of each controller. Specifically, in addition to path <NUM> (corresponding to controller <NUM>), <FIG> also illustrates a path <NUM> that originates at a current location of user <NUM>, follows an arc (which is determined by a change in angle β over time), ends at a terminal location indicated by an anchor point <NUM> (a graphical display element ("X")), and is projected to user <NUM> by headset <NUM>. Path <NUM>, similar to path <NUM>, can represent a travel path that guides movement of user <NUM> in the VR environment and/or it can represent a selection path that selects objects intersecting therewith.

In operation, controllers <NUM>, <NUM> detect respective movements, determine an angle of rotation (α, β) based on the movement, and determine a magnitude of force (e.g., an acceleration force, etc.) associated with the movement and/or the respective positions. Controllers <NUM>, <NUM> further determine a respective path in the VR environment -path <NUM>, <NUM> - that corresponds to the respective angles of rotation and magnitudes of force.

<FIG> illustrate the third person point of view of the VR environment of <FIG>, showing each path terminating in respective anchor points and corresponding VR interactions. In particular, <FIG> illustrates path <NUM> and path <NUM> as a straight line that intersects user <NUM> and respective anchor points <NUM>, <NUM> positioned on a ground surface, while <FIG> illustrates path <NUM> and path <NUM> as a straight line that intersects user <NUM> and respective anchor points <NUM>, <NUM> positioned on a wall surface in VR environment <NUM>.

As discussed above, the paths <NUM>, <NUM> may represent travel paths to guide movement of user <NUM> in the VR environment. While examples falling outside of the scope of the claims having one controller and one travel path (as well as corresponding interactive operations) are discussed above, here, a synergy exists between multiple paths - i.e., path <NUM> and path <NUM> - that supports further interactive operations. In particular, an analogy may be made to operating a kite with two strings, whereby each path represents one string (or a cable) and the kite is represented by a fixed bounding surface (a ground surface in <FIG> and a wall surface in <FIG>). In addition, an axis of rotation 940a may be defined between terminal or anchor points <NUM>, <NUM>. In operation, user <NUM> may control his movement relative to the anchor points (on the ground surface) and input corresponding tugging or pulling motions/gestures into controllers <NUM>, <NUM>. The tugging or pulling motions, which are detected by each controller, cooperate and can move user <NUM> along/between paths <NUM>, <NUM>. In addition, such motions (and/or other defined motions) can rotate or adjust an orientation of user about axis 940a with respect to VR environment <NUM>.

<FIG> illustrates a third person point of view of another VR environment, showing a selection path <NUM> derived from movements by controller <NUM> and controller <NUM>. Here, selection path <NUM> can represent a superposition of path <NUM> and path <NUM>, where portions of both paths cancel and other portions aggregate. Alternatively (or in addition), selection path <NUM> may also represent an average, a difference, a union, or some other combination of paths <NUM> and <NUM>, as is appreciated by those skilled in the art. Moreover, as mentioned above, the interactive control process (employed in conjunction with two controllers) may also incorporate object intersection/collision processes that detect an intersection (or a location proximate to the intersection) between an object (an interactive object <NUM>) and the selection path derived from movement of both controllers. In some embodiments, similar to those discussed above, the selection path <NUM> may represent a travel path that terminates in an anchor point <NUM> for guiding movement of user <NUM> in VR environment <NUM>.

<FIG> illustrate a third person point of view of VR environment <NUM>, showing an initial path (path <NUM>) resulting by movement of controller <NUM> (<FIG>) and an adjustment to the initial path resulting from movement of controller <NUM> (<FIG>). Collectively, <FIG> illustrate additional functionality of the interactive control processes where user <NUM> moves controller <NUM> to generate a path <NUM> (discussed in greater detail above) and moves controller <NUM> to adjust or fine-tune aspects of path <NUM> (e.g., adjust a direction, magnitude, orientation, etc.). For example, as shown, path <NUM> is generated in <FIG> and is located at an initial distance from user <NUM>. User <NUM> further moves and/or rotates controller <NUM> in <FIG> to adjust path <NUM> - e.g., move anchor point <NUM> closer to a current location of user <NUM>.

<FIG> illustrate a first person point of view of <FIG>, respectively. Here, VR environment <NUM> shows a library with a number of books arranged on shelves. User <NUM> moves controller <NUM> (<FIG>) to generate path <NUM>, which terminates on/near authors of books beginning with "mo. User <NUM> further moves controller <NUM> (indicated by a side arrow) to modify, adjust, or otherwise move path <NUM> so it terminates on/near authors of books beginning with "ma. In this fashion, one controller may be used to generate a travel/selection path while a second controller may be used to fine-tune or otherwise adjust the travel/selection path.

<FIG> illustrates an example simplified procedure <NUM> of an interactive control process employed by a VR system (or components thereof). Procedure <NUM> begins at step <NUM> and continues to steps <NUM> and <NUM>, where, as described in greater detail above, a VR system detects movement of a first controller (e.g., controller <NUM>) and a second controller (e.g., controller <NUM>). The VR system further determines, at step <NUM>, for each movement, respective angles of rotation, magnitudes of force applied, accelerations, and the like. Based on the foregoing determinations, the VR system maps, in step <NUM>, each movement to one or more vectors in a three-dimensional (3D) coordinate system.

The VR system also determines, in step <NUM>, a path for the VR environment that corresponds to the one or more vectors (e.g., angles of rotation, magnitudes of force, etc.). For example, the VR system can derive the path based on an average, differences, summations, or other combinations of the vectors, which correspond to respective angles of rotation, magnitudes of forces (changes of force over time), etc., as is appreciated by those skilled in the art.

Procedure <NUM> continues to step <NUM> where, as discussed above, the VR system detects an intersection between portions of the path and an object in the VR environment and selects the object. As mentioned, selecting the object may further cause the object to be indicated as selected (e.g., a bounding box), display menu options associated with the object, retrieving or moving the object to a current position of the user or moving the user toward the object. The VR system further projects (e.g., using headset <NUM>) the path in the VR environment such as the figures illustrate for paths <NUM>/<NUM>.

The techniques described herein, therefore, provide interactive control processes that compliment immersive simulated VR environments without requiring expensive and complex equipment. These interactive controls define simple and intuitive gestures that quickly and efficiently learned by any user.

Claim 1:
A method for interactive control using two controllers (<NUM>, <NUM>) that each support user interaction with and within a virtual reality, VR, environment implemented by a VR system comprising a processor (<NUM>) and a headset (<NUM>) for executing the method, the method comprising:
detecting a first movement of a first of the two controllers associated with the VR environment;
determining an angle of rotation based on the first movement;
determining a magnitude of force associated with the first movement;
determining a path in the VR environment that corresponds to the angle of rotation and the magnitude of force, wherein the path is used as a guide to move the user in the VR environment and/or as a selection tool to select or retrieve objects in the VR environment;
projecting the path in the VR environment by displaying graphical elements that represent the path in the VR environment;
detecting a second movement of a second of the two controllers associated with the VR environment; and
adjusting the path based on the second movement and displaying graphical elements that represent the adjusted path in the VR environment.