Patent Description:
The proposed solution relates to a method as defined by claim <NUM>, an electronic apparatus as defined by claim <NUM> and a non-transitory machine-readable media as defined by claim <NUM>.

VR environments provide users with applications and components which they can use to interact with virtual objects. In some instances, VR environments include representations of real objects with which a user may have contact while immersed in the VR environments. Comparable methods are known from.

Reference will now be made in detail to non-limiting examples of this disclosure, examples of which are illustrated in the accompanying drawings. The examples are described below by referring to the drawings, wherein like reference numerals refer to like elements. When like reference numerals are shown, corresponding description(s) are not repeated and the interested reader is referred to the previously discussed figure(s) for a description of the like element(s).

Turning to <FIG>, a block diagram of an example virtual reality (VR) system <NUM> for creating and interacting with a three-dimensional (3D) VR environment <NUM> in accordance with the teachings of this disclosure is shown. In general, the system <NUM> provides the VR environment <NUM>, and VR content that enable a user to access, view, use and/or interact with the VR environment <NUM>. The VR system <NUM> can provide the user with options for accessing the content, applications, virtual objects, real objects, and VR controls using, for example, actual and projected movement within the VR environment. The example VR system <NUM> of <FIG> includes a head-mounted display (HMD) <NUM> worn by a user <NUM>, and a handheld controller <NUM> held by the user <NUM>. The example VR environment <NUM> shown in <FIG> is a representation of the images being displayed for the user inside their HMD <NUM>.

The example VR environment <NUM>, in accordance with this disclosure, can include a virtual object <NUM> as well as a representation of a real (e.g., non-virtual, physical, actual, etc.) object <NUM> that is normally not seen or visible while the user is operating within the VR environment <NUM> as displayed inside their HMD <NUM>. In other words, the real object <NUM> shown in the VR environment <NUM> represents a physical object while the virtual object <NUM> may not represent a physical object in the physical world. For instance, the real object <NUM> may represent a physical object that is present in a room in which a user is operating their HMD <NUM> and, thus, interacting within the VR environment <NUM>. When the real object <NUM> is not shown to the user in their HMD <NUM>, as the user navigates within the VR environment <NUM>, the user may have unintended or un-anticipated contact with the physical object represented by the real object <NUM>. Such inadvertent contact may decrease customer satisfaction and/or may result in user injury.

Accordingly, the examples disclosed herein adaptively or dynamically display virtual objects, and real objects <NUM> that are representations, within the VR environment <NUM>, of physical objects outside the VR environment <NUM> in order to alert users from within the VR environment <NUM>. In that way, for example, the risk of unintended or un-anticipated contact of the user with the real object can be reduced, thus reducing also the risk of possible injury of the user due to a collision with the real object. For instance, if a projected movement of a user or a portion thereof indicates that the user will move close (e.g., within a threshold criteria or distance, within a small distance) to a physical object in the physical world, the real object <NUM>, which is a representation of the physical object, changes from an un-displayed state, in which the real object <NUM> is not visible (e.g., substantially not visible) in the VR environment <NUM> (e.g., within the VR environment <NUM> on a display within the HMD <NUM>), to a displayed state in which the real object <NUM> is at least partially depicted in the VR environment <NUM>. In this way, what is displayed in the VR environment 102can include both objects in the VR environment <NUM> as well the physical object (represented as real object <NUM>) from the physical space in which the user is operating the HMD <NUM>.

For example, <FIG> illustrates a physical room <NUM> including a physical or real object in the form of a chest <NUM>. The physical room <NUM> represents a physical space in which a user may physically move as they interact with a VR environment displayed inside their HMD. <FIG> illustrates a virtual room <NUM> displayed inside the user's HMD, the virtual room <NUM> including virtual objects in the form of, for example, a virtual bed <NUM>. In <FIG>, the user of the HMD is a distance (not shown) from the bed <NUM>.

If the user of the HMD moves, or is projected to move, along, for example, direction D1 close enough to the foot of the virtual bed <NUM> they also move, or are projected to move, close to the physical chest <NUM> in the physical room <NUM>. This results in a virtual representation of the physical chest <NUM> as virtual chest <NUM> being shown in the VR environment <NUM> inside the user's HMD, as shown in <FIG>.

In this way, the user is made aware inside their HMD that the chest <NUM> is in their projected path of movement so action may be taken to avoid contact. As shown in <FIG>, the virtual chest <NUM> can be shown in a partially visible state.

Example partially visible states can include, but are not limited to, a partially opaque state, a partially transparent state, a displayed state in a wire frame representation, etc. When in the partially visible state another object, real or virtual, and in the VR environment can be at least partially viewed through the displayed representation of the real object.

In some examples, a threshold is used to determine when to at least partially display real objects. More generally speaking, one may specify one or more conditions, including the attainment of such a threshold, under which real objects that represent physical objects may be displayed in a VR environment. Example thresholds include, but are not limited to, a distance between a projected movement and a real object, a projected time to reach the real object, a projected probability of contact, etc. If the user is projected to move away from the real object as determined by calculating a projected movement. The real object may revert to not being visible inside the HMD in response to the calculated projected movement. By using thresholds, real objects can be displayed in partially visible states for a small distance about the user. In some implementations, different thresholds are used to different objects and/or types of objects. In some implementations, any movement of the user in the same room as a physical object may cause the real object <NUM> to be displayed in the VR environment.

Referring back to <FIG>, the example VR system <NUM> includes any number of computing and/or electronic devices that can exchange data over a network <NUM>. The devices may represent clients or servers, and can communicate via the network <NUM> or any other additional and/or alternative network(s). Example client devices include, but are not limited to, a mobile device <NUM> (e.g., a smartphone, a personal digital assistant, a portable media player, etc.), an electronic tablet (not shown), a laptop or netbook <NUM>, a camera (not shown), the HMD <NUM>, a desktop computer <NUM>, the VR handheld controller <NUM>, a gaming device (not shown), and any other electronic or computing devices that can communicate using the network <NUM> or other network(s) with other computing or electronic devices or systems, or that may be used to access VR content or operate within a VR environment. The devices <NUM>, <NUM> and <NUM>-<NUM> may represent client or server devices. The devices <NUM>, <NUM> and <NUM>-<NUM> can execute a client operating system, and one or more client applications that can access, render, provide, or display VR content on a display device included in or in conjunction with each respective device <NUM>, <NUM> and <NUM>-<NUM>.

<FIG> is a schematic diagram of an example HMD <NUM> that may be used, among other things, to implement the example HMD <NUM> of <FIG>. The example HMD <NUM> of <FIG> includes an optics module <NUM>, a processor module <NUM>, and an outward or forward facing camera <NUM>.

The optics module <NUM> includes, but is not limited to, lenses, mirrors, coatings, apertures, etc. that enable a wearer of the HMD <NUM> to view a display <NUM>. The example display <NUM> displays side-by-side images and/or videos for respective eyes of a wearer of the HMD <NUM>. In other examples, two (or more) of the display <NUM> are used to collectively display, respectively, the side-by-side images and/or videos.

The modules <NUM>, <NUM> and <NUM>, and the elements therein may be implemented using any number and/or type(s) of components and/or machine-readable instructions. In some examples, the processor module <NUM> is implemented using a mobile device such as a mobile phone or smartphone that can be communicatively and/or physically coupled to the HMD <NUM>. In some examples, the mobile device <NUM> is placed, located or otherwise implemented or operated in conjunction within the HMD <NUM> or <NUM> to provide a display that can be used as the display <NUM>, and/or a camera that can be used as the camera <NUM> of the HMD <NUM> or <NUM>. The mobile device <NUM> can additionally or alternatively include hardware and/or software for executing VR applications.

To control the HMD <NUM>, the example processor module <NUM> of <FIG> includes a processor <NUM> in the form of a microcontroller, a central processing unit (CPU), an ASIC, a digital signal processor (DSP), an FPGA, a graphics processing unit (GPU), etc. programmed or configured to execute machine-readable instructions stored in a memory <NUM>. In some examples, the instructions, when executed, cause the processor <NUM> to, among other things, change how and when real objects are presented on the display <NUM> inside the HMD <NUM> based on projected physical movement of the user relative to the real objects. In some examples, more than one processor <NUM> and/or memory <NUM> are provided.

The processor <NUM> additionally processes images taken by the camera <NUM> to identify, for example, the locations, dimensions, etc. of real objects in a physical space in which a user is using their HMD. This information can be used to map the real environment in which a user is using their HMD. Additionally or alternatively, other devices can be used to map the locations, dimensions, etc. of real objects in a physical space in which an HMD is being used. In some examples, the map of a physical environment in which a user will use their HMD is pre-scanned, pre-known or otherwise determined a priori. Alternatively and/or additionally, it can be determined in real time as their HMD is worn and moved about.

To communicatively couple the example HMD <NUM> to other devices such as the devices <NUM>-<NUM> of <FIG>, the example HMD <NUM> of <FIG> includes a communication module <NUM>. Example communication modules <NUM> implement communication signals and/or protocols such as Bluetooth®, Wi-Fi®, and universal serial bus (USB).

To record data and/or signals representative of actual movement, the example processor module <NUM> of <FIG> includes a movement tracker <NUM>. The movement tracker <NUM> of <FIG> may record, for example, tracking information determined by the HMD <NUM>, and/or tracking information determined by other devices such as the example devices <NUM>-<NUM>. Example tracking information includes three dimensions of freedom (3DoF) and six degrees of freedom (6DoF) tracking data. Tracking information may be determined using any number and/or type(s) of emitters, sensors and/or cameras. The actual movement tracking data is representative of actual (e.g., physical, real, non-virtual, etc.) movement of a user, an HMD, a handheld controller, a limb, a head, a hand, a foot, a torso, etc. In some examples, the actual movement tracking data is captured in 3D.

To project movement, the example processor module <NUM> of <FIG> includes a movement projector <NUM>. The example movement projector <NUM> of <FIG> determines one or more characteristics of actual movement (e.g., past movement, movement that has already occurred, etc.) based on the actual movement data collected by the movement tracker <NUM>. Example characteristics include shape of a trajectory or vector (line, curve, radius of curve, etc.) velocity, acceleration, etc. The shape of a trajectory may be determined using, for example, a curve fitting algorithm or method.

Based on the determined characteristics, the movement projector <NUM> projects (e.g., predicts, estimates, determines, etc.) movement (e.g., future movement, movement to come, movement that has not yet occurred, etc.). For example, if actual movement has occurred most recently along a linear trajectory at a velocity of X, then an example projected movement would be along that linear trajectory at a distance of (X multiplied by Time) from the current location, where Time is the amount of time (e.g., <NUM> second) over which projections are made. Herein, the time intervals over which projections are made are referred to as projection intervals (or projection time periods) for sake of discussion. Projections may be similarly made for trajectories of other shapes, as well as considering other factors, such as acceleration. When the end of a projection interval (e.g., each <NUM> seconds) is actually reached (e.g., interval time has elapsed), the process is repeated with the projected movement data discarded and replaced by additional actual movement data. Movement projections can be made in two-dimensions (2D) or 3D.

In some implementations, the movement projector <NUM> additionally considers the movement of a handheld controller relative to an HMD. For example, when a user swings their hand about their body as they move (e.g., walk, run, etc.). In such examples, a projected movement trajectory would be computed for each of the HMD and the controller.

To project contact, the example processor module <NUM> of <FIG> includes a contact projector <NUM>. The example contact projector <NUM> of <FIG> compares projected movement data determined by the example movement projector <NUM> with locations and characteristics of real objects to determine whether the user is going to pass close enough to a real object that the real objected should be displayed in an at least partially visible state inside the HMD <NUM>. In examples where a projected movement trajectory is computed for each of an HMD and controller, each of the trajectories could be considered relative to the locations of real objects. In some examples, time rather than distance can be used to trigger the display of a real object. For example, assuming a prediction interval of <NUM> seconds, and a user's reaction time of <NUM> second, then any contact that may occur between now to and one second from now should be at least partially shown.

In some implementations, contact projection accommodates what is happening in a VR application. For example, if a VR application has an object or target that a user tries to reach, contact projection might have a larger threshold criteria in a direction toward that object or target than other directions. In other directions, if a user is moving toward such an object or target it may be more likely that they will accelerate their movement in that direction and, thus, a larger criteria threshold may be used. For example, suppose that there are two physical objects - a chair to the user's left and a table to the user's right - in the room with the user. Suppose further that the user is moving toward the chair. In this case, the contact projector <NUM> generates projected data indicating that the user will likely collide with the chair in three seconds. In this case, if the time threshold is three seconds then an image of the chair may be displayed in the VR environment. Further, because it is unlikely that the user will suddenly turn around to make contact with the table, the time threshold to display the table may be set to one second instead of three seconds.

As an example, turn to <FIG> is a perspective view of a sequence of actual positions <NUM>, <NUM>, <NUM> and <NUM> (in a physical world) of a person <NUM> at respective times T1, T2, T3 and T4. In <FIG>, a user is wearing an HMD and interacting with VR environment they are viewing in their HMD as they move. <FIG> is a top view of the sequence of locations <NUM>-<NUM> of <FIG>. Taken together, the actual positions <NUM>-<NUM> and times T1-T4 represent an actual movement of the person <NUM> along a linear trajectory <NUM> while swinging a fly swatter <NUM> about themselves. For clarity of illustration, only one instance of the person <NUM> and the fly swatter <NUM> are designated.

Between times T1 and T4, the movement tracker <NUM> collects data representing motion of the person <NUM> along the trajectory <NUM> using a sensor included in the HMD <NUM> or <NUM>, and motion of the fly swatter <NUM> based on the handheld controller <NUM>. Motion of the HMD <NUM> or <NUM> may be, for example, determined using a forward-facing camera included in the HMD, and/or emitters included in the HMD that are sensed by sensors (not shown) of the VR system <NUM> of <FIG>.

At time T4 in <FIG>, the movement projector <NUM> processes some or all of the data collected by the movement tractor <NUM> and determines that actual movement has been occurring along the linear trajectory <NUM> at a velocity of X. The movement projector <NUM> accordingly projects that at a time T5 the person will continue to move along the linear trajectory <NUM> at the same velocity because the velocity has been reasonably consistent. Thus, at time T5, the movement projector <NUM> projects they will be at location <NUM>.

The movement projector <NUM> also processes at least some of the data and determines that the fly swatter <NUM> was most recently swung in an arc, and that the fly swatter has been repositioned to be swung again at a fly. The movement projector <NUM> accordingly projects that at the time T5 the person will be mid-swing at position <NUM>. Taken together, the movement projector <NUM> projects that the person will at time T5, be at location <NUM> in mid-swing with the fly swatter <NUM>.

The contact projector <NUM> compares the predicted location <NUM> of the HMD at time T5 with the location of a real object <NUM> in the physical space in which the HMD is being used. The contact projector <NUM> also compares the predicted location <NUM> of the fly swatter <NUM> at time T5 with the location of the real object <NUM>. As seen best in <FIG>, the fly swatter at projected location <NUM> is near to (e.g., within a threshold distance) the real object <NUM>. Accordingly, the real object <NUM> is at least partially displayed in the VR environment displayed inside the HMD. Because the HMD is projected to be relatively far from the real object <NUM>, the HMD does not trigger display of the real object <NUM>.

In some implementations, projection intervals are short enough that only a single projected movement data point is computed for each interval, and only single comparisons relative to real object locations need to be made. By shortening the projection interval, the complexity of movement projection can be reduced as projected movement data will be relatively close to the most recent actual movement data. In such instances, low order extrapolation may be used. However, consideration may need to be given to a sufficiently long interval that allows the user to react to the change in displayed state of a real object.

In some implementations, time is used to trigger the display of a real object. For example, assuming a prediction interval of <NUM> seconds, and a user reaction time of <NUM> second, any contact that may occur within time NOW to time NOW plus <NUM> second should be shown in an at least partially visible state. In some examples, time used to project contact is adapted based on velocity of actual movement. For instance, reaction time may be assumed to increase as the person moves faster to avoid the real object being at least partially displayed too late, or to decrease as the person moves slower to avoid the real object being at partially displayed too early.

In some implementations, probability of contact is utilized. For example, if the likelihood of contact is greater than X%, then the real object should be displayed in an at least a partially visible state.

In some implementations, movement projection is not utilized. Instead, actual positions satisfying an actual distance from a real object triggers the display of the object in an at least partially displayed state.

In some implementations, the movement tracker <NUM>, the movement projector <NUM> and the contact projector <NUM> execute without the knowledge of application developers so that real object contact information can be provided at the system level in an HMD. In this way, real object contact information can be provided without requiring the effort, involvement, knowledge or approval of application developers. In this way, a user obtains advantage of the improvements disclosed herein for all of their VR applications.

In some implementations, the movement projector <NUM> and contact projector <NUM> are combined into and realized by a neural network or artificial intelligence (AI) module <NUM>. In general, the AI module <NUM> of <FIG> uses unstructured learning to learn causal relationships between inputs and outputs (e.g., an AI model). The learned relationships can then be applied to actual movement data inputs to determine projected movement data. The learned relationships can be learned by having the AI module <NUM> trained using sets of actual movement data captured at {T1,T0,. } as inputs and actual movement data captured at T2 as outputs. Subsequent data sets include data captured at {T2,T1,. } and T3; data captured at {T3,T2,T1,. } and T4; etc. In use, actual movement data is input to the AI module <NUM>, and outputs represent which real objects to display in an at least partially displayed state.

In some implementations, a portion of the processor module <NUM> is implemented separate from the HMD <NUM>. For example, the movement tracker <NUM>, the movement projector, the contact projector <NUM> and/or the AI module <NUM> may be implemented by any of the devices <NUM>-<NUM>. Information regarding which real objects should be displayed in an at least partially displayed state could be relayed to the HMD <NUM>.

<FIG> is a flowchart of an example method that may, for example, be implemented as machine-readable instructions carried out by one or more processors, such as the example processors P00 and P50 of <FIG>, to implement the example motion projectors disclosed herein. The example method of <FIG> includes analyzing or processing actual HMD movement data to determine an actual HMD trajectory (block <NUM>), and analyzing or processing actual controller movement data to determine an actual controller trajectory (block <NUM>). A projection of an HMD location (block <NUM>) and a controller location (block <NUM>). Additionally or alternatively, limbs, hands, feet, heads, etc. not necessary associated with a controller may be tracked. Control then exits from the example method of <FIG>.

<FIG> is a flowchart of an example method that may, for example, be implemented as machine-readable instructions carried out by one or more processors, such as the example processors P00 and P50 of <FIG>, to implement the example contact projectors disclosed herein. The example method of <FIG> includes computing distances between a projected HMD location and the location(s) of real objects (block <NUM>). If any distances satisfy a threshold (e.g., less than, less than or equal, equal to, etc.) (block <NUM>), the real objects satisfying the threshold are displayed in an at least partially displayed state (block <NUM>).

Distances between a projected controller location and the location(s) of real objects are computed (block <NUM>). If any distances are less than a threshold (block <NUM>), the real objects satisfying the threshold are displayed in an at least partially displayed state (block <NUM>). Control then exits from the example method of <FIG>.

As disclosed above, other criteria (e.g., time, percentage, etc.) may additionally or alternatively be used to determine which real objects to display in an at least partially displayed state.

<FIG> is a flowchart of an example method that may, for example, be implemented as machine-readable instructions carried out by one or more processors, such as the example processors P00 and P50 of <FIG>, to implement the example AI modules disclosed herein. In addition to projecting contacts, the example method of <FIG> includes adapting the learned relationships as the AI module is used. The example method of <FIG> includes inputting actual motion data to the AI model (block <NUM>) and outputting project contact information (block <NUM>).

If previous projections matched actual projections satisfy a criteria (e.g., number of missed contacts, number of false contacts, contacts displayed too late, contacts displayed too early, etc.) (block <NUM>), the current state of the AI model is saved (block <NUM>) and current actual motion data is saved (block <NUM>). Control then exits from the example method of <FIG>.

If previous projections did not match actual projections within the criteria (block <NUM>), the saved state of the AI model (block <NUM>) and the actual motion data (block <NUM>) are recalled. The AI module is updated using the recalled data as inputs and recent actual data as outputs (block <NUM>). Control then exits from the example method of <FIG>.

One or more of the elements and interfaces disclosed herein may be duplicated, implemented in the parallel, implemented in the singular, combined, divided, re-arranged, omitted, eliminated and/or implemented in any other way. Further, any of the disclosed elements and interfaces may be implemented by the example processor platforms P00 and P50 of <FIG>, and/or one or more circuit(s), programmable processor(s), fuses, application-specific integrated circuit(s) (ASIC(s)), programmable logic device(s) (PLD(s)), field-programmable logic device(s) (FPLD(s)), and/or field-programmable gate array(s) (FPGA(s)), etc. Any of the elements and interfaces disclosed herein may, for example, be implemented as machine-readable instructions carried out by one or more processors. A processor, a controller and/or any other suitable processing device such as those shown in <FIG> may be used, configured and/or programmed to execute and/or carry out the examples disclosed herein. For example, any of these interfaces and elements may be embodied in program code and/or machine-readable instructions stored on a tangible and/or non-transitory computer-readable medium accessible by a processor, a computer and/or other machine having a processor, such as that discussed below in connection with <FIG>. Machine-readable instructions comprise, for example, instructions that cause a processor, a computer and/or a machine having a processor to perform one or more particular processes. The order of execution of methods may be changed, and/or one or more of the blocks and/or interactions described may be changed, eliminated, sub-divided, or combined. Additionally, they may be carried out sequentially and/or carried out in parallel by, for example, separate processing threads, processors, devices, discrete logic, circuits, etc..

The example methods disclosed herein may, for example, be implemented as machine-readable instructions carried out by one or more processors. A processor, a controller and/or any other suitable processing device such as that shown in <FIG> may be used, configured and/or programmed to execute and/or carry out the example methods. For example, they may be embodied in program code and/or machine-readable instructions stored on a tangible and/or non-transitory computer-readable medium accessible by a processor, a computer and/or other machine having a processor, such as that discussed below in connection with <FIG>. Machine-readable instructions comprise, for example, instructions that cause a processor, a computer and/or a machine having a processor to perform one or more particular processes. Many other methods of implementing the example methods may be employed. For example, the order of execution may be changed, and/or one or more of the blocks and/or interactions described may be changed, eliminated, sub-divided, or combined. Additionally, any or the entire example methods may be carried out sequentially and/or carried out in parallel by, for example, separate processing threads, processors, devices, discrete logic, circuits, etc..

As used herein, the term "computer-readable medium" is expressly defined to include any type of computer-readable medium and to expressly exclude propagating signals. Example computer-readable medium include, but are not limited to, one or any combination of a volatile and/or non-volatile memory, a volatile and/or non-volatile memory device, a compact disc (CD), a digital versatile disc (DVD), a read-only memory (ROM), a random-access memory (RAM), a programmable ROM (PROM), an electronically-programmable ROM (EPROM), an electronically-erasable PROM (EEPROM), an optical storage disk, an optical storage device, a magnetic storage disk, a magnetic storage device, a cache, and/or any other storage media in which information is stored for any duration (e.g., for extended time periods, permanently, brief instances, for temporarily buffering, and/or for caching of the information) and that can be accessed by a processor, a computer and/or other machine having a processor.

Returning to <FIG>, the VR system <NUM> may include any number of VR content systems <NUM> storing content and/or VR software modules (e.g., in the form of VR applications <NUM>) that can generate, modify, and/or execute VR scenes. In some examples, the devices <NUM>, <NUM> and <NUM>-<NUM> and the VR content system <NUM> include one or more processors and one or more memory devices, which can execute a client operating system and one or more client applications. The HMD <NUM>, the controller <NUM>, the other devices <NUM>-<NUM> or the VR content system <NUM> may be implemented by the example computing devices P00 and P50 of <FIG>.

The VR applications <NUM> can be configured to execute on any or all of devices <NUM>, <NUM> and <NUM>-<NUM>. The HMD <NUM> can be connected to devices <NUM>-<NUM> to access VR content on VR content system <NUM>, for example. Device <NUM>-<NUM> can be connected (wired or wirelessly) to HMD <NUM>, which can provide VR content for display. A user's VR system can be HMD <NUM> alone, or a combination of device <NUM>-<NUM> and HMD <NUM>.

The HMD <NUM> may represent a VR HMD, glasses, an eyepiece, or any other wearable device capable of displaying VR content. In operation, the HMD <NUM> can execute a VR application <NUM> that can playback received, rendered and/or processed images for a user. In some instances, the VR application <NUM> can be hosted by one or more of the devices <NUM>-<NUM>.

In some implementations, one or more content servers (e.g., VR content system <NUM>) and one or more computer-readable storage devices can communicate with the computing devices <NUM> and <NUM>-<NUM> using the network <NUM> to provide VR content to the devices <NUM> and <NUM>-<NUM>.

In some implementations, the mobile device <NUM> can execute the VR application <NUM> and provide the content for the VR environment. In some implementations, the laptop computing device <NUM> can execute the VR application <NUM> and can provide content from one or more content servers (e.g., VR content server <NUM>). The one or more content servers and one or more computer-readable storage devices can communicate with the mobile device <NUM> and/or laptop computing device <NUM> using the network <NUM> to provide content for display in HMD <NUM>.

In the event that HMD <NUM> is wirelessly coupled to device <NUM> or device <NUM>, the coupling may include use of any wireless communication protocol. A non-exhaustive list of wireless communication protocols that may be used individually or in combination includes, but is not limited to, the Institute of Electrical and Electronics Engineers (IEEE®) family of <NUM>. x standards a. Wi-Fi or wireless local area network (WLAN), Bluetooth, Transmission Control Protocol/Internet Protocol (TCP/IP), a satellite data network, a cellular data network, a Wi-Fi hotspot, the Internet, and a wireless wide area network (WWAN).

In the event that the HMD <NUM> is electrically coupled to device <NUM> or <NUM>, a cable with an appropriate connector on either end for plugging into device <NUM> or <NUM> can be used. A non-exhaustive list of wired communication protocols that may be used individually or in combination includes, but is not limited to, IEEE <NUM>. 3x (Ethernet), a powerline network, the Internet, a coaxial cable data network, a fiber optic data network, a broadband or a dialup modem over a telephone network, a private communications network (e.g., a private local area network (LAN), a leased line, etc.).

A cable can include a Universal Serial Bus (USB) connector on both ends. The USB connectors can be the same USB type connector or the USB connectors can each be a different type of USB connector. The various types of USB connectors can include, but are not limited to, USB A-type connectors, USB B-type connectors, micro-USB A connectors, micro-USB B connectors, micro-USB AB connectors, USB five pin Mini-b connectors, USB four pin Mini-b connectors, USB <NUM> A-type connectors, USB <NUM> B-type connectors, USB <NUM> Micro B connectors, and USB C-type connectors. Similarly, the electrical coupling can include a cable with an appropriate connector on either end for plugging into the HMD <NUM> and device <NUM> or device <NUM>. For example, the cable can include a USB connector on both ends. The USB connectors can be the same USB type connector or the USB connectors can each be a different type of USB connector. Either end of a cable used to couple device <NUM> or <NUM> to HMD <NUM> may be fixedly connected to device <NUM> or <NUM> and/or HMD <NUM>.

<FIG> shows an example of a generic computer device P00 and a generic mobile computer device P50, which may be used with the techniques described here. Computing device P00 is intended to represent various forms of digital computers, such as laptops, desktops, tablets, workstations, personal digital assistants, televisions, servers, blade servers, mainframes, and other appropriate computing devices. Computing device P50 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, and other similar computing devices.

Computing device P00 includes a processor P02, memory P04, a storage device P06, a high-speed interface P08 connecting to memory P04 and high-speed expansion ports P10, and a low speed interface P12 connecting to low speed bus P14 and storage device P06. The processor P02 can be a semiconductor-based processor. The memory P04 can be a semiconductor-based memory. Each of the components P02, P04, P06, P08, P10, and P12, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor P02 can process instructions for execution within the computing device P00, including instructions stored in the memory P04 or on the storage device P06 to display graphical information for a GUI on an external input/output device, such as display P16 coupled to high speed interface P08. Also, multiple computing devices P00 may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

The memory P04 stores information within the computing device P00. In one implementation, the memory P04 is a volatile memory unit or units. In another implementation, the memory P04 is a non-volatile memory unit or units. The memory P04 may also be another form of computer-readable medium, such as a magnetic or optical disk.

The storage device P06 is capable of providing mass storage for the computing device P00. In one implementation, the storage device P06 may be or contain a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. The information carrier is a computer- or machine-readable medium, such as the memory P04, the storage device P06, or memory on processor P02.

The high speed controller P08 manages bandwidth-intensive operations for the computing device P00, while the low speed controller P12 manages lower bandwidth-intensive operations. In one implementation, the high-speed controller P08 is coupled to memory P04, display P16 (e.g., through a graphics processor or accelerator), and to high-speed expansion ports P10, which may accept various expansion cards (not shown). In the implementation, low-speed controller P12 is coupled to storage device P06 and low-speed expansion port P14.

The computing device P00 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server P20, or multiple times in a group of such servers. It may also be implemented as part of a rack server system P24. In addition, it may be implemented in a personal computer such as a laptop computer P22. Alternatively, components from computing device P00 may be combined with other components in a mobile device (not shown), such as device P50. Each of such devices may contain one or more of computing device P00, P50, and an entire system may be made up of multiple computing devices P00, P50 communicating with each other.

Computing device P50 includes a processor P52, memory P64, an input/output device such as a display P54, a communication interface P66, and a transceiver P68, among other components. The device P50 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the components P50, P52, P64, P54, P66, and P68, are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other manners as appropriate.

The processor P52 can execute instructions within the computing device P50, including instructions stored in the memory P64. The processor may provide, for example, for coordination of the other components of the device P50, such as control of user interfaces, applications run by device P50, and wireless communication by device P50.

Processor P52 may communicate with a user through control interface P58 and display interface P56 coupled to a display P54. The display P54 may be, for example, a TFT LCD (Thin-Film-Transistor Liquid Crystal Display) or an OLED (Organic Light Emitting Diode) display, or other appropriate display technology. The display interface P56 may comprise appropriate circuitry for driving the display P54 to present graphical and other information to a user. The control interface P58 may receive commands from a user and convert them for submission to the processor P52. In addition, an external interface P62 may be provided in communication with processor P52, so as to enable near area communication of device P50 with other devices. External interface P62 may provide, for example, for wired communication in some implementations, or for wireless communication in other implementations, and multiple interfaces may also be used.

The memory P64 stores information within the computing device P50. The memory P64 can be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. Expansion memory P74 may also be provided and connected to device P50 through expansion interface P72, which may include, for example, a SIMM (Single In Line Memory Module) card interface. Such expansion memory P74 may provide extra storage space for device P50, or may also store applications or other information for device P50. Specifically, expansion memory P74 may include instructions to carry out or supplement the processes described above, and may include secure information also. Thus, for example, expansion memory P74 may be provide as a security module for device P50, and may be programmed with instructions that permit secure use of device P50.

The information carrier is a computer- or machine-readable medium, such as the memory P64, expansion memory P74, or memory on processor P52 that may be received, for example, over transceiver P68 or external interface P62.

Device P50 may communicate wirelessly through communication interface P66, which may include digital signal processing circuitry where necessary. Communication interface P66 may provide for communications under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication may occur, for example, through radio-frequency transceiver P68. In addition, short-range communication may occur, such as using a Bluetooth, Wi-Fi, or other such transceiver (not shown). In addition, GPS (Global Positioning System) receiver module P70 may provide additional navigation- and location-related wireless data to device P50, which may be used as appropriate by applications running on device P50.

Device P50 may also communicate audibly using audio codec P60, which may receive spoken information from a user and convert it to usable digital information. Audio codec P60 may likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of device P50. Such sound may include sound from voice telephone calls, may include recorded sound (e.g., voice messages, music files, etc.) and may also include sound generated by applications operating on device P50.

The computing device P50 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a cellular telephone P80. It may also be implemented as part of a smart phone P82, personal digital assistant, or other similar mobile device.

In this specification and the appended claims, the singular forms "a," "an" and "the" do not exclude the plural reference unless the context clearly dictates otherwise. Further, conjunctions such as "and," "or," and "and/or" are inclusive unless the context clearly dictates otherwise. For example, "A and/or B" includes A alone, B alone, and A with B. Further, connecting lines or connectors shown in the various figures presented are intended to represent exemplary functional relationships and/or physical or logical couplings between the various elements. It should be noted that many alternative or additional functional relationships, physical connections or logical connections may be present in a practical device. Moreover, no item or component is essential to the practice of the embodiments disclosed herein unless the element is specifically described as "essential" or "critical".

Claim 1:
A method comprising:
projecting a movement of a person (<NUM>) in a virtual reality (VR) environment (<NUM>) based on an actual movement of the person (<NUM>), wherein projecting the movement of the person (<NUM>) comprises:
projecting a movement trajectory of the person (<NUM>) and determining a projected probability of contact with a real object (<NUM>); and
determining a movement along the projected movement trajectory based on at least one of a projected velocity or a projected acceleration;
and if the projected movement and the projected probability satisfy a threshold relative to a position of the real object (<NUM>) in a physical space in which the VR environment (<NUM>) is implemented, changing a representation of the real object (<NUM>) from an un-displayed state to an at least partially displayed state in the VR environment (<NUM>), wherein the real object (<NUM>) in the displayed state is partially visible to the person (<NUM>).