Patent Description:
Mixed reality systems often use a head mounted display (HMD) which allows the user to view the actual environment, but which also includes functionality for displaying virtual objects to the user in the actual environment. This is often accomplished by having sensors in the HMD that are able to sense characteristics of the actual environment (including sensing physical objects in the actual environment), sense user movements, and then based on the sensed characteristics of the actual environment and the sensed user movement, display virtual objects in the actual environment to the user. For example, a mixed reality system may be able to sense a table in an actual environment. The mixed reality system can also detect that a user using a mixed reality system is facing the table. The mixed reality system can cause a virtual object to be displayed to the user on the table. The user can perform various gestures to interact with the virtual object on the table. For example, the user could "pick-up" the virtual object, "move" the virtual object, view different perspectives of the virtual object, etc..

One of the important goals of an HMD is that the HMD provide a user experience that is as natural and realistic as possible. However, current systems have a flaw in that user introduced objects and animation conflict with physical objects in the actual environment. In particular, a user can select a virtual object and select an animation for the virtual object within the mixed reality environment. The user then activates the animation. However, novice users may place the virtual object in the environment in such a way, that when animated, the virtual object interacts with actual objects in unrealistic ways. For example, a virtual planet may be selected along with an orbit animation. The orbit animation may have a radius that causes the virtual ball to pass through a wall. This can be jarring to the user and/or destroy the sense of reality for the user.

<CIT> describes a method for moving a virtual object includes displaying a virtual object and moving the virtual object based on a user input. Based on the user input attempting to move the virtual object in violation of an obstacle, displaying a collision indicator and an input indicator. The collision indicator is moved based on user input and movement constraints imposed by the obstacle. The input indicator is moved based on user input without movement constraints imposed by the obstacle.

Embodiments illustrated herein allow a user to select a virtual object and to select an animation path for placement in a mixed reality environment. Embodiments can display the virtual object in combination with the animation path (without actually animating the object) to illustrate how the object will be animated with respect to actual objects in the mixed reality environment. In particular, a user will be able to see if the animation path moves through an actual object such that interaction of a virtual object with the actual object will seem unrealistic. Thus, a technical problem exists where virtual objects are animated in mixed reality environments in unrealistic ways. This problem can be solved though a technical means of providing feedback to the user, such as visually rendered feedback using displays of devices, to show the user where collisions might occur to allow the user to interact with user input devices to change how objects will be animated in the virtual reality environment.

For example, attention is directed to <FIG> which illustrates a mixed reality environment <NUM>. The mixed reality environment includes an actual object <NUM> which in this example is a wall. A user has selected a virtual object <NUM> for animation in the mixed reality environment <NUM>. In the illustrated example, the virtual object <NUM> is a sphere. The user has also selected an animation path <NUM> defining a path in the mixed reality environment <NUM> in which the virtual object <NUM> will be animated. In particular, a user can use various controllers and head mounted display (HMD) devices to select virtual objects and animation paths to be rendered as shown in <FIG> in a mixed reality environment <NUM>. The user can use the controllers to place the virtual object <NUM> in a particular location in the mixed reality environment <NUM>. The animation path <NUM> is associated with the virtual object <NUM> based on the selected location of the virtual object <NUM>.

As illustrated in <FIG>, the virtual object <NUM> will collide with the actual object <NUM>. If the virtual object <NUM> is animated using the animation path <NUM> as placed in the mixed reality environment <NUM>, the virtual object <NUM> will pass through the actual object <NUM>. In many circumstances, this will be jarring to the user, and upset any perceptions of reality of the mixed reality environment <NUM>. In particular, this may appear as if a solid object is passing through another solid object.

Embodiments illustrated herein allow the user to determine that such an effect will occur. In particular, the animation path <NUM> is shown passing through the actual object <NUM>, without the object being actually animated. Thus, a user can see the animation path <NUM> passing through the actual object <NUM> and move the virtual object <NUM> to avoid collision of the virtual object <NUM> with the actual object <NUM> when the animation associated with the animation path <NUM> is started. In this way, the user can have control so as to implement a realistic animation scenario. For example, as illustrated in <FIG> the virtual object <NUM> and the animation path <NUM> are moved away from the actual object <NUM> such that the virtual object <NUM> will not collide with the actual object <NUM>.

Returning once again to <FIG>, various techniques can be used to illustrate to a user collision of a virtual object <NUM> with an actual object <NUM>.

Note that as will be described in more detail below, embodiments may include sensors that are able to detect and map the location of actual objects in the mixed reality environment <NUM>. In particular, embodiments can use various depths sensors such as infrared emitters and detectors, stereoscopic cameras, ultrasonic emitters and detectors, combinations thereof, etc. For example, embodiments may be able to admit light signals from an HMD which are reflected off of the surfaces of actual objects in the mixed reality environment. Embodiments can measure Doppler shifts, phase shifts, light absorption, and other effects to determine various characteristics about object actual objects in the mixed reality environment <NUM>. In particular, movement of objects may be detected, surface types of objects can be detected, locations can be detected, color (or at least shading) can be detected, etc..

In particular, optical tracking has become a useful and ubiquitous process. In some optical tracking systems, such as time-of-flight cameras, optical tracking involves a transmitter emitting optical signals, such as infrared or laser signals, directed at objects, capturing reflected optical signals, and using the captured, reflected signals, to track various objects. For example, optical tracking may be used to sense distances of objects in an environment. Using dual optical trackers can be useful for <NUM>-D tracking.

This optical tracking may be used for a number of different purposes. For example, optical trackers may be implemented on head mounted displays (HMDs) used in mixed reality or virtual reality. The optical trackers in these scenarios may be used to track locations of objects in an environment with respect to the HMD. This may be useful to help a user avoid and/or locate physical object in a room. Also, this can be used to sense movement over time of the user's head. In other words, such optical tracking may be used to perform head tracking (HeT) operations, which are useful in providing a rich and realistic mixed reality or virtual reality experience by presenting virtual objects to a user based on the user's head movements.

By knowing the location of actual objects in the mixed reality environment <NUM>, embodiments can render for a user, anticipated collisions between virtual objects and actual objects in the mixed reality environment <NUM> when an animation is performed. Thus, for example, in the example illustrated in <FIG>, an HMD displays to a user a rendering of the virtual object <NUM> along with the animation path <NUM>. <FIG> illustrates a collision zone <NUM>. In particular, a device displays an appropriate visual indication of a potential collision if the virtual object <NUM> is animated according to the animation path <NUM> in the mixed reality environment <NUM>. In the example illustrated in <FIG>, the collision zone <NUM> is illustrated with impact lines that are rendered on a display, such as an HMD display or other device display. However, it should be appreciated that in other environments other illustrations may be provided on a display. For example, portions of the animation path <NUM> that do not cause the virtual object <NUM> to collide with the actual object <NUM> may be rendered using a color code, such as a green shading for those portions of the animation path <NUM>. Portions of the animation path in the collision zone <NUM> may be rendered as shaded with a different coloring, such as for example a red shading, indicating that the virtual object <NUM> will collide with the actual object <NUM> along this portion of the animation path <NUM>.

<FIG> further illustrates a hidden zone <NUM>. The hidden zone <NUM> may be rendered by a device such as an HMD to illustrate portions of the animation path <NUM> where the virtual object <NUM> will be partially or completely hidden from view by the actual object <NUM>. In <FIG>, the hidden zone <NUM> is illustrated as being rendered by a device as a set of dotted lines. Although, it should be appreciated that other illustrations may be rendered alternatively or additionally.

Note that in some embodiments, a user may intend for a virtual object to collide with an actual object. For example, consider a case illustrated in <FIG>. In this example, a science teacher may wish to teach her students about the operation of various organs in the context of the students in a mixed reality environment <NUM>. Thus, in the present example, the actual object <NUM> is a person. The virtual object <NUM> is a human heart. The animation path <NUM> is a lateral movement back and forth of the human heart <NUM>. The user may wish to cause an animation where the virtual object <NUM> emerges from the actual object <NUM> along the animation path <NUM>. Given that embodiments can render the animation path <NUM> along with the virtual object <NUM>, the user can position the virtual object <NUM> and the animation path <NUM> to achieve the desired effect. Thus, in the illustrated example, the teacher can create for the student an immersive experience that is more compelling than what might be otherwise available.

Note that in some embodiments, two or more different devices, such as two or more different HMD devices and associated controllers may be used in concert to create certain experiences. In particular, various devices can be networked such that an experience in a mixed reality environment can be created by a user of a first HMD device and experienced by other users with different HMD devices. For example, in the example illustrated in <FIG> the teacher may use the HMD device <NUM>-<NUM> and the controllers <NUM>-<NUM>-R and <NUM>-<NUM>-L. These can be used to place the virtual object <NUM> and the animation path <NUM> in a particular location in the mixed reality environment <NUM> that interacts with the actual object <NUM> in a fashion desired by the user of the HMD <NUM>-<NUM> and controllers <NUM>-<NUM>-R and <NUM>-<NUM>-L. The teacher can then activate the animation once the virtual object <NUM> and animation path <NUM> are placed in an appropriate location. The animation can then be perceived by users of other HMD devices. For example, in <FIG> the actual object <NUM> may be another user wearing an HMD <NUM>-<NUM>. In this way the user, i.e. the actual object <NUM>, can experience an immersive and rich experience so as to perceive how their actual body is organized and functions.

Note that some embodiments allow for tracking of movement of actual objects in the virtual reality environment. Virtual object placement and animation may be anchored to these actual objects. Thus, for example, the virtual object <NUM> and animation path <NUM> may be anchored to the actual object <NUM>, such that when the actual object <NUM> moves in the mixed reality environment <NUM>, the virtual object <NUM> and animation path <NUM> will move in the mixed reality environment. Note that in some circumstances, the animation path will be rendered to allow the user to see the path without the need for animation of the virtual object, whereas in other circumstances the animation path will be un-rendered, such as when the virtual object is animated according to the animation path. Note however that in some circumstances the virtual object can be animated with the animation path rendered.

Some embodiments include interactive functionality that provides suggestions to a user for changes in either or both a virtual object and/or an animation path. For example, attention is now directed to <FIG> which illustrates an alternative state of the mixed reality environment <NUM>. In this example, an HMD displays to a user various suggestions for changes to the virtual object <NUM> and/or the animation path <NUM>. In particular, <FIG> illustrates a drop down box <NUM> for the virtual object <NUM> and a drop down box <NUM> for the animation path <NUM>. The drop down boxes <NUM> and <NUM> provide suggestions for modifications to the virtual object <NUM> and/or the animation path <NUM>. For example, the drop-down box <NUM> may provide a recommendation that the size of the virtual object <NUM> be reduced if such reduction will prevent collision with the actual object <NUM> in the mixed reality environment.

In some embodiments, different textures of objects may be recommended. For example, in some embodiments if a potential collision is detected, embodiments can suggest objects and/or textures of objects that are not as jarring to the user to still allow the particular virtual object to collide with the actual object <NUM>. For example, embodiments may suggest that the object be at least partially transparent which provides an ethereal quality to the virtual object <NUM> which mitigates the jarring effects of the object <NUM> passing through an actual object <NUM>. Alternatively or additionally, different types of objects may be suggested. For example, embodiments may suggest a cloud or other extremely low density object. This can reduce the jarring effects of the virtual object <NUM> passing through the actual object <NUM>.

In yet another alternative embodiment, a device such as the HMD described previously herein may have information about the mixed reality environment <NUM> such that the device has mapped the mixed reality environment <NUM> behind the actual object <NUM> from the perspective of a user. In this fashion, the device can project elements of the mixed reality environment <NUM> onto the actual object <NUM> in a fashion that makes the actual object <NUM> appear somewhat transparent or ethereal. In this way, changes do not need to be made to the virtual object <NUM>, but rather it will appear more natural for the virtual object <NUM> to pass through the actual object <NUM> as the actual object <NUM> appears to be extremely low density, transparent, and/or ethereal. This option may alternatively or additionally be presented to the user as an option for dealing with potential collisions as illustrated.

Additional details are now illustrated. Referring now to <FIG>, in some embodiments and HMD device is configured to project various palettes in the mixed reality environment to a user. For example, <FIG> illustrates an object palette <NUM> and an animation path palette <NUM>. Here a user can use the HMD device and controllers to select an object from the object palette <NUM> and one or more animation paths from the animation path palette <NUM>. These can be drug and dropped into the mixed reality environment <NUM> where the user can then be notified of potential collisions and perform various adjustments to mitigate collisions, or as in the case of the example illustrated in <FIG> properly implement desired collisions.

Alternatively, or additionally, the drop-down box <NUM> may recommend different animation paths and/or changes to the animation path <NUM> to show what changes to the animation path <NUM> or different animation paths would prevent a collision of the virtual object <NUM> with the actual object <NUM>. For example, if the animation path <NUM> is an orbit, the drop-down box <NUM> may suggest an orbit with a smaller radius. Alternatively and/or additionally, the drop-down box <NUM> may suggest a different type of animation path that would prevent collision of the virtual object <NUM> with the actual object <NUM>.

Embodiments can be configured, as described above, to detect actual objects in an environment. For example, in some embodiments system <NUM> is implemented, in whole or in part, as a time-of-flight camera system of a user device (e.g., a head-mounted display (HMD), gaming console, or other computer system) for tracking real-world objects in the vicinity of the user device. In some implementations, tracking real-world objects is useful for head tracking in HMDs (e.g., by detecting the edges or corners of real-world objects over time, a system can determine how the HMD has moved with respect to the real-world objects over time).

Continuing with <FIG>, the system <NUM> illustrates a system that is able to use optical signals to detect distances from the system <NUM> (or any particular part of the system <NUM>) to objects <NUM>. The system <NUM> includes an emitter <NUM>. In some embodiments, the emitter is implemented as one or more light emitting diodes (LEDs), laser diodes, and/or any other illumination unit which is suitable for emitting modulated light. The emitter <NUM> emits an optical signal <NUM>. For example, the optical signal may be a set of laser pulses, infrared light pulses, or another appropriate optical signal. It will be appreciated that the emitter <NUM> can be configured to emit light at any desired wavelength or band of wavelengths (e.g., in the visible spectrum, near IR spectrum, IR spectrum, etc.).

As is evident in <FIG>, optical signal <NUM> is emitted from emitter <NUM> in a plurality of directions, such that different portions of optical signal <NUM> reflect off of different objects <NUM> in the real-world environment of system <NUM>. A first portion of the optical signal <NUM> reflects off of object <NUM>-<NUM>, resulting in reflected optical signal <NUM>-<NUM>, and a second portion of the optical signal <NUM> reflects off of object <NUM>-<NUM>, resulting in reflected optical signal <NUM>-<NUM>.

Both reflected optical signal <NUM>-<NUM> and reflected optical signal <NUM>-<NUM> are received by detector <NUM>. The detector <NUM> may include, for example, various photodiodes, cameras, and/or other sensor hardware that is able to convert the reflected optical signals <NUM>-<NUM> and <NUM>-<NUM> to an input signal <NUM>. In some instances, the detector <NUM> includes a plurality of pixels for detecting reflected optical signals <NUM>-<NUM> and <NUM>-<NUM>. The input signal <NUM> is an electrical signal created by the detector <NUM> from the reflected optical signals <NUM>-<NUM> and <NUM>-<NUM>. Various attributes of the reflected optical signals <NUM>-<NUM> and <NUM>-<NUM>, such as the phase and the active brightness (e.g., an amount of intensity collected) can be determined from the input signal <NUM> received by the detector <NUM>. By analyzing properties of the input signal <NUM> (e.g., compared with or correlated with the properties of the optical signal <NUM> initially emitted by emitter <NUM>), system <NUM> can determine distance between system <NUM> and objects <NUM> (e.g., objects <NUM>-<NUM> and <NUM>-<NUM>, which reflect reflected optical signals <NUM>-<NUM> and <NUM>-<NUM> toward detector <NUM>). In some instances, the distance between system <NUM> and objects <NUM> is determined on a pixel-by-pixel basis (e.g., based on the detections of the various pixels of detector <NUM>).

The reflected optical signals <NUM>-<NUM> and <NUM>-<NUM> will have a shifted phase and changed intensity based on travel to and from, and being reflected off of the objects <NUM>-<NUM> and <NUM>-<NUM>, respectively, as compared to the original phase and intensity of the optical signal <NUM> when transmitted from the emitter <NUM>. The change in phase is related to the distance the signal has traveled. In particular, phase is related to the distance traveled by <MAT>, where φ is the detected phase of the detected optical signal, d is the distanced traversed, f is the modulation frequency of the light, and c is the speed of the light. Thus, the total distance travelled is proportional to the phase of the signal received at the detector <NUM>. The intensity of the signal received at the detector <NUM> will also depend on the distance traveled, in particular because the intensity is proportional to the distance traveled by an inverse square relationship.

It is noted that object <NUM>-<NUM> and <NUM>-<NUM> are at different positions with respect to system <NUM>. In particular, object <NUM>-<NUM> is positioned further from system <NUM> than object <NUM>-<NUM>. Accordingly, the path traversed by reflected optical signal <NUM>-<NUM> is longer than reflected optical signal <NUM>-<NUM>. Thus, the detected phase and active brightness of the two reflected optical signals <NUM>-<NUM> and <NUM>-<NUM> will also be different.

Intensity also changes based on the color of the object reflecting the signal. For example, a dark object will absorb more (and reflect less) of the light and reduce the intensity more than a lighter-colored object. Accordingly, differences in color between object <NUM>-<NUM> and <NUM>-<NUM> (if there are differences) will affect the intensity of the reflected optical signals <NUM>-<NUM> and <NUM>-<NUM> received by detector <NUM>.

As noted above, the change in phase and intensity can be used to determine the distance of the objects <NUM>-<NUM> and <NUM>-<NUM> from the system <NUM>. This can be done by examination of the input signal <NUM>, or other signals created from the input signal <NUM>, as compared with the optical signal <NUM> as emitted by emitter <NUM>.

Although objects <NUM>-<NUM> and <NUM>-<NUM> are shown as distinct objects, it will be appreciated that objects <NUM>-<NUM> and <NUM>-<NUM> can correspond to different parts of a single object. For example, objects <NUM>-<NUM> and <NUM>-<NUM> can correspond to different elements, textures, or portions of a single object or environment, each being positioned at a different distance from system <NUM>. Also, it will be noted that objects <NUM> need not be thought of as having homogeneous features (e.g., color, texturing), and that differences in each individual object would be reflected in the phase and/or intensity of the reflected optical signals received by detector <NUM>. For example, object <NUM>-<NUM> could have a dark portion and a bright portion, and each portion of object <NUM>-<NUM> would reflect a different intensity of light.

Those skilled in the art will recognize that, in some embodiments, emitter <NUM> is configured to simultaneously or sequentially emit an optical signal composed of different modulation frequencies (e.g., <NUM>, <NUM>, <NUM>, or more different modulation frequencies), all of which may be reflected off of objects <NUM> to cause reflected optical signals which are detected by detector <NUM>. Thus, a reflected optical signal can comprise various different modulation frequencies, and an input signal <NUM> can be generated based on some or all of the different modulation frequencies of the reflected optical signal.

<FIG> shows an exemplary input signal, which is a conceptual representation of input signal <NUM> described above with reference to <FIG>. The input signal <NUM> is shown along a vertical axis indicating radial distance (e.g., distance between the system <NUM> and objects <NUM>) and a horizontal axis indicating pixel (e.g., the different optical signal detection pixels of detector <NUM>). In essence, the representation of input signal <NUM> shown in <FIG> demonstrates a profile of the objects <NUM>-<NUM> and <NUM>-<NUM>, based on the reflected optical signals <NUM>-<NUM> and <NUM>-<NUM> received by detector <NUM>.

For example, the input signal <NUM> is shown as having three regions: region <NUM>-<NUM>, region <NUM>-<NUM>, and region <NUM>. Region <NUM>-<NUM> corresponds to the part of the input signal <NUM> that is based on the reflected optical signal <NUM>-<NUM>, which was reflected off of object <NUM>-<NUM> toward detector <NUM>. Similarly, region <NUM>-<NUM> corresponds to the part of the input signal <NUM> that is based on the reflected optical signal <NUM>-<NUM>, which was reflected off of object <NUM>-<NUM> toward detector <NUM>. Accordingly, region <NUM>-<NUM> indicates a nearer radial distance (e.g., distance from system <NUM>), while region <NUM>-<NUM> indicates a further radial distance, since object <NUM>-<NUM> is situated further from system <NUM> than object <NUM>-<NUM>.

Region <NUM> represents a transition region between region <NUM>-<NUM> and region <NUM>-<NUM>. For example, region <NUM> corresponds to the part of input signal <NUM> that is based on optical reflections from the edge, or near the edge, of object <NUM>-<NUM>. Put differently, region <NUM> corresponds to a real-world transition (or edge) between object <NUM>-<NUM> and <NUM>-<NUM>, as detected by detector <NUM>.

Using this functionality, location and movement of physical objects can be determined.

Referring now to <FIG>, a method <NUM> is illustrated. The method <NUM> is practiced in a mixed reality environment where virtual objects are implemented in the context of physical objects. The method includes acts for rendering potential collisions between virtual objects and physical objects if animations are implemented. The method includes receiving user input selecting a virtual object to be animated (act <NUM>).

The method further includes receiving user input selecting an animation path for the virtual object (act <NUM>).

The method further includes receiving user input placing the virtual object to be animated and the animation path in an environment including physical objects (act <NUM>). For example, the virtual object <NUM> and animation path <NUM> are selected by a user and placed in the mixed reality environment where they are rendered by the HMD <NUM> shown in <FIG>.

The method further includes prior to animating the virtual object, displaying the virtual object and the animation path in a fashion that shows the interaction of the virtual object with one or more physical objects in the environment (act <NUM>). An example of this is illustrated in <FIG>, where the collision zone <NUM> can be seen without needing to actually animate the virtual object <NUM>.

The method <NUM> further includes detecting a potential collision between the virtual object and a physical object if the animation were performed and as a result, the method <NUM> may further include highlighting the potential collision. For example, the collision zone rendering illustrated in <FIG> may be performed.

In some embodiments, highlighting the collision includes color coding the animation path to show locations on the animation path where the potential collision occurs. Alternatively or additionally, highlighting the collision comprises displaying a potential collision indicator.

In the embodiment covered by the claimed invention, the method <NUM> includes detecting a potential collision between the virtual object and a physical object if the animation were performed and as a result, automatically providing one or more suggested remedial actions. An example of this is illustrated in <FIG>, where alternate animation paths, objects, and/or object sizes are suggested. Thus, the method may be practiced where the one or more suggested remedial action comprises displaying alternate suggested animation paths. This could be different shapes, different sizes (e.g., shorter lengths, smaller radii, larger radii, etc.). Alternatively or additionally, the method may be practiced where the one or more suggested remedial action comprises displaying alternate suggested virtual objects. For example, different shapes and/or sizes could be suggested.

Having just described the various features and functionalities of some of the disclosed embodiments, attention is now directed to <FIG>, which illustrates an example computer system <NUM> that is used to facilitate the operations described herein. It will be appreciated that, in some instances, the computer system <NUM> shown in <FIG> can be used to carry out the embodiments described herein. In particular, the system can display the various virtual objects, animation paths, menus, palettes, collision indicators, etc., as described above.

The computer system <NUM>, in <FIG>, is embodied as a head-mounted display (HMD) Although the computer system <NUM> is embodied as an HMD, the computer system <NUM> may also be a distributed system that includes one or more connected computing components/devices that are in communication with the HMD. By way of example, the computer system <NUM> may include a projector, desktop computer, a laptop, a tablet, a mobile phone, server, data center and/or any other computer system.

In its most basic configuration, the computer system <NUM> includes various different components. For example, <FIG> shows that computer system <NUM> includes at least one hardware processing unit <NUM> (aka a "processor"), input/output (I/O) interfaces <NUM>, graphics rendering engines <NUM>, one or more sensors <NUM>, and storage <NUM>. More detail on the hardware processing unit <NUM> will be presented momentarily.

The storage <NUM> may be physical system memory, which may be volatile, non-volatile, or some combination of the two. The term "memory" may also be used herein to refer to non-volatile mass storage such as physical storage media. If the computer system <NUM> is distributed, the processing, memory, and/or storage capability may be distributed as well. As used herein, the term "executable module," "executable component," or even "component" can refer to software objects, routines, or methods that may be executed on the computer system <NUM>. The different components, modules, engines, and services described herein may be implemented as objects or processors that execute on the computer system <NUM> (e.g. as separate threads).

The disclosed embodiments may comprise or utilize a special-purpose or general-purpose computer including computer hardware, such as, for example, one or more processors (such the hardware processing unit <NUM>) and system memory (such as storage <NUM>), as discussed in greater detail below. Embodiments also include physical and other computer-readable media for carrying or storing computer-executable instructions and/or data structures. Such computer-readable media can be any available media that can be accessed by a general-purpose or special-purpose computer system. Computer-readable media that store computer-executable instructions in the form of data are physical computer storage media. Thus, by way of example and not limitation, the current embodiments can comprise at least two distinctly different kinds of computer-readable media: computer storage media and transmission media.

Computer storage media are hardware storage devices, such as RAM, ROM, EEPROM, CD-ROM, solid state drives (SSDs) that are based on RAM, Flash memory, phase-change memory (PCM), or other types of memory, or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code means in the form of computer-executable instructions, data, or data structures and that can be accessed by a general-purpose or special-purpose computer.

The computer system <NUM> may also be connected (via a wired or wireless connection) to external sensors <NUM> (e.g., one or more remote cameras, accelerometers, gyroscopes, acoustic sensors, magnetometers, etc.). It will be appreciated that the external sensors include sensor systems (e.g., a sensor system including a light emitter and camera), rather than solely individual sensor apparatuses. For example, the sensors <NUM>/<NUM> can include an emitter <NUM> and/or a detector <NUM>, as described hereinabove with reference to <FIG>. Further, the computer system <NUM> may also be connected through one or more wired or wireless networks <NUM> to remote systems(s) <NUM> that are configured to perform any of the processing described with regard to computer system <NUM>.

During use, a user of the computer system <NUM> is able to perceive information (e.g., a mixed-reality environment) through a display screen that is included among the I/O interface(s) <NUM> and that is visible to the user. The I/O interface(s) <NUM> may include the input elements described herein, which are linked to one or more underlying applications generating information for the mixed-reality scene.

The I/O interface(s) <NUM> and sensors <NUM>/<NUM> also include gesture detection devices, eye trackers, and/or other movement detecting components (e.g., cameras, gyroscopes, accelerometers, magnetometers, acoustic sensors, global positioning systems ("GPS"), etc.) that are able to detect positioning and movement of one or more real-world objects, such as a user's hand, a stylus, and/or any other object(s) that the user may interact with while being immersed in the scene.

The graphics rendering engine <NUM> is configured, with the hardware processing unit <NUM>, to render one or more virtual objects within the scene. As a result, the virtual objects accurately move in response to a movement of the user and/or in response to user input as the user interacts within the virtual scene.

A "network," like the network <NUM> shown in <FIG>, is defined as one or more data links and/or data switches that enable the transport of electronic data between computer systems, modules, and/or other electronic devices. When information is transferred, or provided, over a network (either hardwired, wireless, or a combination of hardwired and wireless) to a computer, the computer properly views the connection as a transmission medium. The computer system <NUM> will include one or more communication channels that are used to communicate with the network <NUM>. Transmissions media include a network that can be used to carry data or desired program code means in the form of computer-executable instructions or in the form of data structures. Further, these computer-executable instructions can be accessed by a general-purpose or special-purpose computer.

Computer-executable (or computer-interpretable) instructions comprise, for example, instructions that cause a general-purpose computer, special-purpose computer, or special-purpose processing device to perform a certain function or group of functions.

Claim 1:
A computer system (<NUM>) comprising:
a head mounted display, HMD for displaying a mixed reality environment;
one or more processors (<NUM>); and
one or more computer-readable media having stored thereon instructions that are executable by the one or more processors to configure the computer system (<NUM>) to render potential collisions between virtual objects (<NUM>) and physical objects (<NUM>) if animations are implemented, including instructions that are executable to configure the computer system (<NUM>) to perform at least the following:
receiving user input selecting a virtual object (<NUM>) to be animated;
receiving user input selecting an animation path (<NUM>) for the virtual object (<NUM>);
receiving user input placing the virtual object (<NUM>) to be animated and the animation path (<NUM>) in an environment including physical objects (<NUM>);
prior to animating the virtual object (<NUM>), displaying the virtual object (<NUM>) and the animation path (<NUM>) to show the interaction of the virtual object (<NUM>) with one or more physical objects (<NUM>) in the environment;
detecting a potential collision between the virtual object (<NUM>) and one of the one or more physical objects (<NUM>) if the animation were performed; and
as a result, automatically providing one or more suggested remedial actions.