Patent Description:
Augmented reality systems, or mixed reality systems, or virtual reality systems, may allow users to view scenes, for example, scenes corresponding to their physical environment, and to augment the scenes of their physical environment with virtual objects, or virtual characters. An augmented reality (AR) system and/or a virtual reality (VR) system may generate a three-dimensional (3D) immersive augmented and/or virtual reality environment. A user may experience this augmented reality, or mixed reality, or virtual reality, environment through interaction with various electronic devices such as for example described in <CIT> and <CIT>. A user may move through and interact with elements in is augmented reality, or mixed reality, or virtual reality, environment through, for example, gestures, manipulation of the electronic device and/or external devices operably coupled to the electronic device. A user experience in this type of augmented reality, or mixed reality, or virtual reality, experience, would be enhanced by germane, pertinent, contextual interaction with the virtual object(s) and/or character(s) placed in the view of scene of the physical environment. Such interaction would enhance the realistic nature of the user's augmented reality, or mixed reality experience, and enhance user presence in the augmented reality, or mixed reality experience.

In one general aspect, a computer-implemented method includes the steps of claim <NUM>.

In another general aspect, a system may include a computing device according to claim <NUM>.

An augmented reality (AR) system, or a mixed reality system, may allow users to place three-dimensional (3D) virtual objects, or augmented reality objects, or mixed reality objects, in a physical environment, and to view the 3D virtual objects/ augmented reality objects/mixed reality objects, for example, within a camera view, on a display portion of a head mounted display device, and the like. In some implementations, the 3D virtual/augmented reality/mixed reality objects may interact with users, and with each other, in a relatively realistic manner, to make the 3D virtual/augmented reality/mixed reality objects feel as though they are physically present in the physical environment of the user, rather than simply virtually present. In some implementations, the interaction of the 3D virtual/augmented reality/mixed reality objects with the user, and with each other, may be contextual, or context appropriate, or consistent, interactive, and reactive to the surrounding environment and changes in the surrounding environment, so as to further enhance the realistic nature of the 3D virtual/augmented reality/mixed reality object placement in the physical environment. Hereinafter, simply for ease of explanation, implementations to be described herein will be referred to as mixed reality systems allowing users to place 3D virtual objects in the physical environment to create a mixed reality environment, or mixed reality experience.

<FIG> illustrate a user with one or more electronic devices that can generate an augmented reality, or mixed reality, or virtual environment. As illustrated in the example shown in <FIG>, in some implementations, the user may view and experience a mixed reality environment on a display portion <NUM> of a handheld device <NUM> that displays a camera view of the physical environment, captured by an imaging device <NUM> of the handheld device <NUM>, together with virtual objects placed by the user in the camera view of the physical environment displayed on the display portion <NUM>. As illustrated in the example shown in <FIG>, in some implementations, the user may view and experience the mixed reality environment via a display portion of a head mounted device <NUM>. In this arrangement, the physical environment may be visible to the user through the head mounted device <NUM>, and the virtual objects may be placed by the user and visible on a display portion of the head mounted device <NUM>, superimposed on the user's view of the physical environment. As illustrated in the example shown in <FIG>, in some implementations, the user may view and experience the mixed reality environment via a display portion of a head mounted device <NUM> which occludes the user's direct visibility of the physical environment. In this arrangement, a pass through image of the physical environment may displayed on a display portion of the head mounted device <NUM>, with virtual objects placed by the user superimposed on the pass through image of the physical environment. Each of the handheld device <NUM> and the head mounted devices <NUM> and <NUM> illustrated in <FIG> are electronic devices including display devices capable of displaying virtual reality objects in an augmented reality, or mixed reality environment. Hereinafter, simply for ease of discussion and illustration, examples of a system providing for interaction of objects in a mixed reality environment, in accordance with implementations described herein, will be presented based on a camera view displayed on a display device of an electronic device similar to what is illustrated in <FIG>. However, the principles to be described herein may be applied to other electronic device(s) and/or systems capable of generating and presenting a mixed reality environment in which users may place 3D virtual objects, for interaction with the virtual objects, and/or for interaction amongst the virtual objects.

<FIG> and <FIG> are third person views of a user in a physical environment <NUM>. In <FIG> and <FIG>, the user is at a position 1000A in the physical environment <NUM>. The position 1000A of the user in the physical environment <NUM> may correspond to a position of an imaging device, or a camera, of an electronic device held by the user in this example. A mixed reality scene <NUM> (for example, corresponding to a camera view of the physical environment <NUM>) may captured by the imaging device of the electronic device. The mixed reality scene <NUM> may be displayed on, for example, a display device of the electronic device, for viewing by the user. In <FIG>, the mixed reality scene <NUM> of the physical environment <NUM> is illustrated in an enlarged state, separated from the electronic device, simply for ease of discussion and illustration. The mixed reality <NUM> visible to the user via the electronic device may represent a portion of the physical environment <NUM> that is captured within a field of view of the imaging device of the electronic device.

The user may place a first virtual object A in the mixed reality scene <NUM>, as shown in <FIG>. In the example shown in <FIG>, the first virtual object A is a first virtual character A is positioned on the ground, at a position <NUM> corresponding to what would be a first distance D1 from the user at the position 1000A. In this example, the distance D1 may represent a distance between the user at the position 1000A and the placement position <NUM> of the first character A at the time of placement of the first character A by the user. In some situations, the user may move within the physical environment <NUM>. As the user approaches the placement position <NUM> of the character A, decreasing the distance between the user and the placement position <NUM> of the character A, the system may detect that the user is nearing the character A. For example, the system may detect that the user is within a threshold distance D2 of the placement position <NUM> of the character A. In other words, the system may detect that a distance between the user and the placement position <NUM> of the character A is less than or equal to the threshold distance D2.

The system may treat the detection of the user within the threshold distance of the character A as an animation trigger. In response to detection of the user within the threshold distance D2 of the placement position <NUM> of the character A (i.e., in response to the detected animation trigger), the system may cause the character A to react to the approach of the user in an appropriate, or natural, or expected manner. For example, in some implementations, in response to the detection of the user within the threshold distance D2 of the placement position <NUM> of the character A (i.e., in response to the detected animation trigger), the system may animate the character A by, for example, causing the character A to react by turning to face the user, as shown in <FIG>.

Similarly, in some implementations, the user may place the first virtual character A at the first placement position <NUM>, as shown in <FIG> and as described above, and then place a second virtual object B, in the form of, for example, a second virtual character B, at a second placement position <NUM>, for example, near the first virtual character A, as shown in <FIG>. The system may detect the placement of the second character B within a threshold placement distance D3 of the first character A. The system may treat the detection of the first character A and the second character B within the threshold placement distance D3 of each other as an animation trigger, and may animate the first character A and/or the second character B, or cause the first character A and/or the second character B to react in an appropriate manner. That is, detection of animation trigger in which the first and second characters A and B are within the threshold distance D3 of each other may cause the system to execute an animation of at least one of the first character A or the second character B. For example, in response to detection of the second character B within the threshold placement distance D3 of the first character A, the system may cause the first character A to turn towards the second character B, as shown in <FIG>, and/or may cause the first character A and the second character B to turn to face each other. In some implementations, animation of the first character A and/or the second character B may continue beyond this type of recognition of the second character B within a threshold distance of the first character A. For example, the first character A and the second character B may continue to interact in an appropriate, or natural, or expected manner by, for example, conversing, moving to new positions 110A, 120A, and the like, as shown in <FIG>. In some implementations, in response to the detected animation trigger/detection of the user within a set threshold distance of the position <NUM> of the first character A and/or within a set threshold distance of the position <NUM> of the second character B, the system may cause first character A and/or the second character B to react in an appropriate manner to the detection of the user within the threshold distance, as described above.

Each of the sequential illustrations of the mixed reality scenes <NUM> (in some implementations, corresponding to camera views <NUM> captured by an imaging device of the electronic device) shown in <FIG> is a still image, representing a snapshot at a point in a particular animation and/or interaction of the first and second virtual characters A and B. However, it may be understood that the animation, or interaction, displayed by the first and second virtual characters A and B in the mixed reality scene <NUM> may be a substantially continuous dynamic animation of 3D virtual object(s), or an intermittent dynamic animation of 3D virtual object(s), providing a relatively realistic representation of the movement of the virtual character.

The position of the electronic device (and of the imaging device, or camera, of the electronic device) in the physical environment may be known and tracked by the system. The known/tracked position of the electronic device/camera in the physical environment may, essentially, correspond to the position of the user in the physical environment. The placement position(s) of the virtual object(s) (the virtual characters A and B in the example described above with respect to <FIG>) may also be known by the system. In this example, the placement position of the virtual object may be a placement position in the mixed reality scene (in some implementations, corresponding to a camera view of the physical environment) that corresponds to a physical position in the physical environment. This correspondence between the placement position of each virtual object in the mixed reality scene and a physical position in the physical environment may allow the system to detect a distance between and/or positioning and/or orientation of the electronic device (i.e., the user) relative to the virtual object(s) placed in the mixed reality scene including the physical environment. This correspondence between the placement position of each virtual object and a physical position in the physical environment may also allow the system to detect a distance between and/or relative positioning and/or relative orientation of different virtual objects placed in the mixed reality scene including the physical environment.

In some implementations, this detection and/or tracking of the positions of each of the virtual objects in the mixed reality scene of the physical environment, and detection and/or tracking of the position of the electronic device/user, may be based on respective individual three-dimensional coordinate positions of the virtual object(s) and the electronic device/user. For example, each virtual object in the mixed reality scene of the physical environment may have an associated three-dimensional coordinate position, for example, an associated (x,y,z) coordinate position. The (x,y,z) coordinate position of each virtual object in the mixed reality scene may correspond to a physical, three-dimensional (x,y,z) coordinate position in the physical environment. Similarly, the electronic device/user may have an associated three-dimensional (x,y,z) coordinate position in the physical environment. The respective three-dimensional (x,y,z) coordinate positions of the virtual object(s) and of the electronic device/user may be intermittently updated, or substantially continuously updated, as the mixed reality scene is updated to reflect movement of the electronic device/user, movement/animation of the virtual object(s), and the like. As the detected three-dimensional (x,y,z) coordinate position(s) of the virtual object(s) and the detected three-dimensional (x,y,z) coordinate position of the electronic device/user are updated, the respective detected three-dimensional (x,y,z) coordinate positions may be used to calculate distances, and update calculated distances, between the electronic device/user and the virtual object(s) and/or between the virtual objects. The system may use distances calculated in this manner to determine when the electronic device/user has moved to a position that is within the set threshold distance of a virtual object or has moved outside of the set threshold distance of the virtual object, when a second virtual object is within the set threshold placement distance of a first virtual object or has moved outside of the set threshold placement distance of the second virtual object, and the like.

In placing virtual object(s), such as, for example, virtual character(s), in the mixed reality scene including the physical environment as described above (in particular, in the camera view of the physical environment, as described above), interaction with the user and/or interaction with each other may make the virtual object(s) feel more present, and more realistic. The virtual object(s) may follow an event-based interaction system, including a set of animations, triggers and effects determined based on, for example, a set of previously defined situational rules. In some implementations, a set of properties may be associated with each virtual object that is placed in the mixed reality scene. Detected animation triggers/triggering conditions may cause the system to execute selected, or identified, animations, based on the type of triggering condition(s) encountered.

In some implementations, the set of properties may include, for example, a set of animation properties. The set of animation properties may include, for example, a plurality of different animations a particular object may support, or may be able to perform, in response to detected animation triggers, conditions, inputs and the like. In some implementations, a plurality of different animations may be associated with each virtual object placed in the physical environment. The system may select, or identify, one of the animations, from the plurality of animations, based on the detected animation triggers, or triggering conditions, for execution by the virtual object in a particular situation or environment.

For example, a placement animation, may be executed by the virtual object upon placement in the mixed reality scene including the physical environment, to exhibit a sense of arrival at the placement position in the mixed reality scene. In the example illustrated in <FIG>, a virtual object C is placed on a real table represented in a mixed reality scene <NUM> (or, camera view <NUM>) including the physical environment <NUM>. Upon placement of the virtual object C on the table, the virtual object C may, for example, open it eyes, glance around to provide an effect of familiarizing itself with its surroundings, and the like.

An idle animation may be executed by the virtual object while the virtual object is displayed in the mixed reality scene, but without any inputs, conditions and the like which would trigger a specific reaction or behavior (such as, for example, the detected approach of the user). In the example illustrated in <FIG>, in absence of detection of, for example, the presence of the electronic device/camera/user and/or another virtual object and/or other condition which would trigger a specific response from the virtual object C, the virtual object C moves back and forth along the surface of the table, consistent with idle behavior characteristics of this particular virtual object C (i.e., idle behavior that may be expected from a crab placed on a tabletop). The execution of this idle animation of the virtual object C in the absence of a specific condition triggering a specific interaction may provide a more realistic effect, causing the virtual object C to appear to be more present in the physical environment <NUM>.

An affected animation, or interactive animation, may be executed by the virtual object when, for example, the system detects the approach of the user and/or another virtual object (i.e., detects the electronic device/camera/user and/or the other virtual object within a set threshold distance of the placement position of the virtual object, as illustrated in the example shown in <FIG>, <FIG> and <FIG>). In some implementations, specific, or custom, affective/interactive animations, may be executed by the virtual object(s) in specific circumstances/in response to specific animation triggers. Specific circumstances may include, for example, a situation in which a first virtual object and a second virtual object are in some way related, and in which certain reaction(s) would be expected in the presence of the first and second virtual objects. In the example illustrated in <FIG>, the virtual object C animates in response to the detection of the electronic device/camera/user within the set threshold distance D2. In this example, the virtual object C turns to face the electronic device/camera/user, and snaps its claws (as may be expected from a crab in this situation). Similarly, in the example illustrated in <FIG>, both the virtual object C and the virtual object D animate in response to the detection of each other within the set threshold placement distance D3. In this example, the animation of the virtual object C and the virtual object D involves a fight, or clawing at each other (as may be expected in this situation).

In some implementations, the set of properties may include behavioral properties such as, for example, logic that triggers specific actions or animations to be performed by the virtual character in response to detected conditions or inputs. The set of behavioral properties may be contextual, in that the actions and/or animations triggered are appropriate and/or expected and/or convincing, taking into consideration the type of virtual object(s) involved, relationships between the virtual object(s) and/or the user, the mixed reality scene, or camera view, in which the virtual object(s) are placed, and the like, as illustrated, for example, in <FIG> and <FIG>. These contextual behavioral properties or characteristics may further enhance the lifelike nature of the virtual object(s) in the mixed reality scene.

For example, in some implementations, the detection of the electronic device, or camera position (essentially corresponding to the position of the user), within the set threshold distance, or set threshold proximity, of the placement position of the virtual object may trigger a set of scripted behaviors, or scripted animations, causing the virtual object to appear to interact with the user in a contextual, appropriate manner as described above. This may include, for example, looking in the direction of the detected electronic device/camera/user (as described above with respect to <FIG> and <FIG>), waving, jumping, and other forms of acknowledgement and interaction with the user. Similarly, detected movement of the electronic device/camera position/user position from within the set threshold distance to outside of the set threshold distance may trigger a different set of scripted behaviors or animations of the virtual object.

Likewise, in some implementations, detection of a second virtual object within the set threshold placement distance, or set threshold placement proximity, of the placement position of a first virtual object may trigger a set of scripted behaviors, or scripted animations, causing the first and second virtual objects to appear to interact with each other in a contextual, appropriate manner. This may include, for example, turning to look at each other (as described above with respect to <FIG>), moving toward each other/gesturing to each other (as described above with respect to <FIG>), the gaze of the second virtual object following the first object as the first object moves, the second virtual object reacting appropriately/contextually to actions of the first virtual object (as described above with respect to <FIG>), laughing, fighting, dancing, and other forms of acknowledgment and interaction that would be a convincing and/or expected interaction between the first and second virtual objects. Similarly, detected movement of second virtual object from within the set threshold distance to outside of the set threshold distance may trigger a different set of scripted behaviors or animations of the first virtual object.

As noted above, each of the sequential illustrations of the mixed reality scenes <NUM>, or in some implementations, camera views <NUM>, shown in <FIG> is a still, or static, two-dimensional image, representing a snapshot at a point in a particular animation and/or interaction of the first and second virtual characters C and D. However, it may be understood that the animation, or interaction, displayed by the first and second virtual characters C and D in the mixed reality scene <NUM> may be a substantially continuous dynamic animation of 3D virtual object(s), or an intermittent dynamic animation of 3D virtual object(s), providing a relatively realistic representation of the movement of the virtual character.

A set of scripted behaviors or animations could be, more simply, randomly executed by the virtual objects in the mixed reality scene, or camera view, of the physical environment, without taking into account camera/user position and/or orientation, and/or virtual object position and/or orientation. However, this approach would not result in as realistic or convincing an effect. Combining these inputs related to the relative position/orientation of the electronic device/camera/user and the virtual object(s) and these sets of properties into a system may cause virtual objects to appear more convincingly present in the mixed reality scene including the physical environment, as the virtual objects interact with users and/or each other, similar to how a real person, a real animal, and other such object would interact in a similar environment, and when confronted with similar circumstances. Reactions, responses, behaviors, interactions and the like triggered in this manner may provide a realistic, convincing mixed reality environment, thus enhancing the user experience.

Another example of the placement of virtual objects in a camera view <NUM> of a physical environment, and interaction of virtual characters selected from a gallery <NUM> of virtual characters, is illustrated in <FIG>. In this particular example, an electronic device <NUM> may include a display <NUM>, and an imaging device <NUM>, or a camera <NUM>. A mixed reality scene <NUM>, or camera view <NUM>, including the physical environment, may be displayed on the display <NUM> based on images captured by the camera <NUM>. The gallery <NUM> may include a plurality of virtual objects, or virtual characters, for selection for selection and placement by the user in the mixed reality scene <NUM>, or camera view <NUM> including the physical environment displayed on the display <NUM>.

A first virtual character 330A may be selected by the user, as shown in <FIG>, and placed at a first placement position 310A in the mixed reality scene <NUM>, or camera view <NUM>, as shown in <FIG>. A second virtual character 330B may then be selected by the user, as shown in <FIG>, for placement at a second placement position 310B in the mixed reality scene <NUM>, or camera view <NUM>, as shown in <FIG>. The placement of the second virtual character 310B within a threshold placement distance of the first virtual character 310A may be detected and interpreted by the system as an animation trigger. That is, as shown in <FIG>, the first virtual character 330A may react to the placement of the second virtual character 330B by, for example, turning toward the second virtual character 330B, smiling, and the like. Thus, the detected animation trigger may cause identification of animation(s) for at least one of the first virtual character 310A or the second virtual character 310B. Additional virtual characters may be added to the mixed reality scene <NUM>, or camera view <NUM>, in a similar manner as described above. Likewise, virtual characters may be removed from the mixed reality scene <NUM>, or camera view <NUM>, in a similar manner.

One, or both, of the first virtual character 330A and/or the second virtual character 330B may animate in response to detection of a placement distance between the first and second virtual characters 330A and 330B that is less than a threshold placement distance D3 (i.e., in response to the detection of this animation trigger). The system may select, or identify, the type of animation, from a plurality of animations, to be executed by the first and/or second virtual characters 310A and 310B based on, for example, the scripted behaviors, triggering conditions, and the like described above. In some implementations, the distance between the first and second virtual characters 330A and 330B may be less than or equal to the set threshold placement distance at initial placement in the camera view <NUM> of the physical environment. In some implementations, the user may move one, or both, of the first virtual character 330A and/or the second virtual character 330B to move the first and second virtual characters to within the set threshold distance D3 to cause one of the first virtual character 330A or the second virtual character 330B to animate.

In the example shown in <FIG>, in response to detection of the first and second virtual characters 330A and 330B at a distance that is less than or equal to the set threshold placement distance D3 (for example, a first animation trigger), the second virtual character 330B animates. In this example, the animation of the second virtual character 330B includes dancing. In response to the animation (dancing) of the second virtual character 330B (for example, a second animation trigger), the first virtual character 330A reacts with a contextual animation, in the form of clapping in this example. In some implementations, an orientation and/or a size, or scale, of the virtual characters in the camera view <NUM> may be adjusted. In some implementations, the adjustment in orientation and/or size/scale may be incorporated into the animation of the virtual character(s). In some implementations, the adjustment in orientation and/or size/scale may be implemented by a user input. <FIG> illustrates an example in which an orientation of the second virtual character 330B is adjusted. <FIG> and <FIG> illustrate an example in which a size, or scale, of the second virtual character 330B is adjusted.

As noted above, each of the sequential illustrations of the mixed reality scenes <NUM>, or camera views <NUM>, shown in <FIG> is a still image, representing a snapshot at a point in a particular animation and/or interaction of the first and second virtual characters 330A and 330B. However, it may be understood that the animation, or interaction, displayed by the first and second virtual characters 330A and 330B in the mixed reality scene <NUM>, or camera view <NUM>, may be a substantially continuous dynamic animation of 3D virtual object(s), or an intermittent dynamic animation of 3D virtual object(s), providing a relatively realistic representation of the movement of the virtual character.

<FIG> is block diagram of an electronic device <NUM> that can generate an augmented reality, or mixed reality environment, and provide for interaction of virtual characters, in accordance with implementations described herein. As shown in <FIG>, the electronic device may include a user interface system <NUM> including an output device and an input device. The output device may include, for example, a display for visual output, a speaker for audio output, and the like. The input device may include, for example, a touch input device that can receive tactile user inputs, a microphone that can receive audible user inputs, and the like. The electronic device <NUM> may also include a sensing system <NUM>. The sensing system <NUM> may include, for example, a light sensor, an audio sensor, an image sensor/imaging device, or camera, a distance/proximity sensor, a positional sensor, and/or other sensors and/or different combination(s) of sensors. Some of the sensors included in the sensing system <NUM> may provide for positional detection and tracking of the electronic device <NUM>. Some of the sensors of the sensing system <NUM> may provide for the capture of images of the physical environment for display on a component of the user interface system <NUM>. The electronic device <NUM> may also include a control system <NUM>. The control system <NUM> may include, for example, a power control device, audio and video control devices, an optical control device, and/or other such devices and/or different combination(s) of devices. The user interface system <NUM>, and/or the sensing system <NUM> and/or the control system <NUM> may include more, or fewer, devices, depending on a particular implementation, and may have a different physical arrangement that shown. The electronic device <NUM> may also include a processor <NUM> in communication with the user interface system <NUM>, the sensing system <NUM> and the control system <NUM>, a memory <NUM>, and a communication module <NUM>. The communication module <NUM> may provide for communication between the electronic device <NUM> and other, external devices.

A method <NUM> of providing for interaction of virtual objects in an augmented, or mixed reality environment, in accordance with implementations described herein, is shown in <FIG>. A user may initiate an augmented reality, or mixed reality, or virtual reality experience, using, for example, an electronic device to display a mixed reality scene including a view of a physical environment (block <NUM>). The mixed reality scene including the view of the physical environment may be, for example, a camera view of the physical environment captured by an imaging device of the electronic device, and displayed on a display device of the electronic device. One or more virtual objects may be placed in the mixed reality scene including the physical environment (block <NUM>). The one or more virtual objects may be placed in the mixed reality scene, or camera view, including the physical environment based on a selection made by the user, and displayed on the display device of the electronic device, superimposed on the mixed reality scene, or camera view.

In response to the detection of a trigger for interaction (block <NUM>), the virtual object displayed in the mixed reality scene, or camera view may be animated for interaction, based on the type of trigger detected (block <NUM>). In some implementations, detection of the electronic device within a set threshold distance, or proximity, of a placement position of the virtual object may trigger an animation for interaction with the user. In some implementations, detection of a new virtual object within a set threshold distance, or proximity, of a placement position of the existing virtual object may trigger an animation for interaction between the new virtual object and the existing virtual object. In some implementations, characteristics associated with the new virtual object may trigger a particular animation for a specific type of interaction between the new virtual object and the existing virtual object. In some implementations, conditions in the environment in which the virtual object is placed may trigger an animation of the virtual object. In some implementations, the animation of the virtual object may be implemented in accordance with a set of rules. The process may continue until it is determined that the trigger for interaction is no longer detected (block <NUM>) and/or the augmented reality/mixed reality experience has been terminated (block <NUM>).

<FIG> shows an example of a computer device <NUM> and a mobile computer device <NUM>, which may be used with the techniques described here.

In addition, short-range communication may occur, such as using a Bluetooth, Wi-Fi, or other such transceiver (not shown).

Various implementations of the systems and techniques described here can be reali7ed in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof.

In some implementations, the computing devices depicted in <FIG> can include sensors that interface with a virtual reality (VR headset/HMD device <NUM>). For example, one or more sensors included on a computing device <NUM> or other computing device depicted in <FIG>, can provide input to VR headset <NUM> or in general, provide input to a VR space. The sensors can include, but are not limited to, a touchscreen, accelerometers, gyroscopes, pressure sensors, biometric sensors, temperature sensors, humidity sensors, and ambient light sensors. The computing device <NUM> can use the sensors to determine an absolute position and/or a detected rotation of the computing device in the VR space that can then be used as input to the VR space. For example, the computing device <NUM> may be incorporated into the VR space as a virtual object, such as a controller, a laser pointer, a keyboard, a weapon, etc. Positioning of the computing device/virtual object by the user when incorporated into the VR space can allow the user to position the computing device so as to view the virtual object in certain manners in the VR space. For example, if the virtual object represents a laser pointer, the user can manipulate the computing device as if it were an actual laser pointer. The user can move the computing device left and right, up and down, in a circle, etc., and use the device in a similar fashion to using a laser pointer.

In some implementations, one or more input devices included on, or connect to, the computing device <NUM> can be used as input to the VR space. The input devices can include, but are not limited to, a touchscreen, a keyboard, one or more buttons, a trackpad, a touchpad, a pointing device, a mouse, a trackball, a joystick, a camera, a microphone, earphones or buds with input functionality, a gaming controller, or other connectable input device. A user interacting with an input device included on the computing device <NUM> when the computing device is incorporated into the VR space can cause a particular action to occur in the VR space.

In some implementations, a touchscreen of the computing device <NUM> can be rendered as a touchpad in VR space. A user can interact with the touchscreen of the computing device <NUM>. The interactions are rendered, in VR headset <NUM> for example, as movements on the rendered touchpad in the VR space. The rendered movements can control virtual objects in the VR space.

In some implementations, one or more output devices included on the computing device <NUM> can provide output and/or feedback to a user of the VR headset <NUM> in the VR space. The output and feedback can be visual, tactical, or audio. The output and/or feedback can include, but is not limited to, vibrations, turning on and off or blinking and/or flashing of one or more lights or strobes, sounding an alarm, playing a chime, playing a song, and playing of an audio file. The output devices can include, but are not limited to, vibration motors, vibration coils, piezoelectric devices, electrostatic devices, light emitting diodes (LEDs), strobes, and speakers.

In some implementations, the computing device <NUM> may appear as another object in a computer-generated, 3D environment. Interactions by the user with the computing device <NUM> (e.g., rotating, shaking, touching a touchscreen, swiping a finger across a touch screen) can be interpreted as interactions with the object in the VR space. In the example of the laser pointer in a VR space, the computing device <NUM> appears as a virtual laser pointer in the computer-generated, 3D environment. As the user manipulates the computing device <NUM>, the user in the VR space sees movement of the laser pointer. The user receives feedback from interactions with the computing device <NUM> in the VR environment on the computing device <NUM> or on the VR headset <NUM>.

In some implementations, a computing device <NUM> may include a touchscreen. For example, a user can interact with the touchscreen in a particular manner that can mimic what happens on the touchscreen with what happens in the VR space. For example, a user may use a pinching-type motion to zoom content displayed on the touchscreen. This pinching-type motion on the touchscreen can cause information provided in the VR space to be zoomed. In another example, the computing device may be rendered as a virtual book in a computer-generated, 3D environment. In the VR space, the pages of the book can be displayed in the VR space and the swiping of a finger of the user across the touchscreen can be interpreted as tuming/flipping a page of the virtual book. As each page is turned/flipped, in addition to seeing the page contents change, the user may be provided with audio feedback, such as the sound of the turning of a page in a book.

In some implementations, one or more input devices in addition to the computing device (e.g., a mouse, a keyboard) can be rendered in a computer-generated, 3D environment. The rendered input devices (e.g., the rendered mouse, the rendered keyboard) can be used as rendered in the VR space to control objects in the VR space.

Computing device <NUM> is intended to represent various forms of digital computers and devices, including, but not limited to laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device <NUM> is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, and other similar computing devices.

A number of embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the scope of the specification.

Claim 1:
A computer-implemented method, comprising:
displaying (<NUM>), by an electronic device, a mixed reality scene;
detecting a selection of a first virtual object for placement in the mixed reality scene, including detecting an initial placement of the first virtual object in the mixed reality scene;
displaying the selected first virtual object at a placement position in the mixed reality scene in response to the detected selection;
detecting (<NUM>) an animation trigger, of a plurality of animation triggers, including:
detecting a distance between the electronic device and the placement position of the first virtual object; and
determining that the detected distance is less than or equal to a threshold distance;
identifying (<NUM>) an animation, from a plurality of animations, for the first virtual object based on the detected animation trigger, wherein a detected movement of the electronic device from within the threshold distance to outside of the threshold distance triggers identifying a different animation, and wherein identifying includes:
identifying an interactive animation in response to the determination that the detected distance that is less than or equal to the threshold distance, and
identifying an idle animation in response to the detection of the initial placement of the first virtual object; and
executing the identified animation of the first virtual object in response to the detected animation trigger, and executing identifying includes:
executing the interactive animation, causing the first virtual object to execute a contextual interaction with a user of the electronic device in response to the determination that the detected distance is less than or equal to the threshold distance, and
executing the idle animation of the first virtual object while the first virtual object is displayed in the mixed reality scene, the idle animation of the first virtual object corresponding to an idle behavior characteristic associated with the first virtual object.