Syndication of direct and indirect interactions in a computer-mediated reality environment

In various embodiments, computerized methods and systems for syndicating direct and indirect interactions with objects in a computer-mediated environment to facilitate precise interactions with the objects in the computer-mediated environment are provided. The system detects a direct interaction with an object in the computer-mediated reality environment. The direct interaction may be a natural or hypernatural interaction. Subsequently, the system may determine various options of indirect interaction with the object related to the direct interaction. The indirect interaction may be generated by a controller. Upon receiving an indirect interaction, the system may modify the object based on the syndication of the direct interaction and the indirect interaction.

BACKGROUND

Computer-mediated reality (CMR) refers to one's perception of reality being mediated through the use of a computing device, e.g., a wearable computer or handheld device. Typically, the computing device mediates the user's visual perception of the environment. Various CMR technologies, such as virtual reality (VR), augmented reality (AR), and mixed reality (MR), enable users to view and perceive computer-generated two-dimensional (2D) objects and three-dimensional (3D) objects, as if the objects were actually present within the user's perceived environment. Advancements in computing have fueled tremendous development efforts to apply these technologies to both entertainment and enterprise marketplaces.

Interaction with objects in a CMR environment is often cumbersome and error prone. Reaching with hands into the CMR environment can be exhaustive, in particular if the hands had to be held in midair. Further, it is difficult for users to accurately express their intention through the interaction with computer-generated objects. Further, users may not receive effective tactile feedback for the interaction, making accurate interaction more difficult, and potentially erroneous. To this end, the user may be unaware of the need to adjust the manner of interaction. In some instances, the computing device may misunderstand the user's intention and erroneously react to the user's interaction, e.g., to mediate the CMR environment differently from the user's preference.

SUMMARY

Embodiments described herein provide methods and systems for syndicating direct and indirect interactions in CMR environments. More specifically, a syndicate of direct and indirect interactions related to an object in a CMR environment is utilized to further mediate the CMR environment.

In various embodiments, a direct interaction (e.g., a natural or hypernatural interaction) with an object in the CMR environment is received by a computing device configured to mediate the CMR environment. The computing device may determine one or more options for also indirectly interacting with the object based at least in part on a characteristic of the direct interaction, e.g., a property of the object, the manner of the interaction, or related contextual information. Based on the characteristic of the direct interaction, the computing device can determine one or more options for indirectly interacting with the object. Optionally, the computing device may augment the CMR environment with a presentation of these options for indirect interaction, e.g., to guide the user in indirectly interacting with the object. Upon receiving a signal indicating at least one option of indirect interaction, the computing device may modify the CMR environment according to the syndication of the direct interaction and the indirect interaction.

To this end, disclosed methods and systems for syndicating direct and indirect interactions enable not only intuitive direct interactions with objects in CMR environments, but also enable syndicated indirect interactions, to enable the computing device to better understand the user's intention, and accordingly quickly modify one or more objects in the CMR environment with precision.

DETAILED DESCRIPTION

In accordance with embodiments described herein, the CMR environment can include any one of a virtual reality (VR) environment, an augmented reality (AR) environment, and a mixed reality (MR) environment, each of which are at least partially rendered by one or more computing devices and provided for immersive display to a user. The term “environment” can reference a portion of or an entire collection of rendered objects available for perceivable display to the user. For instance, in a fully virtual environment (e.g., VR), the environment can reference one or more sets of rendered objects, a rendered background or particle system, or any portion and/or combination thereof. In a partially virtual environment (e.g., AR or MR), the environment can reference one or more sets of rendered objects, a particle system, a real-world space or backdrop, or any portion and/or combination thereof.

CMR technologies can enable the realistic perception of computer-generated 3D objects and, in some configurations, can enable perceived interactivity with these objects. There are three major design approaches that exist in 3D interaction research: naturalism (i.e., “natural”), magic, and hypernatural. “Naturalism attempts to design interaction techniques to work exactly the way the real world works or at least as close as is practically possible. Magic techniques try to enhance usability and performance by giving the user new abilities and non-natural methods for performing tasks. Hypernatural techniques are magic techniques that use natural movements but make them more powerful by giving the user new abilities or intelligent guidance. Increased interaction fidelity is what distinguishes hypernatural techniques from standard magic techniques.” R. P. McMahan, Kopper, R., and Bowman, D. A., “Principles for Designing Effective 3D Interaction Techniques,” inHandbook of Virtual Environments: Design, Implementation, and Applications,2nd ed., K. Hale and Stanney, K. Boca Raton, Fla. CRC Press, 2015, pp. 285-311.

In dynamic and constantly changing CRM environments, it is important to provide users with intuitive modes of interaction with precision in controllability. Each different interaction technique has its advantages and disadvantages. A natural interaction with 3D objects has the advantage of being familiar to users because it corresponds to how we interact with objects in the real world. However, interacting with 3D objects that appear out of arm's reach can be difficult for natural interaction techniques. Additionally, the virtual objects that users are manipulating have no passive or active haptic feedback. At a distance, hypernatural or magic interaction methodologies are generally better than the natural interaction methodology. Ray casting, by way of example, is a feature typically employed by various hypernatural interaction methodologies for interacting with distant objects. With ray casting, a virtual light ray of sorts, projected from a user's hand or head, for example, can enable the user to interact with objects that are far away or out of arm's reach.

For purposes of the present disclosure, the hypernatural and magic interaction methodologies will be grouped together and referenced as hypernatural interaction techniques. It is contemplated, however, that in some embodiments the magic and hypernatural interaction techniques can be distinguished and considered as distinctive interaction techniques, and any reference to a hypernatural interaction technique could be referencing any individual one, or both of, the magic and hypernatural interaction methodologies.

Direct interaction, as discussed in the present disclosure, refers to the interaction that the operator (e.g., a finger) or an extension thereof (e.g., a virtual light ray of the finger) will directly intersect with the target object (e.g., a 3D virtual object) in a CMR environment. Direct interaction may include natural interaction or hypernatural interaction techniques.

However, direct interactions alone are usually ambiguous and also lack precision. As an example, a direct interaction of grabbing a virtual object may be interpreted as a selection or a squeezing of the virtual object. If the user's intent is to squeeze the virtual object, it would be difficult for a computing device to determine the intended force associated with the squeeze. As another example, a direct interaction of touching a virtual object may be interpreted as a selection or a pushing of the virtual object. If the user's intent is to push the virtual object, a computing device may not be able to determine the force with which the virtual object was pushed, and thereby determine the resulting distance that the object was pushed. In both examples, since the virtual object, unlike a physical object, cannot form an action-reaction force pair based on the direct interaction, the user also cannot receive tactile feedback from the direct interaction.

Conversely, indirect interaction, as discussed in the present disclosure, refers to the interaction that the operator (e.g., a controller device) will only indirectly interact with the target object (e.g., a 3D virtual object) in a CMR environment via signals, e.g., electrical signals, optical signals, or radio signals. In various embodiments, indirect interaction is based on controller devices, such as a remote control or a joystick, by way of example. That is, input devices (or “controllers”), such as mice, keyboards, joysticks, virtual reality controllers, touchpads, and the like, are utilized to generate controlling signals to indirectly interact with intended targets in the CMR environment. While controllers are typically utilized for gaming applications, they can be used in any type of application for conducting basic to advanced interactions with rendered objects. Advantageously, controllers can be used to disambiguate the user intention, e.g., via a selection from a host of options. Further, controllers can provide precise control of the magnitude of the controlling signals, e.g., the voltage level or the time of electrical signals.

Embodiments of the present disclosure provide systems and methods for syndicating both direct and indirect interactions with objects in a CMR environment. In other words, a user can interact naturally with a target object in the CMR environment using direct interaction techniques. Meanwhile, the user can also use indirect interaction techniques to more precisely control the reaction of the target object pertaining to the user's intent associated with the direct interaction. In various embodiments, the syndicate of the direct and indirect interactions is used to modify the target object or the CMR environment in general.

By way of example, a direct interaction with an object in the CMR environment may be received by a computing device configured to mediate the CMR environment. The computing device may then determine one or more options of indirect interaction with the object based at least in part on a characteristic of the direct interaction, such as the point of interaction, e.g., the corner, the edge, the surface, or the interior of the object. Based on the characteristic of the direct interaction, the computing device can determine one or more available options for indirectly interacting with the object. For instance, the direct interaction can select a particular object, while the indirect action modifies its position (e.g., facilitates a linear movement) or modifies its orientation (e.g., facilitates a rotational movement).

Optionally, the computing device may augment the CMR environment with a presentation of these options of indirect interaction, e.g., to guide the user on available indirect interactions with the object. These options may be based on the available devices connected or in communication with the head-mounted display (HMD) device, the context, the VR objects selected, etc. Upon receiving a signal indicating at least one option of indirect interaction (e.g., a linear movement), the computing device may modify the CMR environment according to the syndicate of the direct interaction and the indirect interaction. In this case, the syndicate may be a composite command of moving an object of the CMR environment in a certain direction, and in some embodiments, with a specific distance.

With reference now toFIG. 1, an exemplary system for syndicating direct and indirect interactions in a CMR environment is provided, in accordance with some embodiments of the present disclosure. In this CMR environment, head-mounted display (HMD) device120enables user110to see virtual object140. User110may interact with virtual object140directly. By way of example, left hand112may reach the location of feature142of virtual object140, or move around feature142in a certain direction. After HMD device120, and/or a computing device (e.g., stationary sensors in the room, hand wearable devices, etc.) connected or in communication with the HMD device120, detects this direct interaction, HMD device120may cause menu150to be presented to user110. Menu150may present several options of indirect interaction with feature142or virtual object140in general. For example, options of indirect interaction with feature142may include change colors, change orientation, change size (e.g., zoom in or zoom out), change shape, move, rotate, twist, bend, etc. Similar or different options of indirect interaction with virtual object140may also be presented in menu150.

In some embodiments, menu150is optional. After HMD device120detects a direct interaction, HMD device120causes a default behavior associated with the detected direct interaction to be activated. The default behavior associated with the detected direct interaction may depend on the context and the characteristics of the direct interaction.

User110may use another hand114to operate controller130, e.g., to select a menu item from menu150, and to provide a signal of indirect interaction to HMD device120. Although controller130is shown as a dial inFIG. 1, controller130could be in a variety of form factors, e.g., a slate, a joystick, a touchpad, a display, a mouse, a keyboard, a VR controller, and the like. User110may use a display with a touchscreen as a controller. User110may use gestures to interact with controller130, such as hovering over a slate or a smartphone with hand/finger gestures.

In some embodiments, multiple controllers are used separately or simultaneously for indirect interaction. By way of example, each controller is used to manipulate a particular property of the selected virtual object, such as a dial being used to rotate the virtual object, a slider being used to replace the virtual object with the next one in a row, a slate being used to resize the virtual object, and a stylus being used to reshape the virtual object.

In some embodiments, controller130is a wearable device, e.g., a smartwatch. The wearable device may connect to HMD device120to provide indirect interactions. In one embodiment, controller130may be a smartphone, which could be strapped to the user's arm as a special wearable device to indirectly interact with the object. The wearable device can be configured to detect the position or orientation of itself or its host (e.g., a wrist or an arm). For instance, the wearable device could employ built-in sensors (e.g., accelerometer, gyroscope, magnetometer, or GPS sensors). In another instance, the wearable device could emit signals (e.g., light or IR) that can be detected by HMD device120to determine the position and/or orientation of the wearable device relative to HMD device120.

When controller130is a wearable device, the user may use body movement (e.g., the arm movement) to cause the wearable device to move in a certain pattern, cause the wearable device to change position (e.g., facing up or facing down), or cause the wearable device to change its location (e.g., the 3D coordinates) in the CMR environment. Subsequently, the wearable device may send signals to HMD device120to indicate the movement, the position, or the location of the wearable device. Such signals may be used to indirectly interact with objects (e.g., virtual objects).

In some embodiments, controller130may be employed to activate a target menu item, e.g., by pushing a button (not shown) on controller130. Subsequently, user110may use controller130to control the variation and magnitude of the operation, as a way of facilitating the indirect interaction. As an example, user110may use controller130to browse through a series of color options if the indirect interaction of “change colors” is activated. Feature142may change colors dynamically according to, for instance, the present position of controller130. In another example, when an indirect interaction of “slide” is activated and the direct interaction via hand112already indicates a directional movement (e.g., moving right), the operation of controller130may generate a signal of distance to be moved. Optionally, such distance information may be presented along with feature142in the CMR environment, such as overlaying the distance information to menu150. As such, the syndicate of the direct interaction of moving feature142in the right direction, combined with the indirect interaction of a distance to be moved, can enable HMD device120to render a different view, in which feature142moves directionally right with the specified distance.

In some embodiments, indirect interaction is visualized as virtual direct interaction to user110. For example, when using a dial to rotate an object around a vertical axis, a virtual hand may be rendered rotating the object, corresponding to the actual hand rotating the dial. When the user's hand touches the dial, user110sees a virtual hand moving toward the object of choice, and then rotates the objects corresponding to the motion of the controller. The result is an indirect interaction that is visualized as direct manipulation.

In various embodiments, controller130may be connected to HMD device120using wired technologies (e.g., coaxial cable, twisted pair, optical fiber, etc.) or wireless technologies, including a short-range wireless telecommunications connection, e.g., a Wi-Fi® connection, a Bluetooth connection, or the like.

In some embodiments, a syndicate of the gaze of user110and an indirect interaction based on controller130is used to modify the CMR environment. In other words, the gaze of user110is also referred to as a form of direct interaction in this disclosure. By way of example, the gaze of the user110may be used to select an object to interact with, while controller130is employed by the user110to indirectly interact with the object. In this example, the controller130can be utilized to rotate the object that the user is gazing at. In another example, the trajectory of the eye gaze may be interpreted as a direction for moving the object, while indirect interactions based on inputs received by the controller130may be used to determine the precise movement of the object, e.g., the distance.

In some embodiments, controller130may provide haptic feedback to user110. In one example, controller130may provide an act of vibrating in response to hand112touching feature142. Further, controller130may provide different types of vibrations depending on the interaction point of the direct interaction. For instance, if the interaction point is a corner of feature142, controller130will vibrate only once. However, if the interaction point is a surface of feature142, controller130will vibrate multiple times. In another example, assuming user110uses controller130to control the rotation of feature142, haptic feedback may be provided to guide user110without the need for user110to look at controller130. For instance, controller130could tick for every predefined angle of rotation (e.g., every 30 degrees).

In various embodiments, controller130can be used by the dominant hand or the non-dominant hand of user110. In this way, user110may choose to use the dominant hand to directly interact with objects in the CMR environment while using the non-dominant hand to provide indirect interactions with those objects, e.g., via controller130. Similarly, user110may switch dominant and non-dominant hands to facilitate the direct and indirect interactions.

With reference toFIG. 2, interaction syndication system200for syndicating direct and indirect interactions is provided, in accordance with some embodiments of the present disclosure. Interaction syndication system200includes computing device210and controller230. In various embodiments, computing device210can include a head-mounted display (HMD) device, like HMD device120illustrated inFIG. 1. Controller230could be configured in a variety of form factors, such as controller130illustrated inFIG. 1.

In some embodiments, computing device210may include any type of computing device, such as computing device900described below with reference toFIG. 9. As will be described in more detail below, in various embodiments, computing device210can include, among other things, direct interaction component212, indirect interaction component214, interaction syndication component216, sensing component222, and rendering component224. In further embodiments, controller230can include controlling component232and feedback component234. In accordance with embodiments described herein, it is contemplated that the aforementioned components can be implemented in any one or more components or subcomponents of computing device210. For instance, any one of direct interaction component212, indirect interaction component214, and interaction syndication component216may be implemented at least in part within a processor, graphical processing unit (GPU), application code, firmware, and the like.

Further, a “component” as used herein refers to any device, process, service, or any combination thereof. A component may be implemented using hardware, software, firmware, a special-purpose device, or any combination thereof. A component may be integrated into a single device or it may be distributed over multiple devices. The various aspects of a component may be collocated or distributed. The component may be formed from other components and aspects thereof.

Computing device210may include any type of HMD or augmented reality device described below with reference toFIGS. 7 and 8. The augmented reality device is an exemplary HMD device, and other types of augmented reality devices (e.g., projectors) are contemplated in accordance with embodiments of the present disclosure. Computing device210may be a scene-aware device that understands elements surrounding a real-world environment and generates virtual objects as augmented reality images. One non-limiting example of computing device210is the HoloLens®, developed by Microsoft Corporation of Redmond, Wash.

Computing device210can be configured to capture the real-world environment based on components of computing device210, e.g., sensing component222. Sensing component222may include a depth camera and/or sensors that support understanding elements of a scene or environment, for example, generating a 3-D mesh representation of a surrounding real-world environment. Further, sensing component222can use the depth camera and/or sensors to detect direct interactions, e.g., between a physical hand and a virtual object in the CMR environment. Rendering component224may render virtual objects or images based at least in part on the 3-D mesh representation. In this regard, computing device210can specifically include functionality (e.g., augmented reality or mixed reality experiences realized through rendering component224) to modify the CMR environment (e.g., to create or modify a virtual object) based on a syndicate of direct and indirect interactions.

Direct interaction component212processes and analyzes information of detected direct interactions with objects in a CMR environment. By way of example, sensing component222may detect a direct interaction with a virtual object. Subsequently, sensing component222may send various signals related to the direct interaction to direct interaction component212. Accordingly, direct interaction component212can receive input signals that are generated based on a detected direct interaction (e.g., natural or hypernatural) directed to a computer-generated object in a CMR environment. Further, direct interaction component212can determine a characteristic of the detected direct interaction, e.g., based on the received signals.

In accordance with embodiments described herein, sensing component222can include various sensor component(s), e.g., gyroscope, accelerometer, and magnetometer, infrared lights, infrared cameras, motion sensors, light sensors, 3-D scanners, CMOS sensors, GPS radio, etc. In various embodiments, sensing component222can employ the aforementioned sensors, among other things, to identify an intended target (e.g., a virtual object) from the CMR environment, and in some instances, track physical locations and movements (e.g., eye tracking, body movement, finger positions, etc.) of a user, e.g., who wears or otherwise links to computing device210.

Accordingly, from the received signals, direct interaction component212may determine various characteristics associated with the detected direct interaction, such as the gaze position of the user; the locations of various body parts (e.g., hands or fingers) before, during, and after the detected direct interaction; the characteristics of the body movement (e.g., the velocity, the acceleration, the direction, the frequency of the body movement); the interaction point of the virtual object related to the detected direct interaction; or any other characteristics associated with the detected direct interaction.

In some embodiments, direct interaction component212can determine the characteristics of the direct interaction based at least in part on the relative proximity between a moving object (e.g., a hand, a finger) and a virtual object in the CMR environment. In one embodiment, sensing component222can detect the target object in the CMR environment based on a detected gaze of the user. In further embodiments, sensing component222can detect locations of a hand or a finger in the CMR environment. In even further embodiments, sensing component222can detect the distance between the hand or finger and the target object. As an example, if the distance between the finger and the target object becomes less than a predetermined threshold, direct interaction component212may identify the interaction as a direct interaction between the finger and the target object.

In some embodiments, direct interaction component212can recognize a gesture performed by the user. For instance, the user can perform gestures using their extremities (e.g., arms, fingers, legs, head, etc.), which can be detected, by way of example, via an optical input component such as a camera, motion sensor, infrared sensor, and the like. In this regard, more intricate details associated with the user's hand can be analyzed to facilitate the identification of the direct interaction. Specifically, an orientation of the hand, finger positioning and joint angles, gestures (e.g., pointing), or other hand characteristics that are detectable based on an analysis of the received input data can enable direct interaction component212to characterize the direct interaction. Accordingly, direct interaction component212may interpret such gestures as direct interactions with the CMR environment, e.g., with the virtual object presently gazed at by the user. In various embodiments, such gestures can be mapped to a predefined type of direct interaction, e.g., based on a predefined configuration, a configuration set by the user, or a default configuration.

Controller230can include controlling component232and feedback component234, among other components not directly shown inFIG. 2. Similar to controller130inFIG. 1, controller230may be embodied in various form factors, such as a dial, a slate, a joystick, a touchpad, a mouse, a keyboard, a VR controller, a pen-like controller, and the like. In various embodiments, controller230can generate signals for indirect interaction based on user inputs received thereby. In various embodiments, user inputs can be received via at least one button, touchscreen surface, motion sensor (e.g., accelerometer, gyroscope), control interface, or any combination thereof. That is, controller230can be utilized to generate control signals to indirectly interact with intended targets in the CMR environment.

Controlling component232can receive various forms of user input, e.g., via sensors or control interfaces associated with controlling component232. Depending on the form factor of controller230, the received user inputs may be encoded into various analog or digital signals to represent various characteristics (e.g., the duration and the magnitude) of one or more identifiable user inputs. In other words, such signals include measurement information of the user inputs.

Upon receiving such signals from controller230, indirect interaction component214can process and analyze such signals. By way of example, indirect interaction component214may determine whether such signals relate to a detected direct interaction, e.g., by looking up any events of detected direct interactions within a predefined duration or time window. In some embodiments, direct interaction component212can determine an interaction point of the virtual object related to the detected direct interaction, and indirect interaction component214can further determine a modification setting, such as a magnification setting related to the interaction point or a rotation setting of the virtual object associated with and/or relative to the interaction point.

In various embodiments, indirect interaction component214may interpret the signals based on the characteristics of the direct interaction. For example, if the direct interaction is a linear movement action, the signals of indirect interaction may include the distance of such linear movement. If the direct interaction is a rotation action, the signals of indirect interaction may include the degree of such rotation. If the direct interaction is a magnification action, the signals of indirect interaction may include the scale factor of such magnification.

In connection withFIG. 1, controller230may also be used to select a particular form of indirect interaction, e.g., based on menu150inFIG. 1. Advantageously, when there are many options of indirect interaction related to a detected direct interaction, controller230can be used to disambiguate the user intention, e.g., by selecting what the user wants from the menu.

Interaction syndication component216can process information from direct interaction component212and indirect interaction component214, and can further determine how the CMR environment should be modified. Interaction syndication component216may determine a form of the modification based on the direct interaction, and a control parameter for controlling the modification based on the indirect interaction. It may also be said that interaction syndication component216can determine the reaction of the virtual object in view of the direct and indirect interactions related to the virtual object.

Continuing with the examples previously discussed in connection with indirect interaction component214, if the direct interaction is a linear movement action (i.e., the form of the modification) and the indirect interaction further controls the distance of such linear movement (i.e., the control parameter of the form), then the syndicate of these direct and indirect interactions can cause the virtual object to move linearly for the distance corresponding to the indirect interaction. By the same token, if the direct interaction is a rotation action (i.e., the form of the modification) and the indirect interaction further controls the degree of such rotation (i.e., the control parameter of the form), then the syndicate of these direct and indirect interactions can cause the virtual object to rotate a number of degrees corresponding to the indirect interaction. If the direct interaction is a scale action (e.g., shrink or enlargement) and the indirect interaction further controls the scale factor of such magnification, then the syndicate of these direct and indirect interactions may cause the virtual object to scale (i.e., the form of the modification) based on the scale factor (i.e., the control parameter of the form) corresponding to the indirect interaction.

In various embodiments, rendering component224may cause options of direct or indirect interaction to be presented in the CMR environment, such that a user may be presented with a guide listing the available interaction options associated with the target object. Further, rendering component224can modify how virtualized objects are rendered and ultimately perceived by the user, e.g., based at least in part on the syndicate of these direct and indirect interactions. In other words, rendering component224may modify the virtual object, e.g., by modifying the configuration (e.g., the position, size, shape, angle) of the virtual object, transforming (e.g., scale, rotate, skew, stretch, warp) the virtual object, or deleting the virtual object, or even creating new ones, in accordance with the syndicate of these direct and indirect interactions.

In various embodiments, each virtual object rendered by rendering component224is positioned at a corresponding location relative to an HMD, and thereby also to the user wearing the HMD. Moreover, the rendered virtual object can be continuously modified in real time, by rendering component224, to maintain the proper perspective and configuration of rendered virtual objects, e.g., in accordance with the syndicate of these direct and indirect interactions.

In various embodiments, each of direct interaction component212, indirect interaction component214, and interaction syndication component216can generate an indication for feedback based at least in part on the determined characteristic of the detected direct interaction, indirect interaction, or the syndicate of the direct and indirect interactions. The indication for feedback may be transmitted to controller230. In some embodiments, feedback component234may provide feedback to the user. The feedback may be provided in a particular form or in a combination of forms, such as haptic feedback, thermal feedback, or audible feedback. The feedback may be provided via computing device210, controller230, or another device connected to them. In this way, the user may receive confirmation of the detected direct interaction, indirect interaction, or the syndicate of the direct and indirect interactions, and modify those interactions if necessary, all without interruptions (e.g., looking down to the controller230) to continue interactions with the CMR environment.

The specific forms of feedback may be determined based on the detected characteristics of the interaction and the implementation of feedback component234. In one example, controller230may provide an act of vibration in response to a detected direct interaction if a vibration component (e.g., a haptic motor) is installed in feedback component234. In another example, controller230may provide an audible sound for a detected indirect interaction, for instance, a speaker that beeps for every 10 degrees of detected rotation by controller230.

In some embodiments, feedback component234may provide feedback of limits of interaction. For example, for a scaling operation of a virtual object, if the scaling operation cannot be rendered further, such as due to limited free space in the scene, feedback component234may provide feedback of such limitations, e.g., in the form of playing an audible signal, in the form of a haptic feedback (e.g., a click) in controller230, in the form of stopping the operation of controlling component232, etc.

With reference toFIG. 3andFIG. 4, embodiments of the syndication of direct and indirect interactions can be explained in more detail, using exemplary implementations. For purposes of illustrating particular features of the present disclosure, the syndication of direct and indirect interactions with virtual object330is illustrated inFIG. 3.

In various implementations, virtual object330may include one or more control points that facilitate a natural or hypernatural interaction therewith. In this way, a user can directly interact with a virtual object control point to interact with the virtual object. In some further implementations, a selected control point of the intended target can be directly interacted with to facilitate directed interactions or modifications to the intended target based on the selected control point.

Left hand310, in this example, directly contacts virtual object330. In this case, the surface area of virtual object330is an identifiable virtual object control point. Optionally, a menu (not shown) of indirect interactions may be presented to the user. For example, the menu options may include pull or push. The user may select the desired menu item, e.g., push, via controller340. Controller340may have a menu selection button (not shown) for the user to activate the menu item. Alternatively, the menu item may be selected based on the user's eye gaze, e.g., a gaze position indicator (e.g., crosshairs) can be presented in the CMR environment to guide the user in making the selection. In some embodiments, there is a default action for indirect interactions. Without any selection, the user may operate controller340to indirectly interact with virtual object330.

In this implementation, the user operates controller340using right hand320. The operation may include pushing, pulling, or rotating controller340. When the user pushes controller340, controller340can employ sensors to indicate how far and how long controller340has been pushed. In this regard, the controller340can generate signals including the sensed distance. Such signals may then be utilized to define the characteristics of this indirect interaction. By way of example, virtual object330is located at the D1position during the direct interaction. When the user pushes controller340, the signals generated by controller340can be used to define the distance that virtual object330would be pushed. In one embodiment, the syndicate of these direct and indirect interactions may cause virtual object330to be pushed to the D2or D3position depending on the magnitude represented by the signal, which may correspond to the sensed duration of the push or the sensed physical force applied on controller340, and/or a displacement of controller340. In another embodiment, while controller340is being pushed, the movement of virtual object330may be simulated in the CMR environment. For example, the view may be modified to show the real-time movement of virtual object330during the user operation of controller340.

In one embodiment, controller340may have haptic rendering capability, for example, a rotating disc that displays the surface normal at the point of the finger and virtual object330. It may also render the local shape of the surface, and by doing so, guide hand320to operate controller340without the need for the user to look at controller340.

In various embodiments, feedback component342can provide a variety of feedback to the user, in response to the direct interaction, the indirect interaction, or the syndicate of the direct and indirect interactions. As an example, feedback component342may provide one kind of haptic feedback (e.g., a vibration pattern for a brief duration) when the direct interaction is detected. As another example, feedback component342may provide another kind of haptic feedback (e.g., a vibration pattern for an extended duration) during the indirect interaction. As yet another example, feedback component342may provide another kind of haptic feedback (e.g., a vibration pattern having high frequency or amplitude) when the syndicate of the direct and indirect interactions are applied to virtual object330(e.g., when virtual object330finally moved to the desired location).

With reference now toFIG. 4, the graphical illustration depicts the direct interaction between left hand410and virtual object450inside another virtual object430. Meanwhile, right hand420may operate controller440(e.g., a touchpad receiver) to indirectly interact with virtual object450. As a result, virtual object450is enlarged in this example, to become virtual object460based on the syndicate of these direct and indirect interactions.

To initiate the direct interaction with virtual object450, left hand410may use a finger to pierce through virtual object430and virtually “touch” virtual object450. In one embodiment, the CMR environment may be modified, e.g., the color of virtual object450may be changed, to indicate the detected direct interaction. In another embodiment, controller440may provide audible feedback, such as an audible beep, and/or haptic feedback or the like, to indicate the detected direct interaction.

Right hand420may use a gesture, e.g., a reverse pinch, to indirectly interact with virtual object450. The reverse pinch may be measured and translated into a scale factor. As a result, virtual object450may be transformed into virtual object460based on the syndicate of the direct interaction with virtual object450and the indirect interaction indicating the scale factor.

In some embodiments, the indirect interaction via right hand420with controller440is visualized as a virtual direct interaction. For example, suppose that the operation is sculpting in clay. Left hand410may guide the interaction to a specific location, and then fingers of right hand420can move on or gesture the touch surface of controller440to sculpt. The touch surface of controller440is 2D while object450is 3D. In this case, right hand420can define the corresponding plane of surface on object450, for example, using the palm of right hand420as a reference plane to define the corresponding plane of surface on object450. Further, a virtual tool (e.g., a graver) may be rendered to interact with virtual object450, so that the user can have a visual feedback of the sculpting operation in relationship with the indirect interaction with controller440.

Referring now toFIG. 5in light ofFIGS. 1-4,FIG. 5is a flow diagram showing a method500for syndicating direct and indirect interactions. Each block of method500, and other methods described herein, comprises a computing process that may be performed using any combination of hardware, firmware, and/or software. For instance, various functions may be carried out by a processor executing instructions stored in memory. The methods may also be embodied as computer-usable instructions stored on computer storage media. The methods may be provided by a stand-alone application, a service or hosted service (stand-alone or in combination with another hosted service), or a plug-in to another product, to name a few.

At block510, a direct interaction with an object in a CMR environment can be detected, for instance, by direct interaction component212ofFIG. 2. The CMR environment may include virtual objects rendered in a computer-generated world (e.g., VR), or may include virtual objects rendered for augmentation or mixing with the real world (e.g., AR/MR). In various embodiments, the relative proximity (i.e., a distance) between an actor (e.g., a finger or a handheld instrument) and a virtual object may be measured, e.g., by the sensing component222ofFIG. 2. If such distance is below a predetermined threshold of distance for a period longer than a predetermined threshold of time, direct interaction component212may recognize a direct interaction between the detected body part and the virtual object.

As described herein, the relative proximity can be determined based on rendering information associated with the virtual object and sensing information (e.g., via a camera or other optical sensor) associated with the actor. The rendering information can be extracted from, among other things, a transformation matrix or a projection matrix associated with the virtual object. In some instances, the relative proximity can be based further on an approximated or calculated distance between an extension of the user (e.g., the user's hand, foot, or a handheld object) and the virtual object. In various embodiments, the relative proximity is obtained after putting the location of the virtual object and the location of the actor into a same 3-D coordinate system, such that the Euclidean distance between the two locations may be calculated.

In some embodiments, the information of relative proximity, such as the distance from the hand to the virtual object, is used to control the scale of the operation. For example, imagine controlling a ragdoll puppet, the rotation operation may be applied to the entire doll when the hand is far from the doll. However, the rotation operation may be applied to the doll head only when the hand is getting closer to the doll. Similarly, the rotation operation may be applied to an eye of the doll, e.g., to rotate the doll gaze, when the hand is very close to the eye.

At block520, one or more options of indirect interaction with the object may be determined based at least in part on a characteristic of the direct interaction, for instance, by indirect interaction component214ofFIG. 2. The characteristics of the direct interaction may include a property of the virtual object, a property of the controller, the manner of the interaction, contextual information related to the direct interaction, etc. The properties of the virtual object may include the nature of the object (e.g., a virtual animal, a virtual toy, or a virtual piece of furniture), the size, the weight, the orientation, and many other properties of the virtual object. The manner of the interaction may include the point of interaction (e.g., a point or a surface), the action associated with the interaction (e.g., grabbing, pushing, pulling, rotating, piecing), the direction of the interaction (e.g., the direction related to the force), the number of fingers used to interact, etc. Contextual information related to the direct interaction may include the view, the perspective, the previous action of the user, the gaze of the user, the objects surrounding or near the target object, etc.

Based on the characteristic of the direct interaction, one or more options of indirect interactions related to the target object may be determined. As an example, if a finger touches a particular point at a virtual Earth 3D object in a direct interaction, the options of indirect interaction may include moving the virtual object linearly, rotating the virtual object along a latitude crossing the interaction point, rotating the virtual object along a longitude crossing the interaction point, magnifying the interaction point and showing more details, etc.

At block530, a signal associated with one of the one or more options of indirect interaction may be received, e.g., by indirect interaction component214ofFIG. 2. Such signal may be generated from a controller, e.g., a mouse, a joystick, a dial, or a VR controller, just to name a few. In some embodiments, the signal includes information of a selection of a particular option for indirect interaction. Further, the signal may also include information of various properties or measurements of the indirect interaction, e.g., the duration or the magnitude of the indirect interaction.

In some embodiments, indirect interaction techniques disclosed herein can enable real world objects to be used as general controllers. For example, a user is to operate multiple virtual soccer player characters in an animation, and each virtual soccer player is associated with a distinguishable physical object on a table. Further, rotating a physical object may change the animation associated with each player. In this case, the physical objects can be used as general controllers. By way of example, the user may rotate each physical object like a knob to change animation of the corresponding virtual soccer player.

At block540, the CMR environment in relation to the object may be modified based at least in part on the signal. In various embodiments, the signal related to the indirect interaction is interpreted in view of the direct interaction. The syndicate of the direct and indirect interactions is used to modify the CMR environment. In some embodiments, the target object is a virtual object, which may be re-rendered or transformed based on the syndicate of direct and indirect interactions, as illustrated inFIG. 3orFIG. 4.

In some embodiments, the target object is a physical object, e.g., a table. The direct interaction may be a preconfigured gesture, e.g., tapping the table twice. The indirect interaction, in this case, can use a joystick to select a gift from a list of gifts. The syndicate of these direct and indirect interactions in this case may cause the selected gift to be presented on the table as a new virtual object in the CMR environment. In other words, the table is augmented with the virtual gift.

In some embodiments, indirect interaction with a controller affects multiple selected objects. In an implementation of persisting selection, a user may sequentially tap multiple objects with direct interactions to select them. The objects could either stay selected, or stay selected for a certain period of time. The indirect interaction with a controller would then affect the selected multiple objects. This embodiment allows one handed interaction where the same hand can select the objects, and then reach out for the controller to control them.

Referring now toFIG. 6in light ofFIGS. 1-5,FIG. 6is a flow diagram showing a method600for transforming an object based on direct and indirect interactions. Each block of method600, and other methods described herein, comprises a computing process that may be performed using any combination of hardware, firmware, and/or software. For instance, various functions may be carried out by a processor executing instructions stored in memory. The methods may also be embodied as computer-usable instructions stored on computer storage media. The methods may be provided by a stand-alone application, a service or hosted service (stand-alone or in combination with another hosted service), or a plug-in to another product, to name a few. In various embodiments, method600may be performed in relation to block540ofFIG. 5.

At block610, a form of a transformation of the object may be determined based at least in part on the direct interaction. The form of the transformation refers to the operation to be applied to the object, e.g., a selection, a linear movement, an angular movement, a scaling operation, a deformation operation, or other types of operations relevant to the object. A host of relevant operations may be identified from characteristics associated with the direct interaction. A default operation may be configured for the direct interaction based on, e.g., the context, the object selected, and so on.

At block620, a degree of the transformation may be determined based at least in part on the indirect interaction. The degree of the transformation refers to one or more controlling parameters of the operation. For example, if the form of the transformation is a linear movement, the degree of the transformation may be a distance of the linear movement. If the form of the transformation is a rotation operation, the degree of the transformation may be an angle of the rotation. If the form of the transformation is a magnification operation, the degree of the transformation may be a ratio of the magnification. If the form of the transformation is a deformation, the degree of the transformation may be a degree of the deformation. If the form of the transformation is to animate an object over time, the degree of the transformation may be a duration or speed of the animation.

At block630, the object may be transformed based on the form and the degree of the transformation. In various embodiments, modifying the CMR environment comprises transforming the object based at least in part on the form and the degree of the operation. By way of example, the operation is to grow a bar on a bar graph. In this case, the direct interaction indicates which bar to grow, while the indirect interaction indicates how much the bar should grow. As a result, the target bar may be transformed to a different bar (e.g., with a different height) based on the identified form and degree of this transformation.

With reference toFIG. 7, exemplary images of a head-mounted display (HMD) device710are depicted. Rendered virtual objects provided by the HMD device generally appear in rendered space in virtual reality configurations. However, in augmented reality configurations, virtual objects (e.g.,730and740) may appear superimposed on a real-world background and may appear to interact with or be integral with the background. In augmented reality configurations, the background is comprised of a real-world scene, e.g., a scene that a user would perceive without augmented reality images emitted by the HMD device. For example, cube730can appear atop the shelf, while cube740can appear atop the countertop. In various embodiments, left hand750may directly interact with cube740, e.g., to touch an edge of cube740to cause it to rotate, and right hand760may provide indirect interactions with cube740, e.g., to provide the direction and the angle of rotation.

Turning toFIG. 8, a mixed reality HMD device802for augmented reality applications having, among other things, a virtual object rendering component804and an interaction syndication component806is described in accordance with an embodiment described herein. The HMD device802includes a see-through lens810which is placed in front of a user's eye812, similar to an eyeglass lens. It is contemplated that a pair of see-through lenses810can be provided, one for each eye812. The lens810includes an optical display component814, such as a beam splitter (e.g., a half-silvered mirror). The HMD device802includes an augmented reality emitter820that facilitates altering the brightness of augmented reality images. Amongst other components not shown, the HMD device also includes a processor822, memory824, interface826, a bus828, and additional HMD components830. The augmented reality emitter820emits light representing an augmented reality image840exemplified by a light ray842. Light from the real-world scene850, such as a light ray852, reaches the lens810. Additional optics can be used to refocus the augmented reality image840so that it appears to originate from several feet away from the eye812rather than one inch away, where the display component814actually is. The memory824can contain instructions which are executed by the processor822to enable the augmented reality emitter820to perform functions as described. One or more of the processors can be considered to be control circuits. The augmented reality emitter820communicates with the additional HMD components830using the bus828and other suitable communication paths. The augmented reality image840is reflected by the display component814toward a user's eye, as exemplified by a light ray816, so that the user sees an image818. In the image818, a portion of the real-world scene850, such as a countertop, is visible along with the entire augmented reality image840, such as a cube. The user can therefore see a mixed reality image818in which the cube appears to be on the top of the countertop in this example.

In various embodiments, interaction syndication component806, like interaction syndication component216inFIG. 2, may syndicate direct and indirect interactions with a virtual object, e.g., image840, and cause virtual object rendering component804to render a modification to the virtual object or a different view of the virtual object based on the syndicate of the detected direct and indirect interactions with the virtual object.

Having described embodiments of the present invention, an exemplary operating environment in which embodiments of the present invention may be implemented is described below in order to provide a general context for various aspects of the present invention. Referring initially toFIG. 9in particular, an exemplary operating environment for implementing embodiments of the present invention is shown and designated generally as computing device900.

Computing device900is but one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use of the technology described herein. Neither should the computing device900be interpreted as having any dependency or requirement relating to any one or combination of components illustrated.

The technology described herein may be described in the general context of computer code or machine-usable instructions, including computer-executable instructions such as program components, being executed by a computer or other machine. Generally, program components, including routines, programs, objects, components, data structures, and the like, refer to code that performs particular tasks or implements particular abstract data types. The technology described herein may be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, specialty computing devices, etc. Aspects of the technology described herein may also be practiced in distributed computing environments where tasks are performed by remote-processing devices that are connected through a communications network.

With continued reference toFIG. 9, computing device900includes a bus910that directly or indirectly couples the following devices: memory920, one or more processors930, one or more presentation components940, input/output (I/O) ports950, I/O components960, and an illustrative power supply970. Bus910represents what may be one or more busses (such as an address bus, data bus, or a combination thereof). Although the various blocks ofFIG. 9are shown with lines for the sake of clarity, in reality, delineating various components is not so clear, and metaphorically, the lines would more accurately be grey and fuzzy. For example, one may consider a presentation component such as a display device to be an I/O component. Also, processors have memory. The inventors hereof recognize that such is the nature of the art and reiterate that the diagram ofFIG. 9is merely illustrative of an exemplary computing device that can be used in connection with one or more aspects of the technology described herein. Distinction is not made between such categories as “workstation,” “server,” “laptop,” “handheld device,” etc., as all are contemplated within the scope ofFIG. 9and refer to “computer” or “computing device.”

Memory920includes computer storage media in the form of volatile and/or nonvolatile memory. The memory920may be removable, non-removable, or a combination thereof. Exemplary memory includes solid-state memory, hard drives, optical-disc drives, etc. Computing device900includes one or more processors930that read data from various entities such as bus910, memory920, or I/O components960. Presentation component(s)940present data indications to a user or other device. Exemplary presentation components940include a display device, speaker, printing component, vibrating component, etc. I/O ports950allow computing device900to be logically coupled to other devices, including I/O components960, some of which may be built in.

In various embodiments, memory920includes, in particular, temporal and persistent copies of interaction syndication logic922. Interaction syndication logic922includes instructions that, when executed by one or more processors930, result in computing device900performing various functions, such as, but not limited to, method500or600. In various embodiments, interaction syndication logic922includes instructions that, when executed by processor(s)930, result in computing device900performing various functions associated with, but not limited to, direct interaction component212, indirect interaction component214, sensing component222, interaction syndication component216, rendering component224, controlling component232, and feedback component234in connection withFIG. 2.

In some embodiments, one or more processors930may be packaged together with interaction syndication logic922. In some embodiments, one or more processors930may be packaged together with interaction syndication logic922to form a System in Package (SiP). In some embodiments, one or more processors930can be integrated on the same die with interaction syndication logic922. In some embodiments, processors930can be integrated on the same die with interaction syndication logic922to form a System on Chip (SoC).

Illustrative I/O components include a microphone, joystick, game pad, satellite dish, scanner, printer, display device, wireless device, a controller (such as a stylus, a keyboard, and a mouse), a natural user interface (NUI), and the like. In aspects, a pen digitizer (not shown) and accompanying input instrument (also not shown but which may include, by way of example only, a pen or a stylus) are provided in order to digitally capture freehand user input. The connection between the pen digitizer and processor(s)930may be direct or via a coupling utilizing a serial port, parallel port, Universal Serial Bus (USB) port, and/or other interface and/or system bus known in the art. Furthermore, the digitizer input component may be a component separated from an output component such as a display device, or in some aspects, the usable input area of a digitizer may coexist with the display area of a display device, be integrated with the display device, or may exist as a separate device overlaying or otherwise appended to a display device. Any and all such variations, and any combination thereof, are contemplated to be within the scope of aspects of the technology described herein.

Computing device900may include networking interface980. The networking interface980includes a network interface controller (NIC) that transmits and receives data. The networking interface980may use wired technologies (e.g., coaxial cable, twisted pair, optical fiber, etc.) or wireless technologies (e.g., terrestrial microwave, communications satellites, cellular, radio and spread spectrum technologies, etc.). Particularly, the networking interface980may include a wireless terminal adapted to receive communications and media over various wireless networks. Computing device900may communicate via wireless protocols, such as Code Division Multiple Access (CDMA), Global System for Mobiles (GSM), or Time Division Multiple Access (TDMA), as well as others, to communicate with other devices via the networking interface980. The radio communications may be a short-range connection, a long-range connection, or a combination of both a short-range and a long-range wireless telecommunications connection. A short-range connection may include a Wi-Fi® connection to a device (e.g., mobile hotspot) that provides access to a wireless communications network, such as a wireless local area network (WLAN) connection using the 802.11 protocol. A Bluetooth connection to another computing device is a second example of a short-range connection. A long-range connection may include a connection using one or more of CDMA, GPRS, GSM, TDMA, and 802.16 protocols.

For purposes of this disclosure, the word “including” has the same broad meaning as the word “comprising,” and the word “accessing” comprises “receiving,” “referencing,” or “retrieving.” In addition, words such as “a” and “an,” unless otherwise indicated to the contrary, include the plural as well as the singular. Thus, for example, the constraint of “a feature” is satisfied where one or more features are present. Also, the term “or” includes the conjunctive, the disjunctive, and both (a or b thus includes either a or b, as well as a and b).

For purposes of a detailed discussion above, embodiments of the present invention are described with reference to a head-mounted display unit; however, the head-mounted display unit depicted herein is merely exemplary. Components can be configured for performing novel aspects of embodiments, where “configured for” comprises programmed to perform particular tasks or implement particular abstract data types using code. Further, while embodiments of the present invention may generally refer to the head-mounted display unit and the schematics described herein, it is understood that the techniques described may be extended to other implementation contexts.