Patent Description:
Mixed-reality systems are typically configured as head mounted displays that generate and/or render the mixed-reality content. Continued advances in hardware capabilities and rendering technologies have greatly increased the realism of virtual objects and scenes displayed to a user within mixed-reality environments. For example, virtual objects can be placed within a mixed-reality environment in such a way as to give the impression that the virtual object is part of the real world.

Some mixed-reality systems have been configured to track the movement of a user's body parts, such as the user's hands, as the user interacts with virtual objects in the mixed-reality environment. Furthermore, some mixed-reality systems are configured to replicate the user's body parts within the mixed-reality, such that the user is able to view and control virtualized body parts within the mixed-reality environment. For instance, a user's hand can be presented as a hologram occlusion that moves within the mixed-reality environment in direct response to the movements of their own real-world hand. As the user moves their real-world hand, the hand occlusion is also moved, such that it is capable of interacting with other virtual objects within the mixed-reality environment.

Many mixed-reality systems allow users to use their body parts (or tools or other controllers manipulated thereby) to interact with virtual objects in the mixed-reality environment. For instance, some mixed-reality systems allow a user to use their hands (or virtual representations thereof) to grab, push, pull, pick up, slide, press, rotate, or otherwise interact with virtual objects or virtual input elements (such as virtual buttons) within the mixed-reality environment.

Several obstacles exist, however, in facilitating user interaction with such virtual objects or virtual input elements in mixed-reality environments. For instance, physical objects have physical geometries which constrain the way in which the object can be interacted with (e.g., where a physical button cannot be pushed from the back, and where a physical cube cannot be picked up with the back of a user's hand). In contrast, virtual objects do not have such physical constraints, which often gives rise to accidental or unintentional interaction between users and such virtual objects within the mixed-reality environments. For example, a user may unintentionally press a virtual button when the user's hand or controller passes through the virtual button from the back side of the button. Similarly, a user may unintentionally grab or interact with an object when passing their hands through the object in a relaxed position.

Accordingly, there is an ongoing need in the field of mixed-reality for providing improved user interaction with virtual objects or virtual input elements.

The subject matter claimed herein is not limited to aspects that solve any disadvantages or that operate only in environments such as those described above. Rather, this background is only provided to illustrate one technology area where some aspects described herein may be practiced.

<CIT> describes an apparatus and method for processing the manipulation of a three-dimensional (3D) virtual object. The apparatus includes an image input unit, an environment reconstruction unit, a 3D object modeling unit, a space matching unit, and a manipulation processing unit. The image input unit receives image information generated by capturing a surrounding environment including a manipulating object. The environment reconstruction unit reconstructs a 3D virtual reality space. The 3D object modeling unit models a 3D virtual object that is manipulated by the manipulating object, and generates a 3D rendering space. The space matching unit matches the 3D rendering space to the 3D virtual reality space. The manipulation processing unit determines whether the manipulating object is in contact with the surface of the 3D virtual object, and tracks the path of a contact point and processes the motion of the 3D virtual object. <CIT> describes a method for use with a head-mounted display in a physical environment that includes obtaining depth information of the physical environment and capturing a visual image of the physical environment. The method also includes determining a spatial relationship between a user of the head-mounted display and one or more physical objects included in the physical environment based on the depth information. The visual image is then segmented based on the spatial relationship to generate a segmented image that includes the one or more physical objects. The segmented image is then overlaid on a virtual image to display both the virtual image and the one or more physical objects on the head-mounted display. <CIT> describes techniques for user-interaction in augmented reality. In one example, a direct user-interaction method comprises displaying a 3D augmented reality environment having a virtual object and a real first and second object controlled by a user, tracking the position of the objects in 3D using camera images, displaying the virtual object on the first object from the user's viewpoint, and enabling interaction between the second object and the virtual object when the first and second objects are touching. In another example, an augmented reality system comprises a display device that shows an augmented reality environment having a virtual object and a real user's hand, a depth camera that captures depth images of the hand, and a processor. The processor receives the images, tracks the hand pose in six degrees-of-freedom, and enables interaction between the hand and the virtual object. <CIT> describes computer systems, apparatuses, computer-executable methods and one or more non-transitory computer-readable media for receiving gesture input via virtual controls. Examples include a computer-implemented method that includes receiving data indicating a physical environment state, processing the data to determine a physical position of at least one user, determining at least one physical anchor position within the physical environment state, mapping the physical anchor position to a virtual anchor position within a virtual environment state, wherein the virtual environment state includes a plurality of virtual coordinate positions that map to at least a portion of the physical environment state, determining a particular virtual coordinate position for at least one virtual control from the plurality of virtual coordinate positions, and instructing a display device configured to display the virtual environment state to display the virtual control at the particular virtual coordinate position. <CIT> describes an interface that includes selection attractive movement as the selection protocol, where a selection object is used to discriminate between selectable objects and attract a target object toward the selection objects, where the direction and speed of the motion controls, discriminates, attracts, and activates the selected objects. <CIT> describes a system for controlling a menu based augmented reality (AR) Graphical User Interface (GUI) according to predefined head movement positions, comprising a head mounted AR display and one or more hardware processors adapted to execute a code, the code comprising code instructions to present one or more selection menus of a GUI displayed by the head mounted AR display, the selection menu(s) comprising one or more control display objects, code instructions to detect one or more predefined discrete head movement positions of the head mounted display by analyzing sensory data received from one or more orientation sensors monitoring orientation of the head mounted display, each of the predefined discrete head movement positions maps one of a plurality of navigation actions and code instructions to apply a respective navigation action mapped by the detected predefined discrete head movement position(s) on a currently pointed control object of the control display objects. <CIT> describes an information processing system that includes processing circuitry configured to control a movement of a UI object on a display screen from a pre-recognition position toward a post-recognition position in response to recognition of an operation object initiated by a user, wherein the post-recognition position is spatially related to a displayed position of a predetermined displayed feature, and the predetermined displayed feature is an image derived from a camera captured image. <CIT> describes systems and methods for interacting with virtual objects in a three-dimensional space using a wearable system. The wearable system can be programmed to allow a user to interact with virtual objects using a user input device and poses. The wearable system can also automatically determine contextual information such as layout of the virtual objects in the user's environment and switch the user input mode based on the contextual information.

Disclosed aspects include methods and systems for detecting and responding to user-object interactions in mixed-reality environments.

In some aspects, a mixed-reality system detects a controller gesture with an associated controller orientation in a mixed-reality environment. The mixed-reality system then determines an interaction region for the controller gesture and identifies one or more virtual objects within the interaction region. The virtual objects each have an associated orientation affinity. Subsequently, the mixed-reality system determines an orientation similarity score between the controller orientation and the orientation affinity for each virtual object within the interaction region. Then, in response to determining that at least one orientation similarity score exceeds a predetermined threshold, the mixed-reality system executes an interaction between the controller and the virtual object that has the greatest orientation similarity score.

In some aspects, a mixed-reality system identifies a mixed-reality input element (e.g., a selectable button or interface element for an application) that is selectively triggered for providing or processing user input associated with the input element (e.g., selection of the input element for causing a corresponding application to execute a function associated with input accepted/received at the input element). The mixed-reality input element has an associated directional preference for receiving interaction of a user controller (e.g., gesture input) for selectively triggering the input when the corresponding gesture input is received and accepted.

The mixed-reality system also detects a gesture input associated with the user controller (e.g., a user finger) and determines a directional component of the controller gesture as the user controller interacts with the mixed-reality input element by providing the gesture input (e.g., causing the controller to intersect a display surface of the input element). Subsequently, in response to determining that the directional component of the controller gesture matches the directional preference of the mixed-reality input element, within a predefined threshold, the mixed-reality system selectively accepts, triggers and/or processes the gesture input associated with the mixed-reality input element which is sufficient for causing the corresponding application to execute the function associated with the input. Alternatively, if is determined that the directional component of the controller gesture fails to match the directional preference of the mixed-reality input element, within the predefined threshold, the mixed-reality system selectively ignores and fails to accept, trigger and/or process the gesture input.

Alternatively, or additionally, other combinations of computer properties are also used to determine when input is received and accepted or ignored for enabling or refraining from enabling functionality of an input element. For instance, mappings between input element affinities and controller/gesture properties can be used to selectively enable or disable input modes of input elements for receiving and processing input directed at the input elements from controllers, based on the particular controller/gesture properties.

In order to describe the manner in which the above-recited and other advantages and features can be obtained, a more particular description of the subject matter briefly described above will be rendered by reference to specific aspects which are illustrated in the appended drawings. Understanding that these drawings depict only typical aspects and are not therefore to be considered to be limiting in scope, aspects will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:.

Some disclosed aspects include systems and methods configured to detect and facilitate user-object interactions in mixed-reality environments.

Many mixed-reality systems allow users to interact with virtual objects in mixed-reality environments using controllers, which includes a user's own body parts (such as their fingers, hands, arms, feet, legs or other body parts) or virtual representations thereof. User interactions with virtual objects may include, for instance, grabbing, pushing, pulling, picking up, sliding, pressing, rotating, or acting upon virtual objects or virtual input elements (such as virtual buttons) within the mixed-reality environment.

To ameliorate accidental interaction between users and virtual objects or input elements, one possible solution is to have added requirements for enabling users to interact with virtual objects. A system might require a user's hand to be in a particular pose (e.g., in an open position) before allowing interaction with certain virtual objects. In another possible solution, a mixed-reality system might require that a user's hand be in a specific pose for a duration of time before executing an interaction with certain virtual objects.

These solutions, however, may fall short in a number of circumstances. For instance, where several interactable virtual objects and/or virtual input receivers are in close proximity to one another, many mixed-reality systems would have difficulty identifying which particular virtual object or input receiver the user intends to interact with. In such instances, user gestures towards clusters of virtual objects/input receivers may result in unintended user interaction with the virtual content.

Additionally, some virtual objects, such as control buttons may process user input that is unintended, such as when a user traverses a plane associated with the control button in a wrong direction (i.e., a direction other than a direction associated with depressing the virtual button). In particular, while physical objects have physical geometries, which constrain the way in which the objects are interacted with, virtual objects do not have such physical constraints. This lack of physical constraints often gives rise to accidental or unintentional interaction between users and virtual objects. By way of example, a user passes their hand through a virtual button from the back side of the virtual button and unintentionally trigger the button input. Similarly, a user may unintentionally grab or interact with an object when passing their hands through or past the object in a relaxed position.

To help ameliorate some of these problems, various solutions are provided in this disclosure to facilitate interactions between user's and virtual objects in a way that is intuitive and predictable, and which can greatly enhance a user's mixed-reality experience.

In some aspects, a mixed-reality system detects a controller orientation of a controller gesture, determines an interaction region and identifies one or more virtual objects therein, determines an orientation affinity for each virtual object(s) and an orientation similarity score between the controller orientation and the one or more orientation affinities, and executes an interaction between the controller and the virtual object with the highest orientation similarity score (in response to determining that an orientation similarity score exceeds a threshold).

In some aspects, a mixed-reality system identifies a mixed-reality input element and a directional preference therefor, detects a controller gesture and directional component thereof, and, in response to determining that the directional component and the directional preference match, selectively triggers an input associated with the mixed-reality input element.

Those skilled in the art will recognize that the aspects disclosed herein may provide significant benefits over conventional systems and methods for detecting user-object interaction in mixed-reality environments. For example, some disclosed aspects avoid unintended user interaction with virtual objects, particularly when an orientation of a user controller does not correspond to an orientation affinity of a virtual object, or when a directional component of a user's gesture does not comply with a directional preference of a virtual object. In some instances, this functionality proves particularly beneficial in circumstances where a user makes a gesture to interact with a virtual object that is in close proximity to other virtual objects. Therefore, some aspects disclosed herein provide for a seamless interaction pattern between users and virtual objects and/or virtual input elements, thus improving the user experience.

Having just described some of the various high-level features and benefits of the disclosed aspects, attention will now be directed to <FIG>. These figures illustrate various functionalities, examples, supporting illustrations, and methods related to detecting and facilitating/controlling user-object interactions in mixed-reality environments. Subsequently, attention will be directed to <FIG>, which presents an example computer system that may be used to facilitate the disclosed principles.

The following aspects are discussed in the context of a mixed-reality system (e.g., computer system <NUM> illustrated in <FIG>) where a mixed-reality environment including one or more holograms is presented to a user at a mixed-reality display device and where the user may interact with one or more objects. A user-operated control (e.g., a user's hand) for interacting with the mixed-reality environment may be tracked by the mixed-reality system (see <FIG> and the relevant discussion of sensors <NUM> and <NUM>), such that the mixed-reality system detects movement, pose, and other characteristics of the user control. In response to detecting certain movement, poses, and/or other variables/values based on such movement and/or poses of the user control, the mixed-reality system may execute certain commands and/or input to facilitate user-object interactions and to thereby help to control user experiences in mixed-reality environments.

Attention is now directed to <FIG>, which illustrates an aspect wherein a mixed-reality system detects user-object interaction in a mixed-reality environment. <FIG> shows a user controller embodied as a user's hand <NUM> as it performs a gesture in the mixed-reality environment. Here, the gesture corresponds to a grab motion, which the mixed-reality system detects as input for initiating an interaction with a virtual object in the mixed-reality environment.

When the mixed-reality system detects the user's gesture, the mixed-reality system determines an interaction region <NUM> for the user's gesture. The interaction region <NUM> defines which objects the user's gesture is directed toward, thus defining a set of virtual objects that the user may be attempting to interact with (e.g., virtual objects <NUM>, <NUM>, and <NUM>). As illustrated in <FIG>, a dashed conical region extends from the user's hand <NUM> as the user performs the grab gesture, which corresponds to the interaction region <NUM> for the user's gesture.

Upon determining the interaction region <NUM>, the mixed-reality system then identifies one or more virtual objects (e.g., virtual objects <NUM>, <NUM>, <NUM>) within the interaction region <NUM>.

As shown in <FIG>, virtual objects <NUM>, <NUM>, and <NUM> all lie within the interaction region <NUM> and are thus candidates for becoming the subject of the interaction triggered by the user's gesture.

Although the grab region <NUM> is depicted in <FIG> as an oblique conical region extending away from the controller (e.g., hand <NUM>), along an axis of directionality <NUM> corresponding to the controller orientation and/or direction of movement, those skilled in the art will recognize that the grab region <NUM> may be implemented as a region of any shape, such as a spherical, cylindrical, pyramidal, or prismatic region. Furthermore, it will be appreciated that, in some aspects, the mixed-reality system identifies the interaction region based on the pose or other characteristics of the user's hand or controller while making the controller gesture. The interaction region may be defined as a spherical region when the user makes a gesture with their hand in a pose that corresponds to gripping a physical ball. In another example, the interaction region may be defined as a triangular prismatic region when the user makes a gesture with their hand in a pose that corresponds to a pinch with their index finger and thumb.

In yet other aspects, the interaction region is defined dynamically based on characteristics of user controller gesture that initiates the interaction sequence, such as the velocity of the user's controller that performs the controller gesture. By way of example, when a user's hand makes an interaction or grab pose while the hand is moving, the size of the interaction region is augmented in the direction of the hand movement.

In some aspects, the interaction region extends a predetermined distance from the user's hand/controller, such as a few inches to a few feet. The predetermined distance may also be a variable distance that varies according to the gesture of the controller. As the user's fingers expand, the interaction region dynamically expands in width as well as depth. In other aspects, the depth is a fixed distance and the width is the only dimension that expands with the figures.

In some instance, the interaction region is visually presented to the user upon detecting a selection gesture or context associated within a mixed-reality environment. The visual presentation can be a hologram having a predefined level of transparency. IN other aspects, the visual presentation of the interaction region is a hologram that reflects an outline of the interaction region boundaries.

To provide an illustrative example of these principles, <FIG> show a user's hand <NUM> in a pose that corresponds to a pinch gesture with their index finger <NUM> and thumb <NUM>, and an interaction region <NUM> (conical region in <FIG> and triangular prismatic region in <FIG>) is shown extending from the user's hand <NUM> along an axis of directionality <NUM> of the user's hand <NUM>. In some implementations, the shape, size, and/or orientation of the interaction region <NUM> is determined based on the detected pose or other characteristic of the user controller (e.g., hand <NUM>). The orientation of the interaction region <NUM> shown in <FIG> corresponds to the detected axis of directionality <NUM> of the user's hand <NUM> (i.e., the interaction region <NUM> extends in the direction of the axis of directionality <NUM> of the user's hand <NUM>). In some aspects, the axis of directionality <NUM> is based on the pose of the user's hand <NUM> or a motion characteristic of the user's hand <NUM> (e.g., velocity, as described below).

Additionally, in some aspects, the mixed-reality system is configured to select a particular shape (e.g., a spherical, cylindrical, pyramidal, or prismatic shape) for the interaction region <NUM> based on the angular relationship between the fingers or other individual members of the user-operated hand <NUM> or other controller (e.g., an angular relationship between the user's thumb <NUM>, index finger <NUM>, and/or the other fingers).

Furthermore, in some implementations, the size of the interaction region is based on a motion characteristic (e.g., velocity or acceleration) of the user-operated controller (e.g., hand <NUM>). The user's hand <NUM> shown in <FIG> has a velocity associated therewith (denoted by arrow <NUM>), and the size of the interaction region <NUM> is increased based on the detected velocity of the user's hand <NUM>. Still furthermore, in some aspects, the size of the interaction region <NUM> is based on other characteristics of the user-operated controller. By way of example, as shown in <FIG>, the size of the interaction region <NUM> is based on the angular relationship between the user's index finger <NUM> and thumb <NUM>.

In still other aspects, the interaction region <NUM> is defined by other environmental contexts and/or user activity context factors. In one example, when a user is immersed in a role-playing application and becomes inflicted with an impaired state, the interaction/grab region <NUM> may be reduced in size or have an irregularity introduced thereto. In yet another example, when a user immersed in an application with a control or building mode, the interaction region <NUM> may be generally augmented in size to facilitate easier manipulation of the user's environment.

Continuing with <FIG>, the user's hand <NUM> has an arrow extending from the palm of the hand <NUM> in a direction substantially normal to the surface of the palm. The arrow indicates a controller orientation <NUM>. Although the controller orientation <NUM> is illustrated as extending from the palm of the user's hand <NUM>, it should be noted that the controller orientation <NUM> is based on other parts of the controller and/or be based on a pose of the controller. The controller orientation <NUM> may be based on an orientation of a user's index finger when the user's hand is in a pose with the only the index finger extended. In another example, when the controller is embodied as a user-held wand, the controller orientation may extend away from an end of the wand based on the pose of the wand. Furthermore, as with the interaction region, the controller orientation <NUM> of a controller may be determined dynamically (e.g., based on motion or other characteristics of the user's hand or other controller), and in some aspects, the controller orientation <NUM> points in the same direction as the axis of directionality <NUM>.

Similarly, the virtual objects within the interaction region (virtual objects <NUM>, <NUM>, and <NUM>) each have a corresponding arrow which indicates an orientation affinity for each object (orientation affinities <NUM>, <NUM>, and <NUM>, respectively). The orientation affinity for an object, in some implementations, defines an orientation parameter that a user's controller must conform to in order to interact with the object (as discussed in further detail below). As shown, the orientation affinity for each of the virtual objects <NUM>, <NUM>, and <NUM> points in a different direction. Illustratively, in layman's terms, each of the virtual objects <NUM>, <NUM>, and <NUM> "prefer" to be interacted with from different directions.

In some instances, a virtual object has more than one orientation affinity, and the mixed-reality system selects the orientation affinity to be used in carrying out the interaction determination sequence based on the pose or other characteristics of the user controller that performs the user gesture. For example, a virtual object <NUM> may be configured to be interacted with in a variety of ways (e.g., by lifting up, pushing, pulling, etc.) and so may include a variety of orientation affinities (in addition to orientation affinity <NUM>). The mixed reality system selects the appropriate orientation affinity for carrying out the interaction determination sequence based on the pose of the user controller (e.g., whether the user's hand position corresponds to a lifting, pushing, or pulling position) or another characteristic of the user controller (e.g., a motion characteristic, which may trigger the orientation affinity associated with pushing the object, regardless of the controller orientation <NUM> of the user controller/hand <NUM>).

Upon determining the controller orientation <NUM> and the orientation affinity (e.g., <NUM>, <NUM>, and <NUM>) for each virtual object (e.g., <NUM>, <NUM>, and <NUM>) within the interaction region <NUM> for the controller gesture, the mixed-reality system determines an orientation similarity score between the controller orientation and the orientation affinity for each of the virtual objects within the interaction region <NUM>. The mixed-reality system performs this step to identify and/or rank the virtual objects in the interaction region (e.g., candidate objects) in order of the directional similarity they share with the controller orientation <NUM> of the user's gesture. In some aspects, the virtual object(s) with the highest orientation similarity score will be the most logical/selected choice as the subject object(s) of the interaction triggered by the user's gesture within the interaction region <NUM>.

The orientation similarity scores between the controller orientation <NUM> and each of the orientation affinities of the virtual objects (e.g., orientation affinities <NUM>, <NUM>, and <NUM> of virtual objects <NUM>, <NUM>, and <NUM>, respectively) may be determined in a variety of ways. In some aspects, the mixed-reality system determines a dot product between the controller orientation <NUM> and each orientation affinity separately and utilizes the dot product values as the orientation similarity scores. In some aspects, the mixed-reality system utilizes a different method to formulate orientation similarity scores between the controller orientation <NUM> and the various orientation affinities, such as cosine similarity or normalized measures of Euclidean distance, Manhattan distance, or Minkowski distance.

After determining the orientation similarity scores, in some aspects, the mixed-reality system determines whether any of the orientation similarity scores exceed a predetermined threshold. In some instances, requiring a threshold to be met before executing an interaction prevents undesired user-object interactions in mixed-reality environments, in particular where the highest orientation similarity score only indicates low similarity between the controller orientation <NUM> and the orientation affinity.

In some aspects, in response to determining that at least one orientation similarity score exceeds the predetermined orientation similarity threshold, the mixed-reality system executes an interaction between the controller (e.g., hand <NUM>) and the particular virtual object (e.g., <NUM>, <NUM>, or <NUM>) within the interaction region <NUM> that has the greatest or maximum orientation similarity score. Accordingly, the mixed-reality system, in some implementations, selects the virtual object(s) that most likely corresponds to the user's intended object for interaction. This may include selecting only a single virtual object or a plurality of virtual objects that are associated with orientation similarity scores that exceed a predetermined threshold and/or that are within a predetermined variance/deviation from an object associated with a highest orientation similarity score.

In reference to <FIG>, the controller orientation <NUM> for the user's hand <NUM> and the orientation affinities <NUM>, <NUM>, <NUM> for virtual objects <NUM>, <NUM>, <NUM>, respectively, within the interaction region <NUM> are shown. Because the controller orientation <NUM> and the orientation affinity <NUM> of virtual object <NUM> point in substantially the same direction (or, at least, orientation affinity <NUM> is more aligned with controller orientation <NUM> than orientation affinities <NUM> and <NUM>), the orientation similarity score between the controller orientation <NUM> and the orientation affinity <NUM> will be higher than the orientation similarity scores calculated for orientation affinities <NUM> and <NUM>. As such, upon determining that the orientation similarity score between the controller orientation <NUM> and orientation affinity <NUM> exceeds a predetermined threshold, the mixed-reality system executes an interaction between the user's hand virtual object <NUM> (e.g., the user picks up virtual object <NUM> instead of virtual object <NUM> or virtual object <NUM>).

Although the discussion of <FIG> has focused on a grab gesture performed by a user's hand, those skilled in the art will recognize that other controllers (such as handheld controllers) and/or other gestures/poses are within the scope of this disclosure. The gesture for initiating a user-object interaction could amount to a push, slide, pull, press, or rotation gesture, or any combination thereof.

<FIG> illustrates an aspect in which a mixed-reality system uses additional/other constraints in detecting user-object interaction in a mixed-reality environment. In some instances, the user's grab region will include virtual objects that are intended to be interacted with in predefined and/or particular manners. By way of example, <FIG> depicts a user's hand <NUM> performing a semi-open-handed gesture toward two objects, a mixed-reality input element (e.g., button <NUM>) and a virtual box <NUM> (the interaction region is not visually depicted for simplicity).

The physical counterparts of some virtual objects, such as button <NUM>, have physical attributes that govern their real-world behavior. For example, a physical button may not be pressed from its back side. Thus, in some aspects, it is beneficial to include additional/other constraints on mixed-reality buttons or other input elements to prevent unwanted interaction with such objects (e.g., an unintended, accidental press from the back side of a virtual button).

As before, the user's hand <NUM> has a controller orientation <NUM>. The button <NUM> has an orientation affinity <NUM>, and the box <NUM> also has an orientation affinity <NUM>. By performing the processes detailed above with reference to <FIG>, a mixed-reality system determines, in some aspects, that the orientation similarity score between the controller orientation <NUM> and the orientation affinity <NUM> exceeds a predetermined threshold and is greater than the orientation similarity score between the controller orientation <NUM> and the orientation affinity <NUM>. As such, button <NUM> is the most logical choice for becoming the subject of the interaction initiated by the user's gesture.

However, in some aspects, the mixed-reality system takes additional/other measures to ensure that an unintended user-object interaction does not occur. For example, button <NUM> should not be able to be interacted with or pressed from its back side (as with a physical button). As such, if the user moved their hand <NUM> such that their hand <NUM> interfaced with the button <NUM> from its back side, the mixed-reality system should not execute an interaction between the user's hand <NUM> and button <NUM>.

Therefore, in some aspects, the mixed-reality system determines a directional component <NUM> of the controller gesture and a directional preference <NUM> for at least one virtual object (e.g., button <NUM>) within the interaction region. The directional component is based, in some instances, on a detected motion characteristic of the user controller (e.g., hand <NUM>). For example, in some aspects, the directional component <NUM> is based on a velocity (e.g., directional movement) of the controller gesture, as determined by a detecting the position of a portion of the controller (e.g., hand <NUM>) at a time when the controller initially interfaces with a virtual object (e.g., button <NUM>) and at a time thereafter.

The directional preference for the virtual object within the interaction region indicates a direction in which the object should be interacted with to provide an intuitive user experience. For example, where the virtual object is a button, the directional preferences indicates a press direction, wherein the virtual button should only be able to be pressed in the press direction.

Continuing with <FIG>, the directional component <NUM> of the user's hand <NUM> determined by the mixed-reality system is shown. As depicted, directional component <NUM> is based on the velocity of the user's hand. The directional preference <NUM> for the button <NUM> as determined by the mixed-reality system is also shown. Upon determining that the button <NUM> has the highest orientation similarity score that exceeds a threshold and that the directional component <NUM> of the controller (user's hand <NUM>) complies with the directional preference <NUM> of button <NUM>, the mixed-reality system executes an interaction between the user's hand <NUM> and button <NUM>.

Those skilled in the art will recognize that the orientation affinity and the directional preference for a virtual object may be the same or different, depending on the circumstances. For example, a virtual handprint scanner would have its orientation affinity and directional preference in the same direction, whereas cabinet door knob may have its orientation affinity in one direction, but its directional preference in another. Furthermore, it should be noted that an object may have more than one directional preference. For instance, a cabinet door may have a directional preference for opening and a directional preference for closing, as well as additional similarity constraints for determining whether a controller directional component complies with one of the directional preferences. Also, it will be appreciated that the directional preference, controller orientation, and/or the axis of directionality for a controller may point in the same or different directions.

In some implementations, this functionality prevents the user from inadvertently interacting with direction-specific virtual content, such as virtual buttons or virtual doors, or other virtual input receivers.

<FIG> illustrate aspects in which a mixed-reality system uses directional constraints in detecting user-object interaction in a mixed-reality environment. Although the mixed-reality system aspect described hereinabove with reference to <FIG> executed a user-object interaction in a mixed-reality environment based on both orientation similarity scores and compliance with directional preferences, it will be appreciated that a mixed-reality system may facilitate user-object interaction by only focusing on compliance with directional preferences. For example, simple virtual buttons might be agnostic toward the controller orientation of a user controller, focusing only on whether the virtual button is pushed from the correct direction.

Accordingly, <FIG> both illustrate a user controller (embodied as a user's hand <NUM>) with a directional component <NUM> (no controller orientation or interaction region is shown) and a mixed-reality input element (embodied as a virtual button <NUM>) with a selectively triggerable input and an associated directional preference <NUM>. In the illustrated aspects, the user's hand <NUM> performs a gesture, detected by the mixed-reality system, that corresponds with the user's hand <NUM> interacting with the virtual button <NUM>.

The virtual button <NUM> in <FIG> has a directional preference <NUM> that is in substantially the same direction as the directional component <NUM> of the user's hand <NUM>, whereas the virtual button <NUM> in <FIG> has a directional preference <NUM> that substantially opposes the directional component <NUM> of the user's hand <NUM>. Thus, when the user's hand <NUM> of <FIG> interacts with the virtual button <NUM> of <FIG>, the mixed-reality system will detect a match between the directional component <NUM> of the user gesture and the directional preference <NUM> of the virtual button <NUM>, and the system will selectively trigger the input associated with the virtual button <NUM>.

In contrast, when the user's hand <NUM> of <FIG> interacts with the virtual button <NUM> of <FIG>, the mixed-reality system will not detect a match between the directional component <NUM> of the user gesture and the directional preference <NUM> of the virtual button <NUM>, and the system will fail to accept or trigger the input (e.g., gesture) associated with the virtual button <NUM> in response to the user's hand <NUM> touching or crossing a display surface of the virtual button <NUM>.

Attention is now directed to <FIG>, which illustrates an aspect of a mixed-reality system detecting user-object interaction in a mixed-reality environment, according to the present disclosure. <FIG> shows a mixed-reality tablet hologram <NUM> which includes corner affordances <NUM> and side affordances <NUM> which users interact with in order to resize the mixed-reality tablet <NUM>. Users, however, typically move their hand or controller (e.g., hand <NUM>) through the affordances <NUM>, <NUM> in order to interact with the content displayed on the mixed-reality tablet <NUM>. This movement can cause inadvertent interactions with the affordances <NUM>, <NUM> of a mixed-reality tablet <NUM>.

In order to prevent unintended interaction with the affordances <NUM>, <NUM> of the mixed-reality tablet <NUM>, the mixed-reality system aspect shown in <FIG> only executes an interaction between a controller (e.g., a user's hand <NUM>) and a virtual object (e.g., an affordance <NUM>, <NUM> of a mixed-reality tablet <NUM>) upon determining that a pose of the controller corresponds to an interactive pose. Typically, users interact with affordances of mixed-reality tablets by forming a pinching pose with their hand (with their thumb exerting force against one or more fingers).

Accordingly, the mixed-reality system aspect of <FIG> defines a pinching pose as an interactive pose for affordances <NUM>, <NUM> of a mixed-reality tablet <NUM>. Therefore, the mixed-reality system will only execute an interaction between a controller (e.g., a user's hand <NUM>) and an affordance <NUM>, <NUM> when the controller or hand is forming a pinching pose at the time of a user's gesture.

The user hand <NUM> illustrated in <FIG> is gesturing so as to move through a corner affordance <NUM> of the mixed-reality tablet <NUM> (e.g., to interact with virtual content of the mixed-reality tablet <NUM>). The user's hand <NUM> is in a semi-open position, which is not considered an interactive pose with respect to affordances <NUM>, <NUM> of mixed-reality tablet <NUM>. As such, even though the user's hand <NUM> has a controller orientation <NUM> that is substantially in the same direction as an orientation affinity <NUM> of the corner affordance <NUM> and the affordance <NUM> is in an interaction region created by the gesture of the user's hand, the mixed-reality system will not execute an interaction between the user's hand <NUM> and the corner affordance <NUM>.

Continuing with the discussion of interactive poses, <FIG> illustrate some poses that are detectable by a mixed-reality system when detecting user-object interaction in a mixed-reality environment. <FIG> shows a user's hand <NUM> in a closed fist pose. A closed fist might be considered, in some instances, a non-interactive pose for most or all mixed-reality object interaction. Thus, in some implementations, when a user moves their hand <NUM> in a closed fist pose through or in close proximity to interactable virtual objects or virtual input elements, the mixed-reality system will selectively ignore the input and refrain from executing the user-object interactions.

Alternatively, certain poses may be required to process input, such as a hand positioned in a peace sign. In this regard, the controller pose can include controller properties (e.g., velocity, pose, sequence of poses, orientation, etc.) that are mapped to one or more input element affinities or property sets that selectively enable or disable the input receiving mode of the input element to receive or, alternatively, to ignore input from a corresponding controller.

<FIG> illustrates a user's hand <NUM> in a grabbing pose, with the thumb opposing all of the user's fingers (e.g., so as to grab a ball or another object). As described above, a grab pose is defined as an interactive pose for grabbing/picking up certain objects (e.g., boxes, balls, models, etc.). This pose is used to trigger the application/display of an interaction region for selectively identifying objects within that region to be interacted with based on corresponding directional affinities.

As described above in relation to <FIG>, a pinch pose may be defined as the only interactive pose for affordances, and also operates as an interactive pose in other contexts for applying an interaction region and/or for calculating directional and orientation affinities.

<FIG> illustrates a similar pinching pose, but the thumb only opposes the user's index finger (with the other fingers extended). Such a pose is a defined interactive pose where the virtual object is a small object (e.g., a small marble), such that interactions therewith should be constrained to a deliberate hand pose such as the one shown in <FIG>.

<FIG> shows a flow diagram depicting a method <NUM> for detecting user-object interaction in a mixed-reality environment. Method <NUM> includes acts of detecting a controller gesture (<NUM>), determining an interaction region for the controller gesture (<NUM>), identifying one or more virtual objects within the interaction region (<NUM>), determining orientation similarity score(s) (<NUM>), determining that an orientation similarity score exceeds a predetermined threshold (<NUM>), determining a directional component for the controller gesture and a directional preference for a virtual object (<NUM>), and executing an interaction between the controller and the virtual object with the greatest orientation similarity score (<NUM>).

Act <NUM> of method <NUM> includes detecting a controller gesture. In some aspects, the controller gesture amounts to a grab, push, slide, pull, press, rotation, or other gesture directed toward one or more virtual objects in a mixed-reality environment. Furthermore, in some aspects, the gesture has an associated controller orientation, which identifies an interaction orientation for the controller and may be based on various parts of the controller and/or a pose of the controller.

Act <NUM> of method <NUM> includes determining an interaction region for the controller gesture. In some aspects, the interaction region defines which objects the user's gesture is directed toward, thus defining a set of virtual objects that the user is attempting to interact with. In certain implementations, the interaction region corresponds to a conical, spherical, cylindrical, pyramidal, or prismatic region, and the interaction region is identified based on attributes of the user's gesture of act <NUM>, such as pose and/or motion characteristics, and/or other environmental or user activity factors.

Act <NUM> of method <NUM> includes identifying one or more virtual objects within the interaction region. The virtual objects within the interaction region, in some aspects, each include an orientation affinity, which defines for each object an orientation parameter that a user's controller must conform to in order to interact with the object.

Act <NUM> of method <NUM> includes determining orientation similarity score(s), in particular between the controller orientation and the orientation affinity for each one of the virtual objects within the interaction region. The mixed-reality system performs this step to rank the virtual objects in the interaction region (e.g., candidates) in order of the directional similarity they share with the controller orientation of the user's gesture.

The orientation similarity scores between the controller orientation and each of the orientation affinities of the virtual objects may be determined in a variety of ways. In some aspects, the mixed-reality system determines a dot product between the controller orientation and each orientation affinity separately and utilizes the dot product values as the orientation similarity scores. In some aspects, the mixed-reality system utilizes a different method to formulate orientation similarity scores between the controller orientation and the various orientation affinities, such as cosine similarity or normalized measures of Euclidean distance, Manhattan distance, or Minkowski distance.

Act <NUM> of method <NUM> includes determining that an orientation similarity score exceeds a predetermined threshold. In some instances, requiring a threshold to be met before executing an interaction prevents undesired user-object interactions in mixed-reality environments, in particular where the highest orientation similarity score only indicates low similarity between the controller orientation and the orientation affinity.

Act <NUM> of method <NUM> includes determining a directional component for the controller gesture and a directional preference for a virtual object. The directional component is based, in some instances, on a detected motion characteristic of the user controller. For example, in some aspects, the directional component is based on a velocity of the controller gesture, as determined by a detecting the position of a portion of the controller at a time when the controller initially interfaces with a virtual object and at a time thereafter. The directional preference for the virtual object within the interaction region indicates a direction in which the object should be interacted with to provide an intuitive user experience. For example, where the virtual object is a button, the directional preferences indicates a press direction, wherein the virtual button should only be able to be pressed in the press direction.

Act <NUM> of method <NUM> includes executing an interaction between the controller and the virtual object with the greatest orientation similarity score. In some aspects, the mixed-reality system only executes the interaction upon determining that the directional component of the controller gesture complies with the directional preference for at least one of the virtual objects in the interaction region and/or upon determining that the pose of the controller corresponds to an interactive pose. This determination may occur by examining a stored mapping between directional component(s) of controller gesture(s) and directional preference(s) of virtual object(s) in one or more data structures (not shown) stored in the storage of the system (e.g., storage <NUM>) or a remote system <NUM>. In some implementations, the virtual object upon which an interaction is executed is the object that the user intended to initiate an interaction with, rather than an unintended object.

<FIG> shows a flow diagram depicting a method <NUM> for detecting and applying user-object interaction in a mixed-reality environment using directional constraints. Method <NUM> includes acts of identifying a mixed-reality input element and directional preference (<NUM>), detecting a controller gesture (<NUM>), determining a directional component of the controller gesture (<NUM>), determining that the directional component matches the directional preference (<NUM>), and selectively triggering the input element (<NUM>).

Act <NUM> of method <NUM> includes identifying a mixed-reality input element and directional preference, wherein the mixed-reality input element includes an input that is selectively triggered. The directional preference for the virtual object within the interaction region indicates a direction in which the object should be interacted with to provide an intuitive user experience. For example, where the virtual object is a button, the directional preferences indicates a press direction, wherein the virtual button should only be able to be pressed in the press direction.

Act <NUM> of method <NUM> includes detecting a controller gesture. In some aspects, the controller gesture amounts to a grab, push, slide, pull, press, rotation, or other gesture directed toward one or more virtual objects in a mixed-reality environment.

Act <NUM> of method <NUM> includes determining a directional component of the controller gesture. The directional component is based, in some instances, on a detected motion characteristic of the user controller. For example, in some aspects, the directional component is based on a velocity of the controller gesture, as determined by a detecting the position of a portion of the controller at a time when the controller initially interfaces with a virtual object and at a time thereafter.

Act <NUM> of method <NUM> includes determining that the directional component matches the directional preference. In some aspects, determining a match between is a simple binary determination. In other aspects, additional directional constraints are involved, such as a threshold value for a dot product between the directional component and the directional preference.

Act <NUM> of method <NUM> includes and selectively triggering the input element, in particular the input associated with the mixed-reality input element. In some aspects, the input is selectively triggered in response to determining that the directional component of the controller gesture matches the directional preference of the mixed-reality input element.

<FIG> shows a flow diagram depicting a method <NUM> for selectively triggering input with a mixed-reality input element in a mixed-reality environment based on detecting matches between the input element's controller affinity preference set and the interaction property set of an interacting controller. Method <NUM> includes acts of identifying a mixed-reality input element within a mixed-reality environment (<NUM>), detecting a controller gesture (<NUM>), determining a controller interaction property set (<NUM>), determining that the controller interaction property set of the controller gesture matches the controller affinity preference set of the mixed-reality input element (<NUM>), and selectively triggering the input associated with the mixed-reality input element (<NUM>).

Act <NUM> of method <NUM> includes identifying a mixed-reality input element within a mixed-reality environment. In some aspects, the mixed-reality input element includes an input that is selectively triggered, such as a mixed-reality button, keyboard, touch screen, affordance, or other input element. Furthermore, in some aspects, the mixed-reality input element has an associated controller affinity preference set. The controller affinity preference set, in some implementations, defines constraints for receiving interaction of a user controller for selectively triggering the input, such that the input only becomes selectively triggered when the constraints are complied with.

Act <NUM> of method <NUM> includes detecting a controller gesture. In some aspects, the controller gesture corresponds with the user controller interacting with the mixed-reality input element. By way of example, in one instance, the controller gesture is a user manipulating a controller, such as their hand, to interface with or press a mixed-reality button or other input device. In another instance, the controller gesture is a grab gesture for grabbing a mixed-reality object.

Act <NUM> of method <NUM> includes determining a controller interaction property set. In some instances, the controller interaction property set is determined as the user controller interacts with the mixed-reality input element. For example, when a user interfaces with a virtual button, the mixed-reality system determines a controller interaction property set at the time the user interfaces with the virtual button. The controller interaction property set can include any properties described hereinabove, such as a controller orientation, directional component, pose, sequence of poses, motion characteristic, axis of directionality, or other attributes or characteristics of the controller.

Act <NUM> of method <NUM> includes determining that the controller interaction property set of the controller gesture matches the controller affinity preference set of the mixed-reality input element. In some aspects, this is performed by comparing one or more properties of the controller interaction property set with one or more affinity preferences of the controller affinity preference set of the mixed-reality input element. This may include determining whether a threshold value associated with a property and/or affinity preference is exceeded, such as directional similarity, velocity, orientation, directional preference/component, pose characteristics (e.g., whether in an interactive pose or non-interactive pose), sequence of poses, etc..

Act <NUM> of method <NUM> includes selectively triggering the input associated with the mixed-reality input element, such as by triggering the input of a mixed-reality button, touch screen, or other object or input receiver consequent to the detected controller gesture for interacting with the mixed-reality input element. As such, the mixed-reality system is configurable to only selectively trigger the input associated with the mixed-reality input element in response to determining that the controller interaction property set of the controller gesture matches the controller affinity preference set of the mixed-reality input element.

<FIG> shows a flow diagram depicting a method <NUM> for selectively activating or deactivating an input receiving mode of one or more mixed-reality input elements in a mixed-reality environment based on detecting a particular set of controller properties in the mixed-reality environment. Method <NUM> includes acts of identifying a mixed-reality input element within a mixed-reality environment (<NUM>), detecting a controller gesture (<NUM>), determining a particular set of controller properties associated with the controller gesture (<NUM>), and either (a) selectively activating the input receiving mode in response to determining that the particular set of the controller properties matches a first mapped property set associated with activating the input receiving mode of the mixed-reality input element (1008a), or (b) selectively deactivating the input receiving mode in response to determining that the particular set of controller properties fails to match the first mapped property set or that the particular set of controller properties matches a second mapped property set associated with deactivating the input receiving mode of the mixed-reality input element (1008b).

Act <NUM> of method <NUM> includes an act of identifying a mixed-reality input element within a mixed-reality environment. In some aspects, the mixed-reality input element includes an input receiving mode. When the input receiving mode is activated, the system enables user input received at the mixed-reality input element for triggering an input function of the mixed-reality input element based on the user input (such as the pressing of a button, the grabbing of an object, etc.). When the input receiving mode is deactivated, on the other hand, the system ignores user input directed at the mixed-reality input element so that the mixed-reality input element is not used to trigger the input function.

Act <NUM> of method <NUM> includes determining a particular set of controller properties associated with the controller gesture. In some instances, the particular set of controller properties associated with the controller gesture can include any properties described hereinabove, such as a controller orientation, directional component, pose, sequence of poses, motion characteristic (e.g., velocity), axis of directionality, or other attributes or characteristics of the controller.

Act 1008a includes an act selectively activating the input receiving mode in response to determining that the particular set of the controller properties matches a first mapped property set associated with activating the input receiving mode of the mixed-reality input element. Act 1008b includes an act of selectively deactivating the input receiving mode in response to determining that the particular set of controller properties fails to match the first mapped property set or that the particular set of controller properties matches a second mapped property set associated with deactivating the input receiving mode of the mixed-reality input element.

The mapped property sets (e.g., first and second mapped property sets) associated with activating or deactivating the input receiving mode of the mixed-reality input element can include any properties described hereinabove, such as a controller orientation, directional component, pose, sequence of poses, motion characteristic (e.g., velocity), axis of directionality, or other attributes or characteristics of the controller.

In some aspects, the first mapped property set associated with activating the input receiving mode of the mixed-reality input element corresponds to constraints/conditions defined for activating the input receiving mode. Accordingly, the input receiving mode of the mixed-reality input element is only selectively activated when the particular set of controller properties matches the first mapped property set.

Furthermore, in some aspects, the second mapped property set associated with deactivating the input receiving mode of the mixed-reality input element corresponds with constraints/conditions defined for deactivating the input receiving mode. Thus, the input receiving mode of the mixed-reality input element becomes (or remains) deactivated if the conditions for deactivation are met, or the constraints of the first mapped property set are not met.

Additionally, in some aspects, the method further includes determining whether the particular set of controller properties matches a first or second mapped property set of a second mixed-reality input element. This may occur, for example, when two mixed-reality input elements are in close proximity to the controller/gesture (e.g., within an interaction region of a controller) and are concurrently being rendered in the mixed-reality environment.

As with the previously mentioned mixed-reality input element (first mixed-reality input element), the first mapped property set of the second mixed-reality input element is associated with activating an input receiving mode of the second mixed-reality input element, and the second mapped property set of the second mixed-reality input element is associated with deactivating the input receiving mode of the second mixed-reality input element.

In some aspects, upon determining that the particular set of controller properties matches the first mapped property set of the first mixed-reality input element (for activating the input receiving mode of the first mixed-reality input element) and also matches the second mapped property set of the second mixed-reality input element (for deactivating the input receiving mode of the second mixed-reality input element), the system simultaneously and selectively deactivates the input receiving mode of the second mixed-reality input element.

In another scenario, upon determining that the particular set of controller properties matches the second mapped property set of the first mixed-reality input element (for deactivating the input receiving mode of the first mixed-reality input element) and also matches the first mapped property set of the second mixed-reality input element (for activating the input receiving mode of the second mixed-reality input element), the system simultaneously and selectively activates the input receiving mode of the second mixed-reality input element.

<FIG> shows a flow diagram depicting a method <NUM> for determining an interaction region for a controller gesture in a mixed-reality environment for facilitating object interaction in the mixed-reality environment between a user and one or more virtual objects in the interaction region. Method <NUM> includes acts of detecting a controller gesture of a user-operated controller in a mixed-reality environment (<NUM>), identifying an axis of directionality of the controller gesture and a set of one or more pose characteristics of the controller gesture (<NUM>), and determining the interaction of the controller gesture based on the set of one or more pose characteristics and the axis of directionality of the controller gesture (<NUM>).

Act <NUM> of method <NUM> includes detecting a controller gesture of a user-operated controller in a mixed-reality environment. In some aspects, the controller gesture amounts to a grab, push, slide, pull, press, rotation, or other gesture directed toward one or more virtual objects in a mixed-reality environment. Furthermore, in some aspects, the controller gesture is configured for initiating interaction between the user-operated controller and the virtual object(s) in the mixed-reality environment.

Act <NUM> of method <NUM> includes identifying an axis of directionality of the controller gesture and a set of one or more pose characteristics of the controller gesture. In some aspects, the axis of directionality of the controller gesture corresponds to an orientation of the user-operated controller and is based on a pose characteristic of the controller (e.g., an angular relationship between different elements/aspects of the controller, such as fingers of a hand) and/or a motion characteristic of the controller (e.g., velocity of a user's hand). In some aspects, the axis of directionality is in the same direction as the directional component or controller orientation of the controller, as described hereinabove. Furthermore, the set of one or more pose characteristics of the controller gesture can include any characteristics or properties described hereinabove, such as a controller orientation, directional component, pose, sequence of poses, motion characteristic (e.g., velocity), or other attributes or characteristics of the controller.

Act <NUM> of method <NUM> includes determining the interaction region of the controller gesture based on the set of one or more pose characteristics and the axis of directionality of the controller gesture. In some aspects, the interaction region is configured for identifying one or more virtual objects in the mixed-reality environment to be interacted with by the user-operated controller. This is accomplished, in some instances, by configuring the interaction region to define a volumetric region in the mixed-reality environment to identify mixed-reality objects within the volume/region.

The shape of the interaction region (volume), in some aspects, is based on the set of one or more pose characteristics of the controller gesture of the user-operated controller. For instance, different shapes for the interaction region is associated with different pose and/or motion characteristics of the controller/gesture. By way of example, the interaction region may be shaped as a conical or prismatic region upon detecting a first particular pose (e.g., a pinching pose where the user's index thumb and fingers form a particular shape and/or angular relationship) in the set of pose characteristics, and the interaction region may be shaped as a spherical region upon detecting a second particular pose (e.g., a grabbing pose where the user's thumb and fingers form a different shape and/or angular relationship) in the set of pose characteristics.

In some implementations, the size of the interaction region is based on the set of pose characteristics or motion characteristics of the user-operated controller. When the user controller has an associated velocity, the size of the interaction region may be enlarged in the direction of the velocity of the user controller. In another example, when the shape of the interaction region is conical, the aperture of the cone will depend on the angular relationship between the fingers of the user's hand (e.g., the angle between the user's thumb and index finger).

Furthermore, in some aspects, the orientation of the interaction region is based on the axis of directionality of the controller gesture. Thus, in some instances, the interaction region extends from the controller along the axis of directionality, and the orientation of the interaction region corresponds to the axis of directionality of the controller.

The disclosed aspects may, in some instances, provide various advantages over conventional systems and methods for detecting user-object interaction in mixed-reality environments. Some of these advantages include providing users with functionality that ameliorates and/or eliminates unintentional user-object interactions with virtual objects and/or virtual input elements in mixed-reality environments.

Having just described the various features and functionalities of some of the disclosed aspects, attention is now directed to <FIG>, which illustrates an example computer system <NUM> that may be used to facilitate the operations described herein.

The computer system <NUM> may take various different forms. For example, in <FIG>, the computer system <NUM> is embodied as a head-mounted display (HMD). Although the computer system <NUM> may be embodied as a HMD, the computer system <NUM> may also be a distributed system that includes one or more connected computing components/devices that are in communication with the HMD. Accordingly, the computer system <NUM> may be embodied in any form and is not limited strictly to the depiction illustrated in <FIG>. By way of example, the computer system <NUM> may include a projector, desktop computer, a laptop, a tablet, a mobile phone, server, data center and/or any other computer system.

In its most basic configuration, the computer system <NUM> includes various different components. For example, <FIG> shows that computer system <NUM> includes at least one hardware processing unit <NUM> (aka a "processor"), input/output (I/O) interfaces <NUM>, graphics rendering engines <NUM>, one or more sensors <NUM>, and storage <NUM>. More detail on the hardware processing unit <NUM> will be presented momentarily.

The storage <NUM> may be physical system memory, which may be volatile, non-volatile, or some combination of the two. The term "memory" may also be used herein to refer to non-volatile mass storage such as physical storage media. If the computer system <NUM> is distributed, the processing, memory, and/or storage capability may be distributed as well. As used herein, the term "executable module," "executable component," or even "component" can refer to software objects, routines, or methods that may be executed on the computer system <NUM>. The different components, modules, engines, and services described herein may be implemented as objects or processors that execute on the computer system <NUM> (e.g. as separate threads).

The disclosed aspects may comprise or utilize a special-purpose or general-purpose computer including computer hardware, such as, for example, one or more processors (such the hardware processing unit <NUM>) and system memory (such as storage <NUM>), as discussed in greater detail below. Aspects also include physical and other computer-readable media for carrying or storing computer-executable instructions and/or data structures. Such computer-readable media can be any available media that can be accessed by a general-purpose or special-purpose computer system. Computer-readable media that store computer-executable instructions in the form of data are physical computer storage media. Thus, by way of example and not limitation, the current aspects can comprise at least two distinctly different kinds of computer-readable media: computer storage media and transmission media.

Computer storage media are hardware storage devices, such as RAM, ROM, EEPROM, CD-ROM, solid state drives (SSDs) that are based on RAM, Flash memory, phase-change memory (PCM), or other types of memory, or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code means in the form of computer-executable instructions, data, or data structures and that can be accessed by a general-purpose or special-purpose computer.

The computer system <NUM> may also be connected (via a wired or wireless connection) to external sensors <NUM> (e.g., one or more remote cameras, accelerometers, gyroscopes, acoustic sensors, magnetometers, etc.). It will be appreciated that the external sensors include sensor systems (e.g., a sensor system including a light emitter and camera), rather than solely individual sensor apparatuses. Further, the computer system <NUM> may also be connected through one or more wired or wireless networks <NUM> to remote systems(s) <NUM> that are configured to perform any of the processing described with regard to computer system <NUM>.

During use, a user of the computer system <NUM> is able to perceive information (e.g., a mixed-reality environment) through a display screen that is included among the I/O interface(s) <NUM> and that is visible to the user. The I/O interface(s) <NUM> and sensors <NUM>/<NUM> also include gesture detection devices, eye trackers, and/or other movement detecting components (e.g., cameras, gyroscopes, accelerometers, magnetometers, acoustic sensors, global positioning systems ("GPS"), etc.) that are able to detect positioning and movement of one or more real-world objects, such as a user's hand, a stylus, and/or any other object(s) that the user may interact with while being immersed in the scene.

The graphics rendering engine <NUM> is configured, with the hardware processing unit <NUM>, to render one or more virtual objects within the scene. As a result, the virtual objects accurately move in response to a movement of the user and/or in response to user input as the user interacts within the virtual scene.

A "network," like the network <NUM> shown in <FIG>, is defined as one or more data links and/or data switches that enable the transport of electronic data between computer systems, modules, and/or other electronic devices. When information is transferred, or provided, over a network (either hardwired, wireless, or a combination of hardwired and wireless) to a computer, the computer properly views the connection as a transmission medium. The computer system <NUM> will include one or more communication channels that are used to communicate with the network <NUM>. Transmissions media include a network that can be used to carry data or desired program code means in the form of computer-executable instructions or in the form of data structures. Further, these computer-executable instructions can be accessed by a general-purpose or special-purpose computer.

Computer-executable (or computer-interpretable) instructions comprise, for example, instructions that cause a general-purpose computer, special-purpose computer, or special-purpose processing device to perform a certain function or group of functions.

Those skilled in the art will appreciate that the aspects may be practiced in network computing environments with many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile telephones, PDAs, pagers, routers, switches, and the like. The aspects may also be practiced in distributed system environments where local and remote computer systems that are linked (either by hardwired data links, wireless data links, or by a combination of hardwired and wireless data links) through a network each perform tasks (e.g. cloud computing, cloud services and the like).

Additionally or alternatively, the functionality described herein can be performed, at least in part, by one or more hardware logic components (e.g., the hardware processing unit <NUM>). For example, and without limitation, illustrative types of hardware logic components that can be used include Field-Programmable Gate Arrays (FPGAs), Program-Specific or Application-Specific Integrated Circuits (ASICs), Program-Specific Standard Products (ASSPs), System-On-A-Chip Systems (SOCs), Complex Programmable Logic Devices (CPLDs), Central Processing Units (CPUs), and other types of programmable hardware.

Claim 1:
A system (<NUM>) for detecting user-object interaction in a mixed-reality environment, the system comprising:
one or more processors (<NUM>);
one or more computer-readable media (<NUM>) having stored computer-executable instructions that are operable, when executed by the one or more processors, to cause the system to perform the following:
detect a controller gesture of a controller in the mixed-reality environment, wherein the controller gesture has an associated controller orientation and the controller comprises a user's hand (<NUM>);
determine an interaction region for the controller gesture, wherein the interaction region defines a set of virtual objects as candidates for becoming the subject of an interaction triggered by the controller gesture (<NUM>);
identify the set of virtual objects within the interaction region, wherein each virtual object of the set of virtual objects has an associated orientation affinity, the orientation affinity defining an orientation parameter that the controller must conform to in order to interact with the virtual objects (<NUM>);
determine an orientation similarity score between the controller orientation and the orientation affinity for each virtual object of the set of virtual objects within the interaction region (<NUM>); and
in response to determining that at least one orientation similarity score exceeds a predetermined threshold, execute an interaction between the controller and a particular virtual object of the set of virtual objects that has a greatest orientation similarity score out of all orientation similarity scores determined for each virtual object of the set of virtual objects (<NUM>);
wherein the interaction region is configured to have a size that is determined based on a detected pose of the user's hand.