Mixed reality interactions

Embodiments that relate to interacting with a physical object in a mixed reality environment via a head-mounted display are disclosed. In one embodiment a mixed reality interaction program identifies an object based on an image from captured by the display. An interaction context for the object is determined based on an aspect of the mixed reality environment. A profile for the physical object is queried to determine interaction modes for the object. A selected interaction mode is programmatically selected based on the interaction context. A user input directed at the object is received via the display and interpreted to correspond to a virtual action based on the selected interaction mode. The virtual action is executed with respect to a virtual object associated with the physical object to modify an appearance of the virtual object. The modified virtual object is then displayed via the display.

BACKGROUND

Searching for and retrieving relevant information using computing devices is a common task. Typical computing devices that are used to find and display information may include smartphones, tablets, notebook computers, and the like. Such computing devices, however, are typically limited to connecting a user to relevant information via dedicated graphical user interfaces and corresponding display elements, such as web search portals, hyperlinks, etc. These well-known interfaces and display elements generated by a corresponding computing device are tied to the device, and generally embody little or no context linking them to the user or the user's environment. Further, typical computing devices are fairly limited in their capacity to understand an environment in which they are used and/or to link user activity within the environment to information relevant to the user.

SUMMARY

Various embodiments are disclosed herein that relate to systems and methods for interacting with a physical object in a mixed reality environment. For example, one disclosed embodiment provides a method for interacting with a physical object in a mixed reality environment. The method includes providing a head-mounted display device operatively connected to a computing device, with the head-mounted display device including a display system for presenting the mixed reality environment and a plurality of input sensors including a camera for capturing an image of the physical object. A physical object is identified based on the captured image, and an interaction context is determined for the identified physical object based on one or more aspects of the mixed reality environment.

The method includes querying a stored profile for the physical object to determine a plurality of interaction modes for the physical object. The method includes programmatically selecting a selected interaction mode from the plurality of interaction modes based on the interaction context. A user input directed at the physical object is received via one of the input sensors of the head-mounted display device. The user input is interpreted to correspond to a virtual action based on the selected interaction mode.

The method further includes executing the virtual action with respect to a virtual object that is associated with the physical object to thereby modify the virtual object's appearance. The method then displays the virtual object via the head-mounted display device with the modified appearance.

DETAILED DESCRIPTION

FIG. 1shows a schematic view of one embodiment of a mixed reality interaction system10. The mixed reality interaction system10includes a mixed reality interaction program14that may be stored in mass storage18of a computing device22. The mixed reality interaction program14may be loaded into memory28and executed by a processor30of the computing device22to perform one or more of the methods and processes described in more detail below.

The mixed reality interaction system10includes a mixed reality display program32that may generate a virtual environment34for display via a display device, such as the head-mounted display (HMD) device36, to create a mixed reality environment38. As described in more detail below, the virtual environment34may include one or more virtual objects, such as virtual object140, virtual object242, and geo-located target virtual object52. Such virtual objects may include one or more virtual images, such as three-dimensional holographic objects and other virtual objects, as well as two-dimensional virtual images, that are generated and displayed by HMD device36.

The computing device22may take the form of a desktop computing device, a mobile computing device such as a smart phone, laptop, notebook or tablet computer, network computer, home entertainment computer, interactive television, gaming system, or other suitable type of computing device. Additional details regarding the components and computing aspects of the computing device22are described in more detail below with reference toFIG. 5.

The computing device22may be operatively connected with the HMD device36using a wired connection, or may employ a wireless connection via WiFi, Bluetooth, or any other suitable wireless communication protocol. For example, the computing device22may be communicatively coupled to a network16. The network16may take the form of a local area network (LAN), wide area network (WAN), wired network, wireless network, personal area network, or a combination thereof, and may include the Internet.

As described in more detail below, the computing device22may communicate with one or more other computing devices, such as server20, via network16. Additionally, the example illustrated inFIG. 1shows the computing device22as a separate component from the HMD device36. It will be appreciated that in other examples the computing device22may be integrated into the HMD device36.

With reference now also toFIG. 2, one example of an HMD device200in the form of a pair of wearable glasses with a transparent display44is provided. It will be appreciated that in other examples, the HMD device200may take other suitable forms in which a transparent, semi-transparent or non-transparent display is supported in front of a viewer's eye or eyes. It will also be appreciated that the HMD device36shown inFIG. 1may take the form of the HMD device200, as described in more detail below, or any other suitable HMD device. Additionally, many other types and configurations of display devices having various form factors may also be used within the scope of the present disclosure. Such display devices may include hand-held smart phones, tablet computers, and other suitable display devices.

With reference toFIGS. 1 and 2, the HMD device36includes a display system48and transparent display44that enables images such as holographic objects to be delivered to the eyes of a user46. The transparent display44may be configured to visually augment an appearance of a physical environment50to a user46viewing the physical environment through the transparent display. For example, the appearance of the physical environment50may be augmented by graphical content (e.g., one or more pixels each having a respective color and brightness) that is presented via the transparent display44to create a mixed reality environment.

The transparent display44may also be configured to enable a user to view a physical, real-world object, such as physical object154and/or physical object258, in the physical environment50through one or more partially transparent pixels that are displaying a virtual object representation. As shown inFIG. 2, in one example the transparent display44may include image-producing elements located within lenses204(such as, for example, a see-through Organic Light-Emitting Diode (OLED) display). As another example, the transparent display44may include a light modulator on an edge of the lenses204. In this example the lenses204may serve as a light guide for delivering light from the light modulator to the eyes of a user. Such a light guide may enable a user to perceive a 3D holographic image located within the physical environment50that the user is viewing, while also allowing the user to view physical objects in the physical environment, thus creating a mixed reality environment.

The HMD device36may also include various sensors and related systems. For example, the HMD device36may include an eye-tracking system62that utilizes at least one inward facing sensor216. The inward facing sensor216may be an image sensor that is configured to acquire image data in the form of eye-tracking data66from a user's eyes. Provided the user has consented to the acquisition and use of this information, the eye-tracking system62may use this information to track a position and/or movement of the user's eyes.

In one example, the eye-tracking system62includes a gaze detection subsystem configured to detect a direction of gaze of each eye of a user. The gaze detection subsystem may be configured to determine gaze directions of each of a user's eyes in any suitable manner. For example, the gaze detection subsystem may comprise one or more light sources, such as infrared light sources, configured to cause a glint of light to reflect from the cornea of each eye of a user. One or more image sensors may then be configured to capture an image of the user's eyes.

Images of the glints and of the pupils as determined from image data gathered from the image sensors may be used to determine an optical axis of each eye. Using this information, the eye-tracking system62may then determine a direction and/or at what physical object or virtual object the user is gazing. Such eye-tracking data66may then be provided to the computing device22. It will be understood that the gaze detection subsystem may have any suitable number and arrangement of light sources and image sensors.

The HMD device36may also include sensor systems that receive physical environment data60from the physical environment50. For example, the HMD device36may include an optical sensor system68that utilizes at least one outward facing sensor212, such as an optical sensor, to capture image data74. Outward facing sensor212may detect movements within its field of view, such as gesture-based inputs or other movements performed by a user46or by a person or physical object within the field of view. Outward facing sensor212may also capture two-dimensional image information and depth information from physical environment50and physical objects within the environment. For example, outward facing sensor212may include a depth camera, a visible light camera, an infrared light camera, and/or a position tracking camera.

The HMD device36may include depth sensing via one or more depth cameras. In one example, each depth camera may include left and right cameras of a stereoscopic vision system. Time-resolved images from one or more of these depth cameras may be registered to each other and/or to images from another optical sensor such as a visible spectrum camera, and may be combined to yield depth-resolved video.

In other examples a structured light depth camera may be configured to project a structured infrared illumination, and to image the illumination reflected from a scene onto which the illumination is projected. A depth map of the scene may be constructed based on spacings between adjacent features in the various regions of an imaged scene. In still other examples, a depth camera may take the form of a time-of-flight depth camera configured to project a pulsed infrared illumination onto a scene and detect the illumination reflected from the scene. It will be appreciated that any other suitable depth camera may be used within the scope of the present disclosure.

Outward facing sensor212may capture images of the physical environment50in which a user46is situated. In one example, the mixed reality display program32may include a 3D modeling system that uses such input to generate a virtual environment34that models the physical environment50surrounding the user46.

The HMD device36may also include a position sensor system72that utilizes one or more motion sensors220to capture position data76, and thereby enable motion detection, position tracking and/or orientation sensing of the HMD device. For example, the position sensor system72may be utilized to determine a direction, velocity and acceleration of a user's head. The position sensor system72may also be utilized to determine a head pose orientation of a user's head. In one example, position sensor system72may comprise an inertial measurement unit configured as a six-axis or six-degree of freedom position sensor system. This example position sensor system may, for example, include three accelerometers and three gyroscopes to indicate or measure a change in location of the HMD device36within three-dimensional space along three orthogonal axes (e.g., x, y, z), and a change in an orientation of the HMD device about the three orthogonal axes (e.g., roll, pitch, yaw).

Position sensor system72may also support other suitable positioning techniques, such as GPS or other global navigation systems. Further, while specific examples of position sensor systems have been described, it will be appreciated that other suitable position sensor systems may be used. In some examples, motion sensors220may also be employed as user input devices, such that a user may interact with the HMD device36via gestures of the neck and head, or even of the body. The HMD device36may also include a microphone system80that includes one or more microphones224that capture audio data82. In other examples, audio may be presented to the user via one or more speakers228on the HMD device36.

The HMD device36may also include a processor230having a logic subsystem and a storage subsystem, as discussed in more detail below with respect toFIG. 5, that are in communication with the various sensors and systems of the HMD device. In one example, the storage subsystem may include instructions that are executable by the logic subsystem to receive signal inputs from the sensors and forward such inputs to computing device22(in unprocessed or processed form), and to present images to a user via the transparent display44.

It will be appreciated that the HMD device36and related sensors and other components described above and illustrated inFIGS. 1 and 2are provided by way of example. These examples are not intended to be limiting in any manner, as any other suitable sensors, components, and/or combination of sensors and components may be utilized. Therefore it is to be understood that the HMD device36may include additional and/or alternative sensors, cameras, microphones, input devices, output devices, etc. without departing from the scope of this disclosure. Further, the physical configuration of the HMD device36and its various sensors and subcomponents may take a variety of different forms without departing from the scope of this disclosure.

With reference now toFIG. 3, descriptions of example use cases and embodiments of the mixed reality interaction system10will now be provided.FIG. 3provides a schematic illustration of a user304located in a physical environment50that comprises an office308, with the user experiencing a mixed reality environment38via an HMD device36in the form of HMD device200. As discussed in more detail below, inFIG. 3the mixed reality environment38may comprise the office308and one or more physical objects, as well as a virtual environment including one or more virtual objects.

In the example illustrated inFIG. 3, the user304sits at a desk312that includes a framed photograph316of the user's spouse and a keyboard320. The optical sensor system68of the HMD device200may capture image data74from the office308, including image data representing the photograph316and other physical objects in the office, such as table324, book328on the table, basketball332, bookcase334and coat rack338. Image data74of one or more of these physical objects may be provided by the HMD device200to the mixed reality interaction program14.

Using this image data74, the mixed reality interaction program14may identify one or more of these physical objects. For example, the mixed reality interaction program14may identify the photograph316and the face of the user's spouse in the photograph by comparing image data74of the face to stored face image data and corresponding identities located on server20. In this example, the mixed reality interaction program14may include a face detection subsystem that detects face images in the image data74. To detect a face image in the image data74, the face detection subsystem may use any suitable face detection technologies and/or algorithms including local binary patterns (LBP), principal component analysis (PCA), independent component analysis (ICA), evolutionary pursuit (EP), Elastic Bunch Graph Matching (EBGM), or other suitable algorithm or combination of algorithms.

The face detection program may access object profile data92on server20to match image data74including the face in photograph316with one or more images and related user profile information corresponding to user's spouse. It will be appreciated that the face detection program may use any suitable facial recognition techniques to match image data74with stored images of the user's spouse.

The mixed reality interaction program14also determines an interaction context84for the framed photograph316based on one or more aspects of the mixed reality environment38. With reference again toFIG. 1, such aspects of the mixed reality environment38may include one or more data feeds86originating from the mixed reality environment38or externally to the mixed reality environment. Data feeds86may include temporal data, such as time of day, day of week, month, season, etc., social networking data, such as postings, status updates, etc. related to user304, location data, weather data, etc.

In some examples and with reference toFIG. 3, such aspects of the mixed reality environment38may also include information more directly linked to the office308. Such information may include the presence of one or more other persons in the office308, the identities of such persons, an ambient light level, an ambient noise level, etc. In other examples, such aspects of the mixed reality environment38may also include information related to the user304, such as user biometric parameters including, for example, heart rate, pupillary response, hemoglobin saturation, skin conductivity, respiration, perspiration, and brainwave activity.

In one example, the mixed reality interaction program determines that a current time of day is 10:30 am in the mixed reality environment38. It will be appreciated that the time of day may be determined, for example, by an internal clock within computing device22or may be received from an external source via data feeds86. Accordingly, in the present example the mixed reality interaction program14uses the current time of day of 10:30 am to determine a Business-Time interaction context for the photograph316.

Having identified the face of the user's spouse in the photograph316, the mixed reality interaction program14may then query a stored profile of the user's spouse in object profile data92to determine a plurality of interaction modes88that are available for the photograph316. As described in more detail below, each of the plurality of interaction modes88may correspond to a different virtual action90that may be taken with respect to the photograph316. In the present example, the available interaction modes88may include a Family Calendar interaction mode and a Family Reunion Planning interaction mode. In other examples, the mixed reality interaction program14may create a new profile for the photograph316that associates one or more interaction modes with the photograph. For example, the mixed reality interaction program14may enable the user304to create a new profile for the photograph316when, for example, a relationship status has changed.

Next, based on the determined interaction context, the mixed reality interaction program14may programmatically select a selected interaction mode. In the present example, with the determined interaction context being Business-Time, the mixed reality interaction program14programmatically selects a Family Calendar interaction mode. In one example, the Business-Time interaction context may be defined as a time of day falling between 8:00 am and 6:00 pm. The mixed reality interaction program14may be configured to pair the Business-Time interaction context with the Family Calendar interaction mode with respect to the photograph316.

Another interaction context, a Personal-Time interaction context, may be defined as a time of day falling between 6:01 pm and 7:59 am. The mixed reality interaction program14may be configured to pair the Personal-Time interaction context with a Family Reunion Planning interaction mode with respect to the photograph316.

In the present example, the mixed reality interaction program14may receive a user input from user304that is directed at the photograph316. For example, the mixed reality interaction program14may receive eye-tracking data66from the HMD device200indicating that the user304is gazing at the photograph316, as indicated by gaze line336. In other examples, the user input may take one or more other forms including, for example, position data76and image data74. The position data76may include head pose data indicating that the user304is facing the photograph316. The image data74may include image data showing the user304pointing or gesturing at the photograph316. It will be appreciated that other forms of user input may similarly be utilized.

The mixed reality interaction program14may then interpret the user304gazing at the photograph316to correspond to a virtual action90. The virtual action90may be based on the selected interaction mode, in this example the Family Calendar interaction mode. The virtual action90may comprise presenting to the user304a virtual object in the form of the user's family calendar stored in a calendar application. It will be appreciated that the user's family calendar is associated with the photograph316of the user's spouse.

The mixed reality interaction program14may then execute the virtual action90with respect to a virtual instantiation of the user's family calendar to modify an appearance of the calendar. For example, to comfortably display the user's family calendar to the user304, the mixed reality interaction program14may control the mixed reality display program32to scale the virtual instantiation of the calendar to a predetermined size. The mixed reality interaction program14may then display the virtual instantiation of the user's family calendar with the modified appearance, as indicated at340, via the mixed reality display program32and HMD device200.

In other examples, the mixed reality interaction program14may enable the user304to create and add new virtual actions90that are associated with the photograph316. For example, the user304may create a virtual action that includes highlighting family members' birthdays on the family calendar. Thereafter, when the user304gazes at the photograph316, the mixed reality interaction program14will highlight family members' birthdays on the displayed family calendar.

In one example, the mixed reality interaction program14may geo-locate the family calendar340in physical proximity to the photograph316. As shown inFIG. 3, in the present example the calendar340may be displayed just above the photograph316such that when the user304gazes at the photograph, the calendar is presented in an easily viewable location just above the photograph. Being geo-located to the photograph316, the calendar340may remain “tethered” to the photograph and may track the location of the photograph in the office308. Advantageously, if the photograph316is moved to another location in the office308, the user304may still easily recall and view the calendar340by gazing at the photograph316.

In another example, the user304may use his right hand344to point at the basketball332near wall348. The mixed reality interaction program14may identify the basketball, determine an interaction context for the basketball, query a stored profile for the basketball, and programmatically select a selected interaction mode based on the interaction context in a manner similar to that described above for the photograph316. A camera in the HMD device200may capture image data74showing the user's right hand344pointing at the basketball332. The mixed reality interaction program14may interpret this user input to correspond to displaying a website of the user's favorite basketball team.

The mixed reality interaction program14may then modify image data of the website to, for example, enhance the contrast of the displayed website when displayed above the basketball and in front of the wall348. The mixed reality interaction program14may then display a holographic version of the modified website352above the basketball332. The website352may also be geo-located to the basketball332such that it follows the basketball to different locations in the office308, as indicated by basketball332′ and website352′ located near the left rear corner of the office.

In some examples, the mixed reality interaction program14may be unable to determine the identity of a physical object, or may determine an incomplete identity of the object. For example, the mixed reality interaction program14may not recognize a visitor342who enters the office308. The mixed reality interaction program14may also determine that the user304currently has a meeting in the office308scheduled in the user's calendar with a John Doe. Based on this meeting, the mixed reality interaction program14may then query the user304, “Is this John Doe?” via text displayed on the HMD device200. The user304may provide a confirmation of the identity of the visitor342by, for example, nodding, speaking “Yes”, or by any other suitable input.

In another example, where the user304does not confirm that the visitor342is John Doe, the mixed reality interaction program14may present another query to the user requesting the identity of the visitor. When the user304answers and provides an identity of the visitor, such as, “This is John Smith,” the program may then associate the identity John Smith with the visitor. It will also be appreciated that such identity confirmation processes may be used with any other physical objects.

In another example, the mixed reality interaction program14may determine a change in the interaction context84and may correspondingly change the interaction mode86. For example, the user304may be working in the office308late one night. At 7:01 pm the mixed reality interaction program14determines that the interaction context changes from Business-Time to Personal-Time. Accordingly, the mixed reality interaction program14may programmatically select the Family Reunion Planning interaction mode with respect to the photograph316.

The mixed reality interaction program14may then interpret the user304gazing at the photograph316to correspond to a virtual action90that is based on the Family Reunion Planning interaction mode. In this example, the virtual action90may comprise highlighting the dates364of the user's family reunion gathering in the user's family calendar340. The mixed reality interaction program14may then execute the virtual action with respect to the calendar340to modify an appearance of the calendar. For example, the mixed reality interaction program14may control the mixed reality display program32to highlight in flashing color the dates364of the family reunion gathering in the calendar340. The mixed reality interaction program14may then render and display the calendar340with the highlighted dates364via the mixed reality display program32.

In another example, after determining a change in the interaction context84to the Family Reunion Planning interaction mode, the mixed reality interaction program14may execute a different virtual action90with respect to a different virtual object that is associated with the photograph316. For example, the mixed reality interaction program14may display a virtual family reunion To-Do List360via the mixed reality display program32and HMD device200to the user304. As with the family calendar340, the mixed reality interaction program14may control the mixed reality display program32to scale a virtual instantiation of the family reunion To-Do List360to a predetermined size. The mixed reality interaction program14may then display the To-Do List360, via the mixed reality display program32with the modified appearance.

In another example, the user304may desire to manually switch between the Family Calendar interaction mode and the Family Reunion Planning interaction mode. The user may request that the current Family Calendar interaction mode be modified by, for example, speaking “Switch to Family Reunion Planning.” The mixed reality interaction program14interprets this user input as a request to modify the interaction mode, and changes the interaction mode accordingly. The user may then point at the photograph316, which is captured as image data74via the HMD device200. The mixed reality interaction program14interprets this user input as corresponding to a virtual action90based on the Family Reunion Planning interaction mode, such as displaying the virtual family reunion To-Do List360. The mixed reality interaction program14may then execute the virtual action and display the family reunion To-Do List360with a modified appearance as described above.

In another example, the mixed reality interaction program14may display a target virtual object that may function to enable user interaction with another virtual object in a manner similar to that described above. For example, the mixed reality interaction program14may display a geo-located target virtual object52in the form of a holographic wizard370located on table324in the mixed reality environment38. The mixed reality interaction program14may then determine a Business-Time interaction context for the wizard370based on one or more aspects of the mixed reality environment38.

The mixed reality interaction program may query a stored profile of the holographic wizard370in the object profile data92to determine a plurality of interaction modes88that are available for the wizard. In the present example, the available interaction modes88may include a Stock Market interaction mode and a Movie Listings interaction mode. Next, based on the determined interaction context, the mixed reality interaction program14may programmatically select a selected interaction mode. In the present example, with the determined interaction context being Business-Time, the mixed reality interaction program14programmatically selects the Stock Market interaction mode.

In the present example, the mixed reality interaction program14may receive a user input from user304that is directed at the holographic wizard370, such as position data76including head pose data indicating that the user's face is oriented toward the wizard370. The mixed reality interaction program14may then interpret the user304facing the wizard370to correspond to a virtual action90that is based on the selected interaction mode, in this example the Stock Market interaction mode, and is associated with the wizard370. The virtual action90may comprise presenting a stock market summary374via the mixed reality display program32and HMD device200to the user304.

The mixed reality interaction program14may then execute the virtual action90with respect to the stock market summary to, for example, highlight quotes of stocks owned by the user304, such as the XYZ stock quote376. The mixed reality interaction program14may control the mixed reality display program32to highlight the XYZ stock quote376. The mixed reality interaction program14may then display the stock market summary374via the mixed reality display program32.

In another example, the mixed reality interaction program14may also use a user input to create an association between a physical object and a selected interaction mode. With continued reference toFIG. 3, in one example the user304may desire to associate the basketball332with the Anytown Ants website352. The user304may pick up the basketball332and rotate the basketball in the user's hands.

Using image data74, the mixed reality interaction program14may recognize the user's rotation of the basketball332as a trigger to display to the user304the available interaction modes88for the basketball. Such interaction modes88may include, for example, displaying the Anytown Ants website, displaying a scoreboard of current basketball games, and displaying a calendar showing upcoming games in the user's basketball league. The user304may then select the Anytown Ants website interaction mode to associate with the basketball332via user input received by the HMD device200. Accordingly, the mixed reality interaction program14may then be configured to associate the basketball332with the Anytown Ants website interaction mode. Subsequently, the mixed reality interaction program14may execute a virtual action with respect to the Anytown Ants website that is associated with the basketball332when user input directed at the basketball332is received, as described above.

It will be appreciated that the present system may enhance a user's ability to associate particular information sources, portals, etc. with objects in the mixed reality environment38, whether physical objects or virtual objects. For example, by linking the holographic wizard370with a displayed stock market summary, the user may more easily remember to use the wizard to see the stock market summary during business hours. The user may, for example, memorize the phrase, “I'm a Wizard of the stock market” to provide an easily-remembered cognitive link between the holographic wizard370in the user's office308and the stock market summary.

In another example, the mixed reality interaction program14may generate audio feedback that is perceived by the user304as originating from a virtual object or physical object in the office308. The mixed reality interaction program14may utilize a three-dimensional audio positioning technique, such as a head-related transfer function, to generate such audio feedback via speakers228of the HMD200. In one example, the mixed reality interaction program14may generate audio comprising a broadcast of an Anytown Ants basketball game that is perceived by the user304as originating from the virtual website352or the basketball332.

In another example, the mixed reality interaction program14may be configured to take a virtual action with respect to an object that is placed on the table324. With reference toFIG. 3, the mixed reality interaction program14may recognize that book328is resting on the table324. Accordingly, based on the book328resting on the table324, the mixed reality interaction program14may programmatically post a status update to the user's social network that notes that the user304is currently reading book328.

In another example, the mixed reality interaction program14may be configured to take a virtual action when a state of an object changes. For example, the mixed reality interaction program14may identify that the user304has begun drinking his regular cup of morning coffee at 6:30 am. The program may monitor the amount of coffee remaining in the user's cup. If a predetermined amount of coffee is remaining in the cup and the user attempts to check email, the program may prevent or dissuade the user from checking email. For example, upon detecting an attempt to check email, the program may display a message to the user asking, “Wouldn't you like to finish your relaxing cup of coffee first?”

In another example, a document related to a work project managed by the user304may be linked to the user, such as in the object profile data92. A fellow team member of the work project may also wear an HMD device200that is communicatively coupled to a mixed reality interaction program14. When the fellow team member sees the user304, the team member's mixed reality interaction program14recognizes the user304and displays the document related to the work project via the team members' HMD device200.

In another example, a team member may virtually deliver a document to the user304via leaving a physical or virtual object on the user's desk312. For example, a team member of the user304may associate the document with a cardboard cutout of the letter “D.” The team member may leave the cutout D on the user's desk312while the user304is away. Upon returning to the office308, the user's HMD device200may recognize the cutout D. The HMD device200may indicate to the user304that the cutout D is actively linked to virtual information by, for example, animating the cutout D, such as by causing the cutout D to glow when viewed. The user304may request to view the linked virtual information, and the HMD device200may then access the document linked to the cutout D, and may display the document for the user304.

In another example, the user304may provide a user input that applies to multiple virtual objects associated with multiple physical objects. For example and with reference toFIG. 3, the user304may desire to dim all of the virtual objects displayed in the office308. The user304may extend his right hand344outwardly and lower his hand toward the desk312. The mixed reality interaction system10may recognize this gesture, and may correspondingly dim all of the virtual objects displayed via HMD200.

In another example, when the user304gazes at the basketball332, the mixed reality interaction program14may interpret the user's gaze to correspond with displaying over the basketball a virtual volume control knob. The volume control knob may be virtually rotated by the user304to adjust the volume of the broadcast of the Anytown Ants basketball game to which the user is listening. The current volume may also be displayed as a digital volume indicator380. In another example, the user304may pick up and physically rotate the basketball332to correspondingly adjust the volume of the broadcast via the virtual volume control knob.

In another example, while the user304is away from the office308, the visitor342may pick up and rotate the basketball332to lower the volume of the basketball game broadcast that is being delivered by speakers in the room308. When the user304returns to the office308, the mixed reality interaction program14may identify the new orientation of the basketball332and correspondingly lower the volume of the broadcast being delivered to the user via HMD200. The program may also correspondingly modify the volume indicator380.

FIGS. 4A, 4B and 4Cillustrate a flow chart of a method400for interacting with a physical object in a mixed reality environment according to an embodiment of the present disclosure. The following description of method400is provided with reference to the software and hardware components of the mixed reality interaction system10described above and shown inFIGS. 1 and 2. It will be appreciated that method400may also be performed in other contexts using other suitable hardware and software components.

With reference toFIG. 4A, at402the method400includes providing an HMD device operatively connected to a computing device. The HMD device includes a display system for presenting a mixed reality environment and a camera for capturing images of physical objects. At406the method400includes identifying a physical object based on a captured image. At410the method400may include presenting a query to the user to confirm an accuracy of an identity of the object. At412the method400may include, in response to the query, receiving confirmation of the accuracy of the identity of the object.

At414the method400includes determining an interaction context for the identified physical object based on one or more aspects of the mixed reality environment. At416, the aspects of the mixed reality environment may comprise information received from one or more data feeds. At418the method400includes querying a stored profile for the physical object to determine a plurality of interaction modes for the object. At420the method400includes programmatically selecting a selected interaction mode from the plurality of interaction modes based on the interaction context. At422the method400includes receiving a user input directed at the physical object via one of the input sensors of the HMD device. At424the method400includes interpreting the user input to correspond to a virtual action based on the selected interaction mode.

At426the method400includes executing the virtual action with respect to a virtual object associated with the physical object to thereby modify an appearance of the virtual object. And at428the method400includes displaying the virtual object via the HMD device with the modified appearance. With reference now toFIG. 4B, at430and in some examples the method400includes geo-locating the virtual object in physical proximity to the physical object. At432the method400may include using a second user input to associate the physical object with a selected interaction mode.

In another example, at434the method400may include, where the selected interaction mode is a first selected interaction mode and the virtual action is a first virtual action, determining a change in the interaction context. At436the method400may include, based on the change, programmatically selecting a second selected interaction mode from a plurality of interaction mode. At438the method400may include interpreting user input to correspond to a second virtual action based on the second selected interaction mode. At440the method400may include executing the second virtual action with respect to the virtual object associated with the physical object to modify the appearance of the virtual object. And at442the method400may include displaying the virtual object via the HMD device with the modified appearance.

In another example, at444the method400may include executing a second virtual action with respect to a second virtual object that is associated with the physical object to modify the appearance of the second virtual object. At446the method400may include displaying the second virtual object via the HMD device with the modified appearance.

In another example, at448the method400may include receiving a second user input via an input sensor of the HMD device. At450the method400may include, where the selected interaction mode is a first selected interaction mode, interpreting the second user input to correspond to a request to modify the first selected interaction mode. With reference now toFIG. 4C, at452the method400includes, based on the second user input, changing from the first selected interaction mode to a second selected interaction mode. At454the method400may include receiving a third user input via the input sensor of the HMD device. At456the method400may include interpreting the third user input to correspond to a second virtual action based on the second selected interaction mode.

At458the method400may include executing the second virtual action with respect to the virtual object associated with the physical object to modify the appearance of the virtual object. At460the method may include displaying the virtual object via the HMD device with the modified appearance.

In another example, at462the method400may include displaying a geo-located target virtual object in the mixed reality environment. At464the method400may include determining an interaction context for the geo-located target virtual object based on one or more aspects of the mixed reality environment. At466the method may include querying a stored profile for the geo-located target virtual object to determine a plurality of interaction modes for the geo-located target virtual object. At468the method400may include programmatically selecting a selected interaction mode for the geo-located target virtual object based on the interaction context. At470the method400may include receiving a second user input directed at the geo-located target virtual object via an input sensor of the HMD device.

At472the method may include interpreting the second user input to correspond to a second virtual action based on the selected interaction mode. At474the method400may include executing the second virtual action with respect to the geo-located target virtual object to modify the appearance of the geo-located target virtual object. And at476the method400may include displaying the geo-located target virtual object via the HMD device with the modified appearance.

It will be appreciated that method400is provided by way of example and is not meant to be limiting. Therefore, it is to be understood that method400may include additional and/or alternative steps than those illustrated inFIGS. 4A, 4B AND 4C. Further, it is to be understood that method400may be performed in any suitable order. Further still, it is to be understood that one or more steps may be omitted from method400without departing from the scope of this disclosure.

FIG. 5schematically shows a nonlimiting embodiment of a computing system500that may perform one or more of the above described methods and processes. Computing device22may take the form of computing system500. Computing system500is shown in simplified form. It is to be understood that virtually any computer architecture may be used without departing from the scope of this disclosure. In different embodiments, computing system500may take the form of a mainframe computer, server computer, desktop computer, laptop computer, tablet computer, home entertainment computer, network computing device, mobile computing device, mobile communication device, gaming device, etc. As noted above, in some examples the computing system500may be integrated into an HMD device.

As shown inFIG. 5, computing system500includes a logic subsystem504and a storage subsystem508. Computing system500may optionally include a display subsystem512, a communication subsystem516, a sensor subsystem520, an input subsystem522and/or other subsystems and components not shown inFIG. 5. Computing system500may also include computer readable media, with the computer readable media including computer readable storage media and computer readable communication media. Computing system500may also optionally include other user input devices such as keyboards, mice, game controllers, and/or touch screens, for example. Further, in some embodiments the methods and processes described herein may be implemented as a computer application, computer service, computer API, computer library, and/or other computer program product in a computing system that includes one or more computers.

Logic subsystem504may include one or more physical devices configured to execute one or more instructions. For example, the logic subsystem504may be configured to execute one or more instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more devices, or otherwise arrive at a desired result.

The logic subsystem504may include one or more processors that are configured to execute software instructions. Additionally or alternatively, the logic subsystem may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. Processors of the logic subsystem may be single core or multicore, and the programs executed thereon may be configured for parallel or distributed processing. The logic subsystem may optionally include individual components that are distributed throughout two or more devices, which may be remotely located and/or configured for coordinated processing. One or more aspects of the logic subsystem may be virtualized and executed by remotely accessible networked computing devices configured in a cloud computing configuration.

Storage subsystem508may include one or more physical, persistent devices configured to hold data and/or instructions executable by the logic subsystem504to implement the herein described methods and processes. When such methods and processes are implemented, the state of storage subsystem508may be transformed (e.g., to hold different data).

In some embodiments, aspects of logic subsystem504and storage subsystem508may be integrated into one or more common devices through which the functionally described herein may be enacted, at least in part. Such hardware-logic components may include field-programmable gate arrays (FPGAs), program- and application-specific integrated circuits (PASIC/ASICs), program- and application-specific standard products (PSSP/ASSPs), system-on-a-chip (SOC) systems, and complex programmable logic devices (CPLDs), for example.

FIG. 5also shows an aspect of the storage subsystem508in the form of removable computer readable storage media524, which may be used to store data and/or instructions executable to implement the methods and processes described herein. Removable computer-readable storage media524may take the form of CDs, DVDs, HD-DVDs, Blu-Ray Discs, EEPROMs, and/or floppy disks, among others.

It is to be appreciated that storage subsystem508includes one or more physical, persistent devices. In contrast, in some embodiments aspects of the instructions described herein may be propagated in a transitory fashion by a pure signal (e.g., an electromagnetic signal, an optical signal, etc.) that is not held by a physical device for at least a finite duration. Furthermore, data and/or other forms of information pertaining to the present disclosure may be propagated by a pure signal via computer-readable communication media.

When included, display subsystem512may be used to present a visual representation of data held by storage subsystem508. As the above described methods and processes change the data held by the storage subsystem508, and thus transform the state of the storage subsystem, the state of the display subsystem512may likewise be transformed to visually represent changes in the underlying data. The display subsystem512may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with logic subsystem504and/or storage subsystem508in a shared enclosure, or such display devices may be peripheral display devices. The display subsystem512may include, for example, the display system48and transparent display44of the HMD device36.

When included, communication subsystem516may be configured to communicatively couple computing system500with one or more networks and/or one or more other computing devices. Communication subsystem516may include wired and/or wireless communication devices compatible with one or more different communication protocols. As nonlimiting examples, the communication subsystem516may be configured for communication via a wireless telephone network, a wireless local area network, a wired local area network, a wireless wide area network, a wired wide area network, etc. In some embodiments, the communication subsystem may allow computing system500to send and/or receive messages to and/or from other devices via a network such as the Internet.

Sensor subsystem520may include one or more sensors configured to sense different physical phenomenon (e.g., visible light, infrared light, sound, acceleration, orientation, position, etc.) as described above. Sensor subsystem520may be configured to provide sensor data to logic subsystem504, for example. As described above, such data may include eye-tracking information, image information, audio information, ambient lighting information, depth information, position information, motion information, user location information, and/or any other suitable sensor data that may be used to perform the methods and processes described above.

The term “program” may be used to describe an aspect of the mixed reality interaction system10that is implemented to perform one or more particular functions. In some cases, such a program may be instantiated via logic subsystem504executing instructions held by storage subsystem508. It is to be understood that different programs may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same program may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The term “program” is meant to encompass individual or groups of executable files, data files, libraries, drivers, scripts, database records, etc.