Patent Description:
A mixed reality devices can lock the position of a virtual object relative to real-world objects (referred to as "world locking" a virtual object). For example, a virtual flower pot can be placed onto a real-world table and appear as if it is on the surface of the real-world table. The virtual flower pot is spatially locked with respect to the surface of the real-world table, and may leave the user's field of view as the user moves about in their real-world surrounding. If the virtual object is world locked to a moving real-world object, the virtual object may move with the moving real-world object.

Additionally, mixed reality devices can also show notifications and other forms of information in a way that is locked to the display of the mixed reality device, rather than to real-world objects. This viewing mode can be useful for invoking applications such as a web browser, an email client, and so on. In this viewing mode, the virtual object merely overlays the user's field of view independently of real-world objects in their field of view. <CIT> relates to features for supporting an augmented reality shopping experience on a consumer user's mobile device. Information is received from a mobile device over a data connection. The information includes identification data relating to a shopping display to which a mobile device is proximate. Based on the received information, an augmented reality three dimensional model of the shopping display is identified from a plurality of augmented reality three dimensional models of shopping displays stored at or accessible by a backend server. Augmented reality data are provided to the mobile device over the data connection to enable the mobile device to present an augmented reality view of the shopping display on a screen of the mobile device. The augmented reality view includes a two dimensional image of the shopping display captured by a camera associated with the mobile device presented concurrently with a two dimensional overlay such that an augmented reality marker in the two dimensional overlay is associated with a physical item in the shopping display. Selection data indicating selection of the augmented reality marker are received from the mobile device over the data connection and data necessary to present an item page corresponding to the physical item are provided to the mobile device over the data connection in response to the received selection. <CIT> describes a mobile terminal that includes a camera for sequentially acquiring a first image for augmented reality (AR), which include at least one object, a display unit for outputting the first image, and a controller for, if an input for selecting a specific object among the at least one object included in the first image is received, storing information related to the selected specific object. If a virtual reality (VR) mode corresponding to the first image is executed, the controller forms and outputs a second image corresponding to the VR mode, based on the stored information. If update information related to the selected specific object is received from an external server, the controller controls a mark change corresponding to the received update information to be marked on the second image. <CIT> relates to an online shopping system utilizing a virtual reality and augmented reality technology. A user can select an augmented reality mode switching option to activate the AR mode. Virtual images of commodities can be combined with a real scene taken by a camera device of an electronic display device. The real scene is the environment where the user stays. When the user clicks a table at a shopping website, the system loads images of the table according to the identification data thereto under the AR mode. Then, the electronic display device is used to capture the real scene, such as a living room, where the table is projected. The image of the table can be combined with the image of the living room in this AR mode. The AR mode allows the user to look at and feel the decorations in the real scene, for example, the user can try to move the commodity to a favorite position in the living room. <CIT> relates to an augmented reality auction platform. An augmented reality submission includes a hologram to virtually augment a world space object and a compensation offer for presenting the hologram to a viewer of the world space object. The augmented reality submission is selected as a winning submission if the submission satisfies a selection criteria.

A mixed reality device in accordance with the present disclosure can include a display, one or more computer processors, and a computer-readable storage medium comprising instructions for controlling the one or more computer processors. Processing performed by the one or more computer processor can include instantiating a holographic button (virtual <NUM>-dimensional object) associated with an item of interest to a user of the mixed reality device. Item-related data associated with the item can be received from the website of a provider of the item. The item-related data can include holographic data to display the holographic button on a display of the mixed reality device. The holographic button can be world-locked to a fixed spatial location in a real-world surrounding relative to the mixed reality device.

The user of the mixed reality device can be provided with access to the holographic button, thus allowing the user to obtain the item by interacting with the holographic button. For example, the holographic button can be presented on the display to superimpose the holographic button onto a view of the user's real-world surrounding. The mixed reality device can detect a user-made gesture directed to the holographic button, and in response generate and send an order request for the item to the provider.

The following detailed description and accompanying drawings provide further understanding of the nature and advantages of the present disclosure.

With respect to the discussion to follow and in particular to the drawings, it is stressed that the particulars shown represent examples for purposes of illustrative discussion, and are presented in the cause of providing a description of principles and conceptual aspects of the present disclosure. In this regard, no attempt is made to show implementation details beyond what is needed for a fundamental understanding of the present disclosure. The discussion to follow, in conjunction with the drawings, makes apparent to those of skill in the art how embodiments in accordance with the present disclosure can be practiced. Similar or same reference numbers may be used to identify or otherwise refer to similar or same elements in the various drawings and supporting descriptions. In the accompanying drawings:.

The present disclosure relates generally to mixed reality environments. A user in a mixed reality world can drag an image or link from a website to generate/download a 3D model that represents a holographic button, and then drop or place it on a relevant real-world object as perceived in their mixed reality world. The holographic button is associated with an item (goods, data content, services, and the like), and includes item-related data that allows the user to order the item simply by interacting (e.g., air tapping) with the holographic button. The user can instantiate several holographic buttons in their mixed reality world. The user need only interact with a holographic button to initiate an action to order the item associated with that holographic button, without having to navigate to a website to order the item.

It can be appreciated that navigating the Internet in a mixed reality world can be taxing to the user, requiring gestures and textual input to access various websites. User interaction performance and reliability can be enhanced by the use of holographic buttons in accordance with the present disclosure. Since the holographic button is already associated with information for obtaining the item, the amount of user interaction to obtain the item over the Internet using the holographic button is significantly reduced.

It will become clear that the use of holographic buttons in accordance with the present disclosure to obtain goods or services can reduce bandwidth usage by the mixed reality device. Since the holographic button already provides the information needed by the user to initiate a request to order the item, the user can skip steps that require network resources such as accessing the provider's website, navigating to the particular web page that contains the item, and interacting with the web page to initiate an order.

In addition to reducing bandwidth usage, the use of holographic buttons in accordance with the present disclosure can reduce storage requirements in the mixed reality device for storing browser history. When a user is required to search the Internet for items of interest, especially consumable items that require re-ordering, the browser history can become cluttered from those search efforts repeated over time. Associating such items with holographic buttons in accordance with the present invention obviates the need for repeated searches of those items and thus can reduce the amount of browser history that is accumulated in the memory of the mixed reality device.

In the following description, for purposes of explanation, numerous examples and specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be evident, however, to one skilled in the art that the present disclosure as expressed in the claims may include some or all of the features in these examples, alone or in combination with other features described below, and may further include modifications and equivalents of the features and concepts described herein.

<FIG> illustrates the use of a holographic button <NUM> in a mixed reality experience in accordance with the present disclosure. More particularly, the user <NUM> can interact with the holographic button <NUM> to obtain an item <NUM> of interest to the user <NUM>. Since the holographic button <NUM> is virtual, it can visually appear as any image. In some instances, the holographic button <NUM> can appear as some form of a button. In other instances, the holographic button <NUM> can appear as some other image; e.g., the image may be representative of the item <NUM> or may be completely arbitrary.

In the particular use case depicted in <FIG>, the item <NUM> is supplied by a provider <NUM>. The provider <NUM> supports a website <NUM> which the user <NUM> can visit to access the item <NUM>. The item <NUM> can be a good or service that can be ordered over the Internet. The item <NUM> can be data content that can be downloaded over the Internet, and so on. In some cases, the item <NUM> can be a paid for subscription for content or a service that the user <NUM> can request on demand. In other cases, the user <NUM> pays for the item <NUM>, but that is not a necessary component in the present disclosure.

The configuration shown in <FIG> includes a mixed reality device <NUM> that can provide a mixed reality experience to the user <NUM> by immersing the user <NUM> in a mixed reality world. The mixed reality device <NUM> can include any number of applications that can execute on the device. In various embodiments, for example, the mixed reality device <NUM> can include a 3D viewer <NUM>, a web browser <NUM>, and so on. As explained in more detail below, the web browser <NUM> can provide access to the provider's website <NUM>, where the user <NUM> can instantiate a holographic button <NUM> associated with the item <NUM> of interest. The 3D viewer <NUM> can be used to interact with the user <NUM> to obtain the item <NUM> using the holographic button <NUM>.

In some embodiments, the 3D viewer <NUM> can be implemented as a platform-level application, where its functionality can be provided as services accessed via suitable application programming interfaces (APIs). In <FIG>, for example, the provider <NUM> may provide services via site API <NUM> that allow the 3D viewer <NUM> to submit order requests to the provider <NUM>. The 3D viewer <NUM> can incorporate users' payment instruments into the order requests via a payment module <NUM>.

The site API <NUM> can include services to support the provider's website <NUM>. For example, the site API <NUM> can include functionality for receiving, storing, and otherwise managing item-related data 22a for items maintained on the website <NUM>. Item-related data 22a can include ordering information <NUM> that relate to pricing of the item <NUM> (e.g., price, monetary exchange rates, etc.), descriptions and attributes of the item <NUM> (e.g., dimensions, color, size, etc.), and other information that may be needed to fulfill orders for the item <NUM>. In accordance with the present disclosure, the item-related data 22a can also include holographic data <NUM> for the holographic button <NUM>.

The payment module <NUM> can receive, store, and otherwise manage users' payment instruments. The payment module <NUM> can employ a repository <NUM> to store the users' payment instruments. The repository <NUM> can include a payment API 124a to provide secure access to the payment instruments and to conduct payment transactions.

<FIG> illustrates a mixed reality device <NUM> in accordance with some embodiments of the present disclosure. The mixed reality device <NUM> that can take the form of a wearable device, such as shown in <FIG>. Different wearable form factors include goggles, glasses, and the like. An example is the HOLOLENS® wearable mixed reality device developed, manufactured, and sold by the assignee of the present disclosure. It will be appreciated that the mixed reality device <NUM> can be implemented in other formats such as, for example, in a computer tablet, smartphone, or other handheld computing device.

The mixed reality device <NUM> can include a headband <NUM> for wearing the device. The headband <NUM> can be adjustable to provide a suitable fit for the user <NUM>. The headband <NUM> can house or support various components comprising the mixed reality device <NUM>.

The mixed reality device <NUM> can include a computing system <NUM>, which among other things, can display virtual objects (holograms) to the user <NUM> on a display <NUM>. In some embodiments, the display <NUM> can be partially transparent so that the user <NUM> can view their real-world surrounding through the display. The display <NUM> can comprise a left display panel 206a and a right display panel 206b. The left and right display panels 206a, 206b can be implemented using any suitable technology; e.g., using transparent organic light emitting diodes (OLED), edge-mounted light modulators, and so on. In some embodiments where the mixed reality device <NUM> is a handheld device (e.g., computer tablet, smartphone, etc.) the display <NUM> can be the display of the handheld device.

Virtual objects displayed or presented on the display <NUM> are visually superimposed onto the user's field of view such that the virtual objects appear with the real-world objects. The left and right display panels 206a, 206b can provide a stereoscopic effect so as to visually place the virtual object at a desired depth by presenting separate images of the virtual object in the left and right display panels 206a, 206b.

In some embodiments where the mixed reality device <NUM> is a handheld device (e.g., computer tablet, smartphone, etc.) the display <NUM> can be the display of the handheld device. The real-world surrounding can be presented as a live feed on the display <NUM> (e.g., captured by a camera 212b), and virtual objects can be superimposed on the live feed.

The mixed reality device <NUM> can include left and right microphones 208a, 208b for receiving voice input from the user <NUM>; e.g., for voice commands. Left and right speakers 210a, 210b can provide stereo audio to the user <NUM>.

The mixed reality device <NUM> can include optical sensors such as an infra-red (IR) camera 212a and an outward facing optical wavelength RGB camera 212b to capture video and still images. The cameras 212a, 212b can capture the user's real-world surrounding. Additional sensors such as accelerometers, gyroscopes, etc. (not shown) can provide position and orientation information of the mixed reality device <NUM>. The optical sensor information and the position sensor information can be used to perform analysis of the three dimensional real-world surrounding, such as depth analysis and surface reconstruction, to compute or otherwise determine the spatial location of the mixed reality device <NUM> in the user's real-world surrounding.

The optical and position sensor information can also be used to create spatial mapping data that is a virtual representation of the real-world surrounding. In some embodiments, the spatial mapping data may define a three dimensional coordinate space that corresponds to the three dimensional coordinate space of the real-world surrounding. Using the spatial mapping data, the mixed reality device <NUM> can automatically track where the user is looking so that the device can determine where in the user's field of view to insert virtual objects, thus creating a mixed reality world experience for the user <NUM>.

The user <NUM> can interact with virtual objects and real-world objects appearing within their field of view. As used herein, the term "interact" encompasses both physical interaction and verbal interaction. Physical interaction includes the user <NUM> performing predefined gestures using their fingers, hand, head and so on that can be recognized by the mixed reality device <NUM> as a user request for the device to perform a predefined action on the virtual or real-world object. Such predefined gestures can include, but are not limited to, pointing at, grabbing, and pushing virtual objects, hand waving motions, and the like. Other gestures can include an "air tap" on the virtual or real-world object, in which the user <NUM> gazes at the object and forms an "L" with their thumb and forefinger, and making a pinching motion by touching the tip of one to the tip of the other. An interaction can simply consist of the user <NUM> gazing in the direction of the virtual or real-world object.

The user <NUM> can interact with virtual and real-world objects using verbal gestures such as speaking a word or phrase that is recognized by the mixed reality device <NUM> as a user request for the device to perform a predefined action. Verbal gestures can be used in conjunction with physical gestures to interact with virtual and real-world objects in the mixed reality world.

As the user <NUM> moves around within their mixed reality world, a virtual object may be "world-locked"; in other words, the virtual object is spatially locked with respect to the user's real-world surrounding. The virtual object can remain anchored and stationary in the mixed reality world so that the user <NUM> can move around the world-locked virtual object and see it from different perspectives and distances, as if the virtual object physically existed in the user's real-world surrounding (hence "mixed reality").

<FIG> depicts a simplified block diagram of an example of computing system <NUM> according to certain embodiments. As shown in <FIG>, computing system <NUM> includes one or more processors <NUM> that communicate with a number of peripheral devices via a bus subsystem <NUM>. These peripheral devices include a storage subsystem <NUM> (comprising a memory subsystem <NUM> and a file storage subsystem <NUM>), input devices <NUM>, output devices <NUM>, and network access <NUM>.

Bus subsystem <NUM> can provide a mechanism for enabling the various components and subsystems of computing system <NUM> to communicate with each other as intended. Although bus subsystem <NUM> is shown schematically as a single bus, alternative embodiments of the bus subsystem can utilize multiple busses.

Network access <NUM> can include a wireless interface that can implement wireless standards such as WiFi and Bluetooth, in various embodiments. In other embodiments, network access <NUM> may support wired communications. Network access <NUM> can serve as an interface for communicating data between computing system <NUM> and other computer systems or networks; e.g., to access website <NUM> (<FIG>).

Input devices <NUM> include microphones 208a, 208b and cameras 212a, 212b, shown in <FIG>. Output devices <NUM> include display <NUM> and speakers 210a, 210b.

Storage subsystem <NUM> includes a memory subsystem <NUM> and a file/data storage subsystem <NUM>. Subsystems <NUM> and <NUM> represent non-transitory computer-readable storage media that can store program code (e.g., 3D viewer <NUM>, web browser <NUM>) and/or data (e.g., holographic button <NUM>, item-related data 22a) that provide and support the functionality of embodiments in accordance with the present disclosure.

Memory subsystem <NUM> includes a number of memories including a main random access memory (RAM) <NUM> for storage of instructions and data during program execution and a read-only memory (ROM) <NUM> in which fixed instructions are stored. File storage subsystem <NUM> can provide persistent (i.e., non-volatile) storage for program and data files, and can include a solid-state drive.

It should be appreciated that computing system <NUM> is illustrative and many other configurations having more or fewer components are possible.

<FIG> illustrates a high level process for a holographic button <NUM> in a mixed reality world in accordance with the present disclosure. Generally, the holographic button <NUM> facilitates the user <NUM> in obtaining an item <NUM> that the holographic button <NUM> is associated with.

At block <NUM>, a user <NUM> can identify or otherwise become interested in an item <NUM>. Merely as an illustrative example suppose in one example the item is laundry detergent (a good).

At block <NUM>, the user <NUM>, using a mixed reality device (e.g., <NUM>), can obtain a holographic button <NUM> from the provider <NUM> of the item <NUM>. For example, if the laundry detergent is sold at a grocery store, the user can navigate to the grocer's website and download a holographic button corresponding to the laundry detergent from the grocer's website. The holographic button can be any image defined by the grocer; for example, the image may be a box of detergent.

At block <NUM>, the user <NUM> can virtually place the downloaded holographic button <NUM> on or near a real-world object (call it the trigger object). For example, the trigger object might be a washing machine for the laundry detergent example. The holographic button for the laundry detergent can be placed somewhere on the washing machine, or near (e.g., on a wall) the washing machine.

At this point, the holographic button <NUM> has been obtained from the provider's website <NUM> and placed in the user's mixed reality world. The user <NUM> can exit their mixed reality world and at a later time return to the mixed reality world to obtain the item <NUM> using the holographic button <NUM>. For example, if the user <NUM> is doing laundry, they may realize they are low on laundry detergent. The user <NUM> may order additional laundry detergent using the holographic button <NUM> in their mixed reality world.

At block <NUM>, for example, the user <NUM> can re-enter their mixed reality world and interact with the washing machine (e.g., gaze toward the washing machine).

when the user <NUM> interacts with the trigger object the holographic button <NUM> can become visible to the user <NUM>. Interaction with the trigger object may include the user <NUM> physically touching the trigger object, or in some embodiments, simply directing their gaze toward the trigger object. Using gaze direction as an interaction can be useful if it is inconvenient for the user <NUM> to be physically near the trigger object. For example, if the user <NUM> is far away from the washing machine, they need simply to gaze in the direction of the washing machine in their mixed reality world, and in response the mixed reality device can make visible the holographic button <NUM> for laundry detergent.

At block <NUM>, when the holographic button <NUM> becomes available to the user <NUM>, the user <NUM> can interact with it to initiate an action with the provider to obtain the item <NUM>. In some embodiments, the interaction may involve the user making suitable gestures directed to the holographic button <NUM> to initiate the action. In some embodiments, the mixed reality device may interact with the user <NUM> to obtain or authorize a payment instrument.

Thus, once the holographic button for an item has been downloaded and placed, the item can be subsequently re-ordered simply by accessing the holographic button. Placement of the holographic button on or near a relevant object provides contextual cues that allow the user to locate the holographic button. The user need not maintain a list of holographic buttons; instead holographic buttons simply show up in the user's mixed reality world in the proper context (e.g., a laundry detergent button on the washing machine, a cleaning service button on the broom closet door, etc.).

It can be appreciated from the foregoing illustrative description that the use of holographic buttons in accordance with the present disclosure to obtain goods or services can reduce bandwidth usage in the mixed reality device. Since the holographic button already provides the information needed by the user to initiate a request to order the item, the user can skip steps that require network resources such as accessing the provider's website and navigating through to the particular web page that contains the item.

In addition to reducing bandwidth usage, the use of holographic buttons in accordance with the present disclosure can reduce storage requirements for storing browser history. When a user is required to search the Internet for items of interest, especially consumable items that require re-ordering, the browser history can become cluttered from those search efforts. Associating such items with holographic buttons in accordance with the present invention obviates the need for repeated searches for those items and thus can reduce the amount of browser history that is stored in memory.

It can be appreciated that navigating the Internet in a mixed reality world can be taxing to the user, requiring gestures and textual input to access various websites. User interaction performance and reliability can be markedly enhanced by the use of the holographic button of the present disclosure, which can reduce the amount of user interaction to achieve the same goal of obtaining items from the Internet.

Referring to <FIG>, the discussion will now turn to a description of processing in the mixed reality device <NUM> for instantiating a holographic button <NUM> in accordance with the present disclosure. In some embodiments, for example, the mixed reality device <NUM> can include computer executable program code, which when executed by a computing system (e.g., <NUM>, <FIG>), can cause the computing system to perform processing in accordance with <FIG>. The flow of operations performed by the computing system is not necessarily limited to the order of operations shown. The holographic button <NUM> can be more generally referred to as a <NUM>-dimensional (3D) object.

At block <NUM>, the mixed reality device <NUM> can authenticate the user <NUM> when they enter their mixed reality world. In some embodiments, for example, authentication may include displaying a virtual keyboard in the user's field of view and receiving or otherwise detecting gestures made by the user <NUM> that correspond to key taps on the virtual keyboard to enter an authorization code. In other embodiments, the user <NUM> can speak with the mixed reality device <NUM> to become authorized. When the user <NUM> is authorized, the mixed reality device <NUM> can then access the user's user profiles or other similar information.

At block <NUM>, the mixed reality device <NUM> can receive gestures from the user to launch a web browser <NUM>. In response, the mixed reality device <NUM> can superimpose a browser window onto the user's field of view of their real-world surrounding. In some embodiments, the browser window can be displayed in mixed world view mode. In mixed world view, the web browser <NUM> appears on top of the user's field of view, along with any other application window the user <NUM> may have open.

At block <NUM>, the mixed reality device <NUM> can receive input from the user <NUM> to navigate the Internet on the web browser <NUM>. The input can include any combination of gestures, keyboard inputs, voice inputs, and the like made by the user <NUM>. In the context of the present disclosure, the user <NUM> can navigate to the website <NUM> of the provider <NUM> for an item <NUM> of interest to the user <NUM>; e.g., laundry detergent, referring to the illustrative examples mentioned above. The user <NUM> can further navigate the provider's website <NUM> to get to a web page that contains a link 14a (<FIG>) to the item <NUM>. In accordance with the present disclosure, the link 14a can be associated with a holographic button <NUM> of the item <NUM>.

At block <NUM>, the mixed reality device <NUM> can receive user gestures for generating and placing the holographic button <NUM>. For example, the user <NUM> can walk around in their real-world surrounding to determine a relevant real-world object to place the holographic button <NUM>. In the laundry detergent example above, the user <NUM> may walk over to their washing machine. In an office setting example, the item <NUM> might be copier paper in which case the user <NUM> may want to place the holographic button <NUM> on or near the copying machine, and so on. The user <NUM> can make suitable gestures that direct the placement of the holographic button <NUM> on the real-world object.

In some embodiments, for example, the user <NUM> can make gestures to access the link 14a. For instance, the user <NUM> can direct their gaze at the link 14a and use an air tap gesture to select the link 14a and cause the holographic button <NUM> to be generated. In response, the mixed reality device <NUM> can download item-related data 22a associated with the item <NUM> from the provider's website <NUM> used to generate the holographic button <NUM>; e.g., the item-related data 22a can include holographic data <NUM> that defines the holographic button <NUM>. In accordance with the present disclosure, the mixed reality device <NUM> can use the holographic data <NUM> to create a data object for the holographic button <NUM> that can be stored in a memory (e.g., <NUM>, <FIG>) of the mixed reality device <NUM>. The user <NUM> can gesture to place the generated holographic button <NUM> onto the relevant real-world object, for example, by moving their hand to drag the generated holographic button <NUM> toward and releasing the generated holographic button <NUM> to drop it on or near the relevant real-world object.

In other embodiments, the user <NUM> can use a drag-and-drop gesture. For example, the user <NUM> can drag-and-drop the link 14a onto the relevant real-world object. The user <NUM> can direct their gaze at the link 14a and use an air tap gesture to select and hold the link 14a, and move their hand to drag the link 14a to the relevant real-world object and release (drop) the link 14a. The mixed reality device <NUM> can use the holographic data <NUM> to generate the holographic button <NUM>.

At block <NUM>, the mixed reality device <NUM> can receive and store the item-related data 22a associated with the item <NUM> from the provider's website <NUM>, in response to the user's placement action of the holographic button <NUM>. In accordance with the present disclosure, the item-related data 22a can include ordering information <NUM> associated with the item <NUM>, which can be used to create an order request to obtain the item <NUM>. In some embodiments, the downloaded item-related data 22a can be stored in a memory (e.g., <NUM>, <FIG>) of the mixed reality device <NUM> or in remote storage (e.g., cloud-based storage).

At block <NUM>, the mixed reality device <NUM> can receive or otherwise obtain a fixed spatial location corresponding to the drop location of the placement action. For example, when the user <NUM> makes a gesture to drop the holographic button <NUM>, the user's direction of gaze at that moment can be used to determine the drop location. Sensors on the mixed reality device <NUM> can be used to determine the fixed spatial location in the real-world surrounding that corresponds to that drop location.

At block <NUM>, the mixed reality device <NUM> can world-lock the holographic button <NUM> to the fixed spatial location determined above. In some embodiments, for example, the mixed reality device <NUM> can generate spatial mapping data that represents the real-world surrounding, or can access previously generated spatial mapping data. As the user <NUM> manipulates the holographic button <NUM> relative to the real-world surrounding in their field of view, the mixed reality device <NUM> tracks the movement of the holographic button <NUM> using the spatial mapping data. For example, when the user <NUM> drops the holographic button <NUM> on the surface of a real-world object, the mixed reality device <NUM> can associate the holographic button <NUM> with spatial mapping data that corresponds to the fixed spatial location of the surface of that real-world object, thus world-locking the holographic button <NUM> to that real-world object.

The holographic button <NUM> is world-locked in the sense that the holographic button <NUM> maintains its location with respect to the fixed spatial location in the real-world surrounding. When the user <NUM> moves away from that fixed location (e.g., into another room or if they turn their head away), the holographic button <NUM> will remain fixed to that location and eventually disappear from the user's field of view as would any other real-world object.

The holographic button <NUM> is deemed to be instantiated at this point, having been downloaded to the mixed reality device <NUM> and world-locked to a real-world object (vis-à-vis its spatial location in the real-world surrounding). The holographic button <NUM> and its drop location can be stored on the mixed reality device <NUM>, so that even if the user <NUM> walks away from the drop location or exits their mixed reality world (e.g., log out of the mixed reality device <NUM>), the holographic button <NUM> and its drop location persists.

Referring to <FIG>, the discussion will now turn to a description of processing in the mixed reality device <NUM> for using an instantiated holographic button <NUM> in accordance with the present disclosure to order the item <NUM> associated with the holographic button <NUM>. In some embodiments, for example, the mixed reality device <NUM> can include computer executable program code, which when executed by a computing system (e.g., <NUM>, <FIG>), can cause the computing system to perform processing in accordance with <FIG>. The flow of operations performed by the computing system is not necessarily limited to the order of operations shown.

At block <NUM>, the mixed reality device <NUM> can detect user interaction at a fixed spatial location that is associated (tagged) with a previously instantiated holographic button <NUM>. For example, when the user <NUM> enters their mixed reality world, they may approach the real-world object (e.g., washing machine) that was previously tagged with a holographic button <NUM>. Using the spatial mapping data of the real-world environment to track the user <NUM>, the mixed reality device <NUM> can detect when the user <NUM> is interacting with that real-world object.

At block <NUM>, the mixed reality device <NUM> can display the holographic button <NUM> in the user's field of view in response to detecting that the user <NUM> is interacting with a real-world object that was previously tagged with a holographic button <NUM>. For example, the mixed reality device <NUM> can display the holographic button <NUM> on the display <NUM>, thus superimposing the holographic button <NUM> onto the user's perception of their real-world surrounding.

At block <NUM>, the mixed reality device <NUM> can detect gestures made by the user <NUM> and directed to the holographic button <NUM> to initiate an action to obtain the item <NUM> that is associated with the holographic button <NUM>. For example, the user <NUM> can direct their gaze on the holographic button <NUM> and perform an air tap gesture. The user <NUM> will have selected the item <NUM> and initiated activity to obtain the item <NUM> simply by looking at a real-world object (e.g., washing machine) and making an appropriate gesture (e.g., air tap), without having to invoke a web browser, navigate to the provider's website <NUM>, navigate to the web page to get to the item <NUM>, and perform interactions on the web page to order the item <NUM>.

At block <NUM>, the mixed reality device <NUM> can access the ordering information <NUM> associated with the item <NUM>. In some embodiments, the ordering information can be previously stored in the mixed reality device <NUM>, for example, when the user <NUM> downloaded the holographic button <NUM> from the provider's website <NUM>. In other embodiments, the mixed reality device <NUM> can access the ordering information <NUM> from the provider's website <NUM> on an as needed basis (e.g., using the provider's site API <NUM>).

At block <NUM>, the mixed reality device <NUM> can access the user's payment information, such as name, billing address, mailing address, credit card no. , CVV, etc. The payment instrument can be a credit/debit card, a token associated with the credit/debit card, a PayPal® account, a bank account, a physical check (e.g., the mixed reality device <NUM> can read such a check and use it), and so on. In some embodiments, the user's payment information can be linked to their user profile or otherwise stored locally; e.g., in a discreet secure micro-controller or a secure memory partition in memory. In other embodiments, the user's payment information can be stored on a cloud server.

In some embodiments, the user's payment may not be stored anywhere. In those cases, the mixed reality device <NUM> can capture the payment information from the user <NUM> at the time the user <NUM> initiates the action (via the holographic button <NUM>) to obtain the item <NUM>. For example, the mixed reality device <NUM> may instruct the user <NUM> to input their payment information, hold up their credit card so an image of it can be taken, present a check, and the like.

At block <NUM>, the mixed reality device <NUM> can generate an order request to order the item <NUM>. In some embodiments, for example, the order request can include ordering information <NUM> for the item <NUM> and a payment component. The payment component can be an API call to the payment API 124a. For example, the API call can be a concatenation of the data related to the item as well as user authentication data that would be used to conduct a payment transaction.

At block <NUM>, the mixed reality device <NUM> can send the order request to the provider <NUM> of the item <NUM> to fulfill the order. For example, the provider <NUM> can invoke the API call to conduct a payment transaction for the item <NUM> and then fulfill the order. In some cases, the API call can create a payment experience in order for the user <NUM> to confirm their payment information prior to fulfillment of the order. This would be useful, for example, if there are multiple payment instruments associated with the user's account or multiple billing/shipping addresses. The user authentication data could result from the user <NUM> having entered their credentials, or having been authenticated by a PIN or any biometric gesture such as voice recognition or iris scan.

Ordering goods or services (items) using a holographic button in accordance with aspects of the present disclosure can reduce processing resources required by the mixed reality device <NUM>. Bandwidth requirements can be reduced because the user need only navigate to the provider's website once to download a holographic button associated with the item. Once the holographic button has been previously placed in the user's mixed reality world, the user need only look for the holographic button and make a gesture (e.g., air tap) to invoke a series of actions to order the item, without having to re-visit the providers' website.

The use of holographic buttons to order items that require frequent re-ordering can improve user efficiency because the use of holographic buttons obviate the need to navigate the Internet, which can be time consuming and error prone. The increased user efficiency means less transmission time over the wireless communication channel between the mixed reality device and an Internet access point, which can reduce consumption of the device's power source (e.g., battery).

The user can instantiate as many holographic buttons for ordering items as they want and populate them around in their mixed world reality. The user's view of their real-world surrounding through the mixed reality device can remain uncluttered because a holographic button only appears in the user's field of view when the user is interacting with a real-world object that is associated with a holographic button.

Embodiments in accordance with the present disclosure provide a path for the natural evolution of mixed reality e-commerce. Just as markets appeared for traditional trade (e.g., large department stores arose for mass commercialization of goods), online merchants (e.g., Amazon) appeared for e-commerce. Solutions represented by embodiments in accordance with the present disclosure are beginning to appear for mixed reality e-commerce. Traditional "flat" e-commerce is not ideal for interacting in a shoppable mixed world. Embodiments in accordance with the present disclosure can provide accessibility, breadth of goods and services, and convenience of e-commerce with the ease of interaction of physical shopping to enable and facilitate the adoption of mixed reality e-commerce.

The above description illustrates various embodiments of the present disclosure along with examples of how aspects of these embodiments may be implemented. The above examples and embodiments should not be deemed to be the only embodiments, and are presented to illustrate the flexibility and advantages of the present disclosure as defined by the following claims. For example, although certain embodiments have been described with respect to particular process flows and steps, it should be apparent to those skilled in the art that the scope of the present disclosure is not strictly limited to the described flows and steps. Steps described as sequential may be executed in parallel, order of steps may be varied, and steps may be modified, combined, added, or omitted. As another example, although certain embodiments have been described using a particular combination of hardware and software, it should be recognized that other combinations of hardware and software are possible, and that specific operations described as being implemented in software can also be implemented in hardware and vice versa.

Claim 1:
A mixed reality device comprising:
a display;
one or more computer processors; and
a computer-readable storage medium comprising instructions for controlling the one or more computer processors to:
instantiate a holographic button associated with an item from a provider, the holographic button being a 3D virtual object, including:
navigating (<NUM>) by a user a website of the provider to get to a web page that contains a link to the item;
drag-and-dropping (<NUM>) the link onto a real-world object;
receiving (<NUM>) item-related data associated with the item from the website of the provider, the item-related data comprising holographic data to display the holographic button on a display of the mixed reality device; and
world-locking (<NUM>) the holographic button to a fixed spatial location in a real-world surrounding relative to the mixed reality device using spatial mapping data representative of the real-world surrounding, the fixed spatial location corresponding to the drop location;
provide the e user of the mixed reality device with access to the holographic button to obtain the item using the holographic button, including detecting (<NUM>) interactions by the user with the real-world object at the fixed spatial location in the real-world surrounding associated with the holographic button and in response to the detecting the interaction, displaying the holographic button on the display to superimpose (<NUM>) the holographic button onto a view of the user's real-world surrounding, wherein the interactions with the real-world object comprise the user physically touching the real-world object;
detect (<NUM>) a gesture directed to the holographic button made by the user of the mixed reality device; and
in response to detecting a gesture directed to the holographic button:
accessing (<NUM>) ordering information that describes the item to be obtained;
accessing (<NUM>) a payment instrument associated with the user; and
sending (<NUM>) to the provider an order request for the item comprising the ordering information and the payment instrument.