Patent Description:
<CIT> relates to an augmented reality system that provides augmented product and environment information to a wearer of a see through head mounted display. The augmentation information may include advertising, inventory, pricing and other information about products a wearer may be interested in. Interest is determined from wearer actions and a wearer profile.

<CIT> discloses a method and system for generating a customized augmented reality environment, comprising generating a signature for a multimedia content, determining, based on the generated signature, a selection of an augmented reality character, wherein each augmented reality character is associated with a character signature, identifying at least a potential placement location in the multimedia content element, wherein each potential placement location is associated with a portion of the generated signature, correlating between the potential placement location signature portion and the character signature to select, for each augmented reality character, a placement location in the multimedia content element, and generating a customized augmented reality environment, wherein the customized augmented reality environment includes each augmented reality character superimposed on the multimedia content element at the respective placement location.

It is the object of the present invention to reduce interruptions during the incorporation of virtual content items into mixed reality experiences.

The object is solved by the subject matter of the independent claims which define the present invention.

According to one aspect of the present disclosure, a head-mounted display device is provided, including an at least partially see-through display, one or more input devices including one or more image sensors configured to collect imaging data of a physical environment, one or more communication devices, and a processor. The processor may be configured to output for display on the display a mixed reality experience including one or more virtual objects superimposed upon the physical environment. The processor may be further configured to transmit the imaging data to a server computing device via the one or more communication devices. The processor may be further configured to receive, from the server computing device, an identification of a virtual content item display opportunity. The identification of the virtual content item display opportunity may include a determination that the mixed reality experience and one or more physical features identified from the imaging data satisfy one or more spatial and/or temporal constraints. The processor may be further configured to transmit a request for one or more virtual content items to the server computing device based on the identification of the virtual content item display opportunity. The processor may be further configured to receive the one or more virtual content items from the server computing device. The processor may be further configured to output the one or more virtual content items for display on the display at one or more respective dynamically positioned content locations in the physical environment.

The inventor of the subject application has recognized that existing systems and methods for presenting content such as digital advertisements on two dimensional screens are not suitable for three dimensional mixed reality environments. According to existing systems and methods, digital advertisements are typically presented as banner advertisements on webpages, or alternatively as videos. Banner advertisements are presented in a fixed location on a two dimensional screen that is being viewed by a user, however in mixed reality environments the user is free to look in any direction when interacting with the environment so there is no guarantee that a user will be looking in the direction of content such as an advertisement when it is displayed. In addition, banner advertisements are typically visible to users for brief periods of time. However, users of a mixed reality systems frequently interact with the mixed reality environments for longer periods of time in comparison to non-mixed-reality webpages and videos. Thus, a rate at which the users view new advertisements may be lower than would be desirable when methods of displaying banner advertisements on webpages and in videos are applied to mixed reality environments. In addition, banner advertisements may obstruct users' views of mixed reality environments and make it more difficult for the users to interact with physical and/or virtual objects. This obstruction of the users' views of objects may also occur when existing systems and methods for presenting video advertisements are used.

In order to address the problems discussed above, a head-mounted display device <NUM> is provided, as shown in <FIG>. The head-mounted display device <NUM> of <FIG> may include memory <NUM> and/or a processor <NUM>. The memory <NUM> may include volatile and/or non-volatile memory and may be operatively coupled to the processor <NUM>. The head-mounted display device <NUM> may further include one or more input devices, which may be included in the head-mounted display device <NUM> or communicatively coupled to the head-mounted display device <NUM>. The one or more input devices may include one or more image sensors <NUM> configured to collect imaging data <NUM> of a physical environment <NUM>. In some embodiments, the one or more input devices may further include an inward-facing image sensor <NUM> configured to collect inward-facing imaging data <NUM> of a user. The one or more input devices may further include a position sensor <NUM> configured to determine a position of the head-mounted display device <NUM> in the physical environment. In some embodiments, the one or more input devices may further include a microphone <NUM>. One or more other input devices such as a touchscreen, a trackpad, a mouse, a joystick, a keyboard, and/or a button may be included in or communicatively coupled to the head-mounted display device <NUM> in some embodiments. The head-mounted display device <NUM> may further include one or more output devices, which may include an at least partially see-through display <NUM>. The one or more output devices may further include one or more of a speaker <NUM>, a haptic feedback device <NUM>, and/or one or more other output devices. The head-mounted display device <NUM> may further include one or more communication devices <NUM> via which the processor <NUM> of the head-mounted display device <NUM> may send data to and/or receive data from one or more other computing devices. This communication may occur over a network <NUM>. The functions of the memory <NUM> and processor <NUM> of the head-mounted display device <NUM> may be performed at least in part by an offboard computing system with which the head-mounted display device <NUM> communicates via the one or more communication devices <NUM>. In the example of <FIG>, the offboard computing system is a server computing device <NUM>.

The input and output devices that may be included in the head-mounted display device are discussed in further detail below with reference to <FIG> shows an example embodiment of the head-mounted display device <NUM> in which the head-mounted display device <NUM> takes the form of wearable glasses or goggles, but it will be appreciated that other forms are possible. The head-mounted display device <NUM> may include an output device suite including a display <NUM>. In some embodiments, the head-mounted display device <NUM> may be configured in an augmented reality configuration to present an augmented reality environment, and thus the display <NUM> may be an at least partially see-through stereoscopic display configured to visually augment an appearance of a physical environment <NUM> being viewed by the user through the display <NUM>. In some examples, the display <NUM> may include one or more regions that are transparent (e.g. optically clear) and may include one or more regions that are opaque or semi-transparent. In other examples, the display <NUM> may be transparent (e.g. optically clear) across an entire usable display surface of the display <NUM>.

The output device suite of the head-mounted display device <NUM> may, for example, include an image production system that is configured to display one or more virtual objects <NUM> to the user with the display <NUM>. The processor <NUM> may be configured to output for display on the display <NUM> a mixed reality experience <NUM> including one or more virtual objects <NUM> superimposed upon the physical environment <NUM>. In the augmented reality configuration with an at least partially see-through display, the virtual objects <NUM> are visually superimposed onto the physical environment <NUM> that is visible through the display <NUM> so as to be perceived at various depths and locations. In one embodiment, the head-mounted display device <NUM> may use stereoscopy to visually place a virtual object <NUM> at a desired depth by displaying separate images of the virtual object <NUM> to both of the user's eyes. Using this stereoscopy technique, the head-mounted display device <NUM> may control the displayed images of the virtual objects <NUM>, such that the user will perceive that the virtual objects <NUM> exist at a desired depth and location in the viewed physical environment <NUM>.

The output device suite of the head-mounted display device <NUM> may further include one or more speakers <NUM> configured to emit sound. In some embodiments, the head-mounted display device <NUM> may include at least a left speaker 36A and a right speaker 36B situated such that the left speaker 36A may be located proximate the user's left ear and the right speaker 36B may be located proximate the user's right ear when the head-mounted display device <NUM> is worn. Thus, the one or more speakers <NUM> may emit stereo sound output. The output device suite may further include one or more haptic feedback devices <NUM> configured to provide tactile output (e.g. vibration).

The head-mounted display device <NUM> may include an input device suite including one or more input devices. The input device suite of the head-mounted display device <NUM> may include one or more imaging sensors. In one example, the input device suite includes an outward-facing optical sensor <NUM> that may be configured to detect the real-world background from a similar vantage point (e.g., line of sight) as observed by the user through the display <NUM> in an augmented reality configuration. The input device suite may additionally include an inward-facing optical sensor <NUM> that may be configured to detect a gaze direction of the user's eyes. It will be appreciated that the outward facing optical sensor <NUM> and/or the inward-facing optical sensor <NUM> may include one or more component sensors, including an RGB camera and a depth camera. The RGB camera may be a high definition camera or have another resolution. The depth camera may be configured to project non-visible light and capture reflections of the projected light, and based thereon, generate an image comprised of measured depth data for each pixel in the image. This depth data may be combined with color information from the image captured by the RGB camera, into a single image representation including both color data and depth data, if desired.

The input device suite of the head-mounted display device <NUM> may further include a position sensor system that may include one or more position sensors <NUM> such as accelerometer(s), gyroscope(s), magnetometer(s), global positioning system(s), multilateration tracker(s), and/or other sensors that output position data <NUM> as a position, orientation, and/or movement of the relevant sensor. The input device suite may further include one or more microphones <NUM> configured to collect sound data <NUM>.

Optical sensor information received from the one or more imaging sensors and/or position data <NUM> received from position sensors <NUM> may be used to assess a position and orientation of the vantage point of head-mounted display device <NUM> relative to other environmental objects. In some embodiments, the position and orientation of the vantage point may be characterized with six degrees of freedom (e.g., world-space X, Y, Z, pitch, roll, yaw). The vantage point may be characterized globally or independent of the real-world background. The position and/or orientation may be determined by the processor <NUM> of the head-mounted display device <NUM> and/or by an off-board computing system.

Furthermore, the optical sensor information and the position sensor information may be used by the head-mounted display system to perform analysis of the real-world background, such as depth analysis, surface reconstruction, environmental color and lighting analysis, or other suitable operations. In particular, the optical and positional sensor information may be used to create a virtual model of the real-world background. In some embodiments, the position and orientation of the vantage point may be characterized relative to this virtual space. Moreover, the virtual model may be used to determine positions of virtual objects <NUM> in the virtual space and add additional virtual objects <NUM> to be displayed to the user at a desired depth and location. The virtual model is a three-dimensional model and may be referred to as "world space," and may be contrasted with the projection of world space viewable on the display <NUM>, which is referred to as "screen space. " Additionally, the optical sensor information received from the one or more image sensors <NUM> may be used to identify and track objects in the field of view of the one or more image sensors <NUM>. The optical sensors may also be used to identify machine recognizable visual features in the physical environment <NUM> and use the relative movement of those features in successive frames to compute a frame to frame relative pose change for the head mounted display device <NUM> within the world space of the virtual model.

The head-mounted display device <NUM> may further include a communication system including one or more communication devices <NUM>, which may include one or more receivers 16A and/or one or more transmitters 16B. In embodiments in which the head-mounted display device <NUM> communicates with an off-board computing system, the one or more receivers 16A may be configured to receive data from the offboard computing system, and the one or more transmitters 16B may be configured to send data to the off-board computing system. In some embodiments, the head-mounted display device <NUM> may communicate with the off-board computing system via a network, which may be a wireless local- or wide-area network. Additionally or alternatively, the head-mounted display device <NUM> may communicate with the off-board computing system via a wired connection. The head-mounted display device <NUM> may be further configured to communicate with a server computing system via the communication system.

Returning to <FIG>, the processor <NUM> may be further configured to transmit the imaging data <NUM> to a server computing device <NUM> via the one or more communication devices such that the server computing device <NUM> may identify one or more physical features <NUM> in the physical environment <NUM> based on the imaging data <NUM>. In other embodiments, identification of the one or more physical features <NUM> may be performed at least in part at the processor <NUM> of the head-mounted display device <NUM>. In some embodiments, the one or more physical feature identifications <NUM> may be made using a machine learning algorithm. The physical environment <NUM> may be developed based on the one or more physical feature identifications <NUM> of the one or more physical features <NUM>. The virtual model may include one or more geometric features of each of the one or more physical features <NUM>.

In embodiments in which the one or more input devices of the head-mounted display device <NUM> further include an inward-facing image sensor <NUM> configured to collect inward-facing imaging data <NUM> of the user, the processor <NUM> may be further configured to transmit the inward-facing imaging data to the server computing device <NUM>. Additionally or alternatively, in embodiments in which the one or more input devices of the head-mounted display device <NUM> include one or more position sensors <NUM> and/or one or more microphones <NUM>, the processor <NUM> may be further configured to transmit position data <NUM> received from the one or more position sensor <NUM> and/or sound data <NUM> received from the one or more microphones <NUM> respectively to the server computing device <NUM>. In such embodiments, the processor <NUM> may be further configured to receive, from the server computing device <NUM>, a determination of a gaze direction <NUM> of the user based on the inward-facing imaging data <NUM>. In other embodiments, the gaze direction <NUM> may be determined at the processor <NUM>. The gaze direction <NUM> may be correlated with the position and orientation of the head-mounted display device <NUM> itself as determined using position data <NUM> and imaging data <NUM>. In embodiments in which position data <NUM> is transmitted to the server computing device, the processor <NUM> may be further configured to receive an indication of a change in position <NUM> from the server computing device <NUM>.

As discussed above, the mixed reality experience <NUM> may include one or more virtual objects <NUM> superimposed upon the physical environment <NUM>. The one or more virtual objects <NUM> may be received from the server computing device <NUM>. The processor <NUM> and/or the server computing device <NUM> may be configured to modify the one or more virtual objects <NUM> included in the mixed reality experience <NUM> over time according to rules specified by the mixed reality experience <NUM>. For example, the mixed reality experience <NUM> may include a physics engine configured to model responses of the one or more virtual objects <NUM> to features of the physical environment <NUM> and/or events occurring in the physical environment <NUM>. The physics engine may additionally or alternatively model interactions of the one or more virtual objects <NUM> with each other. The processor <NUM> may be further configured to modify the one or more virtual objects <NUM> based on user input received from the one or more input devices. Thus, the mixed reality experience <NUM> may be an interactive experience.

Based on the mixed reality experience <NUM> and/or the identification <NUM> of the one or more physical features <NUM> in the physical environment <NUM>, the server computing device <NUM> may be further configured to identify a virtual content item display opportunity <NUM>. The virtual content item display opportunity <NUM> may be a set of conditions in the physical environment <NUM> and/or the mixed reality experience <NUM> that are conducive to displaying a virtual content item <NUM> on the display <NUM> of the head-mounted display device <NUM>. In some embodiments, the server computing device <NUM> may be configured to identify the virtual content item display opportunity <NUM> at least in part by determining that the one or more identified physical features <NUM> have a predetermined set of geometric characteristics. For example, in an embodiment in which the virtual content item <NUM> is a virtual poster, the virtual content item display opportunity <NUM> may occur when a substantially flat and vertical surface of sufficient size to accommodate the virtual poster is determined to be present in the physical environment <NUM>. As another example, in an embodiment in which the virtual content item <NUM> is a virtual vehicle, the virtual content item display opportunity <NUM> may occur when a road is determined to be present in the physical environment <NUM>.

The server computing device <NUM> may be configured to identify the virtual content item display opportunity <NUM> at least in part by determining that the one or more identified physical features <NUM> satisfy one or more spatial and/or temporal constraints <NUM>. The one or more spatial and/or temporal constraints <NUM> may include one or more position constraints 42A on respective positions at which the one or more virtual content items <NUM> may be displayed. The one or more position constraints 42A may specify, for each virtual content item <NUM>, a subset of the physical environment <NUM> within which the virtual content item <NUM> is eligible to be displayed. For example, the one or more position constraints 42A may specify that a virtual content item <NUM> may be displayed at one or more locations that do not overlap a preexisting virtual object <NUM> or physical feature <NUM>. Additionally or alternatively, the one or more spatial and/or temporal constraints <NUM> may include one or more timing constraints 42B on respective times at which the one or more virtual content items <NUM> may be displayed. For example, the server computing device <NUM> may determine based on the mixed reality experience <NUM> that the user is engaged in an activity (e.g. a video call) that would be undesirable to interrupt. As another example, the timing constraint 42B may specify a maximum duration for which the virtual content item <NUM> may be displayed. The one or more constraints may further include one or more size constraints 42C on respective sizes of the one or more virtual content items <NUM> and/or one or more shape constraints 42D on respective shapes of the one or more virtual content items <NUM>. The one or more size constraints 42C and the one or more shape constraints 42D may, for example, specify that a virtual content item <NUM> may be displayed at one or more surfaces or volumes that can accommodate the virtual content item <NUM> without the virtual content item <NUM> overlapping one or more virtual objects <NUM> and/or physical features <NUM>. Other constraints <NUM> may additionally or alternatively be applied when identifying virtual content display opportunities <NUM>.

When the server computing device <NUM> determines that a virtual content item display opportunity <NUM> occurs, the server computing device <NUM> may identify one or more candidate locations at which the virtual content item <NUM> may be displayed in the physical environment <NUM>. Example candidate locations <NUM> in the physical environment <NUM> are shown in <FIG> according to one example embodiment. The candidate locations <NUM> may be locations at which the virtual content item <NUM> is anchored. In the example of <FIG>, the candidate locations <NUM> are all located on substantially flat horizontal surfaces. However, in other embodiments, one or more candidate locations on vertical surfaces or surfaces of other shapes may be identified. One or more candidate locations may additionally or alternatively be floating candidate locations positioned apart from surfaces in the physical environment <NUM>. <FIG> also shows an example candidate area <NUM>. The virtual content item <NUM> may be anchored at any point within the candidate area <NUM>. In other embodiments, one or more candidate lines and/or candidate volumes may be identified along which or within which a virtual object may be displayed.

In response to receiving the identification of the virtual content item display opportunity <NUM>, the processor <NUM> may be further configured to transmit a request for one or more virtual content items <NUM> to the server computing device <NUM> based on the identification of the virtual content item display opportunity <NUM>. After sending the request, the processor <NUM> may receive the one or more virtual content items <NUM> from the server computing device <NUM>. The request sent from the head-mounted display device <NUM> to the server computing device <NUM> and the response sent from the server computing device <NUM> to the head-mounted display device are discussed in further detail below with reference to <FIG>.

After receiving the one or more virtual content items <NUM>, the processor <NUM> may be further configured to output the one or more virtual content items <NUM> for display on the display <NUM> based on the identification of the virtual content item display opportunity <NUM>, as shown in <FIG>. Each virtual content item <NUM> may be displayed at a respective dynamically positioned content location <NUM> in the physical environment <NUM> selected from among the one or more candidate locations <NUM> by the server computing device <NUM>. The server computing device <NUM> may apply one or more rules and/or heuristics to determine a candidate location <NUM> of the one or more candidate locations <NUM> at which to display the virtual content item <NUM>. In one example, the server computing device <NUM> may assign a score to each candidate location <NUM> and select the candidate location <NUM> that has the highest score. The score of a candidate location <NUM> may be based on one or more properties of the candidate location <NUM> such as distance from the user <NUM>, angular displacement from the gaze direction <NUM>, proximity to one or more virtual objects <NUM>, size of a candidate area <NUM>, user-specified preferences, one or more physical feature identifications <NUM>, and/or one or more other scoring criteria. In some embodiments, the score may be a weighted score expressed as a sum of a plurality of scoring criteria with respective weights. In the example of <FIG>, the virtual content item <NUM>, which depicts a soft drink bottle, is displayed at the candidate location <NUM> closest to the user <NUM>.

In some embodiments, the virtual content item display opportunity <NUM> may occur when an event or activity other than a physical feature <NUM> detected with the image sensor <NUM> is present in the physical environment <NUM> or the mixed reality experience <NUM>. For example as shown in <FIG>, the server computing device <NUM> may determine based on the imaging data <NUM> and/or position data <NUM> received from the position sensor <NUM> that a user <NUM> is running. The virtual content item display opportunity <NUM> may include a physical feature identification <NUM> in addition to an event or activity identification in some embodiments. For example, the server computing device <NUM> may further determine that the physical feature identification <NUM> includes an identification of a road. In response to this determination, the server computing device <NUM> may determine that a virtual content item display opportunity <NUM> exists for a virtual content item <NUM> depicting a motorcycle driving alongside the user <NUM>. In response to receiving the virtual content item display opportunity <NUM> including this determination, the processor <NUM> may be further configured to output the virtual content item <NUM> for display on the display <NUM>. The virtual content item <NUM> may be output for display at a dynamically positioned content location <NUM> in the physical environment <NUM>, which may be located within a candidate area <NUM>. The dynamically positioned content location <NUM> may move as the user <NUM> moves through the physical environment <NUM> so that the virtual content item <NUM> remains near the user. For example, the dynamically positioned content location <NUM> may remain within a predetermined threshold distance of the user <NUM>. The dynamically positioned content location <NUM> may be anchored to a physical feature <NUM> in the physical environment <NUM> such that the virtual content item <NUM> remains located on, or at a predetermined offset from, the physical feature <NUM>. In the embodiment of <FIG>, the physical feature <NUM> to which the virtual content item <NUM> is anchored is the road.

In some embodiments, the processor <NUM> may not output the virtual content item <NUM> to the display <NUM> for display immediately upon receiving the virtual content item <NUM> from the server computing device <NUM>. Instead, the processor <NUM> may wait for a trigger condition <NUM> to occur before the virtual content item <NUM> is displayed. For example, the processor <NUM> may display the virtual content item <NUM> after a predetermined period of time has elapsed. In other embodiments, the trigger condition <NUM> may be an event that occurs in the physical environment <NUM> or the mixed reality experience <NUM>, such as a video ending.

Returning to <FIG>, the processor <NUM> may be further configured to receive, via the one or more input devices, a user interaction <NUM> with a virtual content item <NUM> of the one or more virtual content items <NUM>. The server computing device <NUM> may determine that the data received from the one or more input devices of the head-mounted display device <NUM> includes a user interaction <NUM>. In some embodiments, the user interaction <NUM> may include a gesture selecting the virtual content item <NUM>. Additionally or alternatively, in embodiments in which the processor is configured to determine a gaze direction <NUM> of the user <NUM>, the user interaction <NUM> may include gazing at the virtual content item <NUM>. The user interaction <NUM> may additionally or alternatively include a change in the position of the head-mounted display device <NUM> as detected by the position sensor <NUM>, a voice input as detected by the microphone <NUM>, and/or some other type of input. The user interaction <NUM> may include inputs from a plurality of input devices. Any of the input devices discussed above with reference to <FIG> and <FIG> may be used, alone or in combination, to provide the user interaction <NUM>.

In some embodiments, the processor <NUM> may be further configured to modify at least one virtual object <NUM> in response to the user interaction <NUM>. The at least one virtual object <NUM> modified in response to the user interaction <NUM> may be the virtual content item <NUM>. Additionally or alternatively, the modified virtual object <NUM> may be another virtual object <NUM> included in the mixed reality experience <NUM>. For example, the user <NUM> may select an option to display additional information (e.g. by navigating to a webpage, rotating the virtual content item <NUM>, or expanding a collapsed graphical element). As another example, the user <NUM> may select an option to dismiss the virtual content item <NUM>.

In some embodiments, the processor <NUM> may be further configured to transmit an indication of the user interaction <NUM> to the server computing device <NUM> via the one or more communication devices <NUM>. This indication may be transmitted over the network <NUM>. Alternatively, the processor <NUM> may be configured to transmit sensor data to the server computing device <NUM>, which may determine that the sensor data indicates a user interaction <NUM>. In response to sending the sensor data or the indication of the user interaction <NUM>, the head-mounted display device <NUM> may receive additional data from the server computing device <NUM>, such as one or more assets included in a webpage. The additional data may encode the updated virtual object. Additionally or alternatively, the indication of the user interaction <NUM> may be transmitted to the server computing device <NUM> without the head-mounted display device <NUM> receiving additional data in response. For example, in some embodiments, the one or more virtual content items <NUM> may include one or more advertisements. In such embodiments, the server computing device <NUM> may collect data on rates and types of user interaction <NUM> with the one or more advertisements.

The one or more virtual content items <NUM> may be configured to interact with the one or more virtual objects <NUM> of the mixed reality experience <NUM>. In such interactions, the one or more virtual content items <NUM> may be modified based on one or more properties of the one or more virtual objects <NUM>. Additionally or alternatively, the one or more virtual objects <NUM> may be modified based on one or more properties of the one or more virtual content items <NUM>. In embodiments in which movement of the one or more virtual objects is governed by a physics engine included in the mixed reality experience <NUM>, the physics engine may also control movement of the one or more virtual content items <NUM>. For example, a virtual object <NUM> and a virtual content item <NUM> may collide and bounce off each other. As another example, the user <NUM> may select a portion of the virtual content item <NUM> by manipulating a virtual object <NUM> that acts as a virtual cursor. Thus, the virtual content item <NUM> may be incorporated into the mixed reality experience in a way that leads to less of an interruption in the mixed reality experience <NUM> than existing systems and methods for introducing virtual content items <NUM> into mixed reality experiences <NUM>. In addition, when a plurality of virtual content items <NUM> are displayed, each of the virtual content items <NUM> may interact with any or all of the other virtual content items <NUM> and/or the virtual objects <NUM>. Thus, a plurality of virtual content items <NUM> may be concurrently integrated into the mixed reality experience <NUM>.

<FIG> show the physical environment <NUM>, virtual object <NUM>, and virtual content item <NUM> of <FIG> in an example of an interaction between the virtual object <NUM> and the virtual content item <NUM>. In the example of <FIG>, the processor <NUM> of the head-mounted display device <NUM> receives imaging data <NUM> from the one or more imaging sensors <NUM> and transmits the imaging data <NUM> to the server computing device <NUM>. The processor <NUM> then receives, from the server computing device <NUM>, an indication of a user interaction <NUM> including a gesture <NUM>. The gesture <NUM> is a gesture to select the virtual content item <NUM> and move the virtual content item <NUM> toward the virtual object <NUM>, which represents a bag. In response to this gesture <NUM>, the processor <NUM> may modify the dynamically located position <NUM> of the virtual content item <NUM> and cause the bottle (virtual content item <NUM>) to appear to be placed inside the bag (virtual obj ect <NUM>). Thus, the virtual obj ect <NUM> may interact with the virtual content item <NUM> such that the virtual content item <NUM> is hidden. In the example of <FIG>, the user interaction <NUM> placing the bottle (virtual content item <NUM>) into the bag (virtual object <NUM>) may be an "add to cart" action selecting the soft drink for purchasing.

The server computing device <NUM> is shown in further detail with reference to <FIG>. The server computing device <NUM> may include one or more communication devices <NUM> communicatively coupled to the head-mounted display device <NUM>. The one or more communication devices <NUM> may be communicatively coupled to the head-mounted display device <NUM> over the network <NUM>. The server computing device <NUM> may, in some embodiments, include one or more input devices and/or one or more output devices. The server computing device <NUM> may further include memory <NUM>, which may include volatile and/or non-volatile memory. The server computing device <NUM> may further include a processor <NUM>.

The processor <NUM> of the server computing device <NUM> may be configured to receive, via the one or more communication devices <NUM>, imaging data <NUM> from the head-mounted display device <NUM>. The processor <NUM> of the server computing device <NUM> may further receive one or more of inward-facing imaging data <NUM>, position data <NUM>, and/or sound data <NUM> from the head-mounted display device <NUM>. Based on the imaging data <NUM>, the processor <NUM> of the server computing device <NUM> may be configured to identify the one or more physical features <NUM> in the physical environment of the head-mounted display device <NUM>. The one or more physical features <NUM> may also be identified based on one or more of the inward-facing imaging data <NUM>, the position data <NUM>, the sound data <NUM>, and/or data collected at other input devices of the head-mounted display device <NUM>. The processor <NUM> may be further configured to identify a virtual content item display opportunity <NUM>. The identification of the virtual content item display opportunity <NUM> may include a determination that one or more physical features <NUM> identified at the head-mounted display device <NUM> satisfy one or more spatial and/or temporal constraints <NUM>. For example, the one or more spatial and/or temporal constraints <NUM> may include one or more position constraints 42A, timing constraints 42B, size constraints 42C, and/or shape constraints 42D on a virtual content item <NUM>.

The processor <NUM> of the server computing device <NUM> may be further configured to select one or more virtual content items <NUM> from a plurality of virtual content items <NUM>. The plurality of virtual content items <NUM> may be stored in the memory <NUM>. The one or more virtual content items <NUM> may be selected based on the one or more constraints <NUM> included in the indication of the virtual content item display opportunity <NUM>. For example, the plurality of virtual content items <NUM> may include a first subset of one or more virtual content items <NUM> that satisfy all the constraints <NUM> for at least one virtual content item opportunity <NUM> and a second subset of one or more virtual content items <NUM> that, for each virtual content item display opportunity identified by the head-mounted display device, do not satisfy at least one constraint. The processor <NUM> of the server computing device <NUM> may then select the virtual content item <NUM> from the first subset.

The processor <NUM> of the server computing device <NUM> may be further configured to select one or more respective dynamically positioned content locations <NUM> at which the one or more virtual content items <NUM> are configured to be displayed in a physical environment <NUM> of the head-mounted display device <NUM>. The dynamically positioned content location <NUM> of a virtual content item <NUM> may be selected from among a plurality of candidate locations for the virtual content item <NUM>. In some embodiments, the dynamically positioned content location <NUM> may move over time. The processor <NUM> may further select one or more respective trigger conditions <NUM> for the one or more virtual content items <NUM>.

The processor <NUM> may be further configured to convey the one or more virtual content items <NUM> to the head-mounted display device <NUM>. The processor <NUM> may also convey the one or more respective dynamically positioned content locations <NUM> and/or trigger conditions <NUM> to the head-mounted display device. The one or more virtual content items <NUM>, the one or more dynamically positioned content locations <NUM>, and/or the one or more trigger conditions <NUM> may be conveyed to the head-mounted display device over the network <NUM>.

In some embodiments, the processor <NUM> of the server computing device <NUM> may be further configured to receive, via the one or more communication devices <NUM>, a user interaction <NUM> with a virtual content item <NUM> of the one or more virtual content items <NUM> from the head-mounted display device <NUM>. For example, the user interaction <NUM> may indicate that the user <NUM> has gazed at the virtual content item <NUM> for a predetermined period of time. As another example, the user interaction <NUM> may indicate that the user <NUM> has selected an option to receive additional information from the server computing device <NUM>. In some embodiments, the server computing device <NUM> may store the indication of the user interaction <NUM> in interaction history <NUM>. In embodiments in which the virtual content item <NUM> is an advertisement, data stored in the interaction history <NUM> may be analyzed to determine the effectiveness of the advertisement. For example, the data stored in the interaction history <NUM> may be used as training data for a machine learning algorithm.

<FIG> shows a flowchart of a method <NUM> that may be performed at a head-mounted display device. The method <NUM> may be performed at the head-mounted display device <NUM> of <FIG> or at some other head-mounted display device. At step <NUM>, the method <NUM> may include collecting imaging data of a physical environment. The imaging data may be collected by one or more outward-facing time-of-flight and/or RGB cameras included in the head-mounted display device. Additionally or alternatively, the imaging data may be collected by another sensor device external to the head-mounted display device and communicated to the head-mounted display device. At step <NUM>, the method <NUM> may further include outputting a mixed reality experience for display on an at least partially see-through display. The mixed reality experience may include one or more virtual objects superimposed upon the physical environment. Each virtual object superimposed upon the physical environment may be displayed such that it has an apparent location in the physical environment. At step <NUM>, the method <NUM> may further include transmitting the imaging data to a server computing device. The imaging data may be transmitted to the server computing device via one or more communication devices included in the head-mounted display device.

At step <NUM>, the method <NUM> may further include receiving, from the server computing device, an identification of a virtual content item display opportunity. The virtual content item display opportunity may include a determination that the one or more physical features identified based on the imaging data satisfy one or more spatial and/or temporal constraints. For example, the one or more spatial and/or temporal constraints may include one or more position constraints on respective positions at which the one or more virtual content items are displayed. The one or more spatial and/or temporal constraints may additionally or alternatively include one or more timing constraints on respective times at which the one or more virtual content items are displayed. The one or more spatial and/or temporal constraints may additionally or alternatively include one or more size constraints on respective sizes of the one or more virtual content items and/or one or more shape constraints on respective shapes of the one or more virtual content items. In some embodiments, the virtual content item display opportunity may be identified at least in part by determining that the one or more identified physical features have a predetermined set of geometric characteristics.

In response to identifying the virtual content item display opportunity, the method <NUM> may further include, at step <NUM>, transmitting a request for one or more virtual content items to the server computing device based on the identification of the virtual content item display opportunity. At step <NUM>, the method may further include receiving the one or more virtual content items from the server computing device. In some embodiments, additionally or alternatively to contacting a server computing device and receiving the one or more virtual content items from the server device as in steps <NUM> and <NUM>, the head-mounted display device may access one or more virtual content items stored locally in memory. In such embodiments, the one or more virtual content items may be displayed without contacting a server computing device.

At step <NUM>, the method <NUM> may further include outputting the one or more virtual content items for display on the display at the one or more respective dynamically positioned content locations in the physical environment. The one or more virtual content items may be output from display based on the identification of the virtual content item display opportunity. For example, the one or more virtual content items may be output for display at respective dynamically positioned content locations that satisfy the one or more spatial and/or temporal constraints. In some embodiments, the one or more virtual content items may be configured to interact with the one or more virtual objects of the mixed reality experience. In such embodiments, the one or more virtual content items may be incorporated into the mixed reality experience as newly added virtual objects rather than forming a separate but concurrent mixed reality experience.

<FIG> shows additional steps that may be performed in some embodiments when performing the method <NUM> of <FIG>. At step <NUM>, the method <NUM> may further include receiving, via one or more input devices, a user interaction with a virtual content item of the one or more virtual content items. The one or more input devices may include one or more input devices included in, and/or external to, the head-mounted display device. In some embodiments, receiving the user interaction may include, at step <NUM>, collecting inward-facing imaging data of a user via an inward-facing camera included in the head-mounted display device. The user interaction may include gazing at the virtual content item. The user interaction may additionally or alternatively include, at step <NUM>, collecting position data of the head-mounted display device.

In some embodiments, at step <NUM>, the method <NUM> may further include transmitting an indication of the user interaction to the server computing device via one or more communication devices included in the head-mounted display device. In embodiments in which step <NUM> is performed, step <NUM> may include, at step <NUM>, transmitting the inward-facing imaging data to the server computing device. In embodiments in which step <NUM> is performed, step <NUM> may include, at step <NUM>, transmitting the position data to the server computing device. The server computing device may be the same as or different from the server computing device of steps <NUM> and <NUM>. The indication of the user interaction may be transmitted to the server computing device over a network.

At step <NUM>, the method <NUM> may further include modifying at least one virtual object in response to the user interaction. The at least one virtual object modified in response to the user interaction may include at least one virtual content item of the one or more virtual content items, and/or at least one other virtual object included in the mixed reality experience. In embodiments in which steps <NUM> and <NUM> are performed, step <NUM> may include step <NUM>. At step <NUM>, the method may include receiving, from the server computing device, a determination of a gaze direction of the user based on the inward-facing imaging data. In such embodiments, the user interaction may include gazing at the virtual content item.

In embodiments in which steps <NUM> and <NUM> are performed, step <NUM> may include step <NUM>. At step <NUM>, the method <NUM> may include receiving, from the server computing device, a determination of a change in position of the head-mounted display device based on the position data. In such embodiments, the user interaction may include the change in position.

In embodiments in which the imaging data collected at step <NUM> and transmitted to the server computing device at step <NUM> is determined to include a gesture selecting the virtual content item, step <NUM> may include step <NUM>. At step <NUM>, the method <NUM> may include receiving, from the server computing device, a determination that the imaging data includes a gesture selecting the virtual content item. For example, in response to a user interaction including a dragging gesture, a virtual content item may be relocated to a new position. If the new position is already occupied by a virtual object, that virtual object may also be relocated so that it does not overlap with the virtual content item.

An example implementation of a virtual content item request <NUM> and a virtual content item response <NUM> are provided below with reference to <FIG>. In the following example, the virtual content item is an advertisement. The request <NUM> in the example of <FIG> includes a plurality of parameters. These parameters may include an adtype <NUM> which specifies a type of advertisement to display. For example, the adtype <NUM> may indicate that the advertisement is a tabletop advertisement configured to be displayed on a table or other substantially flat, horizontal surface. Alternatively, the adtype <NUM> may indicate that the advertisement is a wall poster configured to be displayed on a substantially flat vertical surface. In some embodiments, the adtype <NUM> may indicate that the advertisement includes a plurality of virtual content items to be displayed at different dynamically positioned locations. For example, the advertisement may include a virtual person standing on a flat surface and may further include a virtual vehicle driving along a road.

The parameters included in the request <NUM> may further include a list <NUM> of virtual content item display opportunities <NUM> indicating one or more dynamically positioned content locations at which the advertisement may be displayed, such as a tabletop, a chair top, a road, or a wall. The parameters of the request <NUM> may further include a user identification (uid) <NUM> that indicates the user <NUM> of the head-mounted display device <NUM> from which the request <NUM> is sent. In some embodiments, a new uid <NUM> may be generated for each session in which the head-mounted display device <NUM> is used. The request <NUM> may further include a parameter indicating a sceneID <NUM> may include information about the mixed reality experience <NUM> and/or one or more physical features <NUM> identified in the physical environment <NUM>. The parameters included in the request <NUM> may further include a size <NUM> of the advertisement. The size <NUM> may indicate a size category such as small, medium, or large. Additionally or alternatively, the size <NUM> may indicate one or more dimensions of the advertisement. In addition to the parameters listed above, the request <NUM> may include one or more other parameters in some embodiments.

The response <NUM> sent from the server computing device <NUM> to the head-mounted display device <NUM> may include a plurality of parameters, which may include a format <NUM>. The format <NUM> may indicate the advertisement type of the advertisement sent from the server computing device <NUM> to the head-mounted display device <NUM>. The response <NUM> may further include a list of creatives <NUM> per experience. The list of creatives <NUM> per experience may include information specifying an appearance of each virtual content item <NUM> to be displayed at the head-mounted display device <NUM>. The term "creative" as used herein refers to data typically stored in a file that includes content that has been created by an artist. For example, creative <NUM> may include one or more of an image, a color, a three dimensional shape, and a sequence of animated frames or a video. One example of a creative that includes a shape is a three dimensional object of an OBJ file type. Creative <NUM> may further include sound data, haptic feedback data, or other information specifying how the virtual content item <NUM> is presented via another output device of the head-mounted display device <NUM>.

The response <NUM> may further include an assetBundleUrl <NUM> that indicates a unique resource locator (URL) at which the creative <NUM> may be accessed. The response <NUM> may further include a rotation <NUM> indicating a rotation performed on the virtual content item <NUM> relative to a default orientation. Additionally or alternatively, the response <NUM> may include a scale <NUM> indicating a scale factor applied to the virtual content item <NUM> relative to a default size.

The parameters of the response <NUM> may further include a trackingUrl <NUM> indicating one or more URLs that may be accessed in response to user interaction <NUM> with the virtual content item <NUM>. For example, trackingUrl <NUM> may include URLs such as gazeTracking 342A to be accessed when the user <NUM> gazes at the advertisement and/or gesture Tracking 342B to be accessed when the user <NUM> makes a gesture manipulating the advertisement. The trackingUrl <NUM> may further include disposeTracking 342C to be accessed when the advertisement ceases to be displayed, rotateTracking 342D to be accessed when the advertisement is rotated, moveTracking 342E to be accessed when the advertisement is moved, and/or occlusionTracking 342F to be accessed when the advertisement is occluded by a physical or virtual object. Other URLs may be included in trackingURL <NUM> in some embodiments. In addition to the parameters listed above, the response <NUM> may include one or more other parameters in some embodiments.

Example code specifying the response <NUM> is shown below:
<IMG>
<IMG>
<IMG>.

In some embodiments, a virtual content item provider may specify the one or more parameters of the response <NUM> and the one or more constraints <NUM> on the placement of a virtual content item <NUM>, which may be stored in the memory <NUM> of the server computing device <NUM>. For example, the virtual content item provider may be an advertiser who wishes to show an advertisement to a user <NUM> when a predetermined type of virtual content item display opportunity <NUM> occurs. For example, with reference to the example of <FIG>, an advertiser may target an advertisement to be displayed when the user <NUM> is in a physical environment <NUM> that includes a television and at least one horizontal surface. The virtual content item provider may further specify one or more candidate location selection criteria based on which the dynamically positioned location <NUM> of the virtual content item <NUM> may be selected from a plurality of candidate locations <NUM>. The virtual content item provider may additionally or alternatively specify one or more characteristics of a set of users <NUM> to whom the advertisement is targeted. In the example of <FIG>, the virtual content item provider may choose to display the virtual content item <NUM> to the user <NUM> based on an indication that the user <NUM> has previously purchased the soft drink being advertised. In some embodiments, a plurality of virtual content item providers may bid on a virtual content item display opportunity <NUM> as they would bid on other forms of advertising space.

Although the above embodiments are discussed with reference to a head-mounted display device, other embodiments may be implemented with a display device that is not head-mounted. For example, the systems and methods discussed above may alternatively be used with a wall- or stand-mounted display device or a handheld display device. In such embodiments, the display device may provide a mixed-reality experience by displaying an image of the physical environment on the display along with one or more virtual objects. For example, a wall- or stand-mounted display device may act as a "window" into another area of the physical environment, such as an area behind a wall, and may display the one or more virtual objects overlapping the image of the other area of the physical environment. The image of the other area of the physical environment may be received from one or more imaging sensors located in the other area of the physical environment and communicatively coupled to the display device. Similarly, the handheld display device may include an outward-facing imaging sensor configured to image a region of the physical environment behind the display. The one or more imaging sensors may include one or more depth cameras and/or one or more RGB cameras. In other embodiments, the three-dimensional environment displayed on the display device may be a three-dimensional virtual environment.

In addition, in wall- or stand-mounted and handheld embodiments, the display device may include one or more imaging sensors configured to detect a gaze direction and/or body position of the user. For example, the handheld display device may include a user-facing imaging sensor configured to image the user. The one or more imaging sensors configured to detect the gaze direction and/or body position of the user may include one or more depth cameras and/or RGB cameras. In handheld embodiments, the display device may further include one or more position sensors configured to detect the position and orientation of the display device relative to the user. Thus, in one example, the handheld display device may display the region of the physical environment behind the display with one or more virtual objects superimposed onto the image of the physical environment.

Computing system <NUM> may embody the head-mounted display device <NUM> described above and illustrated in <FIG> and <FIG> and/or the server computing device described above and illustrated in <FIG>. Computing system <NUM> may take the form of one or more personal computers, server computers, tablet computers, home-entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smart phone), and/or other computing devices, and wearable computing devices such as smart wristwatches and head mounted augmented reality devices.

According to one aspect of the present disclosure, head-mounted display device is provided, including an at least partially see-through display, one or more input devices including one or more image sensors configured to collect imaging data of a physical environment, one or more communication devices and a processor. The processor may be configured to output for display on the display a mixed reality experience including one or more virtual objects superimposed upon the physical environment. The processor may be further configured to transmit the imaging data to a server computing device via the one or more communication devices. The processor may be further configured to receive, from the server computing device, an identification of a virtual content item display opportunity. The identification of the virtual content item display opportunity may include a determination that the mixed reality experience and one or more physical features identified from the imaging data satisfy one or more spatial and/or temporal constraints. The processor may be further configured to transmit a request for one or more virtual content items to the server computing device based on the identification of the virtual content item display opportunity. The processor may be further configured to receive the one or more virtual content items from the server computing device. The processor may be further configured to output the one or more virtual content items for display on the display at one or more respective dynamically positioned content locations in the physical environment.

According to this aspect, the processor may be further configured to receive, via the one or more input devices, a user interaction with a virtual content item of the one or more virtual content items.

According to this aspect, the processor may be further configured to modify at least one virtual object in response to the user interaction.

According to this aspect, the processor may be further configured to transmit an indication of the user interaction to the server computing device via the one or more communication devices.

According to this aspect, the one or more input devices may include an inward-facing image sensor configured to collect inward-facing imaging data of a user. The processor may be further configured to transmit the inward-facing imaging data to the server computing device. The processor may be further configured to receive, from the server computing device, a determination of a gaze direction of the user based on the inward-facing imaging data. The user interaction may include gazing at the virtual content item.

According to this aspect, the one or more input devices may include a position sensor configured to collect position data of the head-mounted display device. The user interaction may include a change in position of the head-mounted display device. The processor may be further configured to transmit the position data to the server computing device.

According to this aspect, the user interaction may include a gesture selecting the virtual content item.

According to this aspect, the one or more virtual content items may be configured to interact with the one or more virtual objects of the mixed reality experience.

According to this aspect, the identification of the virtual content item display opportunity may include a determination that the one or more identified physical features have a predetermined set of geometric characteristics.

According to this aspect, the one or more spatial and/or temporal constraints may be selected from the group consisting of one or more position constraints on respective positions at which the one or more virtual content items are displayed, one or more timing constraints on respective times at which the one or more virtual content items are displayed, one or more size constraints on respective sizes of the one or more virtual content items, and one or more shape constraints on respective shapes of the one or more virtual content items.

According to this aspect, the one or more virtual content items may include one or more advertisements.

According to another aspect of the present disclosure, a method for use with a head-mounted display device is provided. The method may include collecting imaging data of a physical environment. The method may further include outputting for display on an at least partially see-through display a mixed reality experience including one or more virtual objects superimposed upon the physical environment. The method may further include transmitting the imaging data to a server computing device. The method may further include receiving, from the server computing device, an identification of a virtual content item display opportunity. The identification of the virtual content item display opportunity may include a determination that the mixed reality experience and one or more physical features identified from the imaging data satisfy one or more spatial and/or temporal constraints. The method may further include transmitting a request for one or more virtual content items to the server computing device based on the identification of the virtual content item display opportunity. The method may further include receiving the one or more virtual content items from the server computing device. The method may further include outputting the one or more virtual content items for display on the display at one or more respective dynamically positioned content locations in the physical environment.

According to this aspect, the method may further include receiving, via one or more input devices, a user interaction with a virtual content item of the one or more virtual content items.

According to this aspect, the method may further include transmitting an indication of the user interaction to the server computing device.

According to this aspect, the method may further include collecting inward-facing imaging data of a user. The method may further include transmitting the inward-facing imaging data to the server computing device. The method may further include receiving, from the server computing device, a determination of a gaze direction of the user based on the inward-facing imaging data. The user interaction may include gazing at the virtual content item.

According to another aspect of the present disclosure, a server computing device is provided, including one or more communication devices communicatively coupled to a display device and a processor configured to receive imaging data from the display device. The processor may be further configured to, based on the imaging data, identify one or more physical features in the physical environment of the display device. The processor may be further configured to identify a virtual content item display opportunity at least in part by determining that the one or more physical features identified at the display device satisfy one or more spatial and/or temporal constraints. The processor may be further configured to select, based on the virtual content item display opportunity, one or more virtual content items from a plurality of virtual content items. The processor may be further configured to select, based on the virtual content item display opportunity, one or more respective dynamically positioned content locations at which to display the one or more virtual content items in the physical environment of the display device. The processor may be further configured to convey the one or more virtual content items and one or more respective dynamically positioned content locations to the display device.

According to this aspect, the processor may be further configured to receive a user interaction with a virtual content item of the one or more virtual content items from the display device.

Claim 1:
A head-mounted display device (<NUM>), comprising:
an at least partially see-through display (<NUM>);
one or more input devices including one or more image sensors (<NUM>) configured to collect imaging data of a physical environment;
one or more communication devices (<NUM>); and
a processor (<NUM>) configured to:
output for display on the display a mixed reality experience including one or more virtual objects (<NUM>) superimposed upon the physical environment;
transmit the imaging data to a server computing device (<NUM>) via the one or more communication devices (<NUM>);
receive, from the server computing device (<NUM>), an identification of a virtual content item display opportunity (<NUM>), wherein the identification of the virtual content item display opportunity (<NUM>) includes a determination that the mixed reality experience and one or more physical features (<NUM>) identified from the imaging data satisfy one or more spatial and/or temporal constraints (<NUM>);
transmit a request for one or more virtual content items (<NUM>) to the server computing device (<NUM>) based on the identification of the virtual content item display opportunity (<NUM>);
receive the one or more virtual content items (<NUM>) from the server computing device (<NUM>); and
output, based on the identification of the virtual content item display opportunity (<NUM>), the one or more virtual content items (<NUM>) for display on the display (<NUM>) at one or more respective dynamically positioned content locations (<NUM>) in the physical environment that satisfy the one or more spatial and/or temporal constraints;
wherein the identification of the virtual content item display opportunity (<NUM>) includes a determination that the one or more identified physical features (<NUM>) have a predetermined set of geometric characteristics, and wherein the one or more spatial and/or temporal constraints (<NUM>) comprise one or more size constraints (42C) on respective sizes of the one or more virtual content items (<NUM>).