Generating content for physical elements

First content may be obtained in response to identifying a first physical element of a first object type. The first content may be associated with the first object type. Second content may be obtained in response to identifying a second physical element of a second object type. The second content may be associated with the second object type. The second physical element may be detected as being within a threshold distance of the first physical element. Third content may be generated based on a combination of the first content and the second content. The third content may be associated with a third object type that is different from the first object type and the second object type. The third content may be displayed on the display.

TECHNICAL FIELD

The present disclosure generally relates to generating content for physical elements.

BACKGROUND

Some devices may be capable of generating and presenting content. Some devices that present content include mobile communication devices, such as smartphones. Some devices that present content may present representations of physical elements. Some devices that present representations of physical elements may not provide enough attributes with which a user may interact.

SUMMARY

Various implementations disclosed herein include devices, systems, and methods for generating content for physical elements. In various implementations, a device may include an environmental sensor, a display, a non-transitory memory, and one or more processors coupled with the non-transitory memory. A first content may be obtained in response to identifying a first physical element of a first object type. The first content may be associated with the first object type. A second content may be obtained in response to identifying a second physical element of a second object type. The second content may be associated with the second object type. The second physical element may be detected as being within a threshold distance of the first physical element. A third content may be generated based on a combination of the first content and the second content. The third content may be associated with a third object type that is different from the first object type and the second object type. The third content may be displayed on the display.

In some implementations, a device may include an environmental sensor, a display, a non-transitory memory, and one or more processors coupled with the non-transitory memory. Content may be displayed within or extending from a representation of a physical element. An affordance may be composited in association with a portion of the E content. The affordance may allow manipulation of the portion of the content. An input directed to the affordance may be detected. In response to detecting the input directed to the affordance, a manipulation of the portion of the content may be displayed.

In accordance with some implementations, a device may include one or more processors, a non-transitory memory, and one or more programs. In some implementations, the one or more programs may be stored in the non-transitory memory and may be executed by the one or more processors. In some implementations, the one or more programs may include instructions for performing or causing performance of any of the methods described herein. In accordance with some implementations, a non-transitory computer readable storage medium may have stored therein instructions that, when executed by one or more processors of a device, may cause the device to perform or cause performance of any of the methods described herein. In accordance with some implementations, a device may include one or more processors, a non-transitory memory, and means for performing or causing performance of any of the methods described herein.

DESCRIPTION

A physical environment refers to a physical world that people can sense and/or interact with without aid of electronic devices. The physical environment may include physical features such as a physical surface or a physical object. For example, the physical environment corresponds to a physical park that includes physical trees, physical buildings, and physical people. People can directly sense and/or interact with the physical environment such as through sight, touch, hearing, taste, and smell. In contrast, an extended reality (XR) environment refers to a wholly or partially simulated environment that people sense and/or interact with via an electronic device. For example, the XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and/or the like. With an XR system, a subset of a person's physical motions, or representations thereof, are tracked, and, in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that comports with at least one law of physics. As one example, the XR system may detect head movement and, in response, adjust graphical content and an acoustic field presented to the person in a manner similar to how such views and sounds would change in a physical environment. As another example, the XR system may detect movement of the electronic device presenting the XR environment (e.g., a mobile phone, a tablet, a laptop, or the like) and, in response, adjust graphical content and an acoustic field presented to the person in a manner similar to how such views and sounds would change in a physical environment. In some situations (e.g., for accessibility reasons), the XR system may adjust characteristic(s) of graphical content in the XR environment in response to representations of physical motions (e.g., vocal commands).

The present disclosure provides methods, systems, and/or devices for detecting a physical element and obtaining and/or presenting XR content based on one or more properties of the physical element. In some implementations, when two physical elements with associated XR content are within a threshold distance of one another, XR content is generated based on the XR content associated with the physical elements. In some implementations, contextual XR content is displayed with an XR representation of a physical element. In some implementations, an affordance is composited in association with the XR content. A user may manipulate the XR content.

FIGS.1A-1Eare block diagrams of an example operating environment100in accordance with some implementations. While pertinent features are shown, those of ordinary skill in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity and so as not to obscure more pertinent aspects of the example implementations disclosed herein. To that end, as a non-limiting example, the operating environment100may include a controller102and an electronic device104. In some implementations, the electronic device104includes a smartphone, a tablet, a laptop, or the like. The electronic device104may be carried by or worn by a user10.

As illustrated inFIGS.1A-1E, in various implementations, the electronic device104presents an XR environment106. In some implementations, the XR environment106is generated by the controller102and/or the electronic device104. In some implementations, the XR environment106includes a virtual environment that is a simulated replacement of a physical environment. For example, in some implementations, the XR environment106is simulated by the controller102and/or the electronic device104. In some implementations, the XR environment106may be different from the physical environment in which the electronic device104is located.

In some implementations, the XR environment106includes an augmented environment that is a modified version of a physical environment. For example, in some implementations, the controller102and/or the electronic device104modify (e.g., augment) a representation of the physical environment in which the electronic device104is located to generate the XR environment106. In some implementations, the controller102and/or the electronic device104may generate the XR environment106by adding items to the representation of the physical environment in which the electronic device104is located.

In some implementations, the controller102and/or the electronic device104generate the XR environment106by simulating a replica of the physical environment in which the electronic device104is located. In some implementations, the controller102and/or the electronic device104may generate the XR environment106by removing and/or adding items from the simulated replica of the physical environment in which the electronic device104is located.

The XR environment106may include extended reality (XR) representations of real objects, e.g., physical elements. For example, the XR environment106may include an XR representation of a cereal box108(“cereal box108”, hereinafter for the sake of brevity). In some implementations, when the controller102and/or the electronic device104recognize a physical element as a cereal box, the controller102and/or the electronic device104obtain XR content118. In some implementations, the XR content118includes the XR representation of the cereal box108. The XR representation of the cereal box108may be rendered in a cutaway view. Rendering a cutaway view may facilitate use of the interior of the cereal box108in the XR environment106. In other words, in some implementations, the XR content118is displayed within the XR representation of the cereal box108. More generally, in various implementations, the XR content118is displayed in association with the XR representation of the cereal box108.

The XR environment106may include XR representations of other physical elements, such as an XR representation of a dental floss container110shown inFIG.1B(“dental floss container110”, hereinafter for the sake of brevity) and an XR representation of a lip balm container112shown inFIG.1D(“lip balm container112”, hereinafter for the sake of brevity). In some implementations, when the controller102and/or the electronic device104recognize a physical element as a dental floss container, the controller102and/or the electronic device104obtain XR content120. In some implementations, the XR content120includes the XR representation of the dental floss container110. In some implementations, when the controller102and/or the electronic device104recognize a physical element as a lip balm container, the controller102and/or the electronic device104obtain XR content122. In some implementations, the XR content122includes an XR representation of the lip balm container112. The XR representations may be displayed at locations corresponding to locations of associated physical elements. For example, if, in the physical environment the dental floss container is placed near a side of the cereal box, then the XR representation of the dental floss container110may be displayed near the corresponding side of the XR representation of the cereal box108.

FIG.1Billustrates the XR environment106with an XR representation of another physical element, e.g., the dental floss container110. The controller102and/or the electronic device104may identify a real dental floss container in a physical environment, for example, based on environmental data corresponding to the physical environment. The environmental data may include, e.g., depth information, photographic information, and/or video information.FIG.1Billustrates the dental floss container110located at a distance D1from the cereal box108that is greater than a threshold distance T. The threshold distance T may be a fixed distance, may vary or may be selected based on the size of the first physical element and/or the second physical element.

As shown inFIG.1B, when the distance D1between the dental floss container110and the cereal box108is greater than the threshold distance T, the controller102and/or the electronic device104obtain (e.g., retrieve, synthesize, or generate) and display the XR content120associated with the dental floss container110. For example, in some implementations, the XR content120is retrieved from a content datastore. In some implementations, the XR content120is synthesized based on one or more characteristics (e.g., physical characteristics, chemical characteristics, electrical characteristics, structural characteristics, functional characteristics, etc.) of the physical element. In some implementations, the XR content120is generated based on one or more characteristics of the physical element.

FIG.1Cillustrates the dental floss container110located at a distance D2from the cereal box108that is less than the threshold distance T. In some implementations, when the dental floss container110is closer than the threshold distance T to the cereal box108, XR content130is generated based on a combination of the XR content118and the XR content120, as disclosed in greater detail herein. In some implementations, the XR content130is related to XR representations of both the physical elements (e.g., the cereal box108and the dental floss container110). For example, if the physical element is identified as the dental floss container, the controller102and/or the electronic device104may generate an XR representation of ropes.

FIG.1Dillustrates the XR environment106with an XR representation of another physical element, e.g., the lip balm container112. The controller102and/or the electronic device104may identify the lip balm container in a physical environment, for example, based on environmental data corresponding to the physical environment. As shown inFIG.1D, when the distance D3between the lip balm container112and the cereal box108is greater than the threshold distance T, the controller102and/or the electronic device104obtain (e.g., retrieve, synthesize, or generate) and display XR content122associated with the XR representation of the lip balm container112. As illustrated inFIG.1D, since the distance D3is greater than the threshold distance T, the XR content122is displayed separate from the XR content130.

FIG.1Eillustrates the lip balm container112located at a distance D4from the cereal box108that is less than the threshold distance T. In some implementations, when the lip balm container112is closer than the threshold distance T to the cereal box108, XR content140is generated based on a combination of the XR content122and the XR content130, as disclosed in greater detail herein. In some implementations, the XR content140is unrelated to the physical element. For example, if the physical element is identified as the lip balm container, the controller102and/or the electronic device104may include an XR representation of a power source, such as a battery in the XR content140.

As illustrated inFIG.1E, the controller102and/or the electronic device104determines that a distance D4between the cereal box108and the lip balm container112is within (e.g., less than or equal to) the threshold distance T. In some implementations, the controller102and/or the electronic device104obtain environmental data (e.g., including depth data, image data, and/or a video) to make this determination.

In some implementations, a head-mountable device (HMD), being worn by the user10, presents (e.g., displays) the XR environment106according to various implementations. In some implementations, the HMD includes an integrated display (e.g., a built-in display) that displays the XR environment106. In some implementations, the HMD includes a head-mountable enclosure. In various implementations, the head-mountable enclosure includes an attachment region to which another device with a display can be attached. For example, in some implementations, the electronic device104can be attached to the head-mountable enclosure. In various implementations, the head-mountable enclosure is shaped to form a receptacle for receiving another device that includes a display (e.g., the electronic device104). For example, in some implementations, the electronic device104slides/snaps into or otherwise attaches to the head-mountable enclosure. In some implementations, the display of the device attached to the head-mountable enclosure presents (e.g., displays) the XR environment106. In various implementations, examples of the electronic device104include smartphones, tablets, media players, laptops, etc.

FIG.2is a flowchart representation of a method200of generating XR content in accordance with some implementations. In various implementations, the method200is performed by a device with a non-transitory memory and one or more processors coupled with the non-transitory memory (e.g., the controller102and/or the electronic device104shown inFIG.1A). In some implementations, the method200is performed by processing logic, which may include hardware, firmware, software, or a combination thereof. In some implementations, the method200is performed by a processor executing code stored in a non-transitory computer-readable medium, e.g., a memory.

As represented by block202, in some implementations, the method200includes obtaining (e.g., retrieving, synthesizing, or generating) a first XR content in response to identifying a first physical element of a first object type. For example, in some implementations, the method200includes retrieving the first XR content from a content datastore. In some implementations, the method200includes synthesizing the first XR content based on one or more characteristics (e.g., physical characteristics, chemical characteristics, electrical characteristics, structural characteristics, functional characteristics, etc.) of the physical element. In some implementations, the method200includes generating the first XR content based on one or more characteristics of the physical element.

For example, as shown inFIG.1A, the controller102and/or the electronic device104identifies a cereal box. In some implementations, the controller102and/or the electronic device104identifies the cereal box based on environmental data corresponding to a physical environment. The environmental data may include, for example, depth information, image data, and/or a video. In some implementations, in response to identifying the cereal box, the controller102and/or the electronic device104displays the XR representation of the cereal box108and obtains XR content118.

As represented by block202a, in some implementations, the first XR content is related to the first physical element. For example, the XR content118may be related to a real cereal box represented by the XR cereal box108(e.g., the XR content118may include a video provided by a manufacturer of the cereal). In some implementations, the method200includes obtaining the first XR content based on one or more characteristics of the physical element. For example, in some implementations, the method200includes determining a set of characteristics of the physical element, utilizing the set of characteristics to form a search query for an XR content datastore, and receiving the first XR content as a search result (e.g., if the cereal in the cereal box is in the shape of animal characters, then receiving XR representations of animals). In some implementations, the method200includes providing the set of characteristics as input to a neural network system that generates the first XR content.

As shown in block202b, in some implementations, the first XR content includes a video component and/or an audio component. In some implementations, the method200includes obtaining the video component and the audio component from different sources. For example, in some implementations, the method200includes obtaining the video component from a video content store and the audio component from a music streaming service.

As shown in block202c, the first XR content includes XR representations for a first set of objective-effectuators. In some implementations, the first set of objective-effectuators represent items within the first physical element. For example, each of the first set of objective-effectuators includes a character objective-effectuator that represents a piece of cereal in the cereal box. For example, if some pieces of cereal in the cereal box are in the shape of an animal, then at least some of the first set of objective-effectuators model behavior of the animal.

As represented by block204, in some implementations, the controller102and/or the electronic device104obtains a second XR content in response to identifying a second physical element of a second object type. For example, in some implementations, the method200includes retrieving the second XR content from a content datastore. In some implementations, the method200includes synthesizing the second XR content based on one or more characteristics (e.g., physical characteristics, chemical characteristics, electrical characteristics, structural characteristics, functional characteristics, etc.) of the physical element. In some implementations, the method200includes generating the second XR content based on one or more characteristics of the physical element.

For example, as shown inFIG.1B, the controller102and/or the electronic device104obtains the XR content120in response to identifying a physical dental floss container represented by the XR dental floss container110. In some implementations, the controller102and/or the electronic device104identifies the dental floss container, for example, based on environmental data corresponding to a physical environment. The environmental data may include, e.g., depth information, photographic information, and/or video information. In some implementations, the method200includes obtaining the environmental data using an environmental sensor. In some implementations, the environmental sensor is or includes, for example, a camera or a depth sensor.

Based on this identification, the controller102and/or the electronic device104may generate the XR content120, as shown at block204a. In some implementations, the controller102and/or the electronic device104retrieves the XR content120from a data store based on this identification. In some implementations, the XR content120is related to the dental floss container110. In some implementations, the XR content120is unrelated to the dental floss container110. In some implementations, the XR content120is obtained based on a property (e.g., a physical, chemical, electrical, structural, or functional property) of the dental floss container110.

In some implementations, as shown inFIG.1B, the controller102and/or the electronic device104determines that a distance D1between the cereal box108and the dental floss container110is greater than a threshold distance T. In some implementations, the controller102and/or the electronic device104obtain environmental data (e.g., including depth information, photographic information, and/or video information) to make this determination. The threshold distance T may be a fixed distance. The threshold distance T may vary or may be selected based on the size of the first physical element or the second physical element. For example, the threshold distance T may be a percentage of the size of the first physical element or a percentage of the size of the second physical element.

As represented by block206, in some implementations, the controller102and/or the electronic device104detect that the second physical element is within a threshold distance of the first physical element. For example, in some implementations, the controller102and/or the electronic device104determines that a distance D2between the cereal box108and the dental floss container110is within (e.g., less than or equal to) the threshold distance T, as shown inFIG.1C.

As represented by block206a, in some implementations, the controller102and/or the electronic device104obtain environmental data to make this determination. The environmental data may include, for example, depth information, image data, and/or a video. In some implementations, the threshold distance T is a fixed distance.

As represented by block206b, the threshold distance T may vary or may be selected based on the size of the first physical element or the second physical element. For example, the threshold distance T may be a percentage of the size of the first physical element or a percentage of the size of the second physical element.

As represented by block208, in some implementations, the method200includes generating third XR content based on a combination of the first XR content and the second XR content. The third XR content may be associated with a third object type. The third object type may be different from the first object type and/or the second object type. In some implementations, the third XR content may include a portion of the first XR content that may be selected based on one or more properties associated with the first object type. The third XR content may include a portion of the second XR content that may be selected based on one or more properties associated with the second object type. For example, in some implementations, in response to determining that the first physical element (e.g., the cereal box108) and the second physical element (e.g., the dental floss container110) are within the threshold distance T of one another, the controller102and/or the electronic device104generates XR content130, as shown inFIG.1C.

In some implementations, the XR content130is related to the XR content118and/or the XR content120. For example, the controller102and/or the electronic device104may obtain (e.g., retrieve, synthesize, or generate) XR ropes extending from an XR representation of the cereal box108. In some implementations, the XR content130is retrieved from a content datastore. In some implementations, the XR content130is synthesized based on one or more characteristics (e.g., physical characteristics, chemical characteristics, electrical characteristics, structural characteristics, functional characteristics, etc.) of the physical element. In some implementations, the XR content130is generated based on one or more characteristics of the physical element. In some implementations, the XR environment106may include objective-effectuators that represent characters from different fictional materials (e.g., movies, TV shows, games, comics, and/or novels). In various implementations, the objective-effectuators may represent (e.g., model behavior of) tangible objects. For example, in some implementations, the objective-effectuators may represent equipment (e.g., machinery such as planes, tanks, robots, cars, etc.). In some implementations, the objective-effectuators may model behavior of (e.g., represent) fictional entities (e.g., things such as equipment from fictional materials). In some implementations, the objective-effectuators may model entities from physical entities (e.g., the objective-effectuators may represent things from the real-world), including entities located inside and/or outside of the XR environment106. In some implementations, XR content may be related to an objective for an objective-effectuator.

In various implementations, the objective-effectuators perform one or more actions in order to effectuate (e.g., complete, satisfy, and/or achieve) one or more objectives. In some implementations, the objective-effectuators perform a sequence of actions. In some implementations, the controller102and/or the electronic device104determine the actions that the objective-effectuators are to perform. In some implementations, the actions of the objective-effectuators are within a degree of similarity to actions that the entities (e.g., characters or things) corresponding to the objective-effectuators may perform in the fictional material from which the objective-effectuators are derived. In some implementations, an objective-effectuator is associated with a particular objective, and the objective-effectuator may perform actions that may improve the likelihood of satisfying that particular objective.

In some implementations, the objective-effectuators may be referred to as object representations, for example, because the objective-effectuators may represent various objects (e.g., real-world objects, or fictional objects). In some implementations, an objective-effectuator representing a character may be referred to as a character objective-effectuator. In some implementations, a character objective-effectuator may perform actions to effectuate a character objective. In some implementations, an objective-effectuator representing an equipment may be referred to as an equipment objective-effectuator. In some implementations, an equipment objective-effectuator may perform actions to effectuate an equipment objective. In some implementations, an objective-effectuator representing an environment may be referred to as an environmental objective-effectuator. In some implementations, an environmental objective-effectuator may perform environmental actions to effectuate an environmental objective. In some implementations, XR content is related to an action for an objective-effectuator.

The third XR content may include a mixture of audio and/or video from the first XR content and/or the second XR content. In some implementations, as represented in block208a, the third XR content includes a video component from the first XR content and/or the second XR content. In some implementations, as represented in block208b, the third XR content includes an audio component from the first XR content and/or the second XR content. For example, the third XR content may include audio from the first XR content and video from the second XR content. As another example, the third XR content may include video from the first XR content and audio from the second XR content.

In some implementations, as represented in block208c, the third XR content includes XR representations for a subset of objective-effectuators. For example, in some implementations, the third XR content is generated at least in part by importing one or more objective-effectuators from one XR content into another XR content. One or more objective-effectuators may be imported from the first XR content into the second XR content. As another example, one or more objective-effectuators may be imported from the second XR content into the first XR content. Any imported objective-effectuators may interact with objective-effectuators in the XR content into which they are imported. For example, objective-effectuators imported from the second XR content may interact with objective-effectuators in the first XR content. Similarly, objective-effectuators imported from the first XR content may interact with objective-effectuators in the second XR content.

As represented by block210, in some implementations, the method200includes displaying the third XR content. The third XR content may be displayed, for example, using a display that may be integrated as part of the electronic device104. For example, the user10may be able to see content displayed on the display forming part of the electronic device104. The third XR content may include virtual reality content, mixed reality content, or both. In some implementations, the third XR content may be displayed using an external display.

As represented by block210a, in some implementations, the third XR content is displayed within a boundary of the first physical element (e.g., the cereal box108) and/or the second physical element (e.g., the dental floss container110or the lip balm container112), e.g., within a boundary of an XR representation of the first physical element and/or the second physical element.

In some implementations, the method200includes detecting a user input interacting with a portion of the third XR content, and displaying a manipulation of the portion of the third XR content in response to detecting the user input.

In some implementations, the method200includes compositing an affordance in association with a portion of the third XR content. In some implementations, the affordance allows manipulation of the portion of the third XR content. In some implementations, the method200includes detecting an input directed to the affordance. In some implementations, the method200includes displaying a manipulation of the portion of the third XR content in response to detecting the input directed to the affordance. In some implementations, the affordance includes at least one of a visible affordance, an invisible affordance, an opaque affordance, a translucent affordance, or a transparent affordance.

In some implementations, the portion of the third XR content includes an XR representation of an objective-effectuator. In some implementations, displaying the manipulation of the portion of the third XR content includes animating the XR representation of the objective-effectuator in order to provide an appearance that the XR representation of the objective-effectuator is performing an action.

In some implementations, the portion of the third XR content includes an XR object. In some implementations, displaying the manipulation of the portion of the third XR content comprises moving the XR object (e.g., from a first display location to a second display location) or changing a state of the XR object (e.g., switching between an open state and a closed state).

It should be understood that the particular order in which the operations inFIG.2have been described is merely exemplary and is not intended to indicate that the described order is the only order in which the operations could be performed. One of ordinary skill in the art would recognize various ways to reorder the operations described herein. Additionally, it should be noted that details of other processes described herein with respect to other methods described herein (e.g., method500) are also applicable in an analogous manner to method200described above with respect toFIG.2. For example, the first XR content, the second XR content, the third XR content, etc., described above with reference to method200optionally have one or more of the characteristics of the XR content, the affordance, the input and the manipulation, etc. described herein with reference to other methods described herein (e.g., method500). For brevity, these details are not repeated here.

FIG.3is a block diagram of a device300(e.g., a server system) in accordance with some implementations. In some implementations, the device300implements the controller102and/or the electronic device104shown inFIG.1A. While certain specific features are illustrated, those of ordinary skill in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity, and so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, as a non-limiting example, in some implementations, the device300includes one or more processing units (CPUs)302, a network interface304, a programming interface306, a memory308, and one or more communication buses310for interconnecting these and various other components.

In some implementations, the network interface304is provided to, among other uses, establish and maintain a metadata tunnel between a cloud hosted network management system and at least one private network including one or more compliant devices. In some implementations, the one or more communication buses310include circuitry that interconnects and controls communications between system components. The memory308may include high-speed random access memory, such as DRAM, SRAM, DDR RAM, and/or other random access solid state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, and/or other non-volatile solid state storage devices. The memory308may include one or more storage devices remotely located from the one or more CPUs302. The memory308may include a non-transitory computer readable storage medium.

In some implementations, the memory308or the non-transitory computer readable storage medium of the memory308may store one or more of the following programs, modules and data structures, or a subset thereof, which may include an operating system312, a data obtainer314, a physical element detector316, an XR content generator318, and/or an XR content renderer320. The data obtainer314may include instructions314aand/or heuristics and metadata314bfor obtaining data, for example, from an environmental sensor. The physical element detector316may include instructions316aand/or heuristics and metadata316bfor detecting physical elements based on the obtained data. The XR content generator318may include instructions318aand/or heuristics and metadata318bfor generating XR content based on the detected physical elements. The XR content renderer320may include instructions320aand/or heuristics and metadata320bfor displaying or rendering the generated XR content.

In various implementations, the device300performs the method200shown inFIG.2. In some implementations, the data obtainer314performs the operations described with respect to blocks202and204shown inFIG.2. In some implementations, the physical element detector316performs the operations described with respect to block206shown inFIG.2. In some implementations, the XR content generator318performs the operations described with respect to block208shown inFIG.2. In some implementations, the XR content renderer320performs the operations described with respect to block210.

FIGS.4A-4Care block diagrams of an example operating environment400in accordance with some implementations. The operating environment400may include a controller402and an electronic device404. In some implementations, the electronic device404includes a smartphone, a tablet, a laptop, or the like. In some implementations, the electronic device404includes a head-mountable device (HMD). The electronic device404may be carried by or worn by a user40.

As illustrated inFIGS.4A-4C, in various implementations, the electronic device404presents an XR environment406. In some implementations, the XR environment406is generated by the controller402and/or the electronic device404. In some implementations, the XR environment406includes a virtual environment that is a simulated replacement of a physical environment. For example, in some implementations, the XR environment406is simulated by the controller402and/or the electronic device404. In some implementations, the XR environment406may be different from the real-world environment in which the electronic device404is located.

In some implementations, the XR environment406includes an augmented environment that is a modified version of a physical environment. For example, in some implementations, the controller402and/or the electronic device404modify (e.g., augment) a representation of the physical environment in which the electronic device404is located to generate the XR environment406. In some implementations, the controller402and/or the electronic device404may generate the XR environment406by adding items to the representation of the physical environment in which the electronic device404is located.

In some implementations, the controller402and/or the electronic device404generate the XR environment406by simulating a replica of the real-world scene in which the electronic device404is located. In some implementations, the controller402and/or the electronic device404may generate the XR environment406by removing and/or adding items from the simulated replica of the real-world scene in which the electronic device404is located.

The XR environment406may include XR representations of real-world objects, e.g., physical elements. For example, the XR environment406may include an XR representation of a cereal box408(“XR cereal box408”, hereinafter for the sake of brevity). In some implementations, when the controller402and/or the electronic device404recognize a physical element as a cereal box, the controller402and/or the electronic device404obtain XR content410. In some implementations, the XR content410includes an XR representation of the cereal box408. The XR representation of the cereal box408may be rendered in a cutaway view. Rendering a cutaway view may facilitate use of the interior of the cereal box408in the XR environment406. In other words, in some implementations, the XR content410is displayed within the XR representation of the cereal box408. More generally, in various implementations, the XR content410is displayed in association with the XR representation of the cereal box408.

As shown inFIG.4A, an affordance412may be composited in association with a portion of the XR content410. In some implementations, the affordance412is visible. In some implementations, the affordance412is invisible. In some implementations, the affordance412is opaque. In some implementations, the affordance412is translucent. In some implementations, the affordance412is transparent. In some implementations, the affordance412allows manipulation of the portion of the XR content410.

FIG.4Billustrates an input414directed to the affordance412. In some implementations, the input414includes a user input that is provided by the user40. In some implementations, the input414is generated by the controller402and/or the electronic device404. When the controller402and/or the electronic device404detect the input414, a manipulation of the portion of the XR content410may be displayed.

For example, as shown inFIG.4C, the controller402and/or the electronic device404may display the portion of the XR content being modified or changed, as represented by a modified XR content410a.

FIG.5is a flowchart representation of a method500of displaying a manipulation of XR content in accordance with some implementations. In various implementations, the method500may be performed by a device with a non-transitory memory and one or more processors coupled with the non-transitory memory (e.g., the controller402and/or the electronic device404shown inFIG.4A). In some implementations, the method500may be performed by processing logic, which may include hardware, firmware, software, or a combination thereof. In some implementations, the method500may be performed by a processor executing code that may be stored in a non-transitory computer-readable medium, e.g., a memory.

As represented by block502, in some implementations, the method500includes displaying XR content within or extending from a physical element, such as the cereal box408ofFIG.4A, or an XR representation of a physical element. In some implementations, the method500includes displaying an XR representation of the physical element. The XR content may be displayed wholly or partially within a boundary of the XR representation of the physical element.

As represented by block502a, in some implementations, the method500includes detecting the physical element. For example, in some implementations, the controller402and/or the electronic device404recognizes the physical element as the cereal box408based on environmental data corresponding to a physical environment. The environmental data may include, for example, depth information, image data, and/or a video. In some implementations, the method500includes obtaining the environmental data using an environmental sensor. The environmental sensor may be or may include, for example, a camera or a depth sensor.

As represented by block502b, in some implementations, the method500includes obtaining the XR content. Obtaining XR content may include retrieving, synthesizing, or generating XR content. For example, in some implementations, the XR content is retrieved from a content datastore. In some implementations, the XR content is synthesized based on one or more characteristics (e.g., physical characteristics, chemical characteristics, electrical characteristics, structural characteristics, functional characteristics, etc.) of the physical element. In some implementations, the XR content is generated based on one or more characteristics of the physical element. As illustrated inFIG.4A, the controller402and/or the electronic device404may obtain XR content410.

As represented by block504, in some implementations, the method500includes compositing the affordance412in association with a portion of the XR content. The affordance412may allow manipulation of the portion of the XR content. As represented by block504a, in some implementations, the affordance412is visible. In some implementations, the affordance412is invisible. In some implementations, the affordance412is opaque. In some implementations, the affordance412is translucent. In some implementations, the affordance412is transparent.

In some implementations, as represented by block504b, the portion of the XR content includes an XR representation of an objective-effectuator, such as, for example, a character objective-effectuator, an equipment objective-effectuator, and/or an environmental objective-effectuator. Activating the affordance may trigger the XR representation of the objective-effectuator to perform an action. The action may improve the likelihood of satisfying a particular objective that may be associated with the objective-effectuator. The action may be within a degree of similarity to actions that an entity (e.g., a character or a thing) corresponding to the objective-effectuator may perform in the fictional material from which the objective-effectuator is derived.

As represented by block506, in some implementations, the method500includes detecting the input414directed to the affordance. In some implementations, as represented by block506a, the input414includes a user input that is provided by the user40. For example, the user40may click on or otherwise select or interact with the affordance412. In some implementations, as represented by block506b, the method500includes detecting an input generated by the controller402and/or the electronic device404.

As represented by block508, in some implementations, the method500includes displaying a manipulation of the portion of the XR content in response to detecting this input. For example, as represented by block508a, the affordance may be visible, and the controller402and/or the electronic device404may display an XR representation of an objective-effectuator, e.g., an XR avatar of a person, performing an action, e.g., activating the affordance. As another example, the controller402and/or the electronic device404may display the portion of the XR content being modified or changed, as represented by the modified XR content410aofFIG.4C.

In some implementations, as represented by block508b, the method500includes displaying an XR representation of an objective-effectuator performing a series of actions. In some implementations, the controller402and/or the electronic device404may determine the actions that the objective-effectuators are to perform. In some implementations, the actions of the objective-effectuators may be within a degree of similarity to actions that the characters or things corresponding to the objective-effectuators may perform in the fictional material from which the objective-effectuators are derived. In some implementations, an objective-effectuator may be associated with a particular objective, and the objective-effectuator may perform actions that may improve the likelihood of satisfying that particular objective.

In some implementations, the portion of the XR content includes an XR object. In some implementations, displaying the manipulation of the portion of the XR content includes displaying a movement of the XR object from a first display location to a second display location.

In some implementations, the portion of the XR content includes an XR object. In some implementations, displaying the manipulation of the portion of the XR content includes displaying a change in the XR object from a first state to a second state (e.g., switching between an open state and a closed state).

In some implementations, detecting the input includes detecting that an XR representation of a user has activated the affordance.

In some implementations, detecting the input includes detecting that an XR representation of an objective-effectuator has activated the affordance. For example, in some implementations, detecting the input includes detecting that an XR character has activated the affordance without an explicit command from the user.

It should be understood that the particular order in which the operations inFIG.5have been described is merely exemplary and is not intended to indicate that the described order is the only order in which the operations could be performed. One of ordinary skill in the art would recognize various ways to reorder the operations described herein. Additionally, it should be noted that details of other processes described herein with respect to other methods described herein (e.g., the method200) are also applicable in an analogous manner to method500described above with respect toFIG.5. For example, the XR content, the affordance, the input and the manipulation, etc., described above with reference to method500optionally have one or more of the characteristics of the first XR content, the second XR content, the third XR content, etc. described herein with reference to other methods described herein (e.g., method200). For brevity, these details are not repeated here.

FIG.6is a block diagram of a device600(e.g., a server system) in accordance with some implementations. In some implementations, the device600implements the controller402and/or the electronic device404shown inFIG.4. While certain specific features are illustrated, those of ordinary skill in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity, and so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, as a non-limiting example, in some implementations the device600may include one or more processing units (CPUs)602, a network interface604, a programming interface606, a memory608, and one or more communication buses610for interconnecting these and various other components.

In some implementations, the network interface604may be provided to, among other uses, establish and maintain a metadata tunnel between a cloud hosted network management system and at least one private network including one or more compliant devices. In some implementations, the one or more communication buses610may include circuitry that interconnects and controls communications between system components. The memory608may include high-speed random access memory, such as DRAM, SRAM, DDR RAM, and/or other random access solid state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, and/or other non-volatile solid state storage devices. The memory608may include one or more storage devices remotely located from the one or more CPUs602. The memory608may include a non-transitory computer readable storage medium.

In some implementations, the memory608or the non-transitory computer readable storage medium of the memory608may store one or more of the following programs, modules and data structures, or a subset thereof, which may include an operating system612, a data obtainer614, a physical element detector616, an XR content generator618, an XR content renderer620, and/or an affordance engine622. The data obtainer614may include instructions614aand/or heuristics and metadata614bfor obtaining data, for example, from an environmental sensor. The physical element detector616may include instructions616aand/or heuristics and metadata616bfor detecting physical elements based on the obtained data. The XR content generator618may include instructions618aand/or heuristics and metadata618bfor generating XR content based on the detected physical elements. The XR content renderer620may include instructions620aand/or heuristics and metadata620bfor displaying or rendering the generated XR content. The affordance engine622may include instructions622aand/or heuristics and metadata622bfor compositing an affordance and/or detecting an input directed to an affordance.