Tailoring a computer-generated reality experience based on a recognized object

In various implementations, a method is performed at a device including a non-transitory memory and one or more processors coupled with the non-transitory memory. In some implementations, the method includes detecting a representation of an object that is associated with computer-generated reality (CGR) content. In some implementations, the method includes obtaining a user environment map characterizing a user environment, wherein the user environment is limited by a volumetric region around the device. In some implementations, the method includes mapping a portion of the CGR content associated with the object to a portion of the user environment map. In some implementations, the method includes synthesizing a CGR environment in accordance with the mapping.

TECHNICAL FIELD

The present disclosure generally relates to computer-generated reality (CGR), and in particular, to systems, methods, and devices for tailoring a CGR experience based on a recognized object.

BACKGROUND

Virtual reality (VR) and augmented reality (AR) are becoming more popular due to their remarkable ability to alter a user's perception of the world. For example, VR and AR are used for learning purposes, gaming purposes, content creation purposes, social media and interaction purposes, or the like. These technologies differ in the user's perception of his/her presence. VR transposes the user into a virtual space so their VR perception is different from his/her real-world perception. In contrast, AR takes the user's real-world perception and adds something to it.

These technologies are becoming more commonplace due to, for example, miniaturization of hardware components, improvements to hardware performance, and improvements to software efficiency. As one example, a user may experience VR content by using a head-mounted device (HMD) that encloses the user's field-of-view and is tethered to a computer. As another example, a user may experience AR content by wearing an HMD that still allows the user to see his/her surroundings (e.g., glasses with optical see-through). As yet another example, a user may experience AR content superimposed on a live video feed of the user's environment on a handheld display (e.g., an AR-enabled mobile phone or tablet).

SUMMARY

Various implementations disclosed herein include devices, systems, and methods for synthesizing a CGR environment. In various implementations, a method is performed at a device including a non-transitory memory and one or more processors coupled with the non-transitory memory. In some implementations, the method includes detecting a representation of an object that is associated with computer-generated reality (CGR) content. In some implementations, the method includes obtaining a user environment map characterizing a user environment, wherein the user environment is bounded by depth information characterizing a volumetric region around the device. In some implementations, the method includes mapping a portion of the CGR content associated with the object to a portion of the user environment map. In some implementations, the method includes synthesizing a CGR environment in accordance with the mapping.

DESCRIPTION

Examples of CGR include virtual reality and mixed reality.

Various implementations described herein provide methods and devices for tailoring a computer-generated reality (CGR) experience based on a recognized object. For example, CGR content associated with the recognized object is included in the CGR experience in order to make the CGR experience more relevant to the recognized object. Skinning the CGR experience to include CGR content associated with an object makes the CGR experience appear more relevant to the recognized object.

FIG. 1is a block diagram of an example operating environment100in accordance with some implementations. While pertinent features are shown, those of ordinary skill in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity and so as not to obscure more pertinent aspects of the example implementations disclosed herein. To that end, as a non-limiting example, the operating environment100includes a controller110, a computer-generated reality (CGR) device120that presents a CGR experience, and an optional display device130.

In some implementations, the controller110is configured to manage and coordinate a CGR experience for a user150. In some implementations, the controller110includes a suitable combination of software, firmware, and/or hardware. The controller110is described in greater detail below with respect toFIG. 2. In some implementations, the controller110is a computing device that is local or remote relative to a user environment105where the user150is located. For example, the controller110is a local server located within the user environment105. In another example, the controller110is a remote server located outside of the user environment105(e.g., a cloud server, central server, etc.).

In some implementations, the controller110is communicatively coupled with the CGR device120via one or more wired or wireless communication channels144(e.g., BLUETOOTH, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In some implementations, the controller110is communicatively coupled with the display device130via one or more wired or wireless communication channels142(e.g., BLUETOOTH, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In some implementations, the CGR device120is communicatively coupled with the display device130via one or more wired or wireless communication channels146(e.g., BLUETOOTH, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.).

In some implementations, the CGR device120corresponds to a head-mounted device (HMD), tablet, mobile phone, wearable computing device, or the like. In some implementations, the CGR device120is configured to present a CGR experience to the user150. In some implementations, the CGR device120includes a suitable combination of software, firmware, and/or hardware. The CGR device120is described in greater detail below with respect toFIG. 3. In some implementations, the functionalities of the controller110and/or the display device130are provided by and/or combined with the CGR device120.

According to some implementations, the CGR device120presents a CGR experience to the user150while the user150is virtually and/or physically present within the user environment105. In some implementations, presenting a CGR experience includes presenting an AR experience. In some implementations, while presenting the AR experience, the CGR device120is configured to present AR content and to enable video pass-through of the user environment105(e.g., the CGR device120corresponds to an AR-enabled mobile phone or tablet). In some implementations, while presenting an AR experience, the CGR device120is configured to present AR content and to enable optical see-through of the user environment105(e.g., the CGR device120corresponds to an AR-enabled glasses).

In some implementations, presenting a CGR experience includes presenting a VR experience. In some implementations, while presenting a VR experience, the CGR device120is configured to present VR content and to optionally enable video pass-through of the user environment105(e.g., the CGR device120corresponds to a VR-enabled HMD). As shown inFIG. 1, for example, the user environment105includes chairs162aand162b, a credenza164, a coffee table166, a sofa168, end tables170aand170b, a door172, and a painting174. As shown inFIG. 1, the user150is standing behind the sofa168facing the display device130.

In some implementations, the user150wears the CGR device120on his/her head (e.g., as shown inFIG. 1). As such, the CGR device120includes one or more CGR displays provided to display the CGR content. For example, the CGR device120encloses the field-of-view of the user150. In some implementations, the CGR device120is replaced with a CGR chamber, enclosure, or room configured to present CGR content in which the user150does not wear the CGR device120. In some implementations, the user150holds the CGR device120in his/her hand(s).

In some implementations, the optional display device130is configured to present media content (e.g., video and/or audio content) to the user150. In some implementations, the display device130corresponds to a television (TV) or a computing device such as a desktop computer, kiosk, laptop computer, tablet, mobile phone, wearable computing device, or the like. In some implementations, the display device130includes a suitable combination of software, firmware, and/or hardware. The display device130is described in greater detail below with respect toFIG. 4.

The memory220includes high-speed random-access memory, such as dynamic random-access memory (DRAM), static random-access memory (SRAM), double-data-rate random-access memory (DDR RAM), or other random-access solid-state memory devices. In some implementations, the memory220includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory220optionally includes one or more storage devices remotely located from the one or more processing units202. The memory220comprises a non-transitory computer readable storage medium. In some implementations, the memory220or the non-transitory computer readable storage medium of the memory220stores the following programs, modules and data structures, or a subset thereof including an optional operating system230and a CGR experience engine240.

The operating system230includes procedures for handling various basic system services and for performing hardware dependent tasks. In some implementations, the CGR experience engine240is configured to manage and coordinate one or more CGR experiences for one or more users (e.g., a single CGR experience for one or more users, or multiple CGR experiences for respective groups of one or more users). To that end, in various implementations, the CGR experience engine240includes a data obtainer242, a mapper and locator engine244, a plane detector245, a CGR content obtainer246, a CGR content manager248, and a data transmitter250.

In some implementations, the data obtainer242is configured to obtain data (e.g., presentation data, user interaction data, sensor data, location data, etc.) from at least one of sensors in the user environment105, sensors associated with the controller110, the CGR device120, and the display device130. For example, the data obtainer242obtains sensor data from the CGR device120that includes image data from external facing image sensors of the CGR device120, wherein the image data corresponds to images or a video stream capturing the user environment105. To that end, in various implementations, the data obtainer242includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the object recognizer243recognizes an object in the user environment105. In some implementations, the object recognizer243detects a representation of the object in the data obtained by the data obtainer242. In some implementations, the object recognizer243recognizes objects that are associated with CGR content. For example, in some implementations, the object recognizer243identifies all objects in the user environment105, and filters out objects that are not associated with any CGR content. In some implementations, the object recognizer243recognizes multiple objects with CGR content, and selects the object with the most CGR content. In some implementations, the object recognizer243selects the object with the most recent CGR content. In some implementations, the object recognizer243recognizes a real-world object that is present in the user environment105. In some implementations, the object recognizer243recognizes a virtual object that is present in the user environment105. Referring to the example ofFIG. 1, the object recognizer243detects a representation of the painting174in the data obtained by the data obtainer242because the painting174is associated with CGR content. To that end, in various implementations, the object recognizer243includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the mapper and locator engine244is configured to obtain a user environment map that characterizes the user environment105. For example, in some implementations, the mapper and locator engine244generates the user environment map based on the data obtained by the data obtainer242. In some implementations, the mapper and locator engine244tracks the position/location of the CGR device120or the user150with respect to the user environment105. In some implementations, the user150is physically present in the user environment105. In such implementations, the mapper and locator engine244synthesizes a mesh map of the user environment105based on locality data (e.g., sensor data characterizing the user environment105) from at least one of sensors in the user environment105, sensors associated with the controller110, the CGR device120, and the display device130. In some implementations, the user150is virtually present in the user environment105. In such implementations, the mapper and locator engine244synthesizes the user environment map by identifying boundaries of the user environment105and objects that are present in the user environment105.

In some implementations, the mapper and locator engine244is also configured to determine the location and orientation of the CGR device120or the user150relative to one or more reference points (e.g., an object) in the user environment105(e.g., the center of mass of the object or another point) or the user environment map of the user environment105. According to some implementations, the mapper and locator engine244determines the orientation and location of the CGR device120based on one or more known localization techniques. To that end, in various implementations, the mapper and locator engine244includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the plane detector245is configured to detect planes (e.g., horizontal, vertical, or angled) within the user environment map. In some implementations, the plane detector245detects planes onto which CGR content can be displayed. In some implementations, the plane detector245identifies all planes, and filters out planes that are not suitable for displaying CGR content (e.g., the plane detector245filters out planes that have less area than a threshold area). According to some implementations, the plane detector245detects the planes based on one or more known localization techniques. In some implementations, the plane detector245is also configured to filter planes that do not satisfy spatial criteria (e.g., planes that are smaller than a threshold size). For example, in some implementations, the plane detector245filters out planes based on the CGR content associated with the object recognized by the object recognizer243. For example, the plane detector245filters out planes that are not suitable (e.g., too small or too large) for displaying the CGR content associated with the recognized object. Referring to the example ofFIG. 1, the plane detector245filters out planes that are not suitable for displaying CGR content associated with the painting174. To that end, in various implementations, the plane detector245includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the CGR content obtainer246is configured to obtain (e.g., receive, retrieve, or generate) CGR content associated with the object recognized by the object recognizer243. In some implementations, the CGR content obtainer246obtains the CGR content associated with the recognized object from a datastore that stores CGR content for various objects. Referring to the example ofFIG. 1, the CGR content obtainer246obtains CGR content associated with the painting174. To that end, in various implementations, the CGR content obtainer246includes instructions and/or logic therefor, and heuristics and metadata therefor.

In various implementations, the CGR content manager248maps a portion of the CGR content associated with the object to a portion of the user environment map. In some implementations, the CGR content manager248is configured to select CGR content based on the user environment map. In some implementations, the CGR content manager248selects the CGR content based on the plane detected within the user environment map. For example, the CGR content manager248selects the CGR content based on the user's location and orientation relative to the user environment map and/or the surface area of the planes detected within the user environment map. In some implementations, the CGR content manager248is also configured to manage and coordinate the presentation of the CGR content as the user's orientation and location changes relative to the user environment105or the user interacts with the CGR content. Referring to the example ofFIG. 1, the CGR content manager248maps a portion of the CGR content associated with the painting174to the user environment map. To that end, in various implementations, the CGR content manager248includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the CGR environment synthesizer250synthesizes a CGR environment in accordance with the mapping performed by the CGR content manager248. In some implementations, the CGR environment synthesizer250composites the CGR content selected by the CGR content manager248with the user environment map. In some implementations, the CGR environment synthesizer250modifies an existing CGR environment to include the CGR content selected by the CGR content manager248. Referring to the example ofFIG. 1, the CGR environment synthesizer250synthesizes a CGR environment that includes at least a portion of the CGR content associated with the painting174. To that end, in various implementations, the CGR environment synthesizer250includes instructions and/or logic therefor, and heuristics and metadata therefor.

Although the data obtainer242, the object recognizer243, the mapper and locator engine244, the plane detector245, the CGR content obtainer246, the CGR content manager248, and the CGR environment synthesizer250are shown as residing on a single device (e.g., the controller110), it should be understood that in other implementations, any combination of the data obtainer242, the object recognizer243, the mapper and locator engine244, the plane detector245, the CGR content obtainer246, the CGR content manager248, and the CGR environment synthesizer250may be located in separate computing devices.

FIG. 3is a block diagram of an example of the CGR device120(e.g., an HMD, mobile phone, or tablet) in accordance with some implementations. While certain specific features are illustrated, those skilled in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity, and so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, as a non-limiting example, in some implementations, the CGR device120includes one or more processing units302(e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, and/or the like), one or more input/output (I/O) devices and sensors306, one or more communication interfaces308(e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, and/or the like type interface), one or more programming (e.g., I/O) interfaces310, one or more CGR displays312, one or more optional interior- and/or exterior-facing image sensors314, one or more optional depth sensors316, a memory320, and one or more communication buses304for interconnecting these and various other components.

In some implementations, the one or more communication buses304include circuitry that interconnects and controls communications between system components. In some implementations, the one or more I/O devices and sensors306include at least one of an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., blood pressure monitor, heart rate monitor, blood oxygen sensor, blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptics engine, a heating and/or cooling unit, a skin shear engine, and/or the like.

In some implementations, the one or more CGR displays312are configured to present the CGR experience to the user. In some implementations, the one or more CGR displays312are also configured to present flat video content to the user (e.g., a 2-dimensional or “flat” AVI, FLV, WMV, MOV, MP4, or the like file associated with a TV episode or a movie, or live video pass-through of the user environment105). In some implementations, the one or more CGR displays312correspond to holographic, digital light processing (DLP), liquid-crystal display (LCD), liquid-crystal on silicon (LCoS), organic light-emitting field-effect transitory (OLET), organic light-emitting diode (OLED), surface-conduction electron-emitter display (SED), field-emission display (FED), quantum-dot light-emitting diode (QD-LED), micro-electro-mechanical system (MEMS), and/or the like display types. In some implementations, the one or more CGR displays312correspond to diffractive, reflective, polarized, holographic, etc. waveguide displays. For example, the CGR device120includes a single CGR display. In another example, the CGR device120includes an CGR display for each eye of the user. In some implementations, the one or more CGR displays312are capable of presenting CGR content (e.g., AR, VR and/or MR content).

In some implementations, the one or more optional image sensors314are configured to obtain image data that corresponds to at least a portion of the face of the user that includes the eyes of the user. For example, the one or more optional image sensors314correspond to one or more RGB cameras (e.g., with a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), infrared (IR) image sensors, event-based cameras, and/or the like.

In some implementations, the one or more optional depth sensors316are configured to obtain depth data that corresponds to at least a portion of the face of the user and to synthesize a depth/mesh map of the face of the user, where the mesh map characterizes the facial topography of the user. For example, the one or more optional depth sensors316correspond to a structured light device, a time-of-flight device, and/or the like.

The memory320includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices. In some implementations, the memory320includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory320optionally includes one or more storage devices remotely located from the one or more processing units302. The memory320comprises a non-transitory computer readable storage medium. In some implementations, the memory320or the non-transitory computer readable storage medium of the memory320stores the following programs, modules and data structures, or a subset thereof including an optional operating system330and a CGR presentation engine340.

The operating system330includes procedures for handling various basic system services and for performing hardware dependent tasks. In some implementations, the CGR presentation engine340is configured to present CGR content to the user via the one or more CGR displays312. To that end, in various implementations, the CGR presentation engine340includes a data obtainer342, a CGR presenter344, a user interaction handler346, and a data transmitter350.

In some implementations, the data obtainer342is configured to obtain data (e.g., presentation data, user interaction data, sensor data, location data, etc.) from at least one of sensors in the user environment105, sensors associated with the CGR device120, the controller110, and the display device130. In some implementations, the data obtainer342obtains data indicative of a CGR environment synthesized by the controller110. For example, the data obtainer342obtains data corresponding to the CGR environment synthesized by the CGR environment synthesizer250shown inFIG. 2. Referring to the example ofFIG. 1, in some implementations, the data obtainer342obtains data corresponding to a CGR environment that includes at least a portion of the CGR content associated with the painting174. To that end, in various implementations, the data obtainer342includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the CGR presenter344is configured to present a CGR environment via the one or more CGR displays312. In some implementations, the CGR presenter344presents the CGR environment synthesized by the CGR environment synthesizer250shown inFIG. 2. Referring to the example ofFIG. 1, the CGR presenter344presents a CGR environment that includes CGR content associated with the painting174. In some implementations, the CGR presenter344is also configured to present flat video content via the one or more CGR displays312. To that end, in various implementations, the CGR presenter344includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the user interaction handler346is configured to detect and interpret user interactions with the presented CGR content. For example, in some implementations, the CGR environment includes an affordance for presenting CGR content associated with an object detected in the user environment105. In such implementations, the user interaction handler346detects a user input selecting the affordance, and the CGR presenter344presents the CGR content associated with the detected object in response to detecting the user input selecting the affordance. Referring to the example ofFIG. 1, the user interaction handler346detects a selection of an affordance corresponding to a request to display CGR content associated with the painting174, and the CGR presenter344presents the CGR content associated with the painting174in response to detecting the user input selecting the affordance. To that end, in various implementations, the user interaction handler346includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the data transmitter350is configured to transmit data (e.g., presentation data, location data, user interaction data, etc.) to at least one of the controller110and the display device130. In some implementations, the CGR device120transmits images captured by a front-facing camera to the controller110. To that end, in various implementations, the data transmitter350includes instructions and/or logic therefor, and heuristics and metadata therefor.

Although the data obtainer342, the CGR presenter344, the user interaction handler346, and the data transmitter350are shown as residing on a single device (e.g., the CGR device120), it should be understood that in other implementations, any combination of the data obtainer342, the CGR presenter344, the user interaction handler346, and the data transmitter350may be located in separate computing devices.

In some implementations, the CGR device120includes a head-mountable device (HMD) that is worn by the user150. In some implementations, the HMD includes an integrated display (e.g., a built-in display) that displays a CGR environment. In some implementations, the HMD includes a head-mountable enclosure. In various implementations, the head-mountable enclosure includes an attachment region to which another device with a display can be attached. For example, in some implementations, an electronic device (e.g., a smartphone or a tablet) can be attached to the head-mountable enclosure. In various implementations, the head-mountable enclosure is shaped to form a receptacle for receiving another device that includes a display (e.g., the electronic device). For example, in some implementations, an electronic device slides/snaps into or otherwise attaches to the head-mountable enclosure. In some implementations, the display of the device attached to the head-mountable enclosure presents (e.g., displays) the CGR environment. In various implementations, examples of the electronic device include smartphones, tablets, media players, laptops, etc.

FIG. 4is a block diagram of an example of the optional display device130(e.g., a television (TV) or other display within the user environment105) in accordance with some implementations. While certain specific features are illustrated, those skilled in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity, and so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, as a non-limiting example, in some implementations the display device130includes one or more processing units402(e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, and/or the like), one or more input/output (I/O) devices and sensors406, one or more communication interfaces408(e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, and/or the like type interface), one or more programming (e.g., I/O) interfaces410, a display412, a memory420, and one or more communication buses404for interconnecting these and various other components. In some implementations, the display device130is optionally controlled by a remote-control device, voice commands, the CGR device120, or the like.

In some implementations, the one or more communication buses404include circuitry that interconnects and controls communications between system components. In some implementations, the one or more I/O devices and sensors406include at least one of one or more IR sensors, one or more physical buttons, one or more microphones, one or more speakers, one or more image sensors, one or more depth sensors, and/or the like.

The memory420includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices. In some implementations, the memory420includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory420optionally includes one or more storage devices remotely located from the one or more processing units402. The memory420comprises a non-transitory computer readable storage medium. In some implementations, the memory420or the non-transitory computer readable storage medium of the memory420stores the following programs, modules and data structures, or a subset thereof including an optional operating system430and a presentation engine440.

The operating system430includes procedures for handling various basic system services and for performing hardware dependent tasks. In some implementations, the presentation engine440is configured to present media content (e.g., video and/or audio content) to users via the display412and the one or more I/O devices and sensors406(e.g., one or more speakers). To that end, in various implementations, the presentation engine440includes a data obtainer442, a content presenter444, an interaction handler446, and a data transmitter450.

In some implementations, the data obtainer442is configured to obtain data (e.g., presentation data, user interaction data, etc.) from at least one of sensors in the operating environment105, sensors associated with the display device130, the controller110, and the CGR device120. In some implementations, the data obtainer442obtains data corresponding to the CGR environment synthesized by the CGR environment synthesizer250shown inFIG. 2. Referring to the example ofFIG. 1, the data obtainer442obtains data corresponding to a CGR environment that includes CGR content associated with the painting174. To that end, in various implementations, the data obtainer442includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the content presenter444is configured to render and/display video content via the display412. In some implementations, the content presenter444renders at least a portion of the CGR environment obtained by the data obtainer442. Referring to the example ofFIG. 1, the content presenter444presents a CGR environment that includes CGR content associated with the painting174. To that end, in various implementations, the content presenter444includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the interaction handler446is configured to detect and interpret user interactions with the display device130(e.g., navigation, playback, tuning, volume adjustment, or the like commands). For example, in some implementations, the display device130displays an affordance that triggers modification of the CGR environment based on an object detected in the user environment105. In such implementations, the interaction handler446detects a user input selecting the affordance, and the content presenter444modifies the CGR environment by displaying CGR content associated with the detect object. Referring to the example ofFIG. 1, the interaction handler446detects a selection of an affordance corresponding to a request to display CGR content associated with the painting174, and the content presenter44displays CGR content associated with the painting174in response to detecting selection of the affordance. To that end, in various implementations, the interaction handler446includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the data transmitter450is configured to transmit data (e.g., presentation data, user interaction data, etc.) to at least one of the controller110and the CGR device120. In some implementations, the display device130includes a front-facing camera (e.g., a scene-facing camera), and the display device130transmits images captured by the front-facing camera to the controller110and/or the CGR device120. To that end, in various implementations, the data transmitter450includes instructions and/or logic therefor, and heuristics and metadata therefor.

Although the data obtainer442, the content presenter444, the interaction handler446, and the data transmitter450are shown as residing on a single device (e.g., the display device130), it should be understood that in other implementations, any combination of the data obtainer442, the content presenter444, the interaction handler446, and the data transmitter450may be located in separate computing devices.

FIG. 5illustrates an example CGR content presentation architecture500in accordance with some implementations. While pertinent features are shown, those of ordinary skill in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity and so as not to obscure more pertinent aspects of the example implementations disclosed herein. To that end, as a non-limiting example, the CGR content presentation architecture500detects a representation of an object that is associated with CGR content, obtains a user environment map that characterizes a user environment, maps a portion of the CGR content associated with the object to a portion of the user environment map, and synthesizes a CGR environment in accordance with the mapping.

As shown inFIG. 5, in some implementations, an object detector510(e.g., a portion of the object recognizer243shown inFIG. 2) detects a representation of an object that is associated with CGR content. In some implementations, the object detector510obtains user environment information505, and the object detector510detects the representation of the object in the user environment information505. In some implementations, the user environment information505includes one or more images of a user environment in which the user is physically or virtually present (e.g., the user environment105shown inFIG. 1). In some implementations, detected object information512provides an indication of the detected object.

In some implementations, a map obtainer520(e.g., a portion of the mapper and locator engine244and/or a portion of the plane detector245) obtains a user environment map522that characterizes the user environment. In some implementations, the map obtainer520generates the user environment map522based on the user environment information505. In some implementations, the user150is physically present in the user environment, and the map obtainer520generates a mesh map of the user environment. In some implementations, the user150is virtually present in the user environment, and the map obtainer520generates a map of the user environment that identifies the dimensions of the user environment and the virtual objects that are in the user environment.

In some implementations, a content mapper530(e.g., a portion of the CGR content obtainer246and/or a portion of the CGR content manager248) generates a content mapping532that maps at least a portion of the CGR content associated with the object to a portion of the user environment map522. In some implementations, the content mapping532indicates which of the CGR content associated with the object is to be displayed on which portion of the user environment.

In some implementations, a CGR environment synthesizer540(e.g., a portion of the CGR environment synthesizer250shown inFIG. 2) synthesizes a CGR environment542in accordance with the content mapping532. In some implementations, the CGR environment synthesizer540synthesizes the CGR environment542by compositing the CGR content associated with the object with the user environment map. In some implementations, the CGR environment synthesizer540provides the CGR environment542to a CGR presentation pipeline (e.g., to the CGR presenter344shown inFIG. 3).

FIGS. 6A-6Cillustrate an example CGR presentation scenario600in accordance with some implementations. While pertinent features are shown, those of ordinary skill in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity and so as not to obscure more pertinent aspects of the example implementations disclosed herein.

As shown inFIG. 6A, the user environment105includes the chairs162aand162b, the credenza164, the coffee table166, the sofa168, the end tables170aand170b, the door172and the painting174. As shown inFIG. 6A, the user150is standing behind the sofa168facing the display device130while wearing the CGR device120on his/her head. For example, the CGR device120corresponds to AR-enabled HMD (e.g., glasses, goggles, or the like) with optical see-through of the user environment105.

As shown inFIG. 6A, in state625(e.g., at time7), the user150is standing behind the sofa168. In some implementations, the CGR device120or the display device130displays a subtle (e.g., non-obtrusive) affordance or notification indicating that a CGR experience associated with the painting174is available. Continuing with this example, the controller110and/or the CGR device120detects a command issued by the user150to enter the CGR experience associated with the painting174(e.g., a voice command, gestural command, or the like). In response to detecting the command, for example, the controller110synthesizes a user environment map (e.g., the user environment map522shown inFIG. 5) of the user environment105. In some implementations, the controller100synthesizes a mesh map of the user environment105and detects planes within the mesh map.

As shown inFIG. 6B, in state650(e.g., at time T+1), the controller110identifies planes610a,610b,610c,610d,610e,610f,610g,610h, and610iwithin the user environment105. According to some implementations, the controller110filters planes that do not satisfy spatial criteria. For example, the planes610cand610dassociated with the chairs162aand162b, respectively, do not satisfy a dimensional criterion associated with the spatial criteria (e.g., less than M×N cm2or Y cm2). In other words, the surface area of the planes610cand610dis too small for the placement of CGR content. For example, the planes610fand610gdo not satisfy a line-of-sight criterion associated with the spatial criteria (e.g., more than Z degrees from the focal point of the user150). In other words, the location of the planes610fand610gis too low relative to the focal point of the user150. For example, the plane610hdoes not satisfy a personal radius criterion associated with the spatial criteria (e.g., less than Q cm from the user150). In other words, the plane610his too close to the user150. As such, planes610a,610b,610e, and610isatisfy the spatial criteria.

As shown inFIG. 6C, in state675(e.g., at time T+2), the CGR device120presents CGR content620a(e.g., a video about the artist who created the painting174) on the plane610a, CGR content620bon the plane610b(e.g., comments from famous art critics about the painting174), CGR content620c(e.g., a virtual cover for the credenza164that matches the painting174) on the plane610e, and the CGR content620d(e.g., virtual decorations for the coffee table166that match the painting174) on the plane610i. In some implementations, the CGR content620a,620b,620c, and620dis planar or volumetric. According to some implementations, the controller110selects CGR content associated with the painting174for the detected planes that meet the spatial criteria based on the detected planes and the orientation/location of the user150relative to the user environment map.

FIG. 7is a flowchart representation of a method700of tailoring a CGR experience in accordance with some implementations. In various implementations, the method700is performed by a device with non-transitory memory and one or more processors coupled with the non-transitory memory (e.g., the controller110inFIGS. 1-2, the CGR device120inFIGS. 1 and 3, or a suitable combination thereof). In some implementations, the method700is performed by processing logic, including hardware, firmware, software, or a combination thereof. In some implementations, the method700is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory). Briefly, in some circumstances, the method700includes detecting a representation of an object associated with CGR content, obtaining a user environment map characterizing a user environment, mapping a portion of the CGR content to a portion of the user environment map, and synthesizing a CGR environment in accordance with the mapping.

As represented by block7-1, the method700includes detecting a representation of an object that is associated with CGR content. In some implementations, the method700includes detecting a representation of a real-world object. In some implementations, the method700includes obtaining an image captured by an image sensor, and detecting the representation of the object in the image. In some implementations, the method700includes detecting a representation of a virtual object. In some implementations, the method700includes displaying representations of a plurality of virtual objects, and detecting a selection of a representation of a first virtual object of the plurality of virtual objects.

As represented by block7-2, the method700includes obtaining a user environment map characterizing a user environment. In some implementations, the user environment is bounded by depth information characterizing a volumetric region around the device. In some implementations, the user environment includes a physical environment. As represented by block7-2a, in some implementations, the method700includes synthesizing a mesh map of a geographical area surround the user. In some implementations, the user environment includes virtual environment (e.g., a CGR environment). As represented by block7-2b, in some implementations, the method700includes generating a map of a CGR environment associated with the user (e.g., generating a map of the virtual environment).

As represented by block7-3, the method700includes mapping a portion of the CGR content associated with the object to a portion of the user environment map. As represented by block7-3a, in some implementations, the method700includes identifying an area of the user environment map for displaying the CGR content associated with the object. As represented by block7-3b, in some implementations, the method700includes selecting a particular CGR content item from a plurality of CGR content items based on the user environment map.

As represented by block7-4, the method700includes synthesizing a CGR environment in accordance with the mapping. As represented by block7-4a, in some implementations, the method700includes compositing at least a portion of the CGR content with the user environment map. As represented by block7-4b, in some implementations, the method700includes modifying an existing CGR environment to include the CGR content associated with the object.

In some implementations, the method700includes presenting (e.g., displaying) the CGR content associated with the object. In some implementations, the method700includes displaying the portion of the CGR content in order to occlude at least a portion of a visual presentation of the user environment. In some implementations, the visual presentation of the user environment includes an optical see-through of the user environment. In some implementations, the visual presentation of the user environment includes a video pass-through of the user environment.

In some implementations, the method700includes displaying an affordance to modify a visual presentation of the user environment based on the CGR environment associated with the object. In some implementations, the method700includes detecting a selection of the affordance. In some implementations, the method700includes synthesizing the CGR environment in response to detecting the selection of the affordance.