Patent Description:
Virtual reality (VR) and augmented reality (AR) are becoming more popular due to their remarkable ability to alter a user's perception of the world. For example, VR and AR are used for learning purposes, gaming purposes, content creation purposes, social media and interaction purposes, or the like. These technologies differ in the user's perception of his/her presence. VR transposes the user into a virtual space so their VR perception is different from his/her real-world perception. In contrast, AR takes the user's real-world perception and adds something to it.

These technologies are becoming more commonplace due to, for example, miniaturization of hardware components, improvements to hardware performance, and improvements to software efficiency. As one example, a user may experience AR content superimposed on a live video feed of the user's setting on a handheld display (e.g., an AR-enabled mobile phone or tablet with video pass-through). As another example, a user may experience AR content by wearing a head-mounted device (HMD) or head-mounted enclosure that still allows the user to see his/her surroundings (e.g., glasses with optical see-through). As yet another example, a user may experience VR content by using an HMD that encloses the user's field-of-view and is tethered to a computer.

<CIT> discloses a head-mounted display (HMD) device configured to display an augmented reality image at a display position at an at least partially see-through display of the HMD device.

<CIT> discloses a method and apparatus for displaying augmented reality contents.

<NPL>, XP058043019 discloses a proof-of-concept system that augments the area surrounding a television with projected visualizations to enhance traditional gaming experiences.

In accordance with common practice the various features illustrated in the drawings may not be drawn to scale. Accordingly, the dimensions of the various features may be arbitrarily expanded or reduced for clarity. In addition, some of the drawings may not depict all of the components of a given system, method or device. Finally, like reference numerals may be used to denote like features throughout the specification and figures.

Various implementations disclosed herein include devices, systems, and methods for presenting synthesized reality (SR) content. According to some implementations, the method is performed at a device including non-transitory memory and one or more processors coupled with the non-transitory memory.

Numerous details are described in order to provide a thorough understanding of the example implementations shown in the drawings. However, the drawings merely show some example aspects of the present disclosure and are therefore not to be considered limiting. Those of ordinary skill in the art will appreciate that other effective aspects and/or variants do not include all of the specific details described herein. Moreover, well-known systems, methods, components, devices and circuits have not been described in exhaustive detail so as not to obscure more pertinent aspects of the example implementations described herein.

A physical setting refers to a world that individuals can sense and/or with which individuals can interact without assistance of electronic systems. Physical settings (e.g., a physical forest) include physical elements (e.g., physical trees, physical structures, and physical animals). Individuals can directly interact with and/or sense the physical setting, such as through touch, sight, smell, hearing, and taste.

In contrast, a synthesized reality (SR) setting refers to an entirely or partly computer-created setting that individuals can sense and/or with which individuals can interact via an electronic system. In SR, a subset of an individual's movements is monitored, and, responsive thereto, one or more attributes of one or more virtual objects in the SR setting is changed in a manner that conforms with one or more physical laws. For example, a SR system may detect an individual walking a few paces forward and, responsive thereto, adjust graphics and audio presented to the individual in a manner similar to how such scenery and sounds would change in a physical setting. Modifications to attribute(s) of virtual object(s) in a SR setting also may be made responsive to representations of movement (e.g., audio instructions).

An individual may interact with and/or sense a SR object using any one of his senses, including touch, smell, sight, taste, and sound. For example, an individual may interact with and/or sense aural objects that create a multi-dimensional (e.g., three dimensional) or spatial aural setting, and/or enable aural transparency. Multi-dimensional or spatial aural settings provide an individual with a perception of discrete aural sources in multi-dimensional space. Aural transparency selectively incorporates sounds from the physical setting, either with or without computer-created audio. In some SR settings, an individual may interact with and/or sense only aural objects.

One example of SR is virtual reality (VR). A VR setting refers to a simulated setting that is designed only to include computer-created sensory inputs for at least one of the senses. A VR setting includes multiple virtual objects with which an individual may interact and/or sense. An individual may interact and/or sense virtual objects in the VR setting through a simulation of a subset of the individual's actions within the computer-created setting, and/or through a simulation of the individual or his presence within the computer-created setting.

Another example of SR is mixed reality (MR). A MR setting refers to a simulated setting that is designed to integrate computer-created sensory inputs (e.g., virtual objects) with sensory inputs from the physical setting, or a representation thereof. On a reality spectrum, a mixed reality setting is between, and does not include, a VR setting at one end and an entirely physical setting at the other end.

In some MR settings, computer-created sensory inputs may adapt to changes in sensory inputs from the physical setting. Also, some electronic systems for presenting MR settings may monitor orientation and/or location with respect to the physical setting to enable interaction between virtual objects and real objects (which are physical elements from the physical setting or representations thereof). For example, a system may monitor movements so that a virtual plant appears stationery with respect to a physical building.

One example of mixed reality is augmented reality (AR). An AR setting refers to a simulated setting in which at least one virtual object is superimposed over a physical setting, or a representation thereof. For example, an electronic system may have an opaque display and at least one imaging sensor for capturing images or video of the physical setting, which are representations of the physical setting. The system combines the images or video with virtual objects, and displays the combination on the opaque display. An individual, using the system, views the physical setting indirectly via the images or video of the physical setting, and observes the virtual objects superimposed over the physical setting. When a system uses image sensor(s) to capture images of the physical setting, and presents the AR setting on the opaque display using those images, the displayed images are called a video pass-through. Alternatively, an electronic system for displaying an AR setting may have a transparent or semi-transparent display through which an individual may view the physical setting directly. The system may display virtual objects on the transparent or semi-transparent display, so that an individual, using the system, observes the virtual objects superimposed over the physical setting. In another example, a system may comprise a projection system that projects virtual objects into the physical setting. The virtual objects may be projected, for example, on a physical surface or as a holograph, so that an individual, using the system, observes the virtual objects superimposed over the physical setting.

An augmented reality setting also may refer to a simulated setting in which a representation of a physical setting is altered by computer-created sensory information. For example, a portion of a representation of a physical setting may be graphically altered (e.g., enlarged), such that the altered portion may still be representative of but not a faithfully-reproduced version of the originally captured image(s). As another example, in providing video pass-through, a system may alter at least one of the sensor images to impose a particular viewpoint different than the viewpoint captured by the image sensor(s). As an additional example, a representation of a physical setting may be altered by graphically obscuring or excluding portions thereof.

Another example of mixed reality is augmented virtuality (AV). An AV setting refers to a simulated setting in which a computer-created or virtual setting incorporates at least one sensory input from the physical setting. The sensory input(s) from the physical setting may be representations of at least one characteristic of the physical setting. For example, a virtual object may assume a color of a physical element captured by imaging sensor(s). In another example, a virtual object may exhibit characteristics consistent with actual weather conditions in the physical setting, as identified via imaging, weather-related sensors, and/or online weather data. In yet another example, an augmented reality forest may have virtual trees and structures, but the animals may have features that are accurately reproduced from images taken of physical animals.

Many electronic systems enable an individual to interact with and/or sense various SR settings. One example includes head mounted systems. A head mounted system may have an opaque display and speaker(s). Alternatively, a head mounted system may be designed to receive an external display (e.g., a smartphone). The head mounted system may have imaging sensor(s) and/or microphones for taking images/video and/or capturing audio of the physical setting, respectively. A head mounted system also may have a transparent or semi-transparent display. The transparent or semi-transparent display may incorporate a substrate through which light representative of images is directed to an individual's eyes. The display may incorporate LEDs, OLEDs, a digital light projector, a laser scanning light source, liquid crystal on silicon, or any combination of these technologies. The substrate through which the light is transmitted may be a light waveguide, optical combiner, optical reflector, holographic substrate, or any combination of these substrates. In one embodiment, the transparent or semi-transparent display may transition selectively between an opaque state and a transparent or semi-transparent state. In another example, the electronic system may be a projection-based system. A projection-based system may use retinal projection to project images onto an individual's retina. Alternatively, a projection system also may project virtual objects into a physical setting (e.g., onto a physical surface or as a holograph). Other examples of SR systems include heads up displays, automotive windshields with the ability to display graphics, windows with the ability to display graphics, lenses with the ability to display graphics, headphones or earphones, speaker arrangements, input mechanisms (e.g., controllers having or not having haptic feedback), tablets, smartphones, and desktop or laptop computers.

The implementations described herein provide methods and devices for tailoring a synthesized reality (SR) experience to a physical setting. For example, while a user is watching a movie in his/her living room on a television (TV), the user may wish to experience a more immersive version of the movie where portions of the user's living room may become part of the movie scenery. For example, based on the dimensions of the living room, the furniture within the living room, and the user's orientation/location within the living room, the SR content is overlaid on portions of the user's living room. In some implementations, the SR content corresponds to portions of the movie reconstructed in SR (e.g., background and peripheral scenery from the movie projected onto the walls and/or floor of the living room). In some implementations, the SR content corresponds to auxiliary SR content related to the movie (e.g., maps, graphs, educational information, or the like augmenting the movie). As such, in some implementations, the SR content associated with the movie "skins" at least a portion of the living room (e.g., an at-home holodeck). In some implementations, the SR content is <NUM>-dimensional (e.g., flat), volumetric, and/or a suitable combination thereof.

<FIG> is a block diagram of an example operating architecture 100A in accordance with some implementations. While pertinent features are shown, those of ordinary skill in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity and so as not to obscure more pertinent aspects of the example implementations disclosed herein. To that end, as a non-limiting example, the operating architecture 100A includes an electronic device <NUM> and an optional display device <NUM>.

In some implementations, the electronic device <NUM> is configured to present the SR experience to a user. In some implementations, the electronic device <NUM> includes a suitable combination of software, firmware, and/or hardware. The electronic device <NUM> is described in greater detail below with respect to <FIG>. According to some implementations, the electronic device <NUM> presents a synthesized reality (SR) experience to the user while the user is physically present within a physical setting <NUM> that includes a table <NUM> within the field-of-view <NUM> of the electronic device <NUM>. As such, in some implementations, the user holds the electronic device <NUM> in his/her hand(s). In some implementations, while presenting an augmented reality (AR) experience, the electronic device <NUM> is configured to present AR content (e.g., an AR cylinder <NUM>) and to enable video pass-through of the physical setting <NUM> (e.g., including the table <NUM>) on a display <NUM>.

In some implementations, the display device <NUM> is configured to present media content (e.g., video and/or audio content) to the user. In some implementations, the display device <NUM> corresponds to a television or a computing device such as a desktop computer, kiosk, laptop computer, tablet, mobile phone, wearable computing device, or the like. In some implementations, the display device <NUM> includes a suitable combination of software, firmware, and/or hardware. The display device <NUM> is described in greater detail below with respect to <FIG>.

<FIG> is a block diagram of an example physical setting 100B in accordance with some implementations. While pertinent features are shown, those of ordinary skill in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity and so as not to obscure more pertinent aspects of the example implementations disclosed herein. To that end, as a non-limiting example, the physical setting 100B includes a controller <NUM>, an electronic device <NUM>, and an optional display device <NUM>.

In some implementations, the controller <NUM> is configured to manage and coordinate an SR experience for the user. In some implementations, the controller <NUM> includes a suitable combination of software, firmware, and/or hardware. The controller <NUM> is described in greater detail below with respect to <FIG>. In some implementations, the controller <NUM> is a computing device that is local or remote relative to the physical setting <NUM>. For example, the controller <NUM> is a local server located within the physical setting <NUM>. In another example, the controller <NUM> is a remote server located outside of the physical setting <NUM> (e.g., a cloud server, central server, etc.).

In some implementations, the controller <NUM> is communicatively coupled with the electronic device <NUM> via one or more wired or wireless communication channels <NUM> (e.g., BLUETOOTH, IEEE <NUM>. 11x, IEEE <NUM>. 16x, IEEE <NUM>. In some implementations, the controller <NUM> is communicatively coupled with the display device <NUM> via one or more wired or wireless communication channels <NUM> (e.g., BLUETOOTH, IEEE <NUM>. 11x, IEEE <NUM>. 16x, IEEE <NUM>. In some implementations, the electronic device <NUM> is communicatively coupled with the display device <NUM> via one or more wired or wireless communication channels <NUM> (e.g., BLUETOOTH, IEEE <NUM>. 11x, IEEE <NUM>. 16x, IEEE <NUM>.

In some implementations, the electronic device <NUM> is configured to present the SR experience to the user <NUM>. In some implementations, the electronic device <NUM> includes a suitable combination of software, firmware, and/or hardware. The electronic device <NUM> is described in greater detail below with respect to <FIG>. In some implementations, the functionalities of the controller <NUM> and/or the display device <NUM> are provided by and/or combined with the electronic device <NUM>.

According to some implementations, the electronic device <NUM> presents an SR experience to the user <NUM> while the user <NUM> is virtually and/or physically present within the physical setting <NUM>. In some implementations, while presenting an augmented reality (AR) experience, the electronic device <NUM> is configured to present AR content and to enable optical see-through of the physical setting <NUM> (e.g., the electronic device <NUM> corresponds to an AR-enabled glasses). In some implementations, while presenting a virtual reality (VR) experience, the electronic device <NUM> is configured to present VR content and to optionally enable video pass-through of the physical setting <NUM> (e.g., the electronic device <NUM> corresponds to a VR-enabled HMD). As shown in <FIG>, for example, the physical setting <NUM> includes chairs 162a and 162b, credenza <NUM>, coffee table <NUM>, sofa <NUM>, end tables 170a and 170b, and a door <NUM>. As shown in <FIG>, the user <NUM> is standing behind the sofa <NUM> facing the display device <NUM>.

In some implementations, the user <NUM> wears the electronic device <NUM> on his/her head such as a head-mounted device (HMD). As such, the electronic device <NUM> includes one or more displays provided to display the SR content. For example, the electronic device <NUM> encloses the field-of-view of the user <NUM>. As another example, the electronic device <NUM> slides into or otherwise attaches to a head mounted enclosure. In some implementations, the electronic device <NUM> is replaced with an SR chamber, enclosure, or room configured to present SR content in which the user <NUM> does not wear the electronic device <NUM>. In some implementations, the user <NUM> holds the electronic device <NUM> in his/her hand(s).

In some implementations, the optional display device <NUM> is configured to present media content (e.g., video and/or audio content) to the user <NUM>. In some implementations, the display device <NUM> corresponds to a television (TV) or a computing device such as a desktop computer, kiosk, laptop computer, tablet, mobile phone, wearable computing device, or the like. In some implementations, the display device <NUM> includes a suitable combination of software, firmware, and/or hardware. The display device <NUM> is described in greater detail below with respect to <FIG>.

<FIG> is a block diagram of an example of the controller <NUM> in accordance with some implementations. While certain specific features are illustrated, those skilled in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity, and so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, as a non-limiting example, in some implementations, the controller <NUM> includes one or more processing units <NUM> (e.g., microprocessors, application-specific integrated-circuits (ASICs), field-programmable gate arrays (FPGAs), graphics processing units (GPUs), central processing units (CPUs), processing cores, and/or the like), one or more input/output (I/O) devices <NUM>, one or more communication interfaces <NUM> (e.g., universal serial bus (USB), IEEE <NUM>. 3x, IEEE <NUM>. 11x, IEEE <NUM>. 16x, global system for mobile communications (GSM), code division multiple access (CDMA), time division multiple access (TDMA), global positioning system (GPS), infrared (IR), BLUETOOTH, ZIGBEE, and/or the like type interface), one or more programming (e.g., I/O) interfaces <NUM>, a memory <NUM>, and one or more communication buses <NUM> for interconnecting these and various other components.

In some implementations, the one or more communication buses <NUM> include circuitry that interconnects and controls communications between system components. In some implementations, the one or more I/O devices <NUM> include at least one of a keyboard, a mouse, a touchpad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, and/or the like.

The memory <NUM> includes high-speed random-access memory, such as dynamic random-access memory (DRAM), static random-access memory (SRAM), double-data-rate random-access memory (DDR RAM), or other random-access solid-state memory devices. In some implementations, the memory <NUM> includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory <NUM> optionally includes one or more storage devices remotely located from the one or more processing units <NUM>. The memory <NUM> comprises a non-transitory computer readable storage medium. In some implementations, the memory <NUM> or the non-transitory computer readable storage medium of the memory <NUM> stores the following programs, modules and data struc2tures, or a subset thereof including an optional operating system <NUM> and a synthesized reality (SR) experience engine <NUM>.

The operating system <NUM> includes procedures for handling various basic system services and for performing hardware dependent tasks. In some implementations, the SR experience engine <NUM> is configured to manage and coordinate one or more SR experiences for one or more users (e.g., a single SR experience for one or more users, or multiple SR experiences for respective groups of one or more users). To that end, in various implementations, the SR experience engine <NUM> includes a data obtainer <NUM>, a mapper and locator engine <NUM>, a plane detector <NUM>, an SR content obtainer <NUM>, an SR content manager <NUM>, and a data transmitter <NUM>.

In some implementations, the data obtainer <NUM> is configured to obtain data (e.g., presentation data, user interaction data, sensor data, location data, etc.) from at least one of sensors in the physical setting <NUM>, sensors associated with the controller <NUM>, the electronic device <NUM>, and the display device <NUM>. For example, the data obtainer <NUM> obtains sensor data from the electronic device <NUM> that includes image data from external facing image sensors of the electronic device <NUM>, wherein the image data corresponds to images or a video stream capturing the physical setting <NUM>. To that end, in various implementations, the data obtainer <NUM> includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the mapper and locator engine <NUM> is configured to map the physical setting <NUM> and to track the position/location of the electronic device <NUM> or the user <NUM> with respect to the physical setting <NUM>. As such, in some implementations, the mapper and locator engine <NUM> is configured to synthesize a mesh map of the physical setting <NUM> based on locality data (e.g., sensor data characterizing the physical setting <NUM>) from at least one of sensors in the physical setting <NUM>, sensors associated with the controller <NUM>, the electronic device <NUM>, and the display device <NUM>.

In some implementations, the mapper and locator engine <NUM> is also configured to determine the location and orientation of the electronic device <NUM> or the user <NUM> relative to one or more reference points (e.g., an object) in the physical setting (e.g., the center of mass of the object or another point) or the mesh map of the physical setting <NUM>. According to some implementations, the mapper and locator engine <NUM> determines the orientation and location of the electronic device <NUM> based on one or more known localization techniques. For example, in some implementations, the mapper and locator engine <NUM> determines the orientation and location of the electronic device <NUM> relative to the object based on the techniques described in <CIT>. To that end, in various implementations, the mapper and locator engine <NUM> includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the plane detector <NUM> is configured to detect planes (e.g., horizontal, vertical, or angled) within the mesh map. According to some implementations, the plane detector <NUM> detects the planes based on one or more known localization techniques. For example, in some implementations, the plane detector <NUM> detects the planes based on the techniques described in <CIT>. In some implementations, the plane detector <NUM> is also configured filter planes that do not satisfy spatial criteria. To that end, in various implementations, the plane detector <NUM> includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the SR content obtainer <NUM> is configured to obtain (e.g., receive, retrieve, or generate) SR content. To that end, in various implementations, the SR content obtainer <NUM> includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the SR content manager <NUM> is configured to select SR content based on the mesh map and the plane detected within the mesh map. For example, the SR content manager <NUM> selects the SR content based on the user's location and orientation relative to the mesh map and/or the surface area of the planes detected within the mesh map. In some implementations, the SR content manager <NUM> is also configured to manage and coordinate the presentation of the SR content as the user's orientation and location changes relative to the physical setting or the user interacts with the SR content. To that end, in various implementations, the SR content manager <NUM> includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the data transmitter <NUM> is configured to transmit data (e.g., presentation data, location data, etc.) to at least one of the electronic device <NUM> and the display device <NUM>. To that end, in various implementations, the data transmitter <NUM> includes instructions and/or logic therefor, and heuristics and metadata therefor.

Although the data obtainer <NUM>, the mapper and locator engine <NUM>, the plane detector <NUM>, the SR content obtainer <NUM>, the SR content manager <NUM>, and the data transmitter <NUM> are shown as residing on a single device (e.g., the controller <NUM>), it should be understood that in other implementations, any combination of the data obtainer <NUM>, the mapper and locator engine <NUM>, the plane detector <NUM>, the SR content obtainer <NUM>, the SR content manager <NUM>, and the data transmitter <NUM> may be located in separate computing devices.

Moreover, <FIG> is intended more as a functional description of the various features which are present in a particular embodiment as opposed to a structural schematic of the implementations described herein. As recognized by those of ordinary skill in the art, items shown separately could be combined and some items could be separated. For example, some functional modules shown separately in <FIG> could be implemented in a single module and the various functions of single functional blocks could be implemented by one or more functional blocks in various implementations. The actual number of modules and the division of particular functions and how features are allocated among them will vary from one embodiment to another and, in some implementations, depends in part on the particular combination of hardware, software, and/or firmware chosen for a particular embodiment.

<FIG> is a block diagram of an example of the electronic device <NUM> (e.g., an HMD, mobile phone, or tablet) in accordance with some implementations. While certain specific features are illustrated, those skilled in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity, and so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, as a non-limiting example, in some implementations, the electronic device <NUM> includes one or more processing units <NUM> (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, and/or the like), one or more input/output (I/O) devices and sensors <NUM>, one or more communication interfaces <NUM> (e.g., USB, IEEE <NUM>. 3x, IEEE <NUM>. 11x, IEEE <NUM>. 16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, and/or the like type interface), one or more programming (e.g., I/O) interfaces <NUM>, one or more displays <NUM>, one or more optional interior and/or exterior facing image sensors <NUM>, one or more optional depth sensors <NUM>, a memory <NUM>, and one or more communication buses <NUM> for interconnecting these and various other components.

In some implementations, the one or more communication buses <NUM> include circuitry that interconnects and controls communications between system components. In some implementations, the one or more I/O devices and sensors <NUM> include at least one of an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., blood pressure monitor, heart rate monitor, blood oxygen sensor, blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptics engine, a heating and/or cooling unit, a skin shear engine, and/or the like.

In some implementations, the one or more displays <NUM> are configured to present the SR experience to the user. In some implementations, the one or more displays <NUM> are also configured to present flat video content to the user (e.g., a <NUM>-dimensional or "flat" AVI, FLV, WMV, MOV, MP4, or the like file associated with a TV episode or a movie, or live video pass-through of the physical setting <NUM>). In some implementations, the one or more displays <NUM> correspond to holographic, digital light processing (DLP), liquid-crystal display (LCD), liquid-crystal on silicon (LCoS), organic light-emitting field-effect transitory (OLET), organic light-emitting diode (OLED), surface-conduction electron-emitter display (SED), field-emission display (FED), quantum-dot light-emitting diode (QD-LED), micro-electro-mechanical system (MEMS), and/or the like display types. In some implementations, the one or more displays <NUM> correspond to diffractive, reflective, polarized, holographic, etc. waveguide displays. For example, the electronic device <NUM> includes a single display. In another example, the electronic device <NUM> includes a display for each eye of the user. In some implementations, the one or more displays <NUM> are capable of presenting AR and VR content. In some implementations, the one or more displays <NUM> are capable of presenting AR or VR content.

In some implementations, the one or more optional image sensors <NUM> are configured to obtain image data that corresponds to at least a portion of the face of the user that includes the eyes of the user. For example, the one or more optional image sensors <NUM> correspond to one or more RGB cameras (e.g., with a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), infrared (IR) image sensors, event-based cameras, and/or the like.

In some implementations, the one or more optional depth sensors <NUM> are configured to obtain depth data that corresponds to at least a portion of the face of the user and to synthesize a depth/mesh map of the face of the user, where the mesh map characterizes the facial topography of the user. For example, the one or more optional depth sensors <NUM> correspond to a structured light device, a time-of-flight device, and/or the like.

The memory <NUM> includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices. In some implementations, the memory <NUM> includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory <NUM> optionally includes one or more storage devices remotely located from the one or more processing units <NUM>. The memory <NUM> comprises a non-transitory computer readable storage medium. In some implementations, the memory <NUM> or the non-transitory computer readable storage medium of the memory <NUM> stores the following programs, modules and data structures, or a subset thereof including an optional operating system <NUM> and an SR presentation engine <NUM>.

The operating system <NUM> includes procedures for handling various basic system services and for performing hardware dependent tasks. In some implementations, the SR presentation engine <NUM> is configured to present SR content to the user via the one or more displays <NUM>. To that end, in various implementations, the SR presentation engine <NUM> includes a data obtainer <NUM>, an SR presenter <NUM>, a user interaction handler <NUM>, and a data transmitter <NUM>.

In some implementations, the data obtainer <NUM> is configured to obtain data (e.g., presentation data, user interaction data, sensor data, location data, etc.) from at least one of sensors in the physical setting <NUM>, sensors associated with the electronic device <NUM>, the controller <NUM>, and the display device <NUM>. To that end, in various implementations, the data obtainer <NUM> includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the SR presenter <NUM> is configured to present SR content via the one or more displays <NUM>. In some implementations, the SR presenter <NUM> is also configured to present flat video content via the one or more displays <NUM>. To that end, in various implementations, the SR presenter <NUM> includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the user interaction handler <NUM> is configured to detect and interpret user interactions with the presented SR content. To that end, in various implementations, the user interaction handler <NUM> includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the data transmitter <NUM> is configured to transmit data (e.g., presentation data, location data, user interaction data, etc.) to at least one of the controller <NUM> and the display device <NUM>. To that end, in various implementations, the data transmitter <NUM> includes instructions and/or logic therefor, and heuristics and metadata therefor.

Although the data obtainer <NUM>, the optional orientation determiner <NUM>, the SR presenter <NUM>, the user interaction handler <NUM>, and the data transmitter <NUM> are shown as residing on a single device (e.g., the electronic device <NUM>), it should be understood that in other implementations, any combination of the data obtainer <NUM>, the optional orientation determiner <NUM>, the SR presenter <NUM>, the user interaction handler <NUM>, and the data transmitter <NUM> may be located in separate computing devices.

<FIG> is a block diagram of an example of the optional display device <NUM> (e.g., a television (TV) or other display within the physical setting <NUM>) in accordance with some implementations. While certain specific features are illustrated, those skilled in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity, and so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, as a non-limiting example, in some implementations the display device <NUM> includes one or more processing units <NUM> (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, and/or the like), one or more input/output (I/O) devices and sensors <NUM>, one or more communication interfaces <NUM> (e.g., USB, IEEE <NUM>. 3x, IEEE <NUM>. 11x, IEEE <NUM>. 16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, and/or the like type interface), one or more programming (e.g., I/O) interfaces <NUM>, a display <NUM>, a memory <NUM>, and one or more communication buses <NUM> for interconnecting these and various other components. In some implementations, the display device <NUM> is optionally controlled by a remote-control device, voice commands, the electronic device <NUM>, or the like.

In some implementations, the one or more communication buses <NUM> include circuitry that interconnects and controls communications between system components. In some implementations, the one or more I/O devices and sensors <NUM> include at least one of one or more IR sensors, one or more physical buttons, one or more microphones, one or more speakers, one or more image sensors, one or more depth sensors, and/or the like.

In some implementations, the display <NUM> corresponds to holographic, digital light processing (DLP), liquid-crystal display (LCD), liquid-crystal on silicon (LCoS), organic light-emitting field-effect transitory (OLET), organic light-emitting diode (OLED), surface-conduction electron-emitter display (SED), field-emission display (FED), quantum-dot light-emitting diode (QD-LED), micro-electro-mechanical system (MEMS), and/or the like display types.

The memory <NUM> includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices. In some implementations, the memory <NUM> includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory <NUM> optionally includes one or more storage devices remotely located from the one or more processing units <NUM>. The memory <NUM> comprises a non-transitory computer readable storage medium. In some implementations, the memory <NUM> or the non-transitory computer readable storage medium of the memory <NUM> stores the following programs, modules and data structures, or a subset thereof including an optional operating system <NUM> and a presentation engine <NUM>.

The operating system <NUM> includes procedures for handling various basic system services and for performing hardware dependent tasks. In some implementations, the presentation engine <NUM> is configured to present media content (e.g., video and/or audio content) to users via the display <NUM> and the one or more I/O devices and sensors <NUM> (e.g., one or more speakers). To that end, in various implementations, the presentation engine <NUM> includes a data obtainer <NUM>, a content presenter <NUM>, an interaction handler <NUM>, and a data transmitter <NUM>.

In some implementations, the data obtainer <NUM> is configured to obtain data (e.g., presentation data, user interaction data, etc.) from at least one of sensors in the physical setting <NUM>, sensors associated with the display device <NUM>, the controller <NUM>, and the electronic device <NUM>. To that end, in various implementations, the data obtainer <NUM> includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the content presenter <NUM> is configured to render and/ display video content via the display <NUM>. To that end, in various implementations, the content presenter <NUM> includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the interaction handler <NUM> is configured to detect and interpret user interactions with the display device <NUM> (e.g., navigation, playback, tuning, volume adjustment, or the like commands). To that end, in various implementations, the interaction handler <NUM> includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the data transmitter <NUM> is configured to transmit data (e.g., presentation data, user interaction data, etc.) to at least one of the controller <NUM> and the electronic device <NUM>. To that end, in various implementations, the data transmitter <NUM> includes instructions and/or logic therefor, and heuristics and metadata therefor.

Although the data obtainer <NUM>, the content presenter <NUM>, the interaction handler <NUM>, and the data transmitter <NUM> are shown as residing on a single device (e.g., the display device <NUM>), it should be understood that in other implementations, any combination of the data obtainer <NUM>, the content presenter <NUM>, the interaction handler <NUM>, and the data transmitter <NUM> may be located in separate computing devices.

<FIG> illustrates an example SR content presentation architecture <NUM> in accordance with some implementations. While pertinent features are shown, those of ordinary skill in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity and so as not to obscure more pertinent aspects of the example implementations disclosed herein. To that end, as a non-limiting example, the SR content presentation architecture <NUM> synthesizes a mesh map of the physical setting surrounding a user, selects SR content based on the mesh map, and generates composite SR content tailored to the physical setting by compositing at least a portion of the SR content with the mesh map. As such, the electronic device <NUM> overlays SR content on or "skins" at least a portion of the physical setting <NUM> with SR content (e.g., an at-home holodeck).

As shown in <FIG>, in some implementations, the mapping engine <NUM> (e.g., a portion of the mapper and locator engine <NUM> in <FIG>) obtains locality data <NUM> from one or more sensors associated with the physical setting <NUM>, the controller <NUM>, electronic device <NUM>, and/or the display device <NUM>. In some implementations, the locality data <NUM> characterizes objects and relative spatial information of a volumetric region around a user <NUM> (e.g., an X cm radius centered on the user <NUM> of the electronic device <NUM>).

In some implementations, the locality data <NUM> corresponds to sensor data, such as image data, that enables recognition of humanoids, androids, animals, and/or objects within the physical setting. In some implementations, the locality data <NUM> corresponds to sensor data, such as image data, GPS data, beacon data, IR data, ultrasonic data, LiDAR data, depth data, and/or the like that enables mapping of the physical setting and localization of humanoids, androids, animals, and/or objects within the physical setting.

For example, the locality data <NUM> corresponds to image data from one or more external-facing image sensors of the electronic device <NUM> (e.g., images or a live video stream of the physical setting <NUM> from the perspective of the user <NUM>). For example, the locality data <NUM> corresponds to image data from one or more image sensors within the physical setting <NUM>. In this example, the image sensors within the physical setting may correspond to fixed video cameras (e.g., wall-mounted cameras) or movable devices with attached video cameras (e.g., drones or the like).

As shown in <FIG>, in some implementations, the mapping engine <NUM> synthesizes a mesh map <NUM> of the physical setting <NUM> or a portion thereof (e.g., the volumetric region surrounding the user <NUM>) based on the locality data <NUM>. In some implementations, the plane detector <NUM> (e.g., the plane detector <NUM> in <FIG>) detects one or more planes within the mesh map <NUM> and filters planes that do not satisfy spatial criteria. As shown in <FIG>, in some implementations, the locator engine <NUM> (e.g., a portion of the mapper and locator engine <NUM> in <FIG>) determines the user location <NUM> for the user <NUM> relative to the physical setting <NUM> based on the locality data <NUM> and the mesh map <NUM>.

As shown in <FIG>, in some implementations, the SR content selector engine <NUM> (e.g., a portion of the SR content manager <NUM> in <FIG>) selects SR content <NUM> that satisfies a dimensional variance threshold relative to one or more portions of the mesh map <NUM> (e.g., the planes that satisfy the spatial criteria) from the SR content repository <NUM>. For example, the SR content selector engine <NUM> selects the SR content <NUM> that fits the planes that satisfy the spatial criteria. In other words, the SR content selector engine <NUM> selects the SR content <NUM> that fits the physical setting <NUM> in order to tailor the SR experience to the physical setting <NUM>.

As shown in <FIG>, in some implementations, the composite engine <NUM> (e.g., a portion of the SR content manager <NUM> in <FIG>) generates composite SR content <NUM> by compositing at least a portion of the SR content <NUM> with the mesh map <NUM>. In some implementations, the composite engine <NUM> adapts (e.g., stretches, shrinks, or enlarges) the selected SR content to fit the one or more portions of the mesh map <NUM> (e.g., the planes that satisfy the spatial criteria).

In some implementations, the composite SR content <NUM> is provided to the SR presentation pipeline <NUM> for presentation to the user <NUM>. In some implementations, the composite SR content <NUM> is rendered by the controller <NUM> and transmitted to the electronic device <NUM> as presentation data, where the composite SR content <NUM> is presented via the one or more displays <NUM>.

<FIG> illustrate an example SR presentation scenario <NUM> in accordance with some implementations. While pertinent features are shown, those of ordinary skill in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity and so as not to obscure more pertinent aspects of the example implementations disclosed herein.

As shown in <FIG>, the physical setting <NUM> includes the chairs 162a and 162b, credenza <NUM>, coffee table <NUM>, sofa <NUM>, end tables 170a and 170b, and door <NUM>. As shown in <FIG>, the user is standing behind the sofa <NUM> facing the display device <NUM> while the chairs 162a and 162b, credenza <NUM>, coffee table <NUM>, sofa <NUM>, and end tables 170a and 170b are within the field-of-view <NUM> of the electronic device <NUM>. For example, the electronic device <NUM> corresponds to AR-enabled tablet or mobile phone with video pass-through of the physical setting <NUM> displayed on the display <NUM>.

As shown in <FIG>, in state <NUM> (e.g., at time T), the user is watching video content <NUM> (e.g., a television (TV) episode or movie) on the display device <NUM>. For example, the electronic device <NUM> or the display device <NUM> displays a subtle (e.g., non-obtrusive) affordance or notification indicating that an SR experience associated with the video content <NUM> is available. Continuing with this example, the electronic device <NUM> detects a command issued by user to enter an SR experience associated with the video content <NUM> (e.g., a voice command, gestural command, or the like). In response to detecting the command, for example, the electronic device <NUM> synthesizes a mesh map of the physical setting <NUM> and detects planes within the mesh map.

As shown in <FIG>, in state <NUM> (e.g., at time T+<NUM>), the electronic device <NUM> identifies planes 610a, 610b, 610c, and 610d within the physical setting <NUM>. According to some implementations, the electronic device <NUM> filters planes that do not satisfy spatial criteria. For example, the planes 610b and 610c do not satisfy a line-of-sight criterion associated with the spatial criteria (e.g., more than Z degrees from the focal point of the user <NUM>). In other words, the location of the places 610b and 610c is too low relative to the focal point of the user <NUM>. As such, planes 610a and 610d satisfy the spatial criteria.

As shown in <FIG>, in state <NUM> (e.g., at time T+<NUM>), the electronic device <NUM> presents, on the display <NUM>, SR content 620a (e.g., background scenery associated with the video content <NUM>) on the plane 610a and the SR content 620b (e.g., peripheral scenery associated with the video content <NUM>) on the plane 610d. In some implementations, the SR content 620a and 620b is planar or volumetric. According to some implementations, the electronic device <NUM> selects SR content associated with the video content <NUM> for the detected planes that meet the spatial criteria based on the detected planes and the orientation/location of the user relative to the mesh map.

As one example, the video content <NUM> corresponds to a court room scene within a movie. In this example, the SR content 620a and 620b correspond to an SR reconstruction of at least a portion of the court room scene. Continuing with this example, the SR content 620a corresponds to the judge's bench and the SR content 620b corresponds to opposing lawyers and their teams. As such, continuing with the example, the electronic device <NUM> skins at least a portion of the physical setting <NUM> with the SR content 620a and 620b associated with the video content <NUM> such that the user is able to experience the court room scene within the movie as if it is occurring within his/her living room.

As shown in <FIG>, the physical setting <NUM> includes the chairs 162a and 162b, credenza <NUM>, coffee table <NUM>, sofa <NUM>, end tables 170a and 170b, and door <NUM>. As shown in <FIG>, the user <NUM> is standing behind the sofa <NUM> facing the display device <NUM> while wearing the electronic device <NUM> on his/her head. For example, the electronic device <NUM> corresponds to AR-enabled HMD (e.g., glasses, goggles, or the like) with optical see-through of the physical setting <NUM>.

As shown in <FIG>, in state <NUM> (e.g., at time T), the user <NUM> is watching video content <NUM> (e.g., a television (TV) episode or movie) on the display device <NUM>. For example, the electronic device <NUM> or the display device <NUM> displays a subtle (e.g., non-obtrusive) affordance or notification indicating that an SR experience associated with the video content <NUM> is available. Continuing with this example, the controller <NUM> and/or the electronic device <NUM> detects a command issued by user <NUM> to enter an SR experience associated with the video content <NUM> (e.g., a voice command, gestural command, or the like). In response to detecting the command, for example, the controller <NUM> synthesizes a mesh map of the physical setting <NUM> and detects planes within the mesh map.

As shown in <FIG>, in state <NUM> (e.g., at time T+<NUM>), the controller <NUM> identifies planes 710a, 710b, 710c, 710d, 710e, 710f, <NUM>, <NUM>, and 710i within the physical setting. According to some implementations, the controller <NUM> filters planes that do not satisfy spatial criteria. For example, the planes 710c and 710d associated with the chairs 162a and 162b, respectively, do not satisfy a dimensional criterion associated with the spatial criteria (e.g., less than MxN cm or Y cm<NUM>). In other words, the surface area of the planes 710c and 710d is too small for the placement of SR content. For example, the planes 710f and <NUM> do not satisfy a line-of-sight criterion associated with the spatial criteria (e.g., more than Z degrees from the focal point of the user <NUM>). In other words, the location of the places 710f and <NUM> is too low relative to the focal point of the user <NUM>. For example, the plane <NUM> does not satisfy a personal radius criterion associated with the spatial criteria (e.g., less than Q cm from the user <NUM>). In other words, the plane <NUM> is too close to the user <NUM>. As such, planes 710a, 710b, 710e, and 710i satisfy the spatial criteria.

As shown in <FIG>, in state <NUM> (e.g., at time T+<NUM>), the electronic device <NUM> presents SR content 720a (e.g., background scenery associated with the video content <NUM>) on the plane 710a, SR content 720b on the plane 710b (e.g., background characters and/or objects associated with the video content <NUM>), SR content 720c (e.g., foreground characters and/or objects associated with the video content <NUM>) on the plane 710e, and the SR content 720d (e.g., peripheral scenery associated with the video content <NUM>) on the plane 710i. In some implementations, the SR content 720a, 720b, 720c, and 720d is planar or volumetric. According to some implementations, the controller <NUM> selects SR content associated with the video content <NUM> for the detected planes that meet the spatial criteria based on the detected planes and the orientation/location of the user <NUM> relative to the mesh map.

As one example, the video content <NUM> corresponds to a boxing match scene within a movie. In this example, the SR content 720a, 720b, 720c, and 720d correspond to an SR reconstruction of at least a portion of the boxing match scene. Continuing with this example, the SR content 720a corresponds to the crowd in the background, the SR content 720b corresponds to the referee and the ropes and turnbuckles of the boxing ring, the 7R content 620c corresponds to the fighters sparring within the boxing ring, and the SR content 720d corresponds to the crowd in the background in the periphery. As such, continuing with the example, the electronic device <NUM> skins at least a portion of the physical setting <NUM> with the SR content 720a, 720b, 720c, and 720d associated with the video content <NUM> such that the user <NUM> is able to experience the boxing match scene within the movie as if it is occurring within his/her living room.

<FIG> is a flowchart representation of a method <NUM> of tailoring an SR experience to a physical setting in accordance with some implementations. In various implementations, the method <NUM> is performed by a device with non-transitory memory and one or more processors coupled with the non-transitory memory (e.g., the controller <NUM> in <FIG> and <FIG>, the electronic device <NUM> in <FIG> and <FIG>, or a suitable combination thereof). In some implementations, the method <NUM> is performed by processing logic, including hardware, firmware, software, or a combination thereof. In some implementations, the method <NUM> is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory). Briefly, in some circumstances, the method <NUM> includes: obtaining locality data associated with a volumetric region around a user; synthesizing a mesh map of the volumetric region based on the locality data; selecting SR content based on the mesh map that satisfies a dimensional variance threshold relative to one or more portions of the mesh map; compositing at least a portion of the SR content with the mesh map in order to generate composite SR content; and presenting the composite SR content to the user in order to occlude at least a portion of a visual presentation of the volumetric region.

As represented by block <NUM>-<NUM>, the method <NUM> includes obtaining (e.g., collecting, receiving, or retrieving) locality data associated with a volumetric region around a user. In some implementations, the locality data characterizes objects and relative spatial information for the volumetric region. For example, the volumetric region corresponds to a volumetric region with an X cm radius centered on the user <NUM> of the electronic device <NUM>.

In some implementations, the controller <NUM> and/or the electronic device <NUM>, or a component thereof (e.g., the data obtainer <NUM> in <FIG>) obtains the locality data from at least one of sensors in the physical setting <NUM>, sensors associated with the controller <NUM>, the electronic device <NUM>, and the display device <NUM>. For example, the locality data corresponds to image data from one or more external-facing image sensors of the electronic device <NUM> (e.g., images or a live video stream of the physical setting <NUM> from the perspective of the user <NUM>. In another example, the locality data corresponds to image data from one or more image sensors within the physical setting <NUM>. In this example, the image sensors within the physical setting may correspond to fixed video cameras (e.g., wall-mounted cameras) or movable devices with attached video cameras (e.g., drones or the like). In another example, the locality data corresponds to other sensor data associated with physical setting and/or the user <NUM> from GPS, LiDAR, IR sensors, depth sensors, ultrasonic sensors, and/or the like.

As represented by block <NUM>-<NUM>, the method <NUM> includes synthesizing a mesh map of the volumetric region based on the locality data. In some implementations, the controller <NUM> and/or the electronic device <NUM>, or a component thereof (e.g., the mapper and locator engine <NUM>) synthesizes a mesh map of the physical setting <NUM> or a portion thereof (e.g., the volumetric region surrounding the user <NUM>) based on the locality data. For example, the mesh map defines the dimensions of the volumetric region and objects within the volumetric region such as furniture, walls, other users, and/or the like.

In some implementations, synthesizing the mesh map includes blocking off restricted areas for safety (e.g., balconies). For example, SR content cannot be presented in these restricted areas allowing for the user to see the potential hazard by way of optical see-through via an AR-enabled HMD.

In some implementations, as represented by block <NUM>-2a, the method <NUM> includes detecting planes within the mesh map. In some implementations, the controller <NUM> and/or the electronic device <NUM>, or a component thereof (e.g., the plane detector <NUM>) identifies planes (e.g., horizontal, vertical, or angled) within the mesh map. According to some implementations, the plane detector <NUM> detects the planes based on one or more known localization techniques. For example, in some implementations, the plane detector <NUM> detects the planes based on the techniques described in <CIT>. With reference to <FIG>, for example, the electronic device <NUM> identifies planes 610a, 610b, 610c, and 610d within the physical setting <NUM>. With reference to <FIG>, for example, the controller <NUM> and/or the electronic device <NUM> identifies planes 710a, 710b, 710c, 710d, 710e, 710f, 710f, <NUM>, and 710i within the physical setting.

In some implementations, as represented by block <NUM>-2b, the method <NUM> includes filtering planes that do not satisfy spatial criteria. In some implementations, the controller <NUM> and/or the electronic device <NUM>, or a component thereof (e.g., the plane detector <NUM>) filters planes that do not satisfy spatial criteria. With reference to <FIG>, for example, the controller <NUM> and/or the electronic device <NUM> filters the planes 710c and 710d for failing to satisfy a dimensional criterion associated with the spatial criteria (e.g., less than MxN cm or Y cm<NUM>). With reference to <FIG>, for example, the controller <NUM> and/or the electronic device <NUM> filters the planes 710f and <NUM> for failing to satisfy a line-of-sight criterion associated with the spatial criteria (e.g., more than Z degrees from the focal point of the user <NUM>). With reference to <FIG>, for example, the controller <NUM> and/or the electronic device <NUM> filters the plane <NUM> does for failing to satisfy a personal radius criterion associated with the spatial criteria (e.g., less than Q cm from the user <NUM>). As such, with reference to <FIG>, for example, the controller <NUM> and/or the electronic device <NUM> determines that the planes 710a, 710b, 710e, and 710i satisfy the spatial criteria.

As represented by block <NUM>-<NUM>, the method <NUM> includes selecting SR content based on the mesh map that satisfies a dimensional variance threshold relative to one or more portions of the mesh map. The SR content satisfies a dimensional variance threshold relative to one or more portions of the mesh map. In some implementations, the controller <NUM> and/or the electronic device <NUM>, or a component thereof (e.g., the SR content manager <NUM> in <FIG>) selects SR content that satisfies a dimensional variance threshold relative to one or more portions of the mesh map <NUM> (e.g., SR content that fits the surface area of the planes that satisfy the spatial criteria). In other words, the controller <NUM> and/or the electronic device <NUM> selects the SR content that fits the physical setting <NUM> in order to tailor the SR experience to the physical setting <NUM>.

For example, the controller <NUM> and/or the electronic device <NUM> selects first SR content to be presented on or about a couch, second SR content to be presented on or about a mirror, and third SR content to be presented on or about a wall within the same physical setting. For example, the controller <NUM> and/or the electronic device <NUM> selects different SR content based on open floor space, table surface size, open wall space, couch size and occupation, and/or the like. In some implementations, the selected SR content corresponds to SR reconstructed portions of video content currently being viewed by a user (e.g., characters or scenery from a movie). In some implementations, the selected SR content corresponds to SR content that augments video content currently being viewed by a user (e.g., maps, graphs, educational information, or the like associated with a movie).

In some implementations, selecting the SR content includes first matching selecting portions of the current plot scene within video content that fit onto one or more portions of the mesh map (e.g., the planes that satisfy the spatial criteria) based on the user's current point-of-view relative to the mesh map. For example, if a couch is behind the user but a coffee table is in front of the user, the controller <NUM> and/or the electronic device <NUM> selects SR content that fits the dimensions of the coffee table and forgoes selecting SR content that fits the dimension of the couch due to its position behind the user.

In some implementations, the controller <NUM> and/or the electronic device <NUM>, or a component thereof (e.g., the SR content obtainer <NUM>) obtains (e.g., receives, retrieves, or generates) the SR content. In some implementations, the SR content is obtained from a local library or a remote library (e.g., a remote server, a third-party content provider, or the like). In some implementations, the SR content corresponds associated with video content currently being viewed such as space fighters for a space battle, bystanders for a plot setting in a crowded plaza, automobiles for a plot setting in a traffic jam, or the like. In some implementations, the SR content is an SR reconstruction of a scene in video content currently being viewed. For example, in some implementations, the SR reconstruction of the video content is generated based on the techniques described in <CIT>.

As represented by block <NUM>-<NUM>, the method <NUM> includes compositing at least a portion of the SR content with the mesh map in order to generate composite SR content. In some implementations, the controller <NUM> and/or the electronic device <NUM>, or a component thereof (e.g., the SR content manager <NUM> in <FIG>) composites at least a portion of the SR content with the mesh map in order to generate composite SR content. In some implementations, generating the composite SR content includes skinning a portion of the user's physical setting with the selected SR content of a scene to the user's room. As one example, the controller <NUM> and/or the electronic device <NUM> skins the user's living room with the bridge of a large space cruiser or a shuttle cockpit based on the size of the living room and the furniture within the living room.

As represented by block <NUM>-4a, the method <NUM> includes adapting the SR content. In some implementations, the controller <NUM> and/or the electronic device <NUM>, or a component thereof (e.g., the SR content manager <NUM> in <FIG>) adapts (e.g., stretches, shrinks, or enlarges) the selected SR content to fit the one or more portions of the mesh map (e.g., the planes that satisfy the spatial criteria). In some implementations, the SR content is adapted to fit the mesh map or the planes detected within the volumetric region that satisfy the spatial criteria. (e.g., stretch or shrink the SR content based on the available surface area of the table or the unobstructed open floor space). In some implementations, the amount to which the SR content may be adapted is limited by adaptation constraint criteria (e.g., associated with IP policy right constraints, DRM limitations, or distortion limits).

As represented by block <NUM>-<NUM>, the method <NUM> includes presenting the composite SR content to the user in order to occlude at least a portion of a visual presentation of the volumetric region. In some implementations, the SR content is rendered by the controller <NUM> and transmitted by the controller <NUM> to the SR device where the SR content is presented to the user via the one or more displays <NUM>. In some implementations, the electronic device <NUM> or a component thereof (e.g., the SR presenter <NUM> in <FIG>) presents the SR content via the one or more displays <NUM>. For example, the SR content is overlaid, superimposed, or projected on portions of the physical setting that are in turn occluded by the SR content.

As one example, in <FIG>, the electronic device <NUM> presents, on the display <NUM>, SR content 620a and 620b associated with the video content <NUM>. As another example, in <FIG>, the electronic device <NUM> presents SR content 720a, 720b, 720c, and 720d associated with the video content <NUM>, where the SR content 720a occludes the display device <NUM> shown in <FIG> and the SR content 720d occludes the door <NUM> shown in <FIG>. In some implementations, the SR content is volumetric. For example, the SR content is presented on top of the user's coffee table but not on the user's couch or windows.

In some implementations, the SR experience includes a transition from the user viewing video content on display device <NUM> (e.g., a TV or tablet) to an SR content associated with the video content presented via an HMD. As one example, in <FIG>, in state <NUM>, the display device <NUM> displays video content <NUM> to the user. Continuing with this example, in <FIG>, in state <NUM>, the electronic device <NUM> presents SR content 720a, 720b, 720c, and 720d associated with the video content <NUM>. In some implementations, the video content <NUM> is paused in state <NUM>. In some implementations, the video content <NUM> continues concurrently in state <NUM>.

In some implementations, as represented by block <NUM>-5a, the method <NUM> includes updating the SR content as the user location changes. In some implementations, the controller <NUM> and/or the electronic device <NUM>, or a component thereof (e.g., the SR content manager <NUM> in <FIG>) updates (e.g., replaces, modifies, etc.) the SR content as the user location changes relative to the mesh map. For example, the SR content is updated as the user <NUM> moves about the physical setting <NUM>. According to some implementations, the SR content is volumetric so user can select from almost an infinite set of camera angles by moving about the physical setting. In some implementations, the SR content is updated as the mesh map changes (e.g., a chair or other furniture is moved about the space).

In some implementations, as represented by block <NUM>-5b, the method <NUM> includes updating the SR content as the user interacts with the SR content. In some implementations, the controller <NUM> and/or the electronic device <NUM>, or a component thereof (e.g., the SR content manager <NUM> in <FIG>) updates (e.g., replaces, modifies, etc.) the SR content as the user interacts with the SR content. For example, the SR content is interactive such that the SR changes as the user modifies or otherwise interacts with the SR content. In some implementations, as the user interacts with the SR content, the electronic device <NUM> provides audio, haptic, skin shear, temperature, or the like feedback.

It will also be understood that, although the terms "first," "second," etc. may be used herein to describe various elements, these elements should not be limited by these terms. For example, a first node could be termed a second node, and, similarly, a second node could be termed a first node, which changing the meaning of the description, so long as all occurrences of the "first node" are renamed consistently and all occurrences of the "second node" are renamed consistently. The first node and the second node are both nodes, but they are not the same node.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the claims. As used in the description of the embodiments and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

Claim 1:
A method comprising:
at a device including non-transitory memory and one or more processors coupled with the non-transitory memory:
obtaining locality data from at least one sensor in a physical setting, the locality data characterizing objects and relative spatial information of a volumetric region in the physical setting;
synthesizing a mesh map for the volumetric region based on the locality data; characterized by:
selecting synthesized reality (SR) content that satisfies a dimensional variance threshold relative to one or more portions of the mesh map;
compositing at least a portion of the SR content with the mesh map in order to generate composite SR content;
adapting the composite SR content to fit the one or more portions of the mesh map; and
presenting the adapted composite SR content.