Anchoring virtual content to physical surfaces

Systems and techniques are provided for rendering virtual content in a scene. An example method includes determining a surface plane in a scene of a physical environment; rendering virtual content in a perspective at a location in the scene; rendering a content placement indicator associated with the virtual content on a first space of the surface plane, the content placement indicator indicating the first space for placement of the virtual content; determining one or more locations of one or more objects on a surface associated with the surface plane; and responsive to an input, rendering the content placement indicator on a second space of the surface plane, the second space being determined based on the one or more locations and input, the content placement indicator indicating the second space for placement of the virtual content.

TECHNICAL FIELD

The present disclosure generally relates to organizing and rendering virtual content for extended reality. For example, aspects of the present disclosure relate to techniques and systems for placing virtual content on physical surfaces.

BACKGROUND

Extended reality (e.g., augmented reality, virtual reality, mixed reality, etc.) devices, such as smart glasses and head-mounted displays (HMDs), generally implement cameras and a variety of sensors to track the position of the extended reality (XR) device and other objects within the physical environment. The XR devices can use such tracking information to provide a user of the XR device a realistic XR experience. For example, an XR device can allow a user to experience or interact with immersive virtual environments or content. To provide realistic XR experiences, XR technologies can integrate virtual content with the physical world. In some cases, XR technologies can match the relative pose and movement of objects and devices in the physical world. For example, an XR device can use tracking information to calculate the relative pose of devices, objects, and/or maps of the real-world environment in order to match the relative position and movement of the devices, objects, and/or the real-world environment. Using the pose and movement of one or more devices, objects, and/or the real-world environment, the XR device can render content relative to the real-world environment in a convincing manner. The relative pose information can be used to match virtual content with the user's perceived motion and the spatio-temporal state of the devices, objects, and real-world environment.

BRIEF SUMMARY

In some examples, systems, apparatuses, processes (also referred to as methods), and computer-readable media (collectively referred to as “systems and techniques”) are described for anchoring virtual content to physical surfaces. According to at least one illustrative example, a method is provided for rendering virtual content in a scene of a physical environment. In some examples, the method can include determining at least one surface plane in the scene of the physical environment; rendering, via a computing device, a virtual content item in a first perspective at a first location in the scene; rendering a content placement indicator associated with the virtual content item on a first space of the at least one surface plane, the content placement indicator indicating at least part of the first space for placement of the virtual content item; determining one or more locations of one or more objects on a surface associated with the at least one surface plane; and in response to a user input, rendering the content placement indicator on a second space of the at least one surface plane, wherein the second space of the at least one surface plane is determined based on the one or more locations and the user input, the content placement indicator indicating at least part of the second space for placement of the virtual content item.

According to at least one illustrative example, a non-transitory computer-readable medium is provided for rendering virtual content in a scene of a physical environment. In some aspects, the non-transitory computer-readable medium can include instructions that, when executed by one or more processors, cause the one or more processors to determine at least one surface plane in the scene of the physical environment; render a virtual content item in a first perspective at a first location in the scene; render a content placement indicator associated with the virtual content item on a first space of the at least one surface plane, the content placement indicator indicating at least part of the first space for placement of the virtual content item; determine one or more locations of one or more objects on a surface associated with the at least one surface plane; and in response to a user input, render the content placement indicator on a second space of the at least one surface plane, wherein the second space of the at least one surface plane is determined based on the one or more locations and the user input, the content placement indicator indicating at least part of the second space for placement of the virtual content item.

According to at least one illustrative example, an apparatus is provided for rendering virtual content in a scene of a physical environment. In some aspects, the apparatus can include memory having stored thereon computer-readable instructions and one or more processors configured to determine at least one surface plane in the scene of the physical environment; render a virtual content item in a first perspective at a first location in the scene; render a content placement indicator associated with the virtual content item on a first space of the at least one surface plane, the content placement indicator indicating at least part of the first space for placement of the virtual content item; determine one or more locations of one or more objects on a surface associated with the at least one surface plane; and in response to a user input, render the content placement indicator on a second space of the at least one surface plane, wherein the second space of the at least one surface plane is determined based on the one or more locations and the user input, the content placement indicator indicating at least part of the second space for placement of the virtual content item.

According to another illustrative example, another apparatus for rendering virtual content in a scene of a physical environment can include means for determining at least one surface plane in the scene of the physical environment; rendering, via a computing device, a virtual content item in a first perspective at a first location in the scene; rendering a content placement indicator associated with the virtual content item on a first space of the at least one surface plane, the content placement indicator indicating at least part of the first space for placement of the virtual content item; determining one or more locations of one or more objects on a surface associated with the at least one surface plane; and in response to a user input, rendering the content placement indicator on a second space of the at least one surface plane, wherein the second space of the at least one surface plane is determined based on the one or more locations and the user input, the content placement indicator indicating at least part of the second space for placement of the virtual content item.

In some aspects, the method, non-transitory computer-readable medium, and apparatuses described above can include, in response to an additional user input requesting placement of the virtual content item on the second space, rendering the virtual content item in a second perspective at a second location in the scene based on the content placement indicator. In some examples, the second location can be within the second space. In some cases, the method, non-transitory computer-readable medium, and apparatuses described above can include determining the second perspective based on a perspective of the second space.

In some aspects, the method, non-transitory computer-readable medium, and apparatuses described above can include determining a first surface plane and a second surface plane of the at least one surface plane, the first space being associated with the first surface plane and the second space being associated with the second surface plane; and determining the second perspective based on a perspective of the second surface plane.

In some examples, rendering the content placement indicator on the second space can include determining an overlap of the virtual content item with the second surface plane, and upon determination that the overlap exceeds a first threshold, switching to rendering the content placement indicator on the second space associated with the second surface plane.

In some examples, the first space and the second space are associated with a first surface plane of the at least one surface plane. In some aspects, the method, non-transitory computer-readable medium, and apparatuses described above can include detecting one or more objects, wherein the one or more objects occlude at least part of the first surface plane; and segmenting the first surface plane into at least a first segment and a second segment based at least on an occluded part of the first surface plane. In some examples, the first space is associated with the first segment and the second space is associated with the second segment.

In some cases, segmenting the first surface plane into at least the first segment and the second segment can include determining a fit of the virtual content item rendered in a perspective of the first surface plane in the first segment and the second segment.

In some aspects, the method, non-transitory computer-readable medium, and apparatuses described above can include changing an aspect ratio of the virtual content item between the first segment and the second segment.

In some aspects, the method, non-transitory computer-readable medium, and apparatuses described above can include prior to rendering the content placement indicator on the second space, rendering the virtual content item in a third perspective at a third location in the scene based on the content placement indicator. In some examples, the third location can be within the first space.

In some aspects, the method, non-transitory computer-readable medium, and apparatuses described above can include, in response to the user input, moving the content placement indicator within the first space prior to rendering the content placement indicator on the second space.

In some cases, determining the at least one surface plane in the scene can include applying an offset to the at least one surface plane normally to at least one surface and toward a view point of a user associated with the computing device.

In some cases, determining the at least one surface plane in the scene can include determining one or more edges of the one or more objects in the scene, and determining the at least one surface plane based on the one or more edges of the one or more objects in the scene.

In some cases, rendering the content placement indicator can include rendering a pattern on the at least one surface plane. In some examples, the pattern can include at least one of a highlight, an outline, a color, a shade, a shadow, a hatching, and a gradient.

In some cases, determining the at least one surface plane in the scene can include determining one or more surfaces of the one or more objects in the scene, and determining the at least one surface plane based on the one or more surfaces of the one or more objects in the scene.

In some aspects, an apparatus can be, or can be part of, a camera (e.g., an IP camera), a mobile device (e.g., a mobile telephone or so-called “smartphone,” or other mobile device), a smart wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a multi-camera system, or other device. In some aspects, the apparatus includes a camera or multiple cameras for capturing one or more images. In some aspects, the apparatus further includes a display for displaying one or more images, notifications, and/or other displayable data. In some aspects, the apparatuses described above can include one or more sensors.

DETAILED DESCRIPTION

As previously explained, extended reality (XR) devices can allow a user to experience or interact with immersive virtual environments or content. To provide realistic XR experiences, XR devices can integrate virtual content with a scene or image of the physical world captured by the XR device (e.g., via one or more cameras of the XR device). In some cases, an XR device can perform three-dimensional registration of virtual and real objects. In some cases, the XR device can match the relative pose and movement of objects and the XR device in the physical world. For example, the XR device can use tracking information to calculate the relative pose of the XR device, objects, and/or maps of the real-world environment in order to match the relative position and movement of the XR device, objects, and/or the real-world environment. Using the pose and movement of the XR device, one or more objects, and/or the real-world environment, the XR device can render content relative to the real-world environment in a convincing manner. The relative pose information can be used to match virtual content with the user's perceived motion and the spatio-temporal state of the XR device, the objects, and the real-world environment.

When presenting virtual content to a user in a scene or image of the physical world, an XR device may configure the virtual content to appear to the user as if the rendered virtual content is located in a certain space/region on the physical world. In some cases, it can be difficult for the XR device to determine which spaces/regions in the physical world are suitable (e.g., unobstructed and/or unoccupied by an object in the physical world, free of clutter, etc.) and/or have a suitable surface for depicting/rendering the virtual content. Often, the XR device can have difficulty managing and/or organizing the virtual content rendered by the XR device in a particular environment, and/or avoiding issues when presenting the virtual content in the particular environment such as, for example, obstructions caused by objects in the environment and/or aspects of the environment (e.g., the layout of the environment, the configuration of surfaces and/or spaces in the environment, the placement of any objects in the environment, etc.), clutter caused by the rendered virtual content and objects in the environment, etc.

In some examples, an XR device may try to organize virtual content rendered by the XR device by placing the virtual content in certain spaces on the physical world. For example, an XR device may try to organize virtual content by appending or anchoring the virtual content to a certain area/region in the physical world. In some cases, the XR device can receive a user input requesting a certain organization and/or placement of virtual content. In response to the user input, the XR device can append or anchor the virtual content to a certain area/region in the physical world. In other words, virtual content may be rendered by the XR device such that it has a fixed spatial relationship to a particular space/region in a scene of the physical world. The particular space/region may correspond to one or more real objects in the physical world. As will be described in detail below, the correspondence may be by association with a surface of the one or more objects. In some examples, a three-dimensional orientation of the surface in the scene of the physical world may be used to render a perspective view of the virtual content such that the virtual content appears aligned with the surface in three dimensions. In some cases, the virtual content may be anchored to the surface such that the fixed spatial relationship between the rendered virtual content and the surface is maintained under movement of the user and/or the view. In some cases, the perspective view of the virtual content is adapted to a change of the user's view and/or position in the physical world.

As further described herein, in some examples, the XR device can depict a content placement indicator suggesting and/or identifying a suitable and/or suggested area or region for placing the virtual content in the scene of the physical world. In some cases, a user may manually move (e.g., via a user input provided to the XR device) a content placement indicator for the virtual content. In some examples, the content placement indicator can be moved in response to a user input (e.g., the user dragging a virtual content item across the scene, etc.). The virtual content item can initially be rendered in a first perspective, such as in plan view, at a first location in the scene. The first location may be predefined or manually determined by the user. The virtual content item can be rendered with an initial size which may be predefined and/or user-defined. The initial size may be adapted through user input and/or in response to the user input (e.g., a dragging input, etc.). The content placement indicator can represent and/or identify an area or region depicted within a surface and/or surface plane in the physical environment, where certain virtual content can be placed and rendered by the XR device. The surface plane may be a virtual surface plane associated with an actual surface of an object in the physical environment. In some cases, the surface plane may be aligned with the surface of the object. In some cases, the surface plane may be offset from the surface of the object. The offset may be toward a point a view of the user. In other cases, the surface plane may coincide with the surface of the object.

The content placement indicator can include a visual element such as a visual outline, window, hatching, shape (e.g., bounding box or any other shape), etc., that indicates at least part of a particular space of a surface (and/or surface plane) of the physical environment, where the virtual content item can be placed/rendered for the user. In some cases, the content placement indicator can include a visual element that depicts a fit and placement of the virtual content item on a particular space of a surface in the physical environment. The content placement indicator (e.g., the visual element) can have a same or similar shape and/or configuration as the virtual content. Thus, the content placement indicator (e.g., the visual element) can depict/render to the user how and/or where the virtual content would appear in and/or relative to the physical environment.

When rendering the virtual content in the physical environment, the XR device can place and render the virtual content at/within the content placement indicator. To the user, the virtual content can appear to be located within the physical environment, at the space (e.g., area or region) associated with, e.g., outlined or bounded by, the content placement indicator. In some cases, the process for manually moving a content placement indicator for the virtual content can be cumbersome and imprecise. For example, if the user wants to append or anchor virtual content to a surface in the physical environment that is cluttered (e.g., crowded with objects, messy, limited in available space, etc.) and/or complex (e.g., uneven, rough/coarse/bumpy/textured, elevated, irregular, asymmetrical, curved, three-dimensional, of a certain geometry, etc.), it may be difficult to estimate if the virtual content will fit (with a minimum size, e.g., for readability of text content) in the surface and/or within any suitable space (e.g., a space unoccupied or unobstructed by an object, a space capable of fitting at least a threshold amount of the virtual content, etc.) in the surface. In some cases, it can be difficult to determine whether the virtual content may better fit in a particular surface or surface plane or a different surface or surface plane, or whether placing virtual content within a particular surface may occlude any objects in that surface that the user may not want occluded by the virtual content (e.g., that the user may want to remain visible to the user once the virtual content is placed and rendered on the surface).

Systems, apparatuses, processes (also referred to as methods), and computer-readable media (collectively referred to as “systems and techniques”) are described herein for placing virtual content on and/or relative to surfaces in physical environments. In some examples, an XR device can identify surface planes (planes on/of or relative to respective surfaces of one or more objects) in a physical environment on which the user can anchor (e.g., place/attach) virtual content for rendering by the XR device. To this end, the XR device can identify one or more objects in the physical environment, determine the relative pose of the one or more objects with respect to the XR device, and identify one or more surfaces of the one or more objects. The XR device can provide the user an indication and/or suggestion of the identified surface planes to assist the user in selecting and/or accepting a particular surface plane on which to place/depict virtual content. The indication can be in the form of a content placement indicator which may highlight the entire or only part of the surface plane or may indicate the entire or only part of the surface plane for placement of the virtual content, e.g., by means of a visual outline. The XR device can render the virtual content in a manner that depicts the virtual content on any of such surface planes selected and/or accepted by the user for the placement and rendering of the virtual content.

For example, in some cases, as the user of the XR device moves a rendering of a content placement indicator so as to appear to the user as if the content placement indicator is being moved through/across a scene of the physical environment, the XR device can detect and highlight any suitable surface planes in the physical environment which the virtual content can be anchored to for rendering. The content placement indicator can represent a space on the surface plane for placement of the virtual content within the physical environment. In some examples, the content placement indicator can represent and/or identify an outline, shape, and/or configuration of the virtual content to indicate a corresponding space of a surface plane in the physical environment that would be occupied by the virtual content (e.g., that would be used to render the virtual content) if the virtual content is snapped to (e.g., placed at/within, anchored to, etc.) the content placement indicator at the depicted/identified position of the content placement indicator. The user can use the content placement indicator to determine where to place the virtual content within the surface plane in physical environment.

In some examples, the user can select and/or accept to place the virtual content in/within the content placement indicator at a corresponding location (e.g., at a depicted/identified location) of the content placement indicator within a surface plane which can be highlighted by the XR device. For example, the user can select a current position (e.g., a depicted/identified position) of the content placement indicator to anchor the virtual content to the content placement indicator at the selected position of the content placement indicator in the scene or image of the physical environment (e.g., in/within the surface plane). The XR device can render the virtual content within the content placement indicator at the selected position of the content placement indicator in the scene of the physical environment (e.g., in/within the surface plane). For example, the XR device can render the virtual content within the content placement indicator at a particular location of the surface plane corresponding to the selected position of the content placement indicator.

In some cases, the XR device can highlight (or otherwise visually depict/identify) a surface plane(s) in the physical environment. In some examples, the XR device can highlight an entire or part of a surface plane(s) to provide the content placement indicator. Alternatively or additionally, a (further) content placement indicator, e.g., in the form of a visual outline, may be rendered in/on the surface plane(s) to indicate a particular space of the surface plane for placement of the virtual content item. The XR device can change the surface plane highlighted (and/or the highlighted part of a surface plane) as the XR device moves (e.g., based on a user input, a device setting, a user preference, and/or a dynamic adjustment decision) the content placement indicator through/across the scene of the physical environment. In some examples, the XR device can identify a number of candidate surface planes in the physical environment and highlight a particular surface plane in the physical environment. When the XR device moves the content placement indicator through/across the scene of the physical environment, the XR device can highlight a different surface plane from the candidate surface planes.

The XR device can highlight an entire surface plane or a portion of the surface plane. In some examples, the XR device can highlight an entire surface plane, and present to the user a content placement indicator within the highlighted surface plane. In some cases, the presented content placement indicator can include a smaller highlight (e.g., a highlighted portion/region) within the highlighted surface plane, a visual indicator within the highlighted surface plane, a projected slot/area within the highlighted surface plane, a visual outline (and/or boundaries) of the virtual content within the highlighted surface plane, and/or a representation of a virtual content placement within the highlighted surface plane. The content placement indicator (e.g., the smaller highlight, visual indicator, projected slot/area, visual outline, representation of the virtual content placement, etc.) can show/identify where the virtual content may be placed within the highlighted surface plane. In some cases, the content placement indicator can depict/identify where and/or how the virtual content would be placed and/or fit within the highlighted surface plane.

In some cases, when the user accepts (e.g., via a user input) a placement of the content placement indicator at an indicated location (e.g., a suggested and/or depicted location) within the highlighted surface plane and/or releases the virtual content item (e.g., during/after a positioning/repositioning of the content placement indicator) at the indicated location, the XR device can snap (e.g., anchor/attach/place) the content placement indicator to the indicated location within the highlighted surface plane. In some examples, the XR device can match (e.g., align, size/resize, fit, position/reposition, conform, etc.) the virtual content to the content placement indicator in the indicated location within the highlighted surface plane. For example, the XR device can position (e.g., set an orientation/rotation and/or location) the virtual content along various dimensions in space (e.g., along X, Y, and/or Z axes in space) according to the position of the content placement indicator at the indicated location. To illustrate, the XR device can place the virtual content in space so as to match the position of the virtual content to the position of the content placement indicator (and/or an associated plane) at the indicated location. In some cases, the XR device can size/scale the virtual content according to a size/scale of the content placement indicator. For example, the XR device can adjust a size/scale of the virtual content according to the size/scale of the content placement indicator so the virtual content item fits in/within the content placement indicator.

In some cases, prior to sizing/scaling, positioning and/or adjusting a position of the virtual content as described above, the XR device can configure (e.g., position, size/scale, shape, etc.) the content placement indicator to fit in/within a space of the highlighted surface plane that is associated with the indicated location and/or to match a configuration (e.g., location, orientation/rotation, perspective, size/scale, shape/geometry, etc.) of the space of the highlighted surface plane. For example, the XR device can configure or reconfigure (e.g., position, size/scale, shape/reshape, etc.) the content placement indicator so it fits in/within (and/or conforms to) the space of the highlighted surface plane associated with the indicated location and/or so as to match a configuration (e.g., location, orientation/rotation, perspective, size/scale, shape/geometry, etc.) of the space of the highlighted surface plane. The XR device can then configure or reconfigure (e.g., position/reposition, size/scale, shape/reshape, etc.) the virtual content according to the configuration (e.g., location, orientation/rotation, perspective, size/scale, shape/geometry, etc.) of the content placement indicator. The XR device can configure or reconfigure the virtual content so it fits in/within (and/or conforms to) the content placement indicator, which can also be anchored to the space of the highlighted surface plane associated with the indicated location and/or configured according to a configuration of the space of the highlighted surface plane that is associated with the indicated location.

In some cases, if a surface associated with a highlighted surface plane is cluttered (e.g., crowded with objects, messy, limited in available space, etc.), the XR device can occlude (e.g., hide/conceal, remove, etc.) at least some of the clutter from the highlight (and/or the highlighted surface plane) for easier positioning and/or viewing of the virtual content in/within the highlighted surface plane. Additionally or alternatively, if a surface associated with a highlighted surface plane is complex (e.g., uneven, rough/coarse/bumpy/textured, elevated, irregular, asymmetrical, curved, three-dimensional, of an uncommon shape, etc.), the XR device can, in some cases, adjust the highlighted surface plane based on a reconfigured (e.g., smoothened/flattened, reshaped, etc.) surface associated with the highlighted surface plane for easier positioning and/or viewing of the virtual content in/within the highlighted surface plane. The XR device can place and render the virtual content in/within the adjusted surface plane. In some cases, if a surface associated with a highlighted surface plane is complex, the XR device can position and/or render the virtual content (and/or the content placement indicator) on a raised surface plane parallel to the (reconfigured) surface, located/placed above the surface (e.g., at a threshold distance/height above the surface), etc., for easier positioning and/or viewing of the virtual content (and/or the content placement indicator) in/within the plane.

In some examples, the XR device can render the content placement indicator within the highlighted surface plane in the same aspect ratio as the virtual content item associated with the content placement indicator. In some cases, the XR device can resize the content placement indicator and the virtual content item associated with the content placement indicator to fit one or more available spaces within a highlighted surface plane. In some examples, as the user moves the virtual content item across the highlighted surface plane, the XR device can move the content placement indicator to available spaces within the highlighted surface plane (and, in some cases, within other highlighted surface planes). In some cases, as the user moves the virtual content item across the highlighted surface plane, the XR device may only move the content placement indicator to available spaces that can fit the content placement indicator (e.g., on which at least a threshold amount of the content placement indicator fits). When moving the content placement indicator, the XR device can skip over cluttered or unavailable areas in the highlighted surface plane, and move the content placement indicator toward the next available space or an adjacent available space in the highlighted surface plane (or another available surface plane). In some examples, the XR device can move the content placement indicator toward a next/adjacent available space in the same direction that the user (and/or the XR device) is moving the virtual content item. In some cases, the user can provide an input to the XR device to release and/or anchor the content placement indicator at an available space within the highlighted surface plane. For example, when the user (e.g., through a user input) releases the virtual content item, the XR device can snap the content placement indicator to a space/region of the surface plane corresponding to the location where the user released the virtual content item. When snapping the content placement indicator to a location, the XR device can match the content placement indicator's configuration (e.g., orientation/rotation, perspective, size/scale, shape, etc.) to the configuration of the space associated with that location.

In some cases, if a surface is too cluttered or uneven (e.g., above a threshold), the XR device can render a highlighted surface plane a distance from (e.g., above, in front, etc.) the higher points on (and/or portions of) that surface. When the user releases the virtual content on that surface, the XR device can place the virtual content at the same level as the highlighted surface plane. If there are items on the surface that are a certain distance from/above (e.g., threshold amount higher than) the average height of the rest of the surface and/or the height of other portions of the surface, the XR device can occlude or ignore those items so the highlighted surface plane is not placed too far above or in front of the rest of the surface.

In some cases, the physical environment may have multiple surfaces on which the content placement indicator and/or the virtual content item can be rendered. The XR device can highlight a specific surface plane as the user or the XR device moves the content placement indicator or the virtual content item. In some cases, to prevent flickering of highlighted surface planes when the user or XR device moves the content placement indicator or the virtual content item between surface planes (and/or when the content placement indicator or the virtual content item overlaps multiple surface planes), the XR device may wait to highlight a surface plane (or may not highlight a surface plane) until a threshold amount of the content placement indicator or the virtual content item is over the surface plane (e.g., from the viewpoint of the user).

In some cases, if the XR device determines that there is no space available for the virtual content item in its aspect ratio or the available space is not optimal for placing the virtual content, the XR device can suggest a different aspect ratio that maximizes the available space. For example, the XR device can adjust an aspect ratio of the content placement indicator or the virtual content item to fit or match an available space. The XR device can render the content placement indicator or the virtual content item on the available space according to the adjusted aspect ratio. For example, if the virtual content is configured according to a wide aspect ratio and an available space does not match or cannot fit the wide aspect ratio, the XR device can render the content placement indicator in the available space according to a narrower aspect ratio. The content placement indicator rendered in the narrower aspect ratio can indicate to the user that the virtual content may be rendered on the available space according to the narrower aspect ratio. If the user releases the virtual content or places the virtual content at the location of the content placement indicator, the XR device can change the virtual content to the narrower aspect ratio and render the virtual content at that location according to the narrower aspect ratio.

In some cases, the XR device can implement the techniques described herein in the context of a virtual reality (VR) session. For example, the XR device can implement the content placement indicator in a video pass through VR application where the XR device captures an image(s) of the physical environment (e.g., including one or more surfaces in the physical environment) and uses the captured image(s) to render the physical environment (e.g., including the one or more surfaces) in a VR display to provide visibility of the physical environment around the user. Here, the XR device can render the content placement indicator within the physical environment rendered/depicted by the XR device using the captured image(s) of the physical environment.

Examples of the systems and techniques described herein for processing data are illustrated inFIG.1throughFIG.8and described below.FIG.1is a diagram illustrating an example extended reality (XR) system100, in accordance with some examples of the disclosure. The XR system100can implement the systems and techniques disclosed herein. The XR system100can perform various tasks and operations such as, for example, extended reality tasks and operations (e.g., tracking, mapping, localization, content rendering, pose estimation, object detection/recognition, etc.), image/video processing and/or post-processing, data processing and/or post-processing, computer graphics, machine vision, object modeling and registration, multimedia rendering and/or composition, and/or any other data processing tasks, effects, and/or computations.

As used herein, placing a virtual content item on or within a surface or space in a scene of a physical environment means placing or anchoring the virtual content item on or within a location or area in a three-dimensional (3D) map of the scene that corresponds (the location or area) to the surface or space, such that the virtual content item appears to a user of the XR system100as if the virtual content item is located on or within the surface or space in the scene. Similarly, as used herein, placing a content placement indicator on or within a surface or space in a scene of the physical environment means placing or anchoring the content placement indicator on or within a location or area in a 3D map of the scene that corresponds (the location or area) to the surface or space, such that the virtual content item appears to a user of the XR system100as if the virtual content item is located on or within the surface or space in the scene.

In some examples, the XR system100can perform tracking and localization; pose estimation, mapping of the physical world/environment (e.g., a scene) around the XR system100(e.g., where the XR system100is located); and positioning and rendering of virtual content on a screen, display, and/or visible plane/region as part of an XR experience. For example, the XR system100can generate a map (e.g., a three-dimensional (3D) map) of a scene in the physical world, track a pose (e.g., a location and orientation) of the XR system100relative to the scene (e.g., relative to the 3D map of the scene), position and/or anchor virtual content in a specific location(s) on the map of the scene (e.g., corresponding to a physical location in the scene, such as a surface, a surface plane, a space within a surface or surface plane, etc.), and render the virtual content item and/or a virtual content placement indicator on a display/screen such that the virtual content item and/or a virtual content placement indicator appears to be at a physical location in the scene corresponding to the specific location on the map of the scene where the virtual content item is positioned and/or anchored.

In the example shown inFIG.1, the XR system100includes one or more image sensors102, one or more inertial sensors104(e.g., one or more inertial measurement units), one or more other sensors106(e.g., one or more radio detection and ranging (radar) sensors, light detection and ranging (LIDAR) sensors, acoustic/sound sensors, infrared (IR) sensors, magnetometers, touch sensors, laser rangefinders, light sensors, proximity sensors, motion sensors, active pixel sensors, machine vision sensors, ultrasonic sensors, etc.), storage108, compute components110, an XR engine120, an interface management engine122(e.g., a user interface management engine), an image processing engine124, and a rendering engine126. It should be noted that the components102through126shown inFIG.1are non-limiting examples provided for illustration and explanation purposes, and other examples can include more, less, and/or different components than those shown inFIG.1. For example, in some cases, the XR system100can include one or more display devices, one more other processing engines, one or more receivers (e.g., global positioning systems, global navigation satellite systems, etc.), one or more communications devices (e.g., radio frequency (RF) interfaces and/or any other wireless/wired communications receivers/transmitters), one or more other hardware components, and/or one or more other software and/or hardware components that are not shown inFIG.1. An example architecture and example hardware components that can be implemented by the XR system100are further described below with respect toFIG.8.

Moreover, for simplicity and explanation purposes, the one or more image sensors102will be referenced herein as an image sensor(s)102(e.g., in singular form). However, as previously noted, the XR system100can include a single image sensor or multiple image sensors. Also, references to any of the components of the XR system100in the singular or plural form should not be interpreted as limiting the number of such components implemented by the XR system100to one or more than one. For example, references to a processor in the singular form should not be interpreted as limiting the number of processors implemented by the XR system100to one. One of ordinary skill in the art will recognize that, for any of the components shown inFIG.1, the XR system100can include only one of such component(s) or more than one of such component(s).

The XR system100can be part of, or implemented by, a single computing device or multiple computing devices. In some examples, the XR system100can be part of an electronic device (or devices) such as a camera system (e.g., a digital camera, an IP camera, a video camera, a security camera, etc.), a telephone system (e.g., a smartphone, a cellular telephone, a conferencing system, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a smart television, a display device, a gaming console, a video streaming device, an IoT (Internet-of-Things) device, a smart wearable device (e.g., a head-mounted display (HMD), smart glasses, etc.), or any other suitable electronic device(s).

In some implementations, the one or more image sensors102, the inertial sensor(s)104, the other sensor(s)106, storage108, compute components110, XR engine120, interface management engine122, image processing engine124, and rendering engine126can be part of the same computing device. For example, in some cases, the one or more image sensors102, the inertial sensor(s)104, the other sensor(s)106, storage108, compute components110, XR engine120, interface management engine122, image processing engine124, and rendering engine126can be integrated into a smartphone, laptop, tablet computer, smart wearable device, gaming system, and/or any other computing device. In other implementations, the one or more image sensors102, the inertial sensor(s)104, the other sensor(s)106, storage108, compute components110, XR engine120, interface management engine122, image processing engine124, and rendering engine126can be part of two or more separate computing devices. For example, in some cases, some of the components102through126can be part of, or implemented by, one computing device and the remaining components can be part of, or implemented by, one or more other computing devices.

The image sensor(s)102can include any image and/or video sensors or capturing devices, such as a digital camera sensor, a video camera sensor, a smartphone camera sensor, an image/video capture device on an electronic apparatus such as a television or computer, a camera, etc. In some cases, the image sensor(s)102can be part of a camera or computing device such as an XR device (e.g., an HMD, smart glasses, etc.), a digital camera, a smartphone, a smart television, a game system, etc. In some examples, the image sensor(s)102can be part of a multiple-camera assembly, such as a dual-camera assembly. The image sensor(s)102can capture image and/or video content (e.g., raw image and/or video data), which can then be processed by the compute components110, the XR engine120, the interface management engine122, the image processing engine124, and/or the rendering engine126as described herein.

In some examples, the image sensor(s)102can capture image data and generate frames based on the image data and/or provide the image data or frames to the XR engine120, the interface management engine122, the image processing engine124and/or the rendering engine126for processing. A frame can include a video frame of a video sequence or a still image. A frame can include a pixel array representing a scene. For example, a frame can be a red-green-blue (RGB) frame having red, green, and blue color components per pixel; a luma, chroma-red, chroma-blue (YCbCr) frame having a luma component and two chroma (color) components (chroma-red and chroma-blue) per pixel; or any other suitable type of color or monochrome picture.

The inertial sensor(s)104can detect a specific force and angular rate of the XR system100. In some cases, the inertial sensor(s)104can detect an orientation of the XR system100. The inertial sensor(s) can generate linear acceleration measurements, rotational rate measurements, and/or heading measurements. In some examples, the inertial sensor(s)104can be used to measure the pitch, roll, and yaw of the XR system100. The other sensor(s)106can detect and generate other measurements used by the XR system100. In some cases, the XR engine120can use data and/or measurements from the image sensor(s)102, the inertial sensor(s)104, and/or the other sensor(s)106to track a pose of the XR system100. As previously noted, in other examples, the XR system100can also include other sensors, such as a magnetometer, an acoustic/sound sensors, an IR sensor, a machine vision sensor, a smart scene sensor, a radar sensor, a LIDAR sensor, a light sensor, etc.

The storage108can be any storage device(s) for storing data. Moreover, the storage108can store data from any of the components of the XR system100. For example, the storage108can store data from the image sensor(s)102(e.g., image or video data), data from the inertial sensor(s)104(e.g., measurements), data from the other sensor(s)106(e.g., measurements), data from the compute components110(e.g., processing parameters, preferences, virtual content, rendering content, scene maps, tracking and localization data, object detection data, configurations, XR application data, recognition data, outputs, etc.), data from the XR engine120, data from the interface management engine122, data from the image processing engine124, and/or data from the rendering engine126(e.g., output frames). In some examples, the storage108can include a buffer for storing frames for processing by the compute components110.

The one or more compute components110can include a central processing unit (CPU)112, a graphics processing unit (GPU)114, a digital signal processor (DSP)116, and/or an image signal processor (ISP)118. The compute components110can perform various operations such as image enhancement, computer vision, graphics rendering, extended reality (e.g., tracking, localization, pose estimation, mapping, content anchoring, content rendering, etc.), image/video processing, sensor processing, recognition (e.g., text recognition, facial recognition, object recognition, feature recognition, tracking or pattern recognition, scene recognition, occlusion detection, etc.), machine learning, filtering, object detection, and any of the various operations described herein. In the example shown inFIG.1, the compute components110can implement an XR engine120, an interface management engine122, an image processing engine124, and a rendering engine126. In other examples, the compute components110can also implement one or more other processing engines.

The operations for the XR engine120, the interface management engine122, the image processing engine124, and the rendering engine126(and any other processing engines) can be implemented by any of the compute components110. In one illustrative example, the operations of the rendering engine126can be executed by the GPU114, and the operations of the XR engine120, the interface management engine122, and the image processing engine124can be executed by the CPU112, the DSP116, and/or the ISP118. In some cases, the compute components110can include other electronic circuits or hardware, computer software, firmware, or any combination thereof, to perform any of the various operations described herein.

In some examples, the XR engine120can perform XR operations based on data from the image sensor(s)102, the inertial sensor(s)104, the other sensor(s)106, and/or one or more sensors on the XR system100, such as one or more LIDARS, radars, ultrasonic sensors, IR sensors, etc. In some examples, the XR engine120can perform tracking, localization, pose estimation, mapping, content anchoring operations and/or any other XR operations/functionalities.

The interface management engine122can manage and/or generate interface content presented/rendered by the XR system100, such as XR interfaces and/or interface objects. For example, the interface management engine122can manage and/or generate graphical user interfaces; control objects; visual indicators, such as the content placement indicator described herein, the surface and/or surface plane highlights described herein, etc.; interface elements; etc. In some examples, the interface management engine122can perform various operations to determine and/or manage how, where, and/or when to render user interfaces, visual indicators, control objects; etc., during an XR experience. An XR experience can include use of the XR system100to present XR content (e.g., virtual reality content, augmented reality content, mixed reality content, etc.) to a user associated with the XR system100(e.g., a user wearing the XR system100and/or otherwise using the XR system100for an XR experience.

In some examples, the XR content and experience can be provided by the XR system100through an XR application that provides a specific XR experience such as, for example, an XR gaming experience, an XR classroom experience, an XR shopping experience, an XR entertainment experience, an XR activity (e.g., an operation, a troubleshooting activity, etc.), among other. During the XR experience, the user can view and/or interact with virtual content using the XR system100. In some cases, the user can view and/or interact with the virtual content while also being able to view and/or interact with the physical environment around the user, allowing the user to have an immersive experience between the physical environment and virtual content mixed or integrated with the physical environment.

In some examples, the interface management engine122can use data from the image sensor(s)102, the inertial sensor(s)104, the other sensor(s)106, and/or any other sensors to detect objects (e.g., edges, surfaces, items on surfaces, windows, doors, walls, tables, books, devices, chairs, etc.) in a scene, planes in a scene, etc., identify characteristics of a scene and/or objects in a scene, identify occlusions in a scene, etc. In some examples, the interface management engine122can also use data from other devices or applications, such as data from or reported by other user devices and/or data provided by a user of the XR system100.

In some examples, the interface management engine122can coordinate with the XR engine120and/or the rendering engine126to render user interfaces, control objects, content placement indicators, surface highlights, surface plane highlights, etc. As further described herein, in some examples, the interface management engine122can detect surfaces in a scene on which virtual content can be placed for rendering, and the XR system100can visually identify such surfaces for the user. For example, the interface management engine122can determine a configuration (e.g., size, shape, orientation, perspective, location, gradient, etc.) of a surface in a scene, detect any objects located on the surface, and determine whether there are any available spaces within a surface plane associated with the surface, e.g., aligned with and/or offset from the surface, where a particular virtual content item can be placed for rendering. Available spaces within the surface plane may be determined as such spaces which are not occluded by any of the detected objects. For a surface plane being offset from the surface, part or all of the objects, depending on the value of the offset, can be occluded by the surface plane such that the available (non-occluded) space can be modified by changing the value of the offset.

In some cases, the XR system100(e.g., using the XR engine120, the interface management engine122, the image processing engine124, and/or the rendering engine126) can visually identify the surface plane (e.g., via highlighting, hatching, outlining, changing a visual attribute of the depicted surface to identify the surface as available, etc.) for the user, and the interface management engine122can depict a content placement indicator that appears to the user to be located within an available/non-occluded space on the surface plane. The content placement indicator can visually identify to the user the available space as a candidate location for placing and rendering the particular virtual content item. The interface management engine122can configure a content placement indicator based on the particular virtual content item. For example, the interface management engine122can determine a size and shape of the particular virtual content item, e.g., based on a preceding rendering of the virtual content item and/or a user input, and configure the content placement indicator to have the size and shape of the particular virtual content item. In some cases, the interface management engine122can determine a configuration of the content placement indicator and the associated virtual content item that allows them to fit within the available space and/or maintain a certain position, orientation, etc., relative to the available space and/or a surface plane. For example, the interface management engine122can change the shape and/or aspect ratio of the content placement indicator and the associated virtual content item if/as needed to fit within the available space.

When the XR system100renders the content placement indicator for the user, the content placement indicator can show how the virtual content item would fit and appear if placed within the available space (e.g., if anchored to the available space within the mapped scene and rendered to appear as if the virtual content item is located in/within the available space) where the content placement indicator is depicted to the user. After the content placement indicator is rendered for the user, the user can accept or reject (e.g., via a user input such as an input gesture, an eye gaze interpreted as an input, an input using a controller, or any other input means, e.g., for releasing a dragged virtual content item) the available space as the location for placement of the virtual content. If the user accepts the available space as the location for placement of the virtual content, the XR system100can anchor the virtual content item to the available space and render the virtual content item to appear as if located in the available space. If the user rejects the available space, the interface management engine122can coordinate with the XR system100to render another content placement indicator in a different available space (within the surface plane, or within a different surface plane).

In some examples, if the user attempts to move the content placement indicator to a different location/space, e.g., by moving or dragging the virtual content item across the scene, the interface management engine122can coordinate with the XR system100to depict the content placement indicator in a different available space. In some cases, the different available space can be an adjacent space. In other cases, if the adjacent space is unavailable (e.g., there are one or more objects and/or occlusions in that space), the interface management engine122can coordinate with the XR system100to skip the adjacent space (e.g., refrain from placing and depicting the content placement indicator in/within the adjacent space) and depict the content placement indicator in a different, non-adjacent available space.

The image processing engine124can perform one or more image processing operations. In some examples, the image processing engine124can perform image processing operations based on data from the image sensor(s)102. In some cases, the image processing engine124can perform image processing operations such as, for example, filtering, demosaicing, scaling, color correction, color conversion, segmentation, noise reduction filtering, spatial filtering, artifact correction, etc. The rendering engine126can obtain image data generated and/or processed by the compute components110, the image sensor(s)102, the XR engine120, the interface management engine122, and/or the image processing engine124, and render content (e.g., virtual content, videos, images, etc.) for presentation on a display device.

While the XR system100is shown to include certain components, one of ordinary skill will appreciate that the XR system100can include more or fewer components than those shown inFIG.1. For example, the XR system100can also include, in some instances, one or more memory devices (e.g., RAM, ROM, cache, and/or the like), one or more network interfaces (e.g., wired and/or wireless communications interfaces and the like), one or more display devices, and/or other hardware or processing devices that are not shown inFIG.1. An illustrative example of a computing device and hardware components that can be implemented with the XR system100is described below with respect toFIG.8.

FIG.2Ais a diagram illustrating an example scene200with various surfaces available for placing virtual content for rendering by the XR system100. In this example, the scene200is a physical environment mixed with a virtual content item202that has been rendered by the XR system100within the scene. In particular, the virtual content item202has been placed on a physical surface214(a wall in this example) in the scene200and rendered so as to appear to the user as if the virtual content item202is actually on the physical surface214. The scene200also includes other surfaces, such as surface212which in this example is a top surface of a table210, and surface216which in this example is a side of a bookcase.

Moreover, the user of the XR system100has rendered another virtual content item in a first perspective in a first location; namely virtual content item204in its original configuration/perspective, somewhere in the foreground on the scene200so the virtual content item204appears to be actually on the scene200. To assist the user in identifying a placement for the virtual content item204within the scene200, the XR system100can identify surfaces and/or spaces in the scene200available for placement of the virtual content item204. For example, as the user moves the virtual content item204through an input element220, the XR system100can identify for the user whether a particular surface and/or space within a surface is available for placement of the virtual content item204. The input element220can include any means for providing inputs to the XR system100such as, for example, a hand gesture (e.g., pointing a finger, moving a hand and/or positioning the hand in a certain way, and/or any other hand action or pose) that can be interpreted by the XR system100as an input, a head gesture (e.g., a certain motion and/or pose of the user's head interpreted by the XR system100as an input, etc.), an eye gaze (e.g., a movement of the eyes and/or an eye gaze towards a location that the XR system100can interpret as an input), an input through an input device (e.g., a controller, track pad, mouse, etc.), and/or any other input techniques and/or devices. In this example, the input element220includes a ray projected through ray casting.

As shown, the XR system100has identified a surface plane218as an available surface plane for placing the virtual content item204. The user can then use the input element220to place (e.g., snap, anchor, move, etc.) the virtual content item204within the surface plane218identified by the XR system100. The surface plane218can include the actual surface212of the table210or a plane relative to (e.g., parallel to, etc.) the surface212of the table210.

In the example shown inFIG.2A, the XR system100has rendered a content placement indicator on the surface plane218by highlighting the surface plane218(e.g., rendering a highlighted surface plane) to indicate that the surface plane218is available for placement and rendering of the virtual content item204. In other examples, the XR system100may additionally or alternatively indicate the surface plane218as available for placement in other ways. For example, in some cases, the XR system100can render an outline around the surface plane218as a content placement indicator, render the surface plane218with a visual pattern (e.g., with a color, gradient, shading, transparency, fill, line, text, shadow, reflection, glow, soft edges, virtual object, etc.) as a content placement indicator, render a visual indicator (e.g., an arrow, text, animation, image, visual effect, etc.) as a content placement indicator, and/or can provide any other rendering or visualization that can indicate the surface plane218as available.

FIG.2Bis a diagram illustrating the example scene200fromFIG.2Awith various physical objects on the surface212of the table210. In some examples, the XR system100can help the user avoid placing the virtual content item204on a space in the surface plane218that is occupied (occluded) by one or more objects on the surface212(or refrain from suggesting such placement) and/or to assist in finding available spaces within the surface plane218. For example, the XR system100can detect any available spaces within the surface plane218, such as any spaces that are not occupied (occluded) by objects on the surface212of the table210and/or any spaces that are not cluttered. As will be described further below, the XR system100can detect any available spaces within a surface plane that is offset from the associated surface in a direction normal to the surface based on a detection of any spaces that are not occluded by objects. Such objects may be objects on the surface212or any other objects in the scene.

In some examples, the XR system100can perform surface detection to detect one or more surfaces in the physical environment. The XR system100can also perform edge detection to determine any edges of any surfaces and/or objects in the physical environment. The XR system100can determine a pose of the XR system100in the physical environment. For example, the XR system100can determine a pose, orientation, and/or perspective of the XR system100based on any detected edges, surfaces, planes, and/or objects in the physical environment. In some examples, the XR system100can perform object detection to detect objects in the scene200. For example, the XR system100can perform object detection to detect objects on the surface212of the table210in the scene200. The XR system100can localize and map any detected objects to track and/or understand the location of the detected objects within the scene200(e.g., within a three-dimensional (3D) map of the scene200used by the XR system100to provide XR functionalities/experiences), including any detected objects within the surface212. The XR system100can then determine any spaces within the surface plane218that are available for placing, e.g., by determining portions of the surface plane218occluded by objects and segmenting the surface plane into segments, for instance into rectangular spaces, based on the determined occluded portions such that the segments are not occluded by the objects, and rendering the virtual content item204(e.g., any spaces within the surface plane218that are not occupied/occluded by objects on the surface212or elsewhere). For example, the XR system100can analyze the 3D map of the scene to identify any spaces/locations within the surface plane218that are not associated with mapped objects (e.g., to identify any spaces/locations in the 3D map that correspond to the surface plane218and that do not have a mapped object associated with them).

After detecting any spaces within the surface plane218that are available for placing and rendering the virtual content item204, the XR system100can render one or more content placement indicators indicating one or more available spaces on the surface plane218. In some cases, the XR system100can render only one content placement indicator at any point in time indicating an available space on the surface plane218. The location of this single content placement indicator can be determined based on a proximity to (an anchor point/trip of) the input element220, a proximity to a representation of the virtual content item in a first perspective which may be moved/dragged by means of a user input, an overlap between such a representation and the respective available space, or the like. For example, the XR system100can place the content placement indicator206on an available space208detected by the XR system100. The content placement indicator206can indicate the available space208as a candidate location for placing the virtual content item204. Thus, the content placement indicator206indicates to the user that the available space208is a location available for placing and rendering the virtual content item204. In some cases, the content placement indicator206indicates to the user which location the virtual content item204will automatically be anchored to by the XR system100in response to a user input, such as releasing the dragged virtual content item. If the user selects the content placement indicator206(e.g., if the user accepts placing the virtual content item204on the space208of the surface plane218identified by the content placement indicator206), the XR system100can place the virtual content item204on the space208associated with the content placement indicator206. The XR system100can render the virtual content item204so as to appear as if the virtual content item204is on the space208associated with the content placement indicator206.

In the example shown inFIG.2B, the XR system100has rendered one content placement indicator. However, in some cases, the XR system100can render more than one content placement indicator. For example, the XR system100can render the content placement indicator206and one or more additional content placement indicators associated with one or more additional spaces available for placing and rendering the virtual content item204. The one or more additional spaces can include one or more spaces on the surface plane218highlighted by the XR system100. In some cases, the one or more additional spaces can include spaces available on different surfaces/surface planes in the scene. For example, the one or more additional spaces can include a space available on surface214(which the XR system may or may not highlight for the user), and/or a space available on surface216(which the XR system may or may not highlight for the user).

In some cases, when the user moves the virtual content item204within the scene200(e.g., via input element220), the XR system100can move the content placement indicator206to a specific available space on the surface plane218. Moreover, when moving the content placement indicator206, the XR system100can skip (e.g., refrain from placing/rendering) an object(s) on the surface212. For example, if the XR system100moves the content placement indicator206from the available space209on the surface plane218, rather than placing the content placement indicator206on the adjacent space on the surface plane218that is currently occupied/occluded by the object230, the XR system100can skip over the object230and place the content placement indicator206on the available space208in the surface plane218. Thus, the XR system100can refrain from placing the content placement indicator206on the object230(or the space occupied by the object230). In other words, the XR system100can switch from rendering the content placement indicator in a first space209of the surface plane218to rendering the content placement indicator206in a second space208of the surface plane218. By switching from one rendering location to the next, the XR system can ensure that the user is always presented with a suggesting for placing the virtual content item. As described above, the suggested space indicated by the currently rendered content placement indicator can be determined based on a proximity to the virtual content item204and/or an overlap with the virtual content item204.

In some examples, the XR system100can occlude clutter from the surface plane218. For example, the XR system100can render an occlusion240on a cluttered space of the surface212to occlude the clutter from the surface plane218.

FIG.3Ais a diagram illustrating an example for selecting a surface plane for highlighting when a location of a virtual content item310overlaps different surfaces (or portions thereof) on the scene200. As described above, highlighting or gradual highlighting of a surface can be used as a respective content placement indicator. As shown in this example, a portion312of the virtual content item310rendered in a first perspective in a first location, and possibly moved via user input, overlaps with a portion of the surface212, a portion314of the virtual content item310overlaps with a portion of the surface216, and a portion316of the virtual content item310overlaps with a portion of the surface214. In some examples, rather than highlight surface planes corresponding to all of the surfaces212,214, and216, the XR system100can select a particular surface plane to highlight from the surfaces212,214, and216.

In some cases, the XR system100can determine which surface has the largest amount of overlap with the virtual content item310and select that surface for highlighting (and/or for rendering a highlighted surface plane). For example, the XR system100can determine which of the portions312,314, and316of the virtual content item310that overlap with the surfaces212,214, and216is largest, and select the surface plane associated with the largest portion of the virtual content item204. In this example, the portion312is larger than the portion314and316. Thus, the XR system100can highlight the surface plane218corresponding to the surface212associated with the portion312.

In some cases, the XR system100can determine if any of the surfaces212,214, and216contains a threshold amount or percentage of the virtual content item310. If a threshold amount or percentage of the virtual content item310is on/over a surface, the XR system100can select that surface for highlighting (and/or for highlighting a surface plane). For example, the XR system100can determine if any of the portions312,314, and316of the virtual content item310overlapping with the surfaces212,214, and216satisfies or exceeds a threshold. The threshold can be chosen, e.g. at 50% or more, such that only a single surface plane is highlighted at a time. The XR system100can select the surface associated with the portion of the virtual content item310(e.g., the portion from the portions312,314, and316) that satisfies or exceeds the threshold.

In some examples, the XR system100can switch highlighted surface planes, e.g., as rendered content placement indicators, when the user moves the virtual content item310(e.g., via input element220). For example, as shown inFIG.3B, the user has moved the virtual content item310such that the portion314of the virtual content item310that overlaps with the surface216exceeds the threshold and/or is larger than the portion312of the virtual content item310overlapping with the surface212and the portion316of the virtual content item310overlapping with the surface214. The XR system100can determine that the portion314of the virtual content item310that overlaps with the surface216exceeds the threshold and/or is larger than the portion312of the virtual content item310overlapping with the surface212and the portion314of the virtual content item310overlapping with the surface214, and highlight the surface plane320corresponding to the surface216associated with the portion314of the virtual content item310that exceeds the threshold and/or is larger than the portions312and314of the virtual content item310. Thus, the XR system100can switch from rendering a content placement indicator by highlighting the surface plane218associated with the surface212as shown inFIG.3A, to rendering a content placement indicator by highlighting the surface plane320associated with the surface216as shown inFIG.3B.

In some cases, to prevent flickering between surface plane highlights, the XR system100can set a particular threshold for determining which surface to select as previously described. In some examples, the XR system100can set the threshold to at least 60% to prevent flickering between surface plane highlights. For example, the XR system100can select a surface/surface plane when at least 60% of the virtual content item310overlaps with that surface. To illustrate, if the portion314of the virtual content item310includes at least 60% of the virtual content item310, the XR system100can select the surface plane320corresponding to the surface216that overlaps with the portion314of the virtual content item310.

In some cases, the XR system100can similarly switch between available spaces on a highlighted surface plane. For example, when the XR system100moves a content placement indicator (e.g., content placement indicator206) within a highlighted surface plane, if the content placement indicator overlaps with several available spaces on the highlighted surface plane, the XR system100can select the available space having the largest amount of overlap with the content placement indicator and place/render the content placement indicator in that available space. As another example, when the XR system100moves a content placement indicator (e.g., content placement indicator206) within a highlighted surface plane, if the content placement indicator overlaps with several available spaces on the highlighted surface plane, the XR system100can select the available space having an overlap with the content placement indicator that meets or exceeds a certain threshold, and place/render the content placement indicator in that available space. The described moving/switching of the content placement indicator between available spaces can be performed without highlighting the surface plane. The content placement indicator can further be in the form of a brightness/color grading of the highlight of the surface plane to indicate the respective available space.

In some cases, if the XR system100determines that there is no space available in a surface plane for the virtual content item in its aspect ratio or an available space is not optimal for placing the virtual content, the XR system100can suggest a different aspect ratio for the virtual content item that maximizes the space available and/or fits within an available space. The suggestion can be in the form of changing the content placement indicator from a first to a second aspect ratio upon switching between the available spaces. For example, the XR system100can adjust an aspect ratio of the content placement indicator and/or the virtual content item to fit or match an available space. The XR system100can render the resized content placement indicator and/or the resized virtual content item on the available space according to the adjusted aspect ratio.

FIG.4is a diagram illustrating an example adjustment of an aspect ratio of a content placement indicator404to fit within an available space on the surface plane218. In this example, the surface212of the desk210in the scene200is covered by objects. The XR system100has identified the space420between the objects410,412,414,416, and418as an available space, e.g., via object detection, occlusion detection, and segmentation. However, the virtual content item402does not fit the space420in the current aspect ratio of the virtual content item402(as rendered in the first perspective). Accordingly, the XR system100can render the content placement indicator404in the space420according to an aspect ratio that is narrower than the aspect ratio of the virtual content item402.

For example, the XR system100can adjust the aspect ratio of the content placement indicator404from a wide aspect ratio corresponding to the aspect ratio of the virtual content item402to a narrower aspect ratio that allows the content placement indicator404to fit within the space420. The XR system100can place the content placement indicator404within the space420according to the narrower aspect ratio, and render the content placement indicator404so as to appear located in the narrower aspect ratio and within the space420. When the user releases the virtual content item402and/or snaps (e.g., places/anchors) the virtual content item402to the content placement indicator404(and/or the space420associated with the content placement indicator404), the XR system100can change the aspect ratio of the virtual content item402to match that of the content placement indicator404, and place the virtual content item402in the space420according to the adjusted aspect ratio. The XR system100can render the virtual content item402in the adjusted aspect ratio so as to appear to be located on the space420. Adjustment of size and/or aspect ratio of the content placement indicator404can be conditioned on a minimum height and/or width of the virtual content item after placing/rendering in the current view of the user. In other words, spaces on the surface plane218can be determined as not available/not suitable for rendering the virtual content item based on their size (height and/or width) in the current view being smaller than a minimum (height and/or width) threshold. Such spaces will not explicitly be indicated by the XR system100by rendering a dedicated content placement indicator (such as a gradient highlight and/or visual outline) on the respective spaces.

In some examples, the capability for adjusting the aspect ratio for placing the virtual content item402can be enabled by a user setting or enabled by default. In some cases, the adjusted aspect ratio and/or the configuration/parameters of the adjusted virtual content item402(and/or the adjusted content placement indicator) can be defined by a user setting or can be dictated by the virtual content item402. For example, in some cases, the adjusted aspect ratio of the virtual content item402(and/or whether the aspect ratio of the virtual content item402can be adjusted) can depend on the type of virtual content of the virtual content item402and/or whether the virtual content item402is suitable or capable of resizing. For example, a virtual content item including text content can represent a type of virtual content not suitable for adjustment of the aspect ratio.

In some cases, if a surface does not have available spaces, e.g., non-occluded spaces having at least a minimum size, (and/or has a threshold amount of clutter) or if a surface is a complex surface (e.g., uneven, rough/coarse/bumpy/textured, elevated, irregular, asymmetrical, curved, three-dimensional, of an uncommon shape, etc.), the XR system100can determine a surface plane associated with the surface which is generally aligned with the surface in terms of reflecting the overall perspective (orientation) of the complex surface. The surface plane can be seen as a simplification of the complex surface, e.g., by smoothening/flattening, reshaping, etc. the surface, performing edge detection on the complex surface to determine a perspective of the complex surface, detecting and removing objects on the surface, etc. The surface plane can be determined as a planar representation of the complex surface for easier positioning and/or viewing of the virtual content item in/within the highlighted surface plane. The surface plane can be located at an average surface level of the complex surface or be offset by an offset value in a direction normal to the surface plane/simplified complex surface. The offset can be toward a viewpoint of the user. As a result, the adjusted (offset) surface plane will occlude at least part of the complex surface and therefore, make (more) spaces available for rendering the virtual content item. The XR system100can place and render the virtual content item in/within the adjusted surface plane. In some cases, if the surface is complex, the XR system100can position and/or raise/elevate/offset the highlighted surface plane and the content placement indicator and render them on a raised/elevated plane relative to (e.g., generally parallel to) the complex surface. The highlighted surface plane/content placement indicator can be located/placed above the surface (e.g., at a threshold distance/height above the surface), etc., for easier positioning and/or viewing of the virtual content item (and/or the content placement indicator) in/within the surface plane.

FIG.5Ais a diagram illustrating an example surface plane512and content placement indicator504that are rendered at a certain position relative to a surface510that is complex. As shown, the surface510is uneven because the surface510is created by books on a bookcase that are of different sizes. Thus, the surface510is not flat/even and does not have an available space that is flat or even. In some cases, the perspective or orientation of the surface plane512can be determined based on edge detection (of the books on the shelf) and identifying parallel edges and their common orientation. The XR system100can move the surface plane512and content placement indicator504rendered to be a certain distance (offset) from the surface510toward the viewpoint of the user. For example, the XR system100can move the surface plane512and content placement indicator504to be a threshold distance away from (e.g., above, in front, etc.) the surface510(e.g., from the viewpoint of the user of the XR system100). The XR system100can render the surface plane512and content placement indicator504at the moved position, which is a threshold distance from the surface510.

With reference toFIG.5B, when the user releases the virtual content item502on the content placement indicator504and/or accepts the placement of the virtual content item502within the content placement indicator504, the XR system100can place the virtual content item502in the content placement indicator504and render the virtual content item502at the position associated with the content placement indicator504at the time of releasing the virtual content item502, which is a threshold distance away from the surface510as previously explained. As previously explained, the content placement indicator504may be moved across the surface plane512based on a movement of the virtual content item502in response to user input.

FIG.5Cis a diagram illustrating another example surface plane522and content placement indicator530that are rendered at a certain position relative to (offset from) a surface520that is complex. The surface520in this example is a wall made up of irregularly-shaped (e.g., curved) cement blocks that create an irregular/uneven surface. Thus, the surface520is not flat but rather irregular/uneven. As shown, since the surface520is irregular/uneven, the XR system100can determine a perspective (orientation) of the surface plane522based on a simplified (smoothened, flattened, reshaped, etc.) surface. Additionally or alternatively, edge detection can be performed to determine the perspective (orientation) of the surface plane522. The XR system can move/offset the surface plane522and content placement indicator530rendered to be/appear a certain distance/offset in front of the surface520, i.e., toward a viewpoint of the user. For example, the XR system100can move the surface plane522and content placement indicator530to be a threshold distance in front of the surface520(e.g., from the viewpoint of the user of the XR system100). The XR system100can render the surface plane522and content placement indicator530at the moved/offset position that is a threshold distance in front the surface520.

With reference toFIG.5D, when the user releases the virtual content item502on the content placement indicator530and/or accepts the placement of the virtual content item502within the content placement indicator530, the XR system100can place the virtual content item502in the content placement indicator530and render the virtual content item502at the position associated with the content placement indicator530at the time of releasing the virtual content item, which is a threshold distance in front of the surface520as previously explained.

FIG.6is a diagram illustrating an example rendering of a highlighted surface plane and a content placement indicator in a scene that has a surface with no available spaces (fulfilling the minimum size requirement). In this example, the surface602of a table is cluttered with items and does not have a space available (e.g., that is not occupied/occluded by any items) for placing the virtual content item610. Since the surface602does not have available space, the XR system100can elevate/raise the highlighted surface plane604a certain distance/height above the surface602, as well as the content placement indicator606rendered on the highlighted surface plane604. As in the previous examples, the table top cluttered with items can be considered a complex surface which may be simplified as described above to determine an associated surface plane. The surface plane604can be offset from an average surface level of the complex surface by a specific offset value as described in the following.

In some examples, if there are one or more objects that extend beyond the surface602a threshold distance/amount more than other items on the surface602, the XR system100may place the highlighted surface plane604at certain distance/offset away from the surface602that allows the highlighted surface plane604to partially occlude the one or more objects. In other words, the distance from the surface602may not be sufficient to completely occlude those one or more objects that extend beyond the surface602a threshold distance/amount more than other items on the surface602.

For example, inFIG.6, the candles620and622and the water bottle624on the surface602are significantly taller than the rest of the items on the surface602. Consequently, the candles620and622and the water bottle624extend a significant amount above the rest of the items relative to the surface602. Given the height of the candles620and622and the water bottle624, if the highlighted surface plane604is raised/elevated above the candles620and622and the water bottle624, the raised/elevated position of the highlighted surface plane604may cause the highlighted surface plane604to appear too far above the surface602. In other words, at such height, the highlighted surface plane604may not appear to the user to be on the surface602or close to the surface602, but may rather appear to be floating in the air.

To prevent this, the XR system100can implement a rule to define thresholds and/or ranges for moving (e.g., elevating/raising/offsetting) the highlighted surface plane604relative to the surface602and/or the other items on the surface602. If the farthest point of an object(s) (e.g., the candles620and622and the water bottle624) relative to the surface602(and/or the top or average height of the rest of the items on the surface602) exceeds a defined threshold and/or range, the XR system100can place/render the highlighted surface plane604at a distance from the surface602and/or the rest of the items on the surface602that is within the threshold and/or range, even if a portion of the object(s) (e.g., the candles620and622and the water bottle624) will be occluded by the highlighted surface plane604while another portion of the object(s) extends beyond the highlighted surface plane604(e.g., even if the highlighted surface plane604only partially occludes the object(s)).

For example, inFIG.6, the farthest points of the candles620and622and the water bottle624relative to the surface602(and/or the top or average height of the other items on the surface602) exceed a threshold height640above the surface602. Accordingly, the highlighted surface plane604is placed and rendered within the threshold height640above the surface602. Thus, the highlighted surface plane604partially occludes the candles620and622and the water bottle624but is not elevated/raised to or beyond the surface height630, which includes the farthest points of the candles620and622and the water bottle624relative to the surface602and/or the other items on the surface602. This way, the highlighted surface plane604is not moved too far away from the surface602and/or the other items on the surface602.

Alternatively, the surface plane604can be moved by increasing an offset until at least one space on the moved/offset surface plane becomes available for rendering the virtual content item610. To this end, an initial location of the surface plane can be determined, e.g., at an average height above the surface602. As previously described, parts of the initial surface plane occluded by objects such as the candles620and622and the water bottle624can be determined and available space(s) (fulfilling the minimum size requirement) can be determined based on the non-occluded parts of the initial surface plane. As the initial surface plane is offset/raised relative to the surface602, the initial surface plane if highlighted partially occludes the candles620and622and the water bottle624as well as most, if not all of the other items on the surface602. If it is determined that the initial surface plane does not have any available space (fulfilling the minimum size requirement), the offset of the surface plane can be (repeatedly) increased, e.g., by a predefined amount, and the above described process repeated until at least one space (fulfilling the minimum size requirement) becomes available for rendering the virtual content item. In this way, even complex surfaces can be processed to determine a surface plane suitable for rendering the virtual content item.

FIG.7is a flowchart illustrating an example process700for placing a virtual content item on a surface of a physical scene. At block702, the process700can include determining at least one surface plane in a scene of a physical environment. In some cases, determining the at least one surface plane in the scene can include determining one or more surfaces of the one or more objects in the scene, and determining the at least one surface plane based on the one or more surfaces of the one or more objects in the scene.

At block704, the process700can include rendering a virtual content item in a first perspective at a first location in the scene.

At block706, the process700can include rendering a content placement indicator associated with the virtual content item on a first space of the at least one surface plane. In some examples, the content placement indicator can indicate at least part of the first space for placement of the virtual content item.

At block708, the process700can include determining one or more locations of one or more objects on a surface associated with the at least one surface plane.

At block710, the process700can include, in response to a user input, rendering the content placement indicator on a second space of the at least one surface plane. In some examples, the second space of the at least one surface plane is determined based on the one or more locations and the user input. In some examples, the content placement indicator can indicate at least part of the second space for placement of the virtual content item.

In some aspects, the process700can include, in response to an additional user input requesting placement of the virtual content item on the second space, rendering the virtual content item in a second perspective at a second location in the scene based on the content placement indicator. In some examples, the second location can be within the second space. In some cases, the process700can include determining the second perspective based on a perspective of the second space.

In some aspects, the process700can include determining a first surface plane and a second surface plane of the at least one surface plane, the first space being associated with the first surface plane and the second space being associated with the second surface plane; and determining the second perspective based on a perspective of the second surface plane.

In some examples, rendering the content placement indicator on the second space can include determining an overlap of the virtual content item with the second surface plane, and upon determination that the overlap exceeds a first threshold, switching to rendering the content placement indicator on the second space associated with the second surface plane.

In some examples, the first space and the second space are associated with a first surface plane of the at least one surface plane. In some aspects, the process700can include detecting one or more objects, wherein the one or more objects occlude at least part of the first surface plane; and segmenting the first surface plane into at least a first segment and a second segment based at least on an occluded part of the first surface plane. In some examples, the first space is associated with the first segment and the second space is associated with the second segment.

In some cases, segmenting the first surface plane into at least the first segment and the second segment can include determining a fit of the virtual content item rendered in a perspective of the first surface plane in the first segment and the second segment.

In some aspects, the process700can include changing an aspect ratio of the virtual content item between the first segment and the second segment.

In some aspects, the process700can include prior to rendering the content placement indicator on the second space, rendering the virtual content item in a third perspective at a third location in the scene based on the content placement indicator. In some examples, the third location can be within the first space.

In some aspects, the process700can include, in response to the user input, moving the content placement indicator within the first space prior to rendering the content placement indicator on the second space.

In some cases, determining the at least one surface plane in the scene can include applying an offset to the at least one surface plane normally to at least one surface and toward a view point of a user associated with the computing device.

In some cases, determining the at least one surface plane in the scene can include determining one or more edges of the one or more objects in the scene, and determining the at least one surface plane based on the one or more edges of the one or more objects in the scene.

In some cases, rendering the content placement indicator can include rendering a pattern on the at least one surface plane. In some examples, the pattern can include at least one of a highlight, an outline, a color, a shade, a shadow, a hatching, and a gradient.

In some examples, the processes described herein (e.g., process700, and/or any other process described herein) may be performed by a computing device or apparatus. In one example, the process700can be performed by the XR system100ofFIG.1. In another example, the process700can be performed by the computing system having the computing device architecture800shown inFIG.8. For instance, a computing device with the computing device architecture800shown inFIG.8can implement the operations ofFIG.7and/or the components and/or operations described herein with respect to any ofFIGS.1through7.

The computing device can include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected watch or smartwatch, or other wearable device), a server computer, an autonomous vehicle or computing device of an autonomous vehicle, a robotic device, a laptop computer, a smart television, a camera, and/or any other computing device with the resource capabilities to perform the processes described herein, including the process700and/or any other process described herein. In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and/or other component(s) that are configured to carry out the steps of processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and/or receive the data, any combination thereof, and/or other component(s). The network interface may be configured to communicate and/or receive Internet Protocol (IP) based data or other type of data.

FIG.8is a diagram illustrating an example of a system for implementing certain aspects of the present technology. In particular,FIG.8illustrates an example of computing system800, which can be for example any computing device making up internal computing system, a remote computing system, a camera, or any component thereof in which the components of the system are in communication with each other using connection805. Connection805can be a physical connection using a bus, or a direct connection into processor810, such as in a chipset architecture. Connection805can also be a virtual connection, networked connection, or logical connection.

Example system800includes at least one processing unit (CPU or processor)810and connection805that couples various system components including system memory815, such as read-only memory (ROM)820and random access memory (RAM)825to processor810. Computing system800can include a cache812of high-speed memory connected directly with, in close proximity to, or integrated as part of processor810.

Processor810can include any general purpose processor and a hardware service or software service, such as services832,834, and836stored in storage device830, configured to control processor810as well as a special-purpose processor where software instructions are incorporated into the actual processor design. Processor810may essentially be a completely self-contained computing system, containing multiple cores or processors, a bus, memory controller, cache, etc. A multi-core processor may be symmetric or asymmetric.

The storage device830can include software services, servers, services, etc., that when the code that defines such software is executed by the processor810, it causes the system to perform a function. In some embodiments, a hardware service that performs a particular function can include the software component stored in a computer-readable medium in connection with the necessary hardware components, such as processor810, connection805, output device835, etc., to carry out the function. The term “computer-readable medium” includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other mediums capable of storing, containing, or carrying instruction(s) and/or data. A computer-readable medium may include a non-transitory medium in which data can be stored and that does not include carrier waves and/or transitory electronic signals propagating wirelessly or over wired connections.

Illustrative examples of the disclosure include:

Aspect 1. An apparatus for rendering virtual content in a scene of a physical environment, the apparatus comprising: memory; and one or more processors coupled to the memory, the one or more processors being configured to: determine at least one surface plane in the scene of the physical environment; render a virtual content item in a first perspective at a first location in the scene; render a content placement indicator associated with the virtual content item on a first space of the at least one surface plane, the content placement indicator indicating at least part of the first space for placement of the virtual content item; determine one or more locations of one or more objects on a surface associated with the at least one surface plane; and in response to a user input, render the content placement indicator on a second space of the at least one surface plane, wherein the second space of the at least one surface plane is determined based on the one or more locations and the user input, the content placement indicator indicating at least part of the second space for placement of the virtual content item.

Aspect 2. The apparatus of Aspect 1, wherein the one or more processors are further configured to: in response to an additional user input requesting placement of the virtual content item on the second space, render the virtual content item in a second perspective at a second location in the scene based on the content placement indicator, the second location being within the second space.

Aspect 3. The apparatus of Aspect 2, wherein the one or more processors are further configured to: determine the second perspective based on a perspective of the second space.

Aspect 4. The apparatus of Aspect 2, wherein the one or more processors are further configured to: determine a first surface plane and a second surface plane of the at least one surface plane, the first space being associated with the first surface plane and the second space being associated with the second surface plane; and determine the second perspective based on a perspective of the second surface plane.

Aspect 5. The apparatus of Aspect 4, wherein, to render the content placement indicator on the second space, the one or more processors are configured to: determine an overlap of the virtual content item with the second surface plane; and upon determination that the overlap exceeds a first threshold, switch to rendering the content placement indicator on the second space associated with the second surface plane.

Aspect 6. The apparatus of any of Aspects 1 to 5, wherein the first space and the second space are associated with a first surface plane of the at least one surface plane; and wherein the one or more processors are further configured to: detect one or more objects, wherein the one or more objects occlude at least part of the first surface plane; and segment the first surface plane into at least a first segment and a second segment based at least on an occluded part of the first surface plane; wherein the first space is associated with the first segment and the second space is associated with the second segment.

Aspect 7. The apparatus of Aspect 6, wherein to segment the first surface plane into at least the first segment and the second segment, the one or more processors are configured to determine a fit of the virtual content item rendered in a perspective of the first surface plane in the first segment and the second segment.

Aspect 8. The apparatus of any of Aspects 6 to 7, wherein the one or more processors are further configured to change an aspect ratio of the virtual content item between the first segment and the second segment.

Aspect 9. The apparatus of any of Aspects 1 to 8, wherein the one or more processors are configured to: prior to rendering the content placement indicator on the second space, render the virtual content item in a third perspective at a third location in the scene based on the content placement indicator, the third location being within the first space.

Aspect 10. The apparatus of any of Aspects 1 to 9, wherein the one or more processors are further configured to: in response to the user input, move the content placement indicator within the first space prior to rendering the content placement indicator on the second space.

Aspect 11. The apparatus of any of Aspects 1 to 10, wherein, to determine the at least one surface plane in the scene, the one or more processors are configured to: determine one or more surfaces of the one or more objects in the scene; and determine the at least one surface plane based on the one or more surfaces of the one or more objects in the scene.

Aspect 12. The apparatus of Aspect 11, wherein, to determine the at least one surface plane in the scene, the one or more processors are further configured to: apply an offset to the at least one surface plane normally to at least one surface and toward a view point of a user associated with the apparatus.

Aspect 13. The apparatus of any of Aspects 1 to 12, wherein, to determine the at least one surface plane in the scene, the one or more processors are configured to: determine one or more edges of the one or more objects in the scene; and determine the at least one surface plane based on the one or more edges of the one or more objects in the scene.

Aspect 14. The apparatus of any of Aspects 1 to 13, wherein, to render the content placement indicator, the one or more processors are configured to render a pattern on the at least one surface plane.

Aspect 15. The apparatus of Aspect 14, wherein the pattern comprises at least one of a highlight, an outline, a color, a shade, a shadow, a hatching, and a gradient.

Aspect 16. The apparatus of any of Aspects 1 to 15, wherein the apparatus comprises a mobile device.

Aspect 17. The apparatus of Aspect 16, wherein the mobile device comprises an extended reality device.

Aspect 18. The apparatus of any of Aspects 1 to 17, wherein the first space and the second space comprise spaces on the at least one surface plane that are at least one of unoccupied by any physical objects and free from any physical objects that visually obstruct a visibility of respective portions of the at least one surface plane associated with the first space and the second space.

Aspect 19. The apparatus of any of Aspects 1 to 18, wherein the second space comprises an available space on the at least one surface plane that is occupied by one or more physical objects in the physical scene.

Aspect 20. The apparatus of any of Aspects 1 to 19, wherein the one or more processors are configured to: based on an aspect ratio of the virtual content item, determine that at least a threshold amount of the virtual content item does not fit within the second space; determine that at least the threshold amount of the virtual content item fits within the second space when the virtual content item is configured according to a different aspect ratio; configure the content placement indicator according to the different aspect ratio; and render the content placement indicator on the second space according to the different aspect ratio.

Aspect 21. The apparatus of Aspect 20, wherein the one or more processors are configured to: adjust the aspect ratio of the virtual content item to the different aspect ratio; and in response to an additional input requesting placement of the virtual content item on the second space, render the virtual content item in the different aspect ratio, the virtual content item being rendered on the second space and within the content placement indicator.

Aspect 22. The apparatus of any of Aspects 1 to 21, wherein, to determine the at least one surface plane in the scene, the one or more processors are configured to: determine one or more surfaces of the one or more objects in the scene; and determine the at least one surface plane based on the one or more surfaces of the one or more objects in the scene.

Aspect 23. The apparatus of Aspect 22, wherein the one or more processor are further configured to apply smoothing to the one or more surfaces of the one or more objects in the scene prior to determining the at least one surface plane based on the one or more surfaces of the one or more objects in the scene.

Aspect 24. The apparatus of Aspect 22, wherein, to determine the at least one surface plane in the scene, the one or more processors are configured to: align the at least one surface plane with at least one surface of the one or more surfaces of the one or more objects in the scene.

Aspect 25. The apparatus of any of Aspects 1 to 24, wherein the one or more processors are configured to: render the at least one surface plane relative to the at least one surface.

Aspect 26. The apparatus of Aspect 25, wherein, to render the at least one surface plane, the one or more processors are configured to: determine that an amount of flatness of the surface is below a threshold; and render the at least one surface plane a distance from one or more points on the surface associated with the at least one surface plane.

Aspect 27. The apparatus of Aspect 25, wherein, to render the at least one surface plane, the one or more processors are configured to: determine that a highest point of the surface is a first threshold distance above one or more other points of the surface; and render the at least one surface plane a second threshold distance above the highest point of the surface.

Aspect 28. The apparatus of any of Aspects 1 to 27, wherein, to render the content placement indicator, the one or more processors are configured to render a visual outline on the at least one surface plane.

Aspect 29. The apparatus of Aspect 28, wherein the visual outline comprises at least one of a frame or a bounding box.

Aspect 30. The apparatus of any of Aspects 28 to 29, wherein the one or more processors are further configured to render the visual outline in a perspective of the respective surface plane.

Aspect 31. The apparatus of any of Aspects 1 to 30, wherein, to render the first content placement indicator and/or the second content placement indicator, the one or more processors are configured to render a preview of the virtual content item on the at least one surface plane.

Aspect 32. A method for rendering virtual content in a scene of a physical environment, the method comprising: determining at least one surface plane in the scene of the physical environment; rendering, via a computing device, a virtual content item in a first perspective at a first location in the scene; rendering a content placement indicator associated with the virtual content item on a first space of the at least one surface plane, the content placement indicator indicating at least part of the first space for placement of the virtual content item; determining one or more locations of one or more objects on a surface associated with the at least one surface plane; and in response to a user input, rendering the content placement indicator on a second space of the at least one surface plane, wherein the second space of the at least one surface plane is determined based on the one or more locations and the user input, the content placement indicator indicating at least part of the second space for placement of the virtual content item.

Aspect 33. The method of Aspect 32, further comprising: in response to an additional user input requesting placement of the virtual content item on the second space, rendering the virtual content item in a second perspective at a second location in the scene based on the content placement indicator, the second location being within the second space.

Aspect 34. The method of Aspect 33, further comprising: determining the second perspective based on a perspective of the second space.

Aspect 35. The method of Aspect 33, further comprising: determining a first surface plane and a second surface plane of the at least one surface plane, the first space being associated with the first surface plane and the second space being associated with the second surface plane; and determining the second perspective based on a perspective of the second surface plane.

Aspect 36. The method of Aspect 35, wherein rendering the content placement indicator on the second space comprises: determining an overlap of the virtual content item with the second surface plane; and upon determination that the overlap exceeds a first threshold, switching to rendering the content placement indicator on the second space associated with the second surface plane.

Aspect 37. The method of any of Aspects 32 to 36, wherein the first space and the second space are associated with a first surface plane of the at least one surface plane, wherein the method further comprises: detecting one or more objects, wherein the one or more objects occlude at least part of the first surface plane; and segmenting the first surface plane into at least a first segment and a second segment based at least on an occluded part of the first surface plane; wherein the first space is associated with the first segment and the second space is associated with the second segment.

Aspect 38. The method of Aspect 37, wherein segmenting the first surface plane into at least the first segment and the second segment comprises determining a fit of the virtual content item rendered in a perspective of the first surface plane in the first segment and the second segment.

Aspect 39. The method of any of Aspects 37 to 38, further comprising changing an aspect ratio of the virtual content item between the first segment and the second segment.

Aspect 40. The method of any of Aspects 32 to 39, further comprising: prior to rendering the content placement indicator on the second space, rendering the virtual content item in a third perspective at a third location in the scene based on the content placement indicator, the third location being within the first space.

Aspect 41. The method of any of Aspects 32 to 40, further comprising: in response to the user input, moving the content placement indicator within the first space prior to rendering the content placement indicator on the second space.

Aspect 42. The method of any of Aspects 32 to 41, wherein determining the at least one surface plane in the scene comprises: determining one or more surfaces of the one or more objects in the scene; and determining the at least one surface plane based on the one or more surfaces of the one or more objects in the scene.

Aspect 43. The method of Aspect 42, wherein determining the at least one surface plane in the scene comprises: applying an offset to the at least one surface plane normally to at least one surface and toward a view point of a user associated with the computing device.

Aspect 44. The method of any of Aspects 32 to 43, wherein determining the at least one surface plane in the scene comprises: determining one or more edges of the one or more objects in the scene; and determining the at least one surface plane based on the one or more edges of the one or more objects in the scene.

Aspect 45. The method of any of Aspects 32 to 44, wherein rendering the content placement indicator comprises rendering a pattern on the at least one surface plane.

Aspect 46. The method of Aspect 45, wherein the pattern comprises at least one of a highlight, an outline, a color, a shade, a shadow, a hatching, and a gradient.

Aspect 47. The method of any of Aspects 32 to 46, wherein the first space and the second space comprise spaces on the at least one surface plane that are at least one of unoccupied by any physical objects and free from any physical objects that visually obstruct a visibility of respective portions of the at least one surface plane associated with the first space and the second space.

Aspect 48. The method of any of Aspects 32 to 47, wherein the second space comprises an available space on the at least one surface plane that is occupied by one or more physical objects in the physical scene.

Aspect 49. The method of any of Aspects 32 to 48, further comprising: based on an aspect ratio of the virtual content item, determining that at least a threshold amount of the virtual content item does not fit within the second space; determining that at least the threshold amount of the virtual content item fits within the second space when the virtual content item is configured according to a different aspect ratio; configuring the content placement indicator according to the different aspect ratio; and rendering the content placement indicator on the second space according to the different aspect ratio.

Aspect 50. The method of Aspect 49, further comprising: adjusting the aspect ratio of the virtual content item to the different aspect ratio; and in response to an additional input requesting placement of the virtual content item on the second space, rendering the virtual content item in the different aspect ratio, the virtual content item being rendered on the second space and within the content placement indicator.

Aspect 51. The method of any of Aspects 32 to 50, wherein determining the at least one surface plane in the scene comprises: determining one or more surfaces of the one or more objects in the scene; and determining the at least one surface plane based on the one or more surfaces of the one or more objects in the scene.

Aspect 52. The method of Aspect 51, further comprising applying smoothing to the one or more surfaces of the one or more objects in the scene prior to determining the at least one surface plane based on the one or more surfaces of the one or more objects in the scene.

Aspect 53. The method of Aspect 51, wherein determining the at least one surface plane in the scene comprises: aligning the at least one surface plane with at least one surface of the one or more surfaces of the one or more objects in the scene.

Aspect 54. The method of any of Aspects 32 to 53, further comprising: rendering the at least one surface plane relative to the surface.

Aspect 55. The method of Aspect 54, wherein rendering the at least one surface plane comprises: determining that an amount of flatness of the surface is below a threshold; and rendering the at least one surface plane a distance from one or more points on the surface associated with the at least one surface plane.

Aspect 56. The method of Aspect 54, wherein rendering the at least one surface plane comprises: determining that a highest point of the surface is a first threshold distance above one or more other points of the surface; and rendering the at least one surface plane a second threshold distance above the highest point of the surface.

Aspect 57. The method of any of Aspects 32 to 56, wherein rendering the content placement indicator comprises rendering a visual outline on the at least one surface plane.

Aspect 58. The method of Aspect 57, wherein the visual outline comprises at least one of a frame or a bounding box.

Aspect 59. The method of any of Aspects 57 to 58, further comprising rendering the visual outline in a perspective of the respective surface plane.

Aspect 60. The method of any of Aspects 32 to 59, wherein rendering the content placement indicator comprises rendering a preview of the virtual content item on the at least one surface plane.

Aspect 61. An apparatus comprising means for performing a method according to any of Aspects 32 to 59.

Aspect 62. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform a method according to any of Aspects 32 to 59.

Aspect 63. A method for rendering virtual content in a scene of a physical environment, the method comprising: determining at least one surface plane in the scene of the physical environment; determining that an amount of flatness of a surface associated with the at least one surface is below a threshold; rendering the at least one surface plane a distance from one or more points on the surface associated with the at least one surface plane; rendering a content placement indicator associated with a virtual content item on a first space of the at least one surface plane, the content placement indicator indicating at least part of the first space for placement of the virtual content item; and rendering, via a computing device, the virtual content item in a first perspective at a first location in the scene associated with the first space of the at least one surface.

Aspect 64. An apparatus comprising means for performing a method according to Aspect 63.

Aspect 65. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform a method according to Aspect 63.