Patent Description:
An MR system's HMD typically includes a head tracking camera system having one or more head tracking camera(s) and an inertial measurement unit (IMU). Using these cameras, the head tracking system can determine the HMD's position and pose relative to its surrounding environment. Data from the IMU can be used to augment or supplement the camera data to provide a more reliable position and pose determination.

The HMD's position and pose are both relied upon by an MR system when visually placing/rendering holograms in an MR scene. For instance, using Simultaneous Location And Mapping (SLAM), the MR system's head tracking and IMU units can calculate and determine a user's position as the user moves through space and can provide immediate display corrections for the virtual content in the MR scene.

To improve the virtual content placement process, MR systems also use three-dimensional (3D) sensing technologies to map the space around the HMD. This spatial information is used to provide contextual information to the user (e.g., for obstacle avoidance) and to help accurately place holograms. Unfortunately, the requirements for head tracking cameras and 3D image sensors are quite different and vary among MR systems.

Currently, head tracking is often performed using a stereo camera system that relies on low resolution visible light. For instance, common configurations of these stereo camera systems have a resolution of about <NUM> x <NUM> pixels. With lower resolution camera systems, it is particularly difficult to generate a full surface reconstruction (or a "spatial mapping") of the HMD's environment. Having an incomplete spatial mapping results in poor obstacle detection and avoidance and thus a lower quality user experience. Consequently, there is a significant need to improve how obstacles are identified within an environment, especially when only a lower resolution spatial mapping is available for that environment.

<CIT> describes that a method and apparatus is disclosed for assisting a user, wearing a head mounted display (HMD) that covers a user's field of vision and has a tracker providing information regarding the position and orientation of the HMD, in locating a physical controller located on a physical base station. A processor causes the HMD to display a virtual world, including a virtual representation of the physical base station and physical controller along with a virtual hand that helps guide the user to the physical base station to allow the user to pick up the physical controller. Another embodiment allows an area in the physical world to be defined within which the user should remain, for example to avoid physical obstacles. The processor causes the HMD to display a warning, such as a virtual fence, to alert the user if the user approaches to within a preselected distance of the boundary. <NPL> - describes how a user may set up the Guardian feature for the Oculus Rift S headset. Guardian is the boundary that keeps the user from bumping into walls in VR. It marks the edges of the users play space in VR.

The disclosed embodiments relate to methods, systems, and wearable devices that dynamically generate and render an object bounding fence in an MR scene.

In some embodiments, a sparse spatial mapping of an environment is initially accessed, where the sparse spatial mapping describes the environment in a 3D manner and where the environment includes a particular object. The sparse spatial mapping beneficially includes perimeter edge data describing the object's outer perimeters or boundaries. A gravity vector of a head-mounted device (HMD), which is rendering the MR scene, is also generated. Based on the perimeter edge data and the gravity vector, one or more two-dimensional (2D) boundaries of the object are determined (e.g., a 2D planar area is identified for the object). Then, a bounding fence mesh, a 2D mesh, or 2D spatial mapping of the environment is generated. This bounding fence mesh identifies the 2D boundaries of the object. Additionally, a virtual object is rendered within the MR scene. This virtual object is representative of at least a portion of the bounding fence mesh and also visually illustrates a bounding fence around the object.

The disclosed embodiments relate to methods, systems, and wearable devices that dynamically generate and render an object bounding fence in a mixed-reality (MR) scene/environment.

In some embodiments, a sparse "spatial mapping" is accessed. As used herein, the phrase "spatial mapping" refers to a three-dimensional digital representation of an object or environment, and the phrase "sparse spatial mapping" refers to an incomplete spatial mapping having a reduced number of 3D data points, or rather, reduced surface or texture data relative to a "complete" or "robust spatial mapping. " Although sparse, the sparse spatial mapping does include an adequate amount of perimeter edge data so as to sufficiently describe or identify the perimeters/boundaries of a particular object (e.g., by identifying at least a part of the object's length, width, and height). A gravity vector is also generated. Based on the perimeter edge data and the gravity vector, the object's two-dimensional (2D) boundaries (e.g., length and width) are determined and a bounding fence mesh, a 2D mesh, or 2D spatial mapping of the environment is generated. In some cases, a bounding fence mesh can also be considered to be a virtual 3D object representing an object's perimeter edges. A virtual object is also rendered, where the virtual object is representative of at least a portion of the bounding fence mesh and is provided in order to visually illustrate the object and a bounding fence enveloping/surrounding the object.

It will be appreciated from this disclosure that the disclosed embodiments can be used to help improve the technical field of mapping environments for mixed-reality applications and for projecting holograms in mixed-reality environments in numerous ways, some of which are outlined at a high level in this section while other benefits and improvements are described throughout the remaining portions of this disclosure.

As an initial matter, it is noted that many MR systems in the market today use lower resolution cameras to perform head tracking and depth calculations. Consequently, the resulting spatial mappings, which are often derived from the head tracking data, are also of lower quality and resolution. Using lower resolution spatial mappings often results in a reduced ability for the user to understand an environment and for the MR system to detect obstacles within the environment. As such, it may be the case that users of these traditional MR systems collide with real-world objects while immersed in an MR scene. It is also often the case that traditional MR systems consume a large amount of system bandwidth and have high computational complexity, resulting in high power consumption.

The addition of higher quality cameras or depth sensors represents a significant cost both in terms of hardware and battery expenditure. For very low-cost MR systems (e.g., systems that do not have the budget to afford the additional bill of materials for depth sensors), only a passive stereo camera pair might be used to perform head tracking and to generate depth images. Unfortunately, the low angular resolution of these passive camera systems (e.g., approximately <NUM> pixels per degree) and lack of additional illumination texture to the scene (hence a "passive" system as opposed to an "active" system that may provide additional illumination texture) provides low or insufficient information to generate a full, complete, dense, or robust spatial mapping of the environment. Indeed, in many cases, these low-resolution camera systems are often able to detect (at best) only an object's edge perimeters (i.e. the outer boundaries of an object). As such, traditional systems have been quite inadequate to build a high-density point cloud dataset with high resolution surface reconstruction, resulting in the inability to perform reliable, repeatable, and accurate obstacle avoidance.

In contrast, the disclosed embodiments are able to provide reliable, repeatable, and accurate obstacle avoidance. These benefits are achieved even when only a lower resolution spatial mapping is available. To achieve these benefits, the embodiments translate, transform, or compress three-dimensional (3D) data to generate a much simpler 2D image or 2D spatial mapping of the environment. This translation may be achieved by removing, discarding, or filtering out the object's height dimension relative to a determined gravity vector. To clarify, once the 3D dimension has been removed, then the embodiments are left with a 2D image and can determine whether any given pixel in the 2D image is occupied or not occupied, which is a much simpler binary determination when performing obstacle avoidance.

The resulting 2D image can be thought of as a type of bounding fence mesh, a 2D mesh, or 2D spatial mapping (which was generated using head tracking data, as described above). In some cases, a bounding fence is representative of a geometrical 3D structure representing an object's 2D perimeter edges. The disclosed embodiments also use this bounding fence mesh to generate a "fence" around objects in order to clearly define the environment's play-space or movement area (i.e. the area where a user can move without fear of colliding into an object). Furthermore, the generation of a "fence" is much less computationally expensive than performing the calculations required to generate a full high-density surface reconstruction mesh of the scene.

Accordingly, the disclosed embodiments use fewer computing resources, thereby improving the efficiency of the computing system/device or enabling the experiences to run successfully on a lower-cost computing system (e.g., an HMD). Furthermore, with these reduced computations, the embodiments operate to preserve or expand the battery lifespan of the computing system. In this manner, the disclosed embodiments provide a real-world, practically applicable solution to a technical problem in the computing arts, all while improving computing efficiency and prolonging battery life. Furthermore, the disclosed embodiments utilize a highly compressed way to represent surface reconstruction by storing and using only a single 2D image (i.e. a bounding fence mesh/2D mesh/spatial mapping) as opposed to storing and using an entire 3D mesh.

Attention will now be directed to <FIG>, which illustrates an HMD <NUM> capable of performing the disclosed operations. HMD <NUM> is included as a part of an MR device (which will be illustrated later in connection with <FIG>). The phrases "MR device" and "MR system" can be used interchangeably with one another. In some cases, HMD <NUM> is itself considered an MR device. Therefore, references to HMDs, MR devices, or MR systems generally relate to one another and may be used interchangeably.

HMD <NUM> is shown as including an IMU <NUM>. IMU <NUM> is a type of device that measures force, angular adjustments/rates, orientation, acceleration, velocity, gravitational forces, and sometimes even magnetic fields. To do so, IMU <NUM> may include any number of data acquisition devices, which include any number of accelerometers, gyroscopes, or even magnetometers.

IMU <NUM> can be used to measure a roll rate <NUM>, a yaw rate <NUM>, and a pitch rate <NUM>. It will be appreciated, however, that IMU <NUM> can measure changes in any of the six degrees of freedom, as shown in <FIG>. That is, <FIG> shows an IMU <NUM>, which is representative of IMU <NUM> from <FIG>. IMU <NUM> includes one or more gyroscope(s) <NUM> and one or more accelerometer(s) <NUM>. The ellipsis <NUM> demonstrates how IMU <NUM> may include other types of data acquisition units whose data can be used to determine an HMD's position, orientation, movement, and pose.

IMU <NUM> is able to determine its position in any one or more of the six degrees of freedom <NUM>, which refers to the ability of a body to move in three-dimensional space. Six degrees of freedom <NUM> include surge <NUM> (e.g., forward/backward movement), heave <NUM> (e.g., up/down movement), sway <NUM> (e.g., left/right movement), pitch <NUM> (e.g., movement along a transverse axis), roll <NUM> (e.g., movement along a longitudinal axis), and yaw <NUM> (e.g., movement along a normal axis). Accordingly, IMU <NUM> can be used to measure changes in force and changes in movement, including any acceleration changes. This collected data can be used to help determine a position, pose, and/or perspective of an HMD relative to its environment.

Furthermore, this data, along with the data from the one or more gyroscope(s) <NUM> can be used to determine a gravity vector <NUM> of the HMD <NUM> and for the objects in the scene from <FIG>. As used herein, references to a "gravity vector" refer to a vector that is parallel to the gravity force of the earth. That is, assuming that any particular position on the earth can be thought of as a flat surface, the gravity vector will be perpendicular to the flat surface and will be directed downward. Therefore, regardless of any movement of the HMD <NUM> from <FIG>, IMU <NUM> (and IMU <NUM>) can be used to determine gravity vector <NUM> (i.e. the gravity vector is generated based on data obtained from the IMU <NUM>).

Returning to <FIG>, HMD <NUM> also includes a stereo camera system <NUM>, which includes a first camera <NUM> (e.g., perhaps a head tracking camera) and a second camera <NUM> (also perhaps a head tracking camera). Multiple cameras are typically used for Head Tracking as to increase the effective field of view of the system. Camera <NUM> includes its corresponding field of view (FOV) <NUM> (i.e. the observable area of first camera <NUM>, or rather the observable angle through which first camera <NUM> is able to capture electromagnetic radiation), and camera <NUM> includes its corresponding FOV <NUM>. While only two cameras are illustrated, it will be appreciated that any number of cameras may be included in stereo camera system <NUM> (e.g., <NUM> camera, <NUM>, <NUM>, <NUM>, <NUM>, or more than <NUM> cameras).

Cameras <NUM> and <NUM> can be any type of camera. In some cases, cameras <NUM> and <NUM> may be stereoscopic cameras in which a part of FOVs <NUM> and <NUM> overlap (e.g., see overlap <NUM>) with one another to provide stereoscopic camera operations (e.g., head tracking). In some implementations, cameras <NUM> and <NUM> are able to capture electromagnetic radiation in the visible light spectrum and generate visible light images. In other or additional implementations, cameras <NUM> and <NUM> are able to capture electromagnetic radiation in the infrared (IR) spectrum and generate IR light images. In some cases, cameras <NUM> and <NUM> include a combination of visible light sensors and IR light sensors. In yet other cases, cameras <NUM> and <NUM> can be repurposed or multi-purposed for depth detection functionalities for generating a 3D point cloud of the environment. As an example, when an object is located within overlap <NUM>, the object's depth can be calculated by identifying differences or disparities between the two images that concurrently capture the same object. Because the same object is captured in both images, the disparities can be used to determine the 3D point cloud of the scene. Further details on cameras will be provided later in connection with <FIG>.

<FIG> illustrates a real-world environment <NUM> in which a user <NUM> is located. Here, a user <NUM> is wearing an HMD, like HMD <NUM> of <FIG>. Environment <NUM> is shown as including a number of real-world objects, such as support beam <NUM>, wall <NUM>, shelf <NUM>, shelf <NUM>, camera <NUM>, and support beam <NUM>. <FIG> also shows how the user's HMD is able to determine the gravity vector <NUM> for the real-world environment <NUM>. Furthermore, regardless of whether user <NUM> is viewing or interacting with virtual content in a VR scene or an AR scene, it is desirable to identify objects within the real-world environment <NUM> to ensure that the user <NUM> does not inadvertently collide with those objects.

As an example, <FIG> shows a mixed-reality environment <NUM> that may be projected by an HMD. In this scenario, the mixed-reality environment <NUM> is a type of VR environment because the user's FOV of the real world is entirely occluded. <FIG> shows a user <NUM>, who is representative of user <NUM> from <FIG>. Also shown is HMD <NUM>, which is representative of the HMDs discussed thus far. HMD <NUM> is shown as having a corresponding FOV <NUM>, and mixed-reality environment <NUM> is shown as including any number of virtual images (e.g., virtual image <NUM> and virtual image <NUM>). Also shown is the gravity vector <NUM> corresponding to the real-world environment (but not necessarily corresponding to the mixed-reality environment <NUM>). In some cases, the gravity vector <NUM> may be different than a simulated gravity vector for the mixed-reality environment <NUM>.

<FIG> again shows the real-world environment <NUM>, which is representative of the real-world environments discussed earlier. Here, the user's HMD is shown as projecting an MR scene <NUM>, which is representative of the mixed-reality environment <NUM> of <FIG>. Furthermore, even though the VR scene is tilted (e.g., because the rollercoaster is banking) and the MR scene <NUM> is shown as having a tilt, the gravity vector <NUM> (corresponding to the real-world environment <NUM>) is shown as being unchanged in that the gravity vector <NUM> is always in a same direction as the earth's gravity.

To properly display virtual content and to avoid obstacles, it is beneficial to use camera data obtained from the HMD's cameras (e.g., head or hand tracking cameras). This camera data is used to map out the user's environment in order to determine where and how to place virtual content. Furthermore, this camera data is used to determine the depths and textures of objects within the user's environment as well as the distances of the objects from the user or HMD. In this regard, the camera data is not only useful for placing holograms, but it is also useful to warn the user when the user is about to collide with an object in the real-world.

It will be appreciated that any number and type of camera may be used, either individually or in combination (e.g., multiple cameras of multiple types). <FIG> shows HMD <NUM>, which is representative of the earlier HMDs discussed thus far. HMD <NUM> is shown as including a camera system <NUM>, which may be representative of stereo camera system <NUM> from <FIG> and which may be included as a part of a head or hand tracking camera system.

Camera system <NUM>, which can be used to generate a 3D point cloud of the space/environment, can include one or more of the following different types of cameras: a time of flight camera <NUM> (e.g., an active time-of-flight), an active stereo camera system <NUM> (e.g., an active structure light camera), a passive stereo camera system <NUM>, or a motion stereo camera system <NUM>. The ellipsis <NUM> demonstrates how other types of camera systems may be included as well. For instance, a single pixel laser depth device can be used to scan a room and can contribute in generating depth data for a spatial mapping. As another example, a user's phone may be used as the camera system and can determine a gravity vector. Additionally, other external cameras or sensors may be used to contribute data when generating a spatial mapping. These cameras are beneficially used to determine depth within the user's environment, including any texture and surface data of objects within that environment.

Time of flight camera <NUM> and active stereo camera system <NUM> are typically used to actively scan and illuminate the environment in order to acquire highly detailed, accurate, dense, and robust information describing the environment. For instance, turning briefly to <FIG>, <FIG>, <FIG>, and <FIG>, these figures illustrate how these robust camera systems can operate.

<FIG> shows a real-world environment <NUM> and HMD <NUM>, both of which are representative of the corresponding entities discussed earlier. Here, HMD <NUM> is using its robust cameras to scan <NUM> the real-world environment <NUM> by taking pictures and/or by determining depth measurements of the objects within the real-world environment <NUM>. As the user moves around, as shown in <FIG>, additional scans (e.g., scan <NUM>) can be acquired to obtain additional information describing the real-world environment <NUM>.

In some cases, the scanned information will result in the generation of a robust point cloud <NUM>, as shown in <FIG>. Point cloud information in the robust point cloud <NUM> may have been generated by the HMD itself, from other HMDs in the same environment, from other sensors in the environment, or even from third-party sensor data that was previously acquired and retained. Here, the point data in the robust point cloud <NUM> describes the objects that were in the real-world environment <NUM> (e.g., the support beams, shelves, cameras, walls, etc.).

Using this robust point cloud <NUM>, some embodiments create a surface mesh <NUM> and/or a depth map <NUM>. As used herein, a "3D surface mesh," "surface mesh," or simply "mesh" is a geometric representation or model made up of any number of discrete interconnected faces (e.g., triangles) and/or other interconnected vertices. The combination of these vertices describes the environment's geometric contours, including the contours of any objects within that environment. By generating such a mesh, the embodiments are able to map out the contents of an environment and accurately identify the objects within the environment. Relatedly, depth map <NUM> can include depth values arranged in a map format. As used herein, a "spatial mapping" can include point clouds, surface meshes, and depth maps. It will be appreciated that <FIG>, <FIG>, <FIG>, and <FIG> are for example purposes only and should not be considered binding. Indeed, actual 3D point clouds and spatial mappings will include significantly more 3D points and 3D information than the visualizations provided in these figures.

<FIG> shows how the real-world environment <NUM> can be scanned in order to generate a dense spatial mapping <NUM>. As described earlier, a "spatial mapping" (also called a 3D reconstruction) refers to a 3D representation of an environment. Furthermore, the objects within an environment can be segmented (i.e. "identified") using any type of object recognition or machine learning algorithm such that the spatial mapping is also able to identify and characterize objects.

As shown by the dense spatial mapping <NUM>, many individual objects in the real-world environment <NUM> can be identified (e.g., the support beams, walls, shelves, and even camera are all clearly identifiable in the dense spatial mapping <NUM>). In this regard, these types of camera systems include the ability to generate a highly robust and detailed spatial mapping of an environment.

Returning to <FIG>, the camera system <NUM> is also shown as including a passive stereo camera system <NUM> and a motion stereo camera system <NUM>. These types of camera systems typically do not actively illuminate an environment when scanning. By way of background, motion stereo camera system <NUM> follows a similar principle as a stereo camera system, but instead of having two cameras, only one camera is used. This one camera is moved in order to collect the scanning data. Provided that the environment remains static, the resulting images generated by the motion stereo camera system <NUM> can also be used to compute depth and a spatial mapping. In any event, the resulting spatial mappings from these types of camera systems are typically of a much lower resolution than the spatial mappings generated from the active camera systems described earlier.

To illustrate, <FIG> shows a sparse point cloud <NUM>, which is typically generated by the passive stereo camera system <NUM> and/or the motion stereo camera system <NUM> of <FIG>. This sparse point cloud <NUM> can be used to generate a sparse surface mesh <NUM> or a sparse depth map <NUM> (collectively referred to as a sparse spatial mapping). Accordingly, head tracking data can be used to obtain 3D depth information to thereby generate a single, low-cost sensor set.

<FIG> then shows the resulting spatial mapping. Specifically, <FIG> shows a real-world environment <NUM> and the resulting sparse spatial mapping <NUM>. It will be appreciated that the sparse spatial mapping <NUM> may be generated using a passive stereo camera system, a motion stereo camera system, even an active stereo camera system (e.g., an active structured light camera) or an active time-of-flight camera (e.g., when they are configured to operate at reduced resolution, scanning duration, or power mode, as described later). Accordingly, these camera systems may be included as a part of a head or hand tracking system of an HMD.

When compared to the dense spatial mapping <NUM> of <FIG>, the sparse spatial mapping <NUM> of <FIG> includes significantly less detail. For instance, in the sparse spatial mapping <NUM>, the walls may not be as clearly defined, the support beams may not be as clearly defined, the camera is no longer represented, and even the shelves are less descriptively represented. In <FIG>, each shelf in the shelving units was represented, along with the boxes on those shelves. In <FIG>, however, the shelves and boxes are represented as a single collective unit, without particular distinction (because of the lower resolution scanning data).

In a most extreme embodiment, which does not apply to all embodiments, the terms "dense," "robust," and "sparse" are simply terms of relativity. Dense and robust are terms that mean the resulting dense spatial mapping is relatively more complete or detailed than a sparse spatial mapping. It will be appreciated that in some cases, a dense spatial mapping may also not completely or fully describe the surface and texture of an object, but it will describe the object more completely than a sparse spatial mapping.

In this regard, the sparse spatial mapping <NUM> is relatively less accurate and includes relatively less detail than the dense spatial mapping <NUM>. In some cases, the sparse spatial mapping <NUM> may include <NUM>%, <NUM>%, <NUM>%, <NUM>%, <NUM>%, <NUM>%, <NUM>%, or even <NUM>% of the detail of the dense spatial mapping <NUM>. Accordingly, the sparse spatial mapping <NUM> includes incomplete surface and/or texture data of objects within the environment. Although sparse, the corners and edges of the objects in the environment are generally still detectable and can still be represented within the sparse spatial mapping <NUM> (i.e. perimeter edge data describes the edge perimeters of the objects).

Stated differently, perimeter edge data describes a portion, but not all, of one or more perimeter edge(s) of objects such that the perimeter edge data constitutes some, but incomplete, data. It follows then that the resulting sparse spatial mapping (which uses the perimeter edge data) is also sparse as a result of relying on the incomplete data.

Furthermore, the sparse spatial mapping <NUM> is able to identify at least the edge perimeters of the objects within the environment. To clarify, although the cameras may not be able to detect specific surfaces or textures of the objects, the cameras are at least able to detect the edge perimeters of those objects. With reference to <FIG>, the cameras can detect the edges of the shelving units, though (as represented by the sparse spatial mapping <NUM>), the cameras were not able to adequately distinguish between the boxes on the shelving units nor were the cameras able to distinguish between the different shelving levels. Notwithstanding this limitation, identifying the edges, perimeters, or edge perimeters is sufficient to identify the outer bounds or boundaries of those objects. With regard to obstacle avoidance, this limited edge perimeter data is also now sufficient to enable the HMD to help the user avoid colliding with those objects.

While the above disclosure focused on a scenario in which a passive stereo camera system and/or a motion stereo camera system was used to generate the sparse spatial mapping <NUM>, it will be appreciated that an active stereo camera system and/or a time of flight camera can also be used to generate the sparse spatial mapping <NUM>. For instance, those systems can be configured to operate in a reduced power mode such that less surface and texture data is collected. For instance, the systems can operate at <NUM>%, <NUM>%, <NUM>%, <NUM>%, <NUM>%, <NUM>%, <NUM>%, or even <NUM>% of their normal or default operating power levels when scanning an environment.

Additionally, or alternatively, those systems can be configured to operate in a reduced scan-time mode in which the systems spend less time scanning the room. For instance, the systems can scan at <NUM>%, <NUM>%, <NUM>%, <NUM>%, <NUM>%, <NUM>%, <NUM>%, or even <NUM>% of their normal or default scanning time, duration, or rate to thereby produce a lower resolution spatial mapping. In doing so, these camera systems can be configured to operate at a lower power mode and/or at a reduced compute processing mode, to thereby prolong battery life as needed. In some cases, the systems can be configured to switch to these lower power modes in response to certain events or triggers. For instance, switching modes can occur when the battery level reaches a particular threshold level. In some cases, switching modes can occur in response to certain environmental factors, such as the amount of ambient light in the room. For example, if the amount of ambient light is at a particular level, then active illumination can be turned off and the room can be scanned only in a passive, non-illuminating manner.

Accordingly, some high-end HMD systems can be configured to generate a sparse spatial mapping. Additionally, some low-end HMD systems might be constrained (e.g., hardware constraints) to be able to provide only a sparse spatial mapping. Regardless of the type of hardware used to generate the sparse spatial mapping, the disclosed embodiments are able to beneficially use this sparse spatial mapping to perform obstacle avoidance. It will be appreciated that this sparse spatial mapping can be generated in real-time while the HMD is operating within an environment, or the spatial mapping may have been generated at an earlier time, stored in a repository (e.g., the cloud), and then made available for access. In some cases, a single HMD unit generated the sparse spatial mapping while in other cases multiple HMD units contributed to generating the sparse spatial mapping (e.g., data from multiple HMDs is aggregated/fused together). The data can be acquired all within a single scanning event, or it can be aggregated over time and over multiple different scanning events. In this regard, the disclosed embodiments are highly dynamic and flexible and may be implemented across many different scenarios and circumstances. As such, the disclosed principles should be interpreted broadly.

In accordance with the disclosed principles, the embodiments are able to identify a 3D obstacle avoidance problem (i.e. objects within a room are three-dimensional and thus represent a 3D problem when trying to avoid those objects) and reduce that problem down to a 2D problem and solution. Performing computations on 2D data requires less processing and time than performing computations on 3D data (e.g., because less data is computed and operated on).

After accessing the sparse spatial mapping of the environment, the disclosed embodiments are able to interpret the depth data and generate a two-dimensional mapping of the environment. By "interpret," it is meant that the disclosed embodiments are able to translate the 3D information into 2D information. For instance, a 2D ground plane or visualization of the environment can be created based off of the 3D information in the sparse spatial mapping. In essence, the environment can now be represented from a bird's eye view, where the environment, including all of the objects within the environment, is represented two-dimensionally from a top aerial perspective (i.e. a bird's eye view or a plan view). That is, if the environment were viewed from above, relative to the environment's gravity vector, then the height dimensions of the 3D objects are essentially eliminated, leaving only length and width dimensions. Such a change in perspective results in 3D objects being transformed or translated to now appear as pixels within a 2D ground plane image.

With this 2D ground plane image, every 2D pixel in the 2D image can be classified as being either empty or free. Furthermore, "voxels" (i.e. rectangular cuboids, volumetric pixels, or 3D grids) associated with the floor plane can also be classified as either being empty/free space or as being occupied space (i.e. occupied by an object in the environment) as a result of performing the much simpler 2D pixel determination. Therefore, the disclosed embodiments use the sparse spatial mapping to generate a different representation of the environment. In this regard, instead of making a decision for every rectangular cuboid, 3D grid, or "voxel" in the environment, the disclosed embodiments need only label a pixel (i.e. a 2D image artifact) in a binary manner, either occupied or not occupied. Furthermore, instead of seeking to identify specific contours and features of an object, the disclosed embodiments determine only 2D edge data. Accordingly, the embodiments operate using 2D pixels and intelligently determine whether any particular pixel is occupied (i.e. an object is present at the location corresponding to the pixel) or is not occupied.

In some cases, instead of storing only a binary value, some embodiments additionally store an array of integers. Some of these integers represent the height of the object and may be used to determine the height of a bounding fence that may later be used.

As indicated above, the top aerial perspective (i.e. the bird's eye view) is projected or determined along the gravity vector that was computed using the HMD's IMU data. Accordingly, the disclosed embodiments are able to translate 3D data into 2D data by removing the height dimension along the gravity vector and by viewing the environment from a bird's eye view.

<FIG> shows a bird's eye perspective <NUM> of the real-world environment <NUM> of <FIG>. Head tracking, which is performed by the HMD, also provides the orientation and positioning of the user within the real-world 3D space or environment.

The bird's eye perspective <NUM> was generated using the sparse spatial mapping <NUM> of <FIG>. This bird's eye perspective <NUM> visually illustrates wall <NUM> (which is representative of wall <NUM> from <FIG>), support beam <NUM> (which is representative of support beam <NUM>), shelf <NUM> (which is representative of shelf <NUM>), shelf <NUM> (which is representative of shelf <NUM>), and support beam <NUM> (which is representative of support beam <NUM>). An x-y-z legend is also illustrated to provide bearing on the different perspectives. For instance, the circled x represents a downward z dimension (also the "gravity vector"), which is representative of the gravity vector <NUM> from <FIG>.

In this regard, the height dimensions of the 3D objects have been eliminated, discarded, or filtered from consideration by the disclosed embodiments, thereby translating an incomplete or partial 3D representation of the objects (i.e. the sparse spatial mapping) in an easier-to-work-with 2D representation of those objects. To further elaborate, while the sparse spatial mapping <NUM> of <FIG> was inadequate to clearly distinguish between the boxes and levels on the shelves, it is now unnecessary to perform this distinction because the shelves are viewed from a top aerial perspective. The sparse spatial mapping <NUM> was adequate to identify at least the 2D edge perimeters (as determined from a top/plan view) of the shelves and the other objects within the room. As such, the edge perimeter data is sufficient to represent the objects from a 2D perspective.

The bird's eye perspective <NUM> is also able to visually render a representation of the user via indicator <NUM>. In this case, indicator <NUM> is rendered as an arrow, though other visualizations of the user's relative position and/or orientation within the mapped environment may be used (e.g., a 2D avatar, a triangle, a picture, etc.). The indicator <NUM> can be an animated illustration (e.g., when the user moves, the indicator not only moves to track the user's movements through the 2D environment but can also illustrate a walking animation) or it can be a static illustration (and just track the user's movements through the 2D environment).

In some cases, the indicator <NUM> can also visually portray the direction in which the user is currently facing (i.e. the user's orientation or pose). For instance, indicator <NUM> is shown as an arrow, with the direction of the arrow indicating the direction in which the user is currently facing. This directional visualization can be computed using the HMD's IMUs, head tracking cameras, and/or other direction or compass determining units.

The disclosed principles relate to a new technique for obstacle avoidance. This technique may be implemented without adding additional cost or sensors to the HMD. This technique may also use what is referred to as a "bounding fence," "compute fence," or simply "fence," which provides a low-cost, computationally inexpensive visualization for defining the play-space or movement space (i.e. areas where the user can move without colliding with an object) for MR scenes.

<FIG> illustrates the different bounding fences via the dark bolded areas around the objects. Specifically, <FIG> shows bounding fence <NUM> around wall <NUM>, bounding fence <NUM> around support beam <NUM>, bounding fence <NUM> around shelf <NUM>, bounding fence <NUM> around shelf <NUM>, and bounding fence <NUM> around support beam <NUM>. In some cases, the bounding fences are shaped to correspond to the outer bounds or shapes of the objects they surround. In some cases, especially when multiple corners of the object are grouped or positioned very near one another, the bounding fence visually merges multiple corners to form a single corner. In the bird's eye perspective <NUM>, the bounding fences are visually illustrated to emphasize the metes and bounds of objects within the environment. The bounding fences are also provided to alert the user when the user is near an object.

For instance, as the user travels about the real-world environment, the indicator <NUM> will mimic or track the user's movements within the rendered 2D environment. By looking at the bird's eye perspective <NUM>, the user can determine whether he/she is nearing an object because the object will be highly emphasized via use of the bounding fences. In some cases, additional alerts may be provided, such as an audio alert or even additional visual cues (e.g., text in the HMD, a red splash image or hologram in the HMD, etc.). In some instances, a visualization of the 2D bird's eye perspective <NUM> is visually displayed to a user through their HMD, only in response to user input requesting the display. In other embodiments, the visualization of the 2D bird's eye perspective <NUM> is constantly displayed while the HMD is in certain states/contexts and is displayed within a dedicated portion of the viewing area of the HMD. In yet other embodiments, the visualization of the 2D bird's eye perspective <NUM> is only intermittently and dynamically rendered in response to the user reaching and/or being within a threshold distance from a mapped object in the 2D bird's eye perspective <NUM> and/or within a threshold distance from a mapped object having particular declared attributes that are declared to the HMD-such as by a broadcast from the object or associated beacon or a download of third party content).

In some cases, a buffer region may be provided between the object and the bounding fence. For instance, <FIG> illustrates another bird's eye perspective <NUM>, which is also representative of the real-world environment <NUM> of <FIG>. <FIG> shows the wall <NUM>, support beam <NUM>, shelf <NUM>, shelf <NUM>, support beam <NUM>, an indicator <NUM> representative of the user's position within the environment, and an x-y-z legend.

<FIG> also shows the bounding fences <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> around their respective objects. In contrast to the bird's eye perspective <NUM> of <FIG>, bird's eye perspective <NUM> now includes a buffer region around each object. This buffer region is provided to cushion or buffer the fence from its corresponding object. Providing such a buffer may also help the user in navigating the environment and may help the user from immediately striking or colliding with an object in the event the user minorly or minimally breaches the bounding fence.

To illustrate, <FIG> shows buffer <NUM> and buffer <NUM> around wall <NUM>. Buffers <NUM> and <NUM> expand the area defined by wall <NUM> and operate as a buffer between bounding fence <NUM> and wall <NUM>. That is, instead of using the exact edges of wall <NUM>, the buffers <NUM> and <NUM> operate to virtually enlarge the wall <NUM>, thus providing a safety region where, if the user does somewhat breach the bounding fence, the user will not immediately strike the object. In this regard, the area defined by an actual object may be smaller than the area defined by that object's corresponding bounding fence. It will be appreciated that the buffers <NUM> and <NUM> may be set to any value (e.g., <NUM> inch larger than the bounds of the object, <NUM> inches, <NUM> inches, <NUM> inches, <NUM> inches, <NUM> inches, <NUM> foot, <NUM> feet, <NUM> feet, and so on). In some cases, larger objects may have a relatively larger buffer than smaller objects, or vice versa, in which a smaller object may have a relatively larger buffer region than a larger object. The object's determined type, dimension, or configuration may determine or may influence the size of that object's bounding fence, including its buffer regions.

Accordingly, the disclosed embodiments are able to generate any number of bounding fences, which are defined by the 2D boundaries of their corresponding objects to form 2D planar areas surrounding those objects, where the planar areas are oriented relative to the gravity vector. In some cases, a buffer is provided between a bounding fence and the 2D boundaries of the object. Consequently, an area defined by the bounding fence may be larger than an area defined by the 2D boundaries of the object. It will be appreciated that one or more objects may have buffers while one or more other objects may not have buffers. Determining which objects will have buffers can be dependent on the object's object type (which may be detected through object identification and reference tables or which may be declared), on the size of the object, on the MR scene experience (e.g., will the user be moving around a lot), or even on the user's detected behavior (e.g., is the user prone or likely to bump into something). The size of the buffer can also be dynamically determined and may be different for different objects, even within the same MR scene. In some cases, the buffer can be visually modified or formatted to round out corners/edges or even to aggregate closely proximate edges to thereby form a single curved edge as opposed to multiple discrete edges.

<FIG> illustrates a mixed-reality environment <NUM>, which is representative of the mixed-reality environment <NUM> of <FIG>. Here, a bird's eye perspective <NUM> is also visually rendered within the mixed-reality environment <NUM>, where the bird's eye perspective <NUM> is representative of any of the earlier bird's eye perspectives/views discussed thus far and where the bird's eye perspective <NUM> is projected parallel to/along with the gravity vector. Bird's eye perspective <NUM> also specifically includes an indicator <NUM> illustrating the user's actual position within the real-world environment. In this regard, even though the user may not be able to directly see the real-world environment (e.g., in a VR case), the user can still be made aware of his/her surroundings by consulting the bird's eye perspective <NUM>.

The bird's eye perspective <NUM> is only selectively rendered in response to certain conditions, circumstances, events, or triggers. In this regard, rendering the bird's eye perspective <NUM> (i.e. a type of "virtual object") may be performed only in response to a triggering event. For instance, the bird's eye perspective <NUM> may, as a default, not be displayed. According to the invention, when the HMD determines that the user (or HMD) is located within a predetermined or pre-established threshold distance to an object, then the HMD triggers the display of the bird's eye perspective <NUM>. As such, proximity detection or a likelihood of collision may cause the bird's eye perspective <NUM> to be rendered. The bird's eye perspective <NUM> can be placed anywhere within the mixed-reality environment <NUM> and is not limited to only the bottom right-hand corner. Indeed, the bird's eye perspective <NUM> can also be placed so as to overlap one or more other holograms/virtual images.

Additionally, the size of the bird's eye perspective <NUM> can vary or be dynamically adjusted. In some cases, the size can be modified based on the user's proximity to an object, where the size progressively gets larger as the user progressively moves nearer to an object and where the size progressively gets smaller as the user progressively moves away from the object. The embodiments can terminate the display of the bird's eye perspective <NUM> in response to determining that the user/HMD is no longer within the distance threshold. In some cases, a maximum size and a minimum size of the bird's eye perspective <NUM> may be imposed, where the bird's eye perspective <NUM> is not permitted to become larger than the maximum size or smaller than the minimum size. Some embodiments, on the other hand, refrain from having size restrictions.

<FIG> illustrated a scenario in which a 2D bounding fence was visualized. In addition to performing the 3D-to-2D translation described earlier, some embodiments are also able to render a visualization of the bounding fence as a separate hologram, as shown in <FIG>. Similar to how the bird's eye perspective was selectively displayed or terminated from display in response to one or more triggers, the bounding fences may also be selectively displayed or terminated from display in response to the same triggers discussed earlier.

Specifically, <FIG> illustrates a mixed-reality environment <NUM>, which is similar to the real-world environment <NUM> of <FIG> but which now includes holograms, and a user <NUM> in that environment. Mixed-reality environment <NUM> includes the following real-world objects: support beam <NUM>, shelf <NUM>, shelf <NUM>, and support beam <NUM>. In addition to these real-world objects, mixed-reality environment <NUM> also includes a number of holograms. These holograms include fence <NUM>, fence <NUM>, fence <NUM>, fence <NUM>, and hologram <NUM> (i.e. a dragon). <FIG> also shows the gravity vector <NUM>.

In accordance with the disclosed principles, the embodiments are able to generate and display a bounding fence around the objects in the mixed-reality environment <NUM> to alert the user of the objects. Similar to the earlier discussion, these bounding fence holograms can be displayed continuously or can be displayed in response to certain stimuli or triggering conditions.

As depicted in <FIG>, in some cases, not part of invention, a bounding fence includes a rectangular cuboid or a 3D voxel whose length and width are defined by the 2D boundaries of the object. Furthermore, the height of this rectangular cuboid / 3D voxel can extend upwardly. In some cases, not part of the invention, the height extends upwardly in an infinite or unbounded direction perpendicular to the 2D planar area defined by the 2D boundaries of the object and parallel to the gravity vector. In other situations, not part of the invention, however, the height extends vertically in a bounded direction perpendicular to the 2D planar area and parallel to the gravity vector. For instance, the height may extend at least to a height of the object so that the rectangular cuboid / 3D voxel entirely envelopes the object. Such a scenario is illustrated in <FIG>. Of course, the fence's height can extend even further than the object's height, such as in cases where a vertical buffer is used. In such situations, the volume occupied by the object is smaller than the volume occupied by the fence.

<FIG> shows a mixed-reality environment <NUM>, which is representative of the mixed-reality environment <NUM> of <FIG>. Also shown are a fence <NUM>, with its corresponding height <NUM>, fence <NUM>, with its corresponding height <NUM>, and fence <NUM>. In contrast to the fences in <FIG>, fences <NUM> and <NUM> have bounded heights. Here, height <NUM> extends at least to the height of its corresponding shelf object and, in some circumstances, may extend somewhat further depending on the buffer provided for the shelf obj ect.

Similarly, height <NUM> extends at least to the height of that fence's corresponding shelf object and, in some circumstances, not part of the invention, may extend somewhat further depending on the corresponding buffer. In some cases, a first object in the environment may be associated with an unbounded (height-wise) bounding fence while a second object in the environment may be associated with a bounded (height-wise) bounding fence. Bounding or restricting the height of the fence may be useful in scenarios where the object is short enough that a user can simply walk over the object without exerting much effort.

In some cases, not part of the invention, a bounding fence may overlap with another bounding fence or may overlap another virtual image/hologram. Additionally, objects may also extend from a ceiling downward. As such, bounding fences may originate at a ceiling or upward location and may extend downward a determined distance. In some implementations, an object may extend outward from a side wall. As a result, bounding fences may originate on a side wall and may extend laterally outward (i.e. perpendicular to the gravity vector).

The visual appearance of the bounding fences can vary or change as well. In some cases, the bounding fences are at least partially transparent so that the underlying object is at least partially visible through the bounding fence. For instance, the bounding fence may be <NUM>%, <NUM>%, <NUM>%, <NUM>%, <NUM>%, <NUM>%, <NUM>%, <NUM>%, or even <NUM>% transparent. In other cases, the bounding fence may not be transparent and instead may entirely or completely occlude the underlying object. A descriptive text label may also be visually presented near the fence. For instance, when the bounding fence entirely occludes the underlying object, such as the shelf unit, the HMD may visually render the following text with the bounding fence: "Shelving Unit. " In some cases, the text may entirely or partially overlap the bounding fence. In other cases, the text may not overlap the bounding fence but instead may be visually rendered near or proximate to the bounding fence.

As shown in <FIG>, there are at least two virtual objects/holograms. For instance, fence <NUM> may constitute a first virtual object and fence <NUM> may constitute a second virtual object. Similar to the fence <NUM>, the fence <NUM> is rendered in the mixed-reality scene/environment and is operating as a second bounding fence for a second object (e.g., the shelving unit) in the environment. Furthermore, as represented by the diagonal lines for fence <NUM>, fence <NUM> may be visually distinguished from fence <NUM>, which includes a dot pattern background. Of course, any type of visual distinction may be used. In some cases, the distinction may occur through use of different colors, patterns, animations, or even holographic textures. For instance, fence <NUM> may include an ocean-wave-like surface extending from the floor to the ceiling. In some cases, the visual distinction may include an animation (e.g., lines flowing in a certain direction, rain falling from the top, etc.). As such, visually distinguishing the second virtual object (e.g., the bounding fence) from a first virtual object (e.g., another bounding fence) may be performed with at least one of a different color, animation, pattern, or texture.

In some implementations, different object types may be assigned different bounding fence visualizations. As a consequence, visually distinguishing the different bounding fences may be based on the determined types of the objects.

<FIG> and <FIG> illustrate example scenarios in which the occurrence of a triggering event or condition causes the HMD to display the bounding fences. Specifically, <FIG> shows an augmented-reality environment <NUM> and a gravity vector <NUM>. In scenario 1710A, the user 1715A is viewing content within a FOV 1720A. Here, user 1715A is not sufficiently proximate to the object <NUM> (i.e. a cupboard) to trigger the display of a bounding fence.

In contrast, in scenario 1710B, the user 1715B (viewing content via FOV 1720B) is now physically closer to the object and is within a particular distance threshold <NUM>. Because of this closer proximity, the user's HMD was triggered to display a bounding fence <NUM> around the object. In this particular scenario, the bounding fence <NUM> is not transparent but rather is opaque. Furthermore, in this particular scenario, the HMD is also rendering descriptive text (e.g., "CUPBOARD") to describe the underlying object, as described earlier. By displaying bounding fence <NUM>, the user 1715B will be alerted as to the presence and proximity of the underlying cupboard object.

<FIG> illustrates a similar circumstance, but in the context of a virtual-reality environment <NUM>, which includes a gravity vector <NUM>. In scenario 1810A, the user 1815A is viewing content through a FOV 1820A. Specifically, the user is viewing the VR scene 1825A. In this scenario, the user 1815A is not sufficiently near the object 1830A. Consequently, the VR scene 1825A is not displaying a bird's eye perspective of the real-world environment.

In scenario 1810B, user 1815B is viewing content in the FOV 1820B. This content includes the VR scene 1825B. Furthermore, the real-world environment includes object 1830B. Because user 1815B is physically within a distance threshold <NUM> of the object 1830B, the user's HMD was triggered to display the bird's eye perspective <NUM> in the VR scene 1825B. As such, the user 1815B may be alerted that he/she is physically near a real-world object. This alert will allow the user 1815B to avoid colliding with object 1830B.

The following discussion now refers to a number of method acts that may be performed.

<FIG> illustrates a flowchart of an example method <NUM> for dynamically generating and rendering an object bounding fence (e.g., any of the bounding fences discussed thus far) in an MR scene/environment. Initially, method <NUM> includes an act <NUM> of accessing a sparse spatial mapping of an environment. As described earlier, this sparse spatial mapping is considered to be "sparse" because it includes incomplete surface and texture data for real-world objects located within a user's real-world environment. Although the sparse spatial mapping includes incomplete data, it nevertheless still includes a sufficient amount of perimeter edge data, which describes one or more perimeter edge(s) of an object located within the environment, in order to adequately detect the boundaries of the object.

Method <NUM> then includes an act <NUM> of generating a gravity vector of a head-mounted device (HMD). This HMD is operating in the environment and is displaying a mixed-reality scene. Furthermore, the HMD may include any number of IMUs, which may be used to determine the gravity vector.

Based on the perimeter edge data and the gravity vector, method <NUM> then includes an act <NUM> of determining two-dimensional (2D) boundaries of the object within the environment. In this regard, the embodiments operate to generate a 2D representation of a 3D object. Additionally, method <NUM> includes an act <NUM> of generating a bounding fence mesh (e.g., a 2D mesh or a 2D spatial mapping) of the environment. This bounding fence mesh identifies the 2D boundaries of the object within the environment. The bird's eye perspective <NUM> of <FIG> can constitute this 2D spatial mapping.

Finally, method <NUM> includes an act <NUM> of rendering, within the mixed-reality scene, a virtual object that is representative of at least a portion of the bounding fence mesh and that visually illustrates a bounding fence around the object. In some implementations, the virtual object is a visualization of the bird's eye perspective (i.e. a 2D bird's eye view or a plan view) discussed throughout this disclosure (e.g., bird's eye perspective <NUM> of <FIG>). In some cases, the virtual object additionally includes one or more rectangular cuboids / 3D voxels, such as fences <NUM>, <NUM>, <NUM>, and <NUM> in <FIG>.

Accordingly, the disclosed embodiments provide for an improved technique to identify objects within an environment. This improved technique also helps users avoid those objects. In doing so, the user's experience with the HMD is significantly improved. Additionally, by using a sparse spatial mapping and by translating a 3D problem into a 2D problem and then providing a 2D solution to the 2D problem, the embodiments enable the HMD to use less processing and less power. As such, by practicing the disclosed principles, the battery lifespan of the HMD can be lengthened, which lengthening will also improve the user's experience.

It will be appreciated that as new areas of an environment are scanned by the HMD's camera system, then bounding fences can also be displayed for any objects in those new areas. As an example, suppose a user is backing up towards a wall. In this example, the space between the user and the wall had already been previously mapped. Using the mapping data, the embodiments are able to display the bird's eye perspective and/or other bounding fences. In this regard, the embodiments are able to generate a bounding fence mesh in which the HMD generates an outline of the plan or layout of the environment (i.e. a type of map of the environment), including objects within that environment. All of this information can be utilized in order to provide a 2D view of that environment to the user. Accordingly, the user can be made aware of objects located to his/her blind spots, including areas to his/her left, right, and back.

In some implementations, the embodiments may utilize a distributed and shared spatial mapping of the user's environment, where multiple users may be in the same environment and where these multiple users may all be contributing data to the shared spatial mapping (i.e. the data is being fused together). Furthermore, the users' HMDs can communicate with one another or detect one another, and indicators representative of the users (e.g., indicator <NUM>) can each be displayed on the visualization of the 2D bird's eye perspective.

Attention will now be directed to <FIG> which illustrates an example computer system <NUM> that may include and/or be used to perform the operations described herein. In particular, this computer system <NUM> may be in the form of the MR systems/devices that were described earlier. As such, the computer system may be one of the following: a virtual-reality system or an augmented-reality system.

Computer system <NUM> may take various different forms. For example, in <FIG>, computer system <NUM> may be embodied as a tablet 2000A, a desktop 2000B, or an HMD 2000C (with a corresponding wearable display), such as those described throughout this disclosure. The ellipsis 2000D demonstrates that computer system <NUM> may be embodied in any form.

Computer system <NUM> may also be a distributed system that includes one or more connected computing components/devices that are in communication with computer system <NUM>, a laptop computer, a mobile phone, a server, a data center, and/or any other computer system. The ellipsis 2000D also indicates that other system subcomponents may be included or attached with the computer system <NUM>, including, for example, sensors that are configured to detect sensor data such as user attributes (e.g., heart rate sensors), as well as sensors like cameras and other sensors that are configured to detect sensor data such as environmental conditions and location/positioning (e.g., clocks, pressure sensors, temperature sensors, gyroscopes, accelerometers and so forth), all of which sensor data may comprise different types of information used during application of the disclosed embodiments. Some of the embodiments are implemented as handheld devices or handheld depth cameras. Some embodiments are also operable in robotics, drones, ambient settings, and any type of mobile phone.

In its most basic configuration, computer system <NUM> includes various different components. <FIG> shows that computer system <NUM> includes at least one processor(s) <NUM> (aka a "hardware processing unit"), input/output ("I/O") <NUM>, camera system <NUM> (which is representative of camera system <NUM> of <FIG>), IMU(s) <NUM>, a boundary detection <NUM>, and storage <NUM>.

I/O <NUM> may include any number of input/output devices, including wearable or handheld devices. I/O <NUM> may also include a wearable display, which may be used to render virtual content. Camera system <NUM> may include any number of cameras, including head tracking, hand tracking, depth detection, or any other type of camera. These cameras may be configured in the manner described earlier, and the camera system <NUM> may perform any of the disclosed scanning or head tracking operations. Similarly, IMU(s) <NUM> are configured in the manner discussed earlier.

Boundary detection <NUM> is able to use the camera data from the camera system <NUM> to generate a surface mesh, or spatial mapping, of an environment. In this regard, the boundary detection <NUM>, the camera system <NUM>, and/or the processor(s) <NUM> may be configured to perform the disclosed operations.

Storage <NUM> is shown as including executable code/instructions <NUM>. The executable code/instructions <NUM> represent instructions that are executable by computer system <NUM> to perform the disclosed operations, such as those described in the method of <FIG>.

Storage <NUM> may be physical system memory, which may be volatile, non-volatile, or some combination of the two. The term "memory" may also be used herein to refer to non-volatile mass storage such as physical storage media. If computer system <NUM> is distributed, the processing, memory, and/or storage capability may be distributed as well. As used herein, the term "executable module," "executable component," or even "component" can refer to software objects, routines, or methods that may be executed on computer system <NUM>. The different components, modules, engines, and services described herein may be implemented as objects or processors that execute on computer system <NUM> (e.g. as separate threads).

The disclosed embodiments may comprise or utilize a special-purpose or general-purpose computer including computer hardware, such as, for example, one or more processors (such as processor(s) <NUM>) and system memory (such as storage <NUM>), as discussed in greater detail below. Embodiments also include physical and other computer-readable media for carrying or storing computer-executable instructions and/or data structures. Such computer-readable media can be any available media that can be accessed by a general-purpose or special-purpose computer system. Computer-readable media that store computer-executable instructions in the form of data are physical computer storage media. Thus, by way of example and not limitation, the current embodiments can comprise at least two distinctly different kinds of computer-readable media: computer storage media and transmission media.

Computer storage media are computer-readable hardware storage devices, such as RAM, ROM, EEPROM, CD-ROM, solid state drives ("SSD") that are based on RAM, Flash memory, phase-change memory ("PCM"), or other types of memory, or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code means in the form of computer-executable instructions, data, or data structures and that can be accessed by a general-purpose or special-purpose computer.

Computer system <NUM> may also be connected (via a wired or wireless connection) to external sensors (e.g., one or more remote cameras, accelerometers, gyroscopes, acoustic sensors, magnetometers, etc.) or devices via a network <NUM>. For example, computer system <NUM> can communicate with a handheld device <NUM> that includes spatial mapping data <NUM>. This spatial mapping data <NUM> may be used to augment or supplement any spatial mapping data accessed or generated by computer system <NUM>.

Furthermore, computer system <NUM> may also be connected through one or more wired or wireless networks <NUM> to remote/separate computer systems(s) that are configured to perform any of the processing described with regard to computer system <NUM>.

During use, a user of computer system <NUM> is able to perceive information (e.g., an MR scene/environment (including VR or AR)) through a display screen that is included with the I/O <NUM> of computer system <NUM> and that is visible to the user. The I/O <NUM> and sensors with the I/O <NUM> also include gesture detection devices, eye trackers, and/or other movement detecting components (e.g., cameras, gyroscopes, accelerometers, magnetometers, acoustic sensors, global positioning systems ("GPS"), etc.) that are able to detect positioning and movement of one or more real-world objects, such as a user's hand, a stylus, and/or any other object(s) that the user may interact with while being immersed in the mixed-reality environment.

A graphics rendering engine may also be configured, with processor(s) <NUM>, to render one or more virtual objects within an MR scene. As a result, the virtual objects accurately move in response to a movement of the user and/or in response to user input as the user interacts within the virtual scene.

A "network," like the network <NUM> shown in <FIG>, is defined as one or more data links and/or data switches that enable the transport of electronic data between computer systems, modules, and/or other electronic devices. When information is transferred, or provided, over a network (either hardwired, wireless, or a combination of hardwired and wireless) to a computer, the computer properly views the connection as a transmission medium. Computer system <NUM> will include one or more communication channels that are used to communicate with the network <NUM>. Transmissions media include a network that can be used to carry data or desired program code means in the form of computer-executable instructions or in the form of data structures. Further, these computer-executable instructions can be accessed by a general-purpose or special-purpose computer.

Computer-executable (or computer-interpretable) instructions comprise, for example, instructions that cause a general-purpose computer, special-purpose computer, or special-purpose processing device to perform a certain function or group of functions.

Additionally, or alternatively, the functionality described herein can be performed, at least in part, by one or more hardware logic components (e.g., the processor(s) <NUM>). For example, and without limitation, illustrative types of hardware logic components that can be used include Field-Programmable Gate Arrays ("FPGA"), Program-Specific or Application-Specific Integrated Circuits ("ASIC"), Program-Specific Standard Products ("ASSP"), System-On-A-Chip Systems ("SOC"), Complex Programmable Logic Devices ("CPLD"), Central Processing Units ("CPU"), and other types of programmable hardware.

Claim 1:
A method (<NUM>) for dynamically generating and rendering an object bounding fence in a mixed-reality scene, the method comprising:
accessing (<NUM>) a spatial mapping (<NUM>) of an environment (<NUM>), the spatial mapping including perimeter edge data describing one or more perimeter edge(s) of an object (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) located within the environment; wherein
the perimeter edge data describes a portion, but not all, of the one or more perimeter edge(s) of the object such that the perimeter edge data constitutes incomplete data, and wherein the spatial mapping is a sparse spatial mapping (<NUM>) as a result of including the incomplete data for the object;
generating (<NUM>) a gravity vector (<NUM>) of a head-mounted device (HMD) (<NUM>) that is operating in the environment and that is displaying a mixed-reality scene;
based on the perimeter edge data and the gravity vector, determining (<NUM>) two-dimensional (2D) boundaries of the object within the environment;
generating (<NUM>) a bounding fence mesh (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) of the environment, the bounding fence mesh identifying the 2D boundaries of the object within the environment; and
rendering (<NUM>), within the mixed-reality scene, a virtual object (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) that is representative of at least a portion of the bounding fence mesh and that visually illustrates a bounding fence around the object;
wherein a visualization of a 2D bird's eye view (<NUM>) of the environment is generated using the spatial mapping of the environment and is displayed within the mixed reality scene; wherein when the HMD determines that the HMD is located within a predetermined or pre-established threshold distance to the object, then the HMD triggers the display of the bird's eye view (<NUM>).