Patent Description:
The accompanying drawings illustrate implementations of the concepts conveyed in the present patent. Features of the illustrated implementations can be more readily understood by reference to the following description taken in conjunction with the accompanying drawings. Like reference numbers in the various drawings are used wherever feasible to indicate like elements. Further, the left-most numeral of each reference number conveys the figure and associated discussion where the reference number is first introduced.

This discussion relates to the use of cameras to provide environment information relating to a physical environment, such as a facility, and further relates to efficient ways to process environment information to detect planar surfaces without overburdening available computing resources that may be available to a computing device intended to process the environment information. The environment information can be used for various purposes, such as to recreate the environment in a virtual manner for training purposes, or by providing enhancements to the environment through augmented reality. Moreover, enhanced visualization data can be provided to create a mixed reality environment where a user can interact or co-exist with virtual, computer-generated mixed reality content.

Such mixed reality content can include graphics that are representative of objects, people, biometric data, effects, etc., that are not physically present in the environment. Mixed reality content can include two-dimensional graphics, three dimensional objects, and/or content associated with applications. For example, a mixed reality environment can augment a physical, real-world scene (e.g., an office) and/or a physical, real-world person with mixed reality content, such as computer-generated graphics (e.g., a cat, a chair, an advertisement on a wall, etc.) in the physical, real-world scene.

However, the scale of the environment information can be problematic for small mixed reality devices, such as those powered by batteries. For example, a user wearing a mixed reality device may receive a surface representation generated from environment information that corresponds to an entire building or factory. However, to place any mixed reality elements, traditionally the mixed reality device must process the surface representation corresponding to the entire building or factory in order to determine appropriate planar surfaces within the surface representation.

Due to the scale of the surface representation created from the environment information, this may prove too computationally intensive for the mixed reality device. For instance, the battery resources may be used unacceptably fast. Alternatively, the intense computation may cause delays that make an augmented reality experience less believable. For instance, if the user rolls a virtual ball in the environment, the ball may stop or glitch until the surface representation computations are completed. Moreover, as computational processing moves to shared resources on the Internet, the amount of data needing to be sent over a wireless transmission from the mixed reality device may need to be reduced for efficient operation. Thus, the following description solves the technical problem of performing environment information processing in manageable sized sub-volumes of the environment, and seamlessly stitching together certain environmental aspects that were separated into sub-volumes to ensure an accurate representation of the environment. These and other aspects are described below.

<FIG> collectively relate to an environment <NUM>, which may represent, for example, an office building or other such facility, but in some instances may include outdoor environments. With reference to <FIG>, a camera <NUM> is used to capture environment information <NUM> (depicted in <FIG>) of portions of the environment <NUM> and/or of scenes generally. (Note that for ease of explanation in introductory <FIG>, the term "environment information" is used broadly and can include two-dimensional images, three-dimensional images, visible light images and/or non-visible light images, among others).

In this example, camera 102A (facing away from reader) is manifest as an element of a head mounted mixed reality device <NUM> worn by a user <NUM>. Camera 102B is manifest as one of the sensors on a semi-autonomous facilities robot <NUM>. Other camera manifestations are contemplated and additional examples are described relative to <FIG>. Note that the location and orientation of the camera <NUM> can be tracked. For instance, the earth coordinate system location of the camera can be known. For example, the location of the camera can be tracked via global navigation satellite system coordinates, such as global positioning system (GPS) coordinates (e.g., the camera is at a particular xyz location). Further, the orientation of the camera can be tracked. For example, micro-electromechanical sensors (MEMS) can track the orientation relative to six axes (e.g. the camera is facing in a specific horizontal direction and at a specific vertical direction).

As depicted in <FIG>, user <NUM> may be walking a path to arrive at office <NUM>, which is depicted in <FIG>. The cameras <NUM> can capture environment information, such as multiple images of environment <NUM> from multiple different viewpoints, and the images can be fused together to create surface representations of portions of the environment <NUM> and/or of scenes generally as the user progresses toward office <NUM>. The environment information can include depth information received from depth images (e.g., how far regions of the image are from the camera), which can be aligned to a three-dimensional earth coordinate system given that the location and orientation of the camera is known, and may be represented as, for example, point clouds or surface meshes. It is to be appreciated that there are techniques for generating depth information from non-depth images, such as two-dimensional images. For instance, depth information can be generated from pairs of two-dimensional images. Thus, depth information can be available without traditional depth images.

Referring to <FIG>, environment information <NUM> of office <NUM> is depicted as a portion of environment <NUM>. Environment information <NUM>, as generated by cameras <NUM>, may include information related to various physical objects located within office <NUM>, such as a chair <NUM> having chair arms <NUM>, desk <NUM>, walls <NUM>, floor <NUM>, and ceiling <NUM>. These various physical objects have planar regions associated with them, such as planar region <NUM> corresponding to a section of the plane defined by floor <NUM>.

Furthermore, the owner of environment <NUM> or office <NUM> may desire that whenever user <NUM> wearing mixed reality device <NUM> enters a particular room in the environment, such as office <NUM>, a holographic representation of an object can be placed on one of the planar regions, such as planar region <NUM>. To accurately place a hologram on planar region <NUM>, mixed reality device <NUM> must correctly determine the position of planar region <NUM>, in relation to other planar regions within office <NUM>.

However, as noted earlier, to accurately identify the planar regions, a device may require processing the entire surface representation generated from environment information corresponding to environment <NUM>. As this can introduce difficulties on low power mixed reality devices, or on network/cloud-based processing, embodiments of the invention disclosed herein may allow environment <NUM> to instead be partitioned into a virtual grid of sub-volumes according to the three-dimensional coordinate space. Then, environment information corresponding to environment <NUM> is used to generate a surface representation in each of the sub-volumes, and the smaller surface representations can then be processed individually for determining planar regions, in order to reduce power requirements.

<FIG> depicts an example overhead view corresponding to environment <NUM> of <FIG>, which shows user <NUM> progressing toward office <NUM>. As described above, environment <NUM> is divided into a plurality of sub-volumes <NUM>, and a surface representation corresponding to environment information <NUM> is generated in each of the plurality of sub-volumes. By creating the surface representation in sub-volumes <NUM>, processing requirements with respect to planar surface detection can be reduced, as a plane detection process can be performed with respect to only certain sub-volumes of interest. Specifically, each sub-volume can be processed individually to determine planar regions within each sub-volume. However, in certain instances, the plane detection process is executed on all sub-volumes in parallel depending on the capabilities of, for example, mixed reality device <NUM>.

For example, when user <NUM> enters office 110A, plane detection can be performed on the specific sub-volumes that encompass the area of office 110A, resulting in a reduced amount of processing for mixed reality device <NUM>. Specifically, mixed reality device <NUM> does not need to detect planar regions of office 110B, in order for mixed reality device <NUM> to appropriately a hologram on floor <NUM> of office 110A. Note that for purposes of explanation, the sub-volumes <NUM> are shown to be of a representative size and shape, but can be of any specific size and shape, and therefore environment <NUM> is partitioned into many more sub-volumes than what is depicted. The individual sub-volume size can, be defined depending on the processing and/or wireless capabilities of mixed reality device <NUM>.

<FIG> depicts an area specific to office <NUM>, where a three-dimensional environment corresponding to office <NUM> has been partitioned into a plurality of sub-volumes <NUM>, depicted as cubical volumes having equal size. While <FIG> is shown with spacing between sub-volumes <NUM> (for ease of understanding of the drawings), it is to be appreciated that sub-volumes <NUM> may be created such that each sub-volume is slightly overlapping to ensure that the entire surface volume of the environment <NUM> is captured. Moreover, in certain configurations, the sub-volumes may each be of a fixed size and cubical in shape as depicted in <FIG>, but different sizes or shapes are equally usable depending on user preferences. Once the sub-volumes <NUM> are created, a surface representation corresponding to environment information is generated in the plurality of sub-volumes <NUM>.

As shown in <FIG>, the surface of desk <NUM> may span several of the sub-volumes <NUM>. When a plane detection process is performed (as described in further detail below relative to <FIG>) on individual sub-volumes, the plane detection process identifies planar regions that constitute planar fragments of a contiguous planar surface. For example, the surface of desk <NUM>, as depicted in <FIG>, spans several sub-volumes. Therefore, when the plane detection process is performed, a planar surface (e.g., surface of desk <NUM>) that spans multiple sub-volumes can become fragmented, resulting in desk planar fragments 130A, 130B, and 130C. Similarly, certain surfaces of chair <NUM> and chair arm <NUM> may span several of the sub-volumes <NUM>.

This aspect is depicted in <FIG>, where the surface of desk <NUM> has desk planar fragments 130A, 130B, and 130C as a result of the partitioning into sub-volumes. However, desk planar fragments 130A, 130B, and 130C constitute portions of a contiguous planar surface <NUM> (shown for ease of illustration as a freeform area) associated with the surface of desk <NUM>.

As desk planar fragments 130A, 130B, and 130C constitute contiguous planer surface <NUM> associated with the surface of desk <NUM>, additional processing is performed to aggregate any plane fragments that constitute a contiguous planar surface. Therefore, while segmenting planar detection into sub-volumes assists in reducing processing requirements, a plane aggregation process (as described in further detail below relative to <FIG>) is performed to determine whether a planar region has been split into planar fragments dispersed across sub-volumes.

Generally, when processing a particular sub-volume having a planar fragment, the plane aggregation algorithm checks neighboring sub-volumes based on the relative positions of the sub-volumes, and determine planar fragments having similar plane equations to the current planar fragment. When certain planar fragments are determined to have a similar plane equation, they are marked as candidate planar fragments for aggregation into a contiguous planar surface. In some instances, the similarity of planar fragments are based on a comparison of a plane equation associated with a seed plane and each planar fragment, as described in greater detail below with regard to <FIG>.

For example, with reference to <FIG>, the plane aggregation process may determine that desk planar fragments 130A, 130B, and 130C are all similar in plane equation, and will mark the planar fragments as candidates for aggregation. However, the plane aggregation process also determines that planar fragments associated with the top of chair arm <NUM> have a similar plane equation to, for example, desk planar fragment 130A (due to the chair height being set such that the chair arms are level with the desk), and as a result, assumes that the planar fragments associated with the top of the chair arm constitute candidate planar fragments that should be aggregated with desk planar fragment 130A to form a contiguous planar surface.

However, as this would be an incorrect joining of planar fragments (i.e., a chair arm should not be considered a common plane with a desk), the plane aggregation algorithm performs a connected components analysis to detect planar fragments across sub-volumes that have similar planar equations, but do not fall within a set threshold to the other planar fragment. In certain instances, the threshold can be based on a variety of different parameters, but in this example the threshold may be based on a distance between planar fragments.

When it is determined that a planar fragment is not within the threshold distance of another planar fragment, this plane can be removed from the listing of candidate planar fragments that will be aggregated together to form the contiguous planar surface. This threshold distance can be set by a user, depending on the sensitivity desired out of the plane aggregation process. For example, a user may define the threshold distance to be one meter if the process should be overly inclusive, or may set a threshold distance to one centimeter or even one millimeter for more exact definitions of planar surfaces. It is to be appreciated that any set distance can be used.

<FIG> depicts a top-down view showing a plurality of sub-volumes <NUM>, where a planar surface corresponding to desk <NUM> is divided into desk planar fragments 130A and 130B, and planar surfaces corresponding to chair arm <NUM> is divided into chair arm planar fragments 134A and 134B. As noted earlier, in some instances a planar fragment, such as chair arm planar fragment 134A, have a similar plane equation to a planar fragment associated with a different object, such as desk planar fragment 130A, but these planar fragments should not be aggregated due to belonging to different objects.

As such, the connected components analysis checks whether the planar fragments are within the threshold distance from each other. This can be performed by creating a two-dimensional projection image based at least on the three-dimensional surface points in one planar fragment and the three-dimensional surface points associated with candidate planar fragments that are marked for possible aggregation. Based on the projection image, pixels representing the surface points can be compared to determine whether they fall within the threshold.

As depicted in <FIG>, a projection image <NUM> is shown as a large grid for illustration purposes, but in actual implementation, the projection image can be created by rasterizing the surface points into pixels. The projection image can be generated such that the distance between pixels of a planar surface equate to the threshold distance defined by the user. As such, if there exists a section of the projection image where there is a perceivable gap (i.e., using projection image <NUM> as an example, any empty grid cells between two pixels associated with planar fragments), the connected components analysis process can determine that at least a threshold distance exists between the planar fragments. In creating the projection image, each candidate planar fragment can be associated with a different marker, such as a semantic label, color, or other such identifying indicia.

For example, as depicted in <FIG>, which represents the two-dimensional projection image corresponding to <FIG>, a number of pixel regions associated with different candidate planar fragments can be shown. For illustration purposes, desk planar fragment 130A is shown with diagonal hatching, while desk planar fragment 130B is shown with cross hatching, and chair arm planar fragments 134A and 134B are shown with square hatching.

As an example, if the threshold distance is set to one centimeter, then each pixel (or cell, as depicted for ease of illustration in <FIG>) of projection image <NUM> can be spaced apart according to this threshold distance. Using this spacing, the connected components analysis can parse the projection image and determine that desk planar fragment 130A directly connects with desk planar fragment 130B within the defined threshold amount, as there is no perceived gap in the projection image (i.e., no empty pixel). Therefore, the connected components analysis can determine that desk planar fragments 130A and 130B not only include similar plane equations, but that they are within the threshold distance for aggregation, so will be joined together in the final contiguous planar surface.

In contrast, the connected components analysis can determine that there are gaps (i.e., an empty pixel and therefore at least one centimeter of space) between desk planar fragment 130A, and both chair arm planar fragments 134A and 134B. As such, while the chair arm planar fragments 134A and 134B have a similar plane equation to desk planar fragments 130A and 130B, because the projection image indicates a gap between the projected points, the connected components analysis determines that the planar fragments are not within the threshold distance, and should not be included in the final contiguous planar surface that is aggregated together.

In certain instances, semantic labels associated with planar fragments can be checked as part of the connected components analysis. That is, upon determining that chair arm fragments 134A and 134B should not be included in the final contiguous planar surface, a secondary check can be performed based on any semantic labelling associated with the planar fragments. In this instance, the semantic labels associated with chair arm planar fragments 134A and 134B may be labelled as "chair," while the semantic labels associated with desk planar fragments 130A and 130B may be labelled as "desk," therefore confirming that the analysis properly excluded chair arm planar fragments 134A and 134B from the final contiguous planar surface. However, in other instances, if the semantic labels for fragments set for removal also reflect "desk," a user may realize that the threshold distance being used by the connected components analysis may be set at too close a distance.

Once the final contiguous planar surface is determined, data can then be output enabling usage of the contiguous planar surface. This resulting data can be utilized in a number of ways for displaying content to a user. In one instance, the process can send a collection of all surface points associated with the contiguous planar surface to the mixed reality device <NUM> so that the device can accurately process holographic information in an environment. For example, with reference to <FIG>, once the contiguous planar surface corresponding to floor <NUM> is determined, a cat hologram <NUM> can be placed on the contiguous planar surface. Alternatively, a bounding box could be provided to the mixed reality device <NUM>, or to an external server or device that can perform additional processing on the various contiguous planar surfaces.

In addition to the aspects discussed above, the environment information of environment <NUM> may change over time. For example, while a chair may be present in a hallway of the environment on one day, the chair may be moved to a different section of the environment the next day. It can be impractical to constantly require mixed reality device <NUM> to process every sub-volume in environment <NUM> in an attempt to determine whether there have been any temporal changes in the sub-volumes. As such, mixed reality device <NUM> can load stored planar data from memory (e.g., internal memory or a database stored on, for example, a hard disk or other computer-readable storage medium) that correspond to specific sub-volumes surrounding the user <NUM> wearing mixed reality device <NUM>, such as a spherical area surrounding mixed reality device <NUM>.

For example, a sphere having a fixed distance can be used, where any sub-volumes that fall within the spherical distance from the user <NUM> can be loaded from memory, and prior environment information associated with those sub-volumes are compared to current or subsequent environment information associated with those sub-volumes. When changes in the environment information are detected, mixed reality device <NUM> may determine whether the amount of changes between the prior environment information and the subsequent environment information exceed a threshold number of changes. It is to be appreciated that any shape other than a sphere can be used, or mixed reality device <NUM> may be programmed to only load N number of sub-volumes at a time, such as one sub-volume in each cardinal direction based on the current position of user <NUM>.

Such a threshold number of changes can be user defined and dependent on the needs of the exactness of planar regions associated with environment <NUM>. If it is determined that the changes exceed the threshold, mixed reality device <NUM> can re-process the planar region data associated with any sub-volume that exhibits changes that exceed the threshold number of changes or deviations. While a prior environment information may show a desk being three feet from wall <NUM>, the current environment information may depict the desk being four feet from wall <NUM>. Based on the threshold value, this may be within the tolerance bounds of acceptable changes, and mixed reality device <NUM> can avoid having to re-process the planar data associated with the desk.

<FIG> offers a detailed overview of a method <NUM> for performing the sub-volume processing described earlier. Initially, at block <NUM>, a three-dimensional environment representing environment <NUM> is partitioned into a plurality of sub-volumes. In partitioning the three-dimensional environment into sub-volumes, a bounding box of a fixed size and/or volume can be set with respect to coordinates of the three-dimensional environment. Once the bounding box is set, sections of the three-dimensional environment that fall within the bounding box can be partitioned off as sub-volumes.

In certain instances, minimum amounts of surfaces may be defined, where a partitioned sub-volume may be required to have at least a threshold number of surfaces to be included in sub-volumes resulting from the partitioning. For example, a user may not be interested in any floor surfaces for purposes of determining planar surfaces, and during partitioning, specific sub-volumes can be checked to determine a sub-volume solely has floor surfaces. In this instance, the sub-volume containing only floor surfaces could be excluded from any additional processing.

At block <NUM>, environment information, such as depth measurements received from cameras <NUM>, can be used to generate a surface representation in each individual sub-volume. Specifically, by using the environment information received from cameras <NUM>, a surface representation corresponding to the bounding box of each sub-volume can be generated in the sub-volumes.

Alternatively, in certain instances, blocks <NUM> and <NUM> could be collapsed into a single step, whereby user <NUM> wearing mixed reality device <NUM> can move about environment <NUM>, and during this movement, bounding boxes of a fixed size and/or volume can be defined that surround user <NUM>. The bounding boxes can be defined based on the current location of user <NUM>, and surface representations corresponding to the bounding boxes can be generated based on data received from, for example, camera 102A.

Furthermore, as user <NUM> moves about environment <NUM>, additional sub-volumes can be created dynamically surrounding user <NUM>. Additionally, as user <NUM> moves about environment <NUM>, certain bounding boxes created around user <NUM> correspond to sections of environment <NUM> that lack surfaces entirely. In this case, additional processing with respect to that bounding box can be cancelled because there are no planar regions to detect.

Once the surface representations are generated, at block <NUM>, planar fragments in the sub-volumes are detected. Specifically, a sub-volume plane detection algorithm detects planar regions in each sub-volume's surface representation, where each planar region can be identified by surface points that lie on the plane, and an associated three-dimensional plane equation, and certain planar regions are marked as potential planar fragments that are part of a contiguous planar surface.

The sub-volume plane detection algorithm used at block <NUM> first determines the existence of planar regions within each sub-volume. Specifically, a k-Nearest Neighbors algorithm can be performed in order to compute the k nearest surface points for each of the surface points in the sub-volume, where k represents a predefined constant specifying the number of neighbors to access. These k-nearest neighbors may be based on, for example, mesh triangle connectivity information, or by using a k-d tree data structure. Next, a surface normal can be computed for each three-dimensional surface point within the sub-volume by fitting a plane to that surface point's k-nearest neighbors, and each of the three-dimensional surface points are then sorted according to how well the fitted plane of the surface point matches with the point's k-nearest neighbors. It is to be appreciated that k can be defined as any number, and while a small value of k can result in faster processing due to having to check only a limited number of neighbors, a large value of k can potentially capture a greater number of similar surface points, thereby improving the detection of planar regions.

Based on the sorting of the three-dimensional surface points, the point that is most planar (i.e., has the best plane fit to the point's k-nearest neighbors) can be selected as a seed point. The sub-volume plane detection algorithm can then initiate a breadth-first traversal (BFT), where the k-nearest neighbors of any inliers of the seed point are checked to discover any additional surface points that should be considered inliers to the seed point. In this instance, a surface point can be considered an inlier to the seed point plane if it is within a certain distance to the seed point plane, and its surface normal is within a certain angle to the seed point plane's surface normal.

Once the BFT terminates (i.e., no additional inliers could be added), a refined plane can be fit to the seed point and the entire set of inlier points. Next, the sub-volume plane detection algorithm can initiate another BFT on the refined plane to attempt to determine whether any additional inlier points can be added to the refined plane, and this process may be repeated until no additional inlier points can be added to a particular seed point plane. This entire process can be performed on each surface point in a sub-volume, and through iterative processing, the various planar surfaces in the sub-volume can be discovered.

However, as noted earlier, because a planar surface can span multiple sub-volumes, at block <NUM>, a plane aggregation algorithm can be performed to determine candidate planar fragments for aggregating as a contiguous planar surface. A plane aggregation algorithm is described in more detail below relative to <FIG>. As a result of the plane aggregation algorithm, candidate planar fragments are marked for possible aggregation.

Next, at block <NUM>, a connected components analysis can be performed to confirm the contiguousness of the candidate planar fragments. Specifically, the connected components analysis can utilize a two-dimensional projection image that can be created by rasterizing the three-dimensional surface points of candidate planar fragments. The two-dimensional projection can be used by the connected components analysis to detect contiguousness of the planar regions that are candidates for aggregating. As described earlier, while two surfaces have similar planar equations, and therefore be candidates for aggregating, they may in fact be separated by at least a threshold distance, and by using the two-dimensional projection image in the manner set forth above with respect to <FIG>, this threshold distance can be checked between points of candidate planar fragments.

At block <NUM>, the candidate planar fragments that have been found to be contiguous as a result of the connected components analysis is aggregated to form the contiguous planar surface. The various planar fragments that are being aggregated are assigned common semantic labels indicating that they are part of a contiguous surface, notwithstanding the fact that they occupy different sub-volumes.

Finally, at block <NUM>, data representing the contiguous planar regions can be output. For example, a set of planar points, or a bounding box containing the various planar points for the contiguous planar region are output to, e.g., a user <NUM> wearing mixed reality device <NUM>.

<FIG> shows another method or technique <NUM>, which describes steps performed by the plane aggregation algorithm. First, at block <NUM>, the k-nearest neighbors of a sub-volume can be determined based on the sub-volume position and the positions of neighboring sub-volumes.

Next, at block <NUM> a listing of planar fragments in the sub-volume and neighboring sub-volumes can be sorted according to the number of inliers to each planar fragment, where the planar fragment with the highest number of inlier planes can be selected as a seed plane. A planar fragment is considered an inlier to the seed plane if the angle between their planar surface normal is below a threshold value, and the distance (along the normal direction) between the two planes' average surface points is below a threshold value.

At block <NUM>, the seed plane can be used as a starting point to initiate a BFT that follows the k-nearest neighbors of the sub-volume to discover additional inlier planes with a similar plane equation to the seed plane within a threshold level of similarity. Similar inliers planes can be marked as candidate planar fragments for aggregation.

Once the BFT propagates through the various neighboring sub-volumes and terminates (i.e., no more planes are found for aggregating), at block <NUM>, the seed plane equation can be recomputed using the entire set of points associated with the inlier planes to create a refined plane equation.

Once the seed plane equation is refined, at block <NUM>, a check can be performed to determine whether there are any additional planes that can be processed based on the refined seed plane equation. If there are additional planes, then a BFT can be initiated again based on the refined plane equation in an attempt to continue growing the planar region, and the process loops back to block <NUM> to perform additional inlier plane processing. This iterative processing between blocks <NUM> to <NUM> continues until there are no more neighboring sub-volumes with additional planar fragments to process.

If there are no additional planes to add based on the refined seed plane equation, then at block <NUM>, the candidate planar fragments marked for aggregation are provided to the connected components analysis to confirm whether the candidate planar fragments should be aggregated into the contiguous planar surface, described in detail above with regard to <FIG>.

<FIG> depicts a system <NUM> that can be used to process surface representations in sub-volumes, and further aggregate planar fragments into contiguous planar surfaces. For purposes of explanation, system <NUM> can include four devices <NUM>(<NUM>), <NUM>(<NUM>), <NUM>(<NUM>), and <NUM>(<NUM>). Device <NUM>(<NUM>) is manifest as an autonomous robot that is similar to robot <NUM> of <FIG>. Device <NUM>(<NUM>) is manifest as a head mounted mixed reality device, similar to mixed reality device <NUM> of <FIG>, and device <NUM>(<NUM>) is manifest as a tablet-type device that can employ a camera. Devices <NUM>(<NUM>)-<NUM>(<NUM>) can include cameras <NUM>. Any of these devices can be free-standing and/or can communicate with other devices, such as server-type devices <NUM>(<NUM>). Individual devices <NUM> can include camera <NUM>, other sensor <NUM>, a scene processing component <NUM>, a processor <NUM>, and/or memory/storage <NUM>.

<FIG> depicts two device configurations <NUM> that can be employed by devices <NUM>. Individual devices <NUM> can employ either of configurations <NUM>(<NUM>) or <NUM>(<NUM>), or an alternate configuration. (Due to space constraints on the drawing page, one instance of each device configuration is illustrated rather than illustrating the device configurations relative to each device <NUM>). Briefly, device configuration <NUM>(<NUM>) represents an operating system (OS) centric configuration. Device configuration <NUM>(<NUM>) represents a system on a chip (SOC) configuration. Device configuration <NUM>(<NUM>) is organized into one or more applications <NUM>, operating system <NUM>, and hardware <NUM>. Device configuration <NUM>(<NUM>) is organized into shared resources <NUM>, dedicated resources <NUM>, and an interface <NUM> therebetween.

In some configurations, each of devices <NUM> can have an instance of the scene processing component <NUM>. However, the functionalities that can be performed by scene processing component <NUM> may be the same or they may be different from one another. For instance, in some cases, each device's scene processing component <NUM> can be robust and provide all of the functionality described above and below (e.g., a device-centric implementation). In other cases, some devices can employ a less robust instance of the scene processing component <NUM> that relies on some functionality to be performed by another device. For instance, device <NUM>(<NUM>) may have more processing resources than device <NUM>(<NUM>). In such a configuration, some scene processing component functions may be performed on device <NUM>(<NUM>) rather than device <NUM>(<NUM>), or scene processing may be split among the devices (i.e., some processing locally and some on the cloud) depending on device capability and/or network capability.

In some configurations, devices <NUM>, such as via scene processing component <NUM>, can be configured to perform certain processing in relation to memory <NUM> storing planar data. Devices <NUM> may perform processing such as capturing, via camera <NUM>, current environment information, and can further determine whether any sub-volumes need updating due to changes between prior environment information and current environment information by loading from memory/storage <NUM> a subset portion of the stored planar data associated with the prior environment information. The devices can further be configured to perform additional processing, such as identifying sub-volumes that include changes in planar data between the current environment information and the prior environment information, and can then update the stored planar data associated with the identified sub-volumes.

The term "device," "computer," or "computing device" as used herein can mean any type of device that has some amount of processing capability and/or storage capability. Processing capability can be provided by one or more processors that can execute data in the form of computer-readable instructions to provide a functionality. Data, such as computer-readable instructions and/or user-related data, can be stored on storage, such as storage that can be internal or external to the device. The storage can include any one or more of volatile or non-volatile memory, hard drives, flash storage devices, and/or optical storage devices (e.g., CDs, DVDs etc.), remote storage (e.g., cloud-based storage), among others. As used herein, the term "computer-readable media" can include signals. In contrast, the term "computer-readable storage media" excludes signals. Computer-readable storage media includes "computer-readable storage devices. " Examples of computer-readable storage devices include volatile storage media, such as RAM, and non-volatile storage media, such as hard drives, optical discs, and flash memory, among others.

Examples of devices <NUM> can include traditional computing devices, such as personal computers, desktop computers, servers, notebook computers, vehicles, smart cameras, surveillance devices/systems, safety devices/systems, wearable smart devices, appliances, and other developing and/or yet to be developed device types, etc..

As mentioned above, device configuration <NUM>(<NUM>) can be thought of as a system on a chip (SOC) type design. In such a case, functionality provided by the device can be integrated on a single SOC or multiple coupled SOCs. One or more processors <NUM> can be configured to coordinate with shared resources <NUM>, such as memory/storage <NUM>, etc., and/or one or more dedicated resources <NUM>, such as hardware blocks configured to perform certain specific functionality. Thus, the term "processor" as used herein can also refer to central processing units (CPUs), graphical processing units (GPUs), field programable gate arrays (FPGAs), controllers, microcontrollers, processor cores, or other types of processing devices.

Generally, any of the functions described herein can be implemented using software, firmware, hardware (e.g., fixed-logic circuitry), or a combination of these implementations. The term "component" as used herein generally represents software, firmware, hardware, whole devices or networks, or a combination thereof. In the case of a software implementation, for instance, these may represent program code that performs specified tasks when executed on a processor (e.g., CPU or CPUs). The program code can be stored in one or more computer-readable memory devices, such as computer-readable storage media. The features and techniques of the component are platform-independent, meaning that they may be implemented on a variety of commercial computing platforms having a variety of processing configurations.

To summarize some of the aspects described above, some implementations can reduce processing requirements imposed on a mixed reality device by enabling the device to partition a three-dimensional environment into a plurality of sub-volumes. Once the environment is partitioned into sub-volumes, surface representations associated with environment information can be generated in the sub-volumes, and processing can be performed separately on the sub-volumes to detect planar regions that are located within each of the sub-volumes.

Once the planar regions associated with each of the sub-volumes are detected, plane aggregation can be performed, which can attempt to determine whether certain planar regions are fragments of a contiguous planar region or surface. Once these planar fragments are detected, the plane aggregation algorithm can also determine whether the planar fragments should be aggregated into a contiguous planar surface by checking the planar fragments against a threshold value. If the planar fragments fall within the threshold value, they can be included in the fragments that are aggregated into the contiguous planar surface.

Thus, by generating smaller surface representations in individual sub-volumes, the processing requirements of a device can be reduced/minimized, as only individual sub-volume amounts of plane data are required to be processed at any one point of time.

The order in which the disclosed methods are described is not intended to be construed as a limitation, and any number of the described blocks can be combined in any order to implement the method, or an alternate method. Furthermore, the methods can be implemented in any suitable hardware, software, firmware, or combination thereof, such that a computing device can implement the method. In one case, the methods are stored on one or more computer-readable storage media as a set of instructions such that execution by a processor of a computing device causes the computing device to perform the method.

Claim 1:
A method (<NUM>) for generating contiguous planar surfaces of a three-dimensional environment, the method comprising:
partitioning (<NUM>) the three-dimensional environment including objects into a plurality of sub-volumes;
generating (<NUM>) a surface representation in each of the plurality of sub-volumes based on captured environment information;
detecting (<NUM>) a plurality of planar fragments inside each of the plurality of sub-volumes, wherein detecting (<NUM>) the plurality of planar fragments is performed independently and in parallel for each sub-volume, and wherein detecting the plurality of planar fragments comprises:
determining whether a first planar fragment has a similar plane equation as a second planar fragment; and
marking the first planar fragment and the second planar fragment as the candidate planar fragments for aggregating based at least on having a similar plane equation;
aggregating (<NUM>) the candidate planar fragments to form a contiguous planar surface spanning multiple sub-volumes, wherein the candidate planar fragments are aggregated to form the contiguous planar surface when their plane equations are within a threshold level of similarity; and
outputting (<NUM>) data representing the contiguous planar surface..