Normal estimation for a planar surface

Various implementations disclosed herein include devices, systems, and methods for normal estimation using a directional measurement, such as a gravity vector. In various implementations, a device includes a non-transitory memory and one or more processors coupled with the non-transitory memory. In some implementations, a method includes identifying planar surfaces in an environment represented by an image. Each planar surface is associated with a respective orientation. A directional vector associated with the environment is determined. A subset of the planar surfaces that have a threshold orientation relative to the directional vector is identified. For each planar surface in the subset of the planar surfaces, a normal vector for the planar surface is determined based on the orientation of the planar surface and the directional vector.

TECHNICAL FIELD

The present disclosure generally relates to normal estimation for a planar surface.

BACKGROUND

Normal estimation continues to be of interest in machine vision. For example, normal information may be used, e.g., with semantic class information, to identify objects in an environment. Normal information is also used in depth estimation. Normal estimation tends to be a resource-intensive operation.

SUMMARY

Various implementations disclosed herein include devices, systems, and methods for normal estimation using a directional measurement, such as a gravity vector. In various implementations, a device includes a non-transitory memory and one or more processors coupled with the non-transitory memory. In some implementations, a method includes identifying planar surfaces in an environment represented by an image. Each planar surface is associated with a respective orientation. In some implementations, the method includes determining a directional vector associated with the environment. In some implementations, the method includes identifying a subset of the planar surfaces that have a threshold orientation relative to the directional vector. For each planar surface in the subset of the planar surfaces, a normal vector for the planar surface is determined based on the orientation of the planar surface and the directional vector.

In accordance with some implementations, a device includes one or more processors, a non-transitory memory, and one or more programs. In some implementations, the one or more programs are stored in the non-transitory memory and are executed by the one or more processors. In some implementations, the one or more programs include instructions for performing or causing performance of any of the methods described herein. In accordance with some implementations, a non-transitory computer readable storage medium has stored therein instructions that, when executed by one or more processors of a device, cause the device to perform or cause performance of any of the methods described herein. In accordance with some implementations, a device includes one or more processors, a non-transitory memory, and means for performing or causing performance of any of the methods described herein.

DESCRIPTION

A physical environment refers to a physical world that people can sense and/or interact with without aid of electronic devices. The physical environment may include physical features such as a physical surface or a physical object. For example, the physical environment corresponds to a physical park that includes physical trees, physical buildings, and physical people. People can directly sense and/or interact with the physical environment such as through sight, touch, hearing, taste, and smell. In contrast, an extended reality (XR) environment refers to a wholly or partially simulated environment that people sense and/or interact with via an electronic device. For example, the XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and/or the like. With an XR system, a subset of a person's physical motions, or representations thereof, are tracked, and, in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that comports with at least one law of physics. As one example, the XR system may detect head movement and, in response, adjust graphical content and an acoustic field presented to the person in a manner similar to how such views and sounds would change in a physical environment. As another example, the XR system may detect movement of the electronic device presenting the XR environment (e.g., a mobile phone, a tablet, a laptop, or the like) and, in response, adjust graphical content and an acoustic field presented to the person in a manner similar to how such views and sounds would change in a physical environment. In some situations (e.g., for accessibility reasons), the XR system may adjust characteristic(s) of graphical content in the XR environment in response to representations of physical motions (e.g., vocal commands).

Some machine vision systems perform normal estimation (e.g., determine a vector that is normal to a surface) for a variety of purposes. For example, normal information and semantic class information may be used in a region growing algorithm to identify physical articles in a physical environment. Normal information may also be used for depth estimation. Some machine vision systems perform normal estimation on a pixel-by-pixel basis. However, performing normal estimation in this way can be computationally expensive and can be prone to error.

The present disclosure provides methods, systems, and/or devices for normal estimation using a directional measurement, such as a gravity vector. In various implementations, a device includes a non-transitory memory and one or more processors coupled with the non-transitory memory. In some implementations, a method includes identifying planar surfaces in an environment (e.g., a physical environment or an XR environment) represented by an image. Each planar surface is associated with a respective orientation. A directional vector (e.g., a gravity vector) associated with the environment is determined. In some implementations, the method includes identifying a subset of the planar surfaces that have a threshold orientation relative to the directional vector. For example, planar surfaces that are parallel to or orthogonal to the directional vector may be identified. In some implementations, the method includes determining a normal vector for each such planar surface based on the orientation of the planar surface and the directional vector. In some implementations, if a planar surface is horizontal, the normal vector is determined to be aligned with the direction of gravity. On the other hand, if a planar surface is vertical, the normal vector may be determined to be orthogonal to the direction of gravity. In some implementations, using a directional vector, such as a gravity vector, to determine the normal vector for certain surfaces, reduces the number of surfaces for which computationally intensive normal estimation techniques are used. For example, determining normal vectors for horizontal and vertical surfaces based on the directional vector (e.g., based on the gravity vector) reduces the need to determine the normal vectors for horizontal and vertical surfaces on a pixel-by-pixel basis thereby reducing a utilization of computing resources. Reducing the utilization of computing resources tends to enhance an operability of a battery-operated device by extending a battery life of the battery-operated device. Reducing the utilization of computing resources tends to improve an operational efficiency of a device by reducing an amount of heat that the device generates.

FIG.1illustrates an exemplary operating environment100in accordance with some implementations. While pertinent features are shown, those of ordinary skill in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity and so as not to obscure more pertinent aspects of the example implementations disclosed herein. To that end, as a non-limiting example, the operating environment100includes an electronic device102and a controller104. In some implementations, the electronic device102is or includes a smartphone, a tablet, a laptop computer, and/or a desktop computer. The electronic device102may be worn by or carried by a user106.

As illustrated inFIG.1, in some implementations, the electronic device102and/or the controller104obtains an image108that corresponds to a physical environment. In some implementations, the image108corresponds to an XR environment. In some implementations, the image108is captured by an image sensor, such as a camera110associated with the electronic device102and/or the controller104. In some implementations, the image108is obtained from another device that is in communication with the electronic device102and/or the controller104.

In some implementations, the image108is a still image. In some implementations, the image108is an image frame forming part of a video feed. The image108includes a plurality of pixels. Pixels in the image108may correspond to features in a physical environment. For example, a set of pixels in the image108may correspond to an object112in the physical environment.

In some implementations, the physical environment has one or more planar surfaces. For example, the object112may have multiple planar surfaces, and each planar surface may be associated with a respective normal vector. A planar surface114may be associated with a normal vector116that is orthogonal to the planar surface114. A planar surface118may be associated with a normal vector120that is orthogonal to the planar surface118. A planar surface122may be associated with a normal vector124that is orthogonal to the planar surface122.

In some implementations, the electronic device102and/or the controller104identifies one or more of the planar surfaces in the physical environment. For example, the electronic device102and/or the controller104may identify the planar surface114, the planar surface118, and/or the planar surface122. In some implementations, the planar surfaces114,118, and122correspond with respective sets of pixels in the image108.

In some implementations, the electronic device102and/or the controller104determines a directional vector126that is associated with the physical environment. The directional vector126may be a gravity vector. In some implementations, the electronic device102and/or the controller104includes an inertial measurement unit (IMU)128. In some implementations, a gravity measurement is derived from the IMU128. The electronic device102and/or the controller104may synthesize the directional vector126(e.g., a gravity vector) based on the gravity measurement.

In some implementations, the electronic device102and/or the controller104determines the directional vector126based on one or more spatial lines in the physical environment. For example, the electronic device102and/or the controller104may derive a gravity measurement from the image108. In some implementations, the image108includes a set of pixels130, e.g., a line, that corresponds to a vertical line in the physical environment, such as a portion of a window or a door. In some implementations, the directional vector126(e.g., the direction of gravity) is determined based on (e.g., inferred from) the orientation of the vertical line. As another example, the directional vector126may be determined based on an orientation of a physical article, e.g., a window or a door, that is represented by the set of pixels130. In some implementations, the image108includes a set of pixels that corresponds to a horizontal line in the physical environment, and the directional vector126is determined based on (e.g., inferred from) the orientation of the horizontal line.

In some implementations, the electronic device102and/or the controller104identifies a subset of the planar surfaces that have a threshold orientation relative to the directional vector126. For example, the electronic device102and/or the controller104may identify planar surfaces that are parallel to the directional vector126, such as the planar surface118. The electronic device102and/or the controller104may identify planar surfaces, such as the planar surface114, that are orthogonal to the directional vector126. It will be appreciated that the physical environment may include multiple surfaces that are parallel to the directional vector126and/or multiple surfaces that are orthogonal to the directional vector126.

In some implementations, for each planar surface that was identified, the electronic device102and/or the controller104determines a normal vector for the planar surface based on the orientation of the planar surface and the directional vector126. For example, in the case of a planar surface that is parallel to the directional vector126, such as the planar surface118, the electronic device102and/or the controller104may determine that the normal vector120is orthogonal to the directional vector126. In the case of a planar surface that is orthogonal to the directional vector126, such as the planar surface114, the electronic device102and/or the controller104may determine that the normal vector116is parallel to the directional vector126.

It will be appreciated that some planar surfaces in the physical environment, such as the planar surface122, are neither parallel to the directional vector126nor orthogonal to the directional vector126. In some implementations, the electronic device102and/or the controller104uses another normal estimation technique (e.g., pixel-by-pixel analysis) to determine the normal vector, e.g., the normal vector124, for such planar surfaces.

In some implementations, the electronic device102is replaced by or is attached to a head-mountable device (HMD) worn by the user106. The HMD presents (e.g., displays) an XR environment according to various implementations. In some implementations, the HMD includes an integrated display (e.g., a built-in display) that displays the XR environment. In some implementations, the HMD includes a head-mountable enclosure. In various implementations, the head-mountable enclosure includes an attachment region to which another device with a display can be attached. For example, in some implementations, the electronic device102can be attached to the head-mountable enclosure. In various implementations, the head-mountable enclosure is shaped to form a receptacle for receiving another device that includes a display (e.g., the electronic device102). For example, in some implementations, the electronic device102slides/snaps into or otherwise attaches to the head-mountable enclosure. In some implementations, the display of the device attached to the head-mountable enclosure presents (e.g., displays) the XR environment. In various implementations, examples of the electronic device102include smartphones, tablets, media players, laptops, etc.

FIG.2illustrates an example system200that performs normal estimation according to various implementations. In some implementations, a normal estimation device202obtains an image204. The normal estimation device202may obtain the image204from an image sensor206or other component of a device in which the normal estimation device202is integrated, such as, for example, the electronic device102ofFIG.1. In some implementations, the normal estimation device202obtains the image204from a device208that is external to a device in which the normal estimation device202is integrated. For example, if the normal estimation device202is integrated in the electronic device102ofFIG.1, the normal estimation device202may obtain the image204from another electronic device with which the electronic device102is in communication.

The image204comprises a plurality of pixels210. The pixels210may correspond to features in a physical environment. For example, a set of pixels in the image204may correspond to an object212in the physical environment. In some implementations, the normal estimation device202performs semantic segmentation and/or instance segmentation on the image204to distinguish the pixels corresponding to the object212from pixels corresponding to a background, e.g., pixels not corresponding to the object212.

Some pixels correspond to a boundary between the object212and the background. For example, the normal estimation device202may determine that pixels210a,210b, and210ccorrespond to the boundary of the object212. In some implementations, the normal estimation device202identifies planar surfaces in the physical environment. The identified planar surfaces may be associated with pixels that correspond to the boundary of the object212. For example, planar surfaces214a,214b, and214cmay be associated with the pixels210a,210b, and210c, respectively.

In some implementations, the normal estimation device202determines a directional vector, such as a gravity vector, that is associated with the physical environment. In some implementations, an IMU216provides a gravity measurement to the normal estimation device202. The normal estimation device202may synthesize the gravity vector based on the gravity measurement.

In some implementations, the normal estimation device202determines the directional vector based on one or more spatial lines, such as a vertical line, in the physical environment. For example, the normal estimation device202may identify a set of pixels in the image204that corresponds to a vertical line or a horizontal line in the physical environment. Semantic segmentation and/or instance segmentation may be applied to associate the identified set of pixels with an object type (e.g., a door). The normal estimation device202may determine an orientation of the line based on the object type. For example, if the normal estimation device202associates the identified set of pixels with a door, the normal estimation device202may determine that the identified set of pixels corresponds to a vertical line. On the other hand, if the normal estimation device202instead associates the identified set of pixels with a tabletop, the normal estimation device202may determine that the identified set of pixels corresponds to a horizontal line. In some implementations, the normal estimation device202determines the directional vector based on the orientation of the line.

In some implementations, the normal estimation device202identifies a subset of the planar surfaces that have a threshold orientation relative to the directional vector. For each such planar surface having the threshold orientation, the normal estimation device202may determine a normal vector based on the orientation of the planar surface and the directional vector.

For example, the normal estimation device202may identify planar surfaces that are parallel to the directional vector. If the directional vector is a gravity vector, the planar surface214amay be parallel to the gravity vector. In some implementations, the normal estimation device202determines that a normal vector218aextends from the pixel210ain a direction orthogonal to the gravity vector and outward from the object212.

As another example, the normal estimation device202may identify planar surfaces that are orthogonal to the directional vector. If the directional vector is a gravity vector, for example, the planar surface214bmay be orthogonal to the gravity vector. In some implementations, the normal estimation device202determines that a normal vector218bextends from the pixel210bin a direction parallel to the gravity vector and outward from the object212.

In some implementations, the normal estimation device202identifies planar surfaces that are neither parallel to nor orthogonal to the directional vector. For example, if the directional vector is a gravity vector, the planar surface214cmay be neither parallel to nor orthogonal to the gravity vector. In some implementations, the normal estimation device202uses another normal estimation technique to determine a normal vector218cthat extends from the pixel210coutward from the object212.

In various implementations, the normal estimation device202performs a first normal estimation operation for planar surfaces that are either parallel to or orthogonal to the directional vector. In some implementations, the first normal estimation operation utilizes a first amount of computing resources. In some implementations, performing the first normal estimation operation includes determining that a planar surface is parallel to the directional vector, and assigning the planar surface a normal vector that is orthogonal to the directional vector. In some implementations, performing the first normal estimation operation includes determining that a planar surface is orthogonal to the directional vector, and assigning the planar surface a normal vector that is parallel to the directional vector. In various implementations, the normal estimation device202performs a second normal estimation operation for planar surfaces that are neither parallel to nor orthogonal to the directional vector. In some implementations, the second normal estimation operation utilizes a second amount of computing resources that is greater than the first amount of computing resources. The second normal estimation operation is different from the first normal estimation operation. In some implementations, the second normal estimation operation includes performing a pixel-by-pixel analysis of the planar surface.

FIG.3is a block diagram of an example normal estimation device300in accordance with some implementations. In some implementations, the normal estimation device300implements the normal estimation device202shown inFIG.2. In some implementations, the normal estimation device300obtains an image302.

In some implementations, an image obtainer310obtains the image302. The image obtainer310may obtain the image302from a component of a device in which the normal estimation device300is integrated. For example, if the normal estimation device300is integrated in the electronic device102ofFIG.1, the electronic device102may have an image sensor, such as a camera, that may capture the image302and provide the image302to the image obtainer310. In some implementations, the image obtainer310obtains the image302from another device. For example, if the normal estimation device300is integrated in the electronic device102ofFIG.1, the image obtainer310may obtain the image302from a device with which the electronic device102is in communication.

In some implementations, the image302comprises a plurality of pixels. The pixels may correspond to features in a physical environment. For example, a set of pixels in the image302may correspond to an object in the physical environment. In some implementations, a pixel analyzer320performs semantic segmentation and/or instance segmentation on the image302to distinguish the pixels corresponding to the object from pixels corresponding to a background, e.g., pixels not corresponding to the object.

In some implementations, a surface identifier330identifies planar surfaces in the physical environment. For example, the pixel analyzer320may identify pixels that correspond to a boundary of the object. In some implementations, the pixel analyzer320uses an edge detection technique, e.g., an edge-aware image filter, to determine the boundary of the object. The surface identifier330may identify planar surfaces that are associated with pixels corresponding to the boundary of the object.

Each planar surface is associated with a respective orientation. Some planar surfaces have a horizontal orientation, e.g., orthogonal to a direction of gravity. Other planar surfaces have a vertical orientation, e.g., parallel to the direction of gravity. Still other planar surfaces have an orientation that is neither horizontal nor vertical.

Each planar surface is also associated with a respective normal vector that is orthogonal to the planar surface. For example, a planar surface with a horizontal orientation has a normal vector that has a vertical orientation. A planar surface with a vertical orientation has a normal vector that has a horizontal orientation. A planar surface that has an orientation that is neither horizontal nor vertical has a normal vector with an orientation that is neither horizontal nor vertical.

In some implementations, the normal estimation device300determines the normal vector for a planar surface based on the orientation of the planar surface and on a directional vector, such as a gravity vector. For example, in some implementations, a direction determiner340determines a directional vector342that is associated with the physical environment. The direction determiner340may determine the directional vector342based on a gravity measurement344from an IMU346. The direction determiner340may synthesize the directional vector342based on the gravity measurement344.

In some implementations, the direction determiner340determines the directional vector342based on one or more spatial lines, such as a vertical line, in the physical environment. For example, the pixel analyzer320may identify a set of pixels in the image302that corresponds to a vertical line or a horizontal line in the physical environment. The pixel analyzer320may perform semantic segmentation and/or instance segmentation on the image302to associate the identified set of pixels with an object type (e.g., a door). In some implementations, the pixel analyzer320determines an orientation of the line based on the object type. For example, if the identified set of pixels is associated with a door, the pixel analyzer320may determine that the identified set of pixels corresponds to a vertical line. On the other hand, if the identified set of pixels is instead associated with a tabletop, the pixel analyzer320may determine that the identified set of pixels corresponds to a horizontal line.

In some implementations, the direction determiner340determines the directional vector342based on the orientation of the line. For example, if the pixel analyzer320determines that the line is vertical, the direction determiner340may determine that the directional vector342is parallel to the line. On the other hand, if the pixel analyzer320determines that the line is horizontal, the direction determiner340may determine that the directional vector342is orthogonal to the line.

In some implementations, the normal estimation device300reduces the number of planar surfaces for which potentially computationally expensive and/or error-prone normal estimation techniques are used to determine normal vectors. For example, a normal information generator350may identify a subset of the planar surfaces that have a threshold orientation relative to the directional vector342. For each such planar surface having the threshold orientation, the normal information generator350may determine normal information352(e.g., a normal vector) based on the orientation of the planar surface and the directional vector342. For example, the normal information generator may identify planar surfaces that are parallel to the directional vector and determine that such planar surfaces have normal vectors that are orthogonal to the directional vector342. As another example, the normal information generator350may identify planar surfaces that are orthogonal to the directional vector and determine that such planar surfaces have normal vectors that are parallel to the gravity vector.

In some implementations, the normal information generator350identifies planar surfaces that are neither parallel to nor orthogonal to the directional vector. The normal information generator350may use another normal estimation technique to determine normal information352(e.g., a normal vector) for such planar surfaces.

FIGS.4A-4Care a flowchart representation of a method400for performing normal estimation in accordance with some implementations. In various implementations, the method400is performed by a device (e.g., the electronic device102and/or the controller104shown inFIG.1, and/or the system200shown inFIG.2). In some implementations, the method400is performed by processing logic, including hardware, firmware, software, or a combination thereof. In some implementations, the method400is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory). Briefly, in various implementations, the method400includes identifying planar surfaces in an environment represented by an image, determining a directional vector associated with the environment, identifying planar surfaces that have a threshold orientation relative to the directional vector, and, for each such planar surface, determining a normal vector based on the orientation of the planar surface and the directional vector.

As represented by block410, in various implementations, the method400includes identifying planar surfaces in an environment represented by an image. Each planar surface is associated with a respective orientation. Referring now toFIG.4B, as represented by block410a, in some implementations, the method400includes obtaining the image, which includes a plurality of pixels. As represented by block410b, the image may be obtained from an image sensor, such as a camera associated with the electronic device102ofFIG.1. In some implementations, the system200applies semantic segmentation and/or instance segmentation to identify pixels that correspond to an object in the environment.

As represented by block410c, the planar surfaces may be associated with a physical article in the environment, such as the object corresponding to the set of pixels. As represented by block410d, each planar surface may be associated with a point on the physical article. For example, certain pixels may be associated with a boundary of the physical article (e.g., between the physical article and the background), and planar surfaces may be associated with points on the physical article that correspond with those pixels. As represented by block410e, in some implementations, the point on the physical article may be detected using a depth sensor.

In some implementations, as represented by block410f, the method400includes determining the respective orientation associated with each planar surface. For example, when a directional vector has been determined, the system200may classify the orientations of each planar surface relative to the directional vector as parallel to the directional vector, orthogonal to the directional vector, or neither parallel to nor orthogonal to the directional vector. In some implementations, the system200determines the orientations of the planar surfaces without reference to the directional vector.

As represented by block420, in various implementations, the method400includes determining a directional vector associated with the environment. As represented by block420a, in some implementations, the directional vector is a gravity vector. In some implementations, as represented by block420b, the system200obtains the gravity vector. As represented by block420c, the system200may obtain the gravity vector by determining a gravity measurement and synthesizing the gravity vector based on the gravity measurement.

In some implementations, as represented by block420d, the system200includes an IMU. As represented by block420e, the system200may obtain the gravity vector from the IMU. For example, the system200may obtain a gravity measurement from the IMU and determine the gravity vector based on the gravity measurement.

In some implementations, as represented by block420f, the method400includes determining the gravity vector based on an orientation of a physical article in the environment, such as a door or a tabletop. As represented by block420g, the physical article may be represented by a set of pixels in the image. The system200may apply semantic segmentation and/or instance segmentation to associate the set of pixels with an object type (e.g., a door). The system200may determine the gravity vector based on the object type. For example, if the physical article is a door, the system200may determine that the gravity vector is parallel to the set of pixels representing the door. On the other hand, if the physical article is a tabletop, the system200may determine that the gravity vector is orthogonal to the set of pixels representing the tabletop.

In some implementations, as represented by block420h, the method400includes determining the directional vector based on a set of pixels in the image. As represented by block420i, the set of pixels may represent a spatial line in the environment. For example, the set of pixels may represent a vertical line in the environment. In some implementations, the system200determines the directional vector based on the orientation of the line. For example, if the line is vertical, the system200determines that the directional vector is parallel to the line. On the other hand, if the line is horizontal, the system200determines that the directional vector is orthogonal to the line.

As represented by block430, in various implementations, the method400includes identifying a subset of the planar surfaces that have a threshold orientation relative to the directional vector. Referring now toFIG.4C, in some implementations, as represented by block430a, the threshold orientation is parallel to the directional vector. For example, if the directional vector is a gravity vector, the system200may identify a subset of the planar surfaces that have a vertical orientation. As represented by block430b, in some implementations, the threshold orientation is orthogonal to the directional vector. For example, if the directional vector is a gravity vector, the system200may identify a subset of the planar surfaces that have a horizontal orientation. In some implementations, the system200identifies at least two subsets of planar surfaces, e.g., a first subset of planar surfaces having a horizontal orientation and a second subset of planar surfaces having a vertical orientation. In some implementations, the system200identifies another subset of planar surfaces having neither a horizontal orientation nor a vertical orientation, e.g., planar surfaces that are neither parallel to nor orthogonal to the directional vector.

As represented by block440, in various implementations, the method400includes determining, for each planar surface in the subset of planar surfaces, a normal vector for the planar surface based on the orientation of the planar surface and the directional vector. In some implementations, as represented by block440a, on a condition that a planar surface is horizontal, the system200determines that the normal vector is aligned with (e.g., parallel to) the directional vector (e.g., the normal vector is vertical). For example, the normal vector may originate from the planar surface and extend outward from an object associated with the planar surface in a direction parallel to the directional vector, e.g., in a vertical direction. In some implementations, as represented by block440b, on a condition that a planar surface is vertical, the system200determines that the normal vector is orthogonal to (e.g., perpendicular to) the directional vector (e.g., normal vector is horizontal). For example, the normal vector may originate from the planar surface and extend outward from an object associated with the planar surface in a direction orthogonal to the directional vector, e.g., in a horizontal direction. In some implementations, the system200identifies planar surfaces that are neither parallel to nor orthogonal to the directional vector. The system200may use another normal estimation technique to determine the normal vector for such planar surfaces.

Using a directional vector, such as a gravity vector, to determine a normal vector may reduce the number of planar surfaces for which potentially computationally expensive and/or error-prone normal estimation techniques are used to determine normal vectors. Accordingly, computational resources may be used for other tasks. Assigning a normal vector to a planar surface that has a threshold orientation relative to the directional vector reduces the need to perform a computationally-intensive operation (e.g., pixel-by-pixel analysis of the planar surface) to determine the normal vector. Reducing the need to perform computationally-intensive normal estimation operations tends to enhance an operability of a battery-operated device by extending a battery life of the battery-operated device. Reducing a number of computationally-intensive normal estimation operations tends to improve an operational efficiency of a device by reducing an amount of heat that the device generates.

FIG.5is a block diagram of a server system500enabled with one or more components of a device (e.g., the electronic device102and/or the controller104shown inFIG.1) in accordance with some implementations. While certain specific features are illustrated, those of ordinary skill in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity, and so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, as a non-limiting example, in some implementations the server system500includes one or more processing units (CPUs)501, a network interface502, a programming interface503, a memory504, and one or more communication buses505for interconnecting these and various other components.

In some implementations, the network interface502is provided to, among other uses, establish, and maintain a metadata tunnel between a cloud hosted network management system and at least one private network including one or more compliant devices. In some implementations, the one or more communication buses505include circuitry that interconnects and controls communications between system components. The memory504includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. The memory504optionally includes one or more storage devices remotely located from the one or more CPUs501. The memory504comprises a non-transitory computer readable storage medium.

In some implementations, the memory504or the non-transitory computer readable storage medium of the memory504stores the following programs, modules and data structures, or a subset thereof including an optional operating system506, the image obtainer310, the pixel analyzer320, the surface identifier330, the direction determiner340, and/or the normal information generator350. As described herein, the image obtainer310may include instructions310aand/or heuristics and metadata310bfor obtaining an image. As described herein, the pixel analyzer320may include instructions320aand/or heuristics and metadata320bfor determining correspondences between pixels in the obtained image and features in a physical environment, such as objects or spatial lines. As described herein, the surface identifier330may include instructions330aand/or heuristics and metadata330bfor identifying planar surfaces in the physical environment. As described herein, the direction determiner340may include instructions340aand/or heuristics and metadata340bfor determining a directional measurement, such as a gravity measurement or a gravity vector, for example, based on the obtained image or based on measurements from an IMU. As described herein, the normal information generator350may include instructions350aand/or heuristics and metadata350bfor generating (e.g., estimating) normal information, such as a normal vector to a planar surface, based on an orientation of the planar surface and the directional vector.

It will be appreciated thatFIG.5is intended as a functional description of the various features which may be present in a particular implementation as opposed to a structural schematic of the implementations described herein. As recognized by those of ordinary skill in the art, items shown separately could be combined and some items could be separated. For example, some functional blocks shown separately inFIG.5could be implemented as a single block, and the various functions of single functional blocks could be implemented by one or more functional blocks in various implementations. The actual number of blocks and the division of particular functions and how features are allocated among them will vary from one implementation to another and, in some implementations, depends in part on the particular combination of hardware, software, and/or firmware chosen for a particular implementation.