Methods and apparatuses for calculating building heights from mono imagery

A technique is directed to methods and systems for calculating building heights from mono imagery. In some implementations, a building height calculation system performs orthorectification of an image of buildings against a digital terrain model to remove effects of terrain distortion from the image. The building height calculation system can execute an edge detection algorithm on the image to identify the edges of the building in the image. The edges can provide a rooftop vector of the building. The building height calculation system can execute, using image data at input, a machine learning algorithm to determine the footprint vector of the building in the image. The building height is calculated based on a camera angle, a distance from the camera to the building, and a pixel offset from the footprint vector to the rooftop vector.

BACKGROUND

In some cases, to find out the height of a building an individual must access records (e.g., building plans, city documents, etc.) but this is not practical on a larger scale, such as measuring all buildings in a city. Building heights cannot currently be measured with mono two-dimensional (2D) aerial imagery or satellite imagery, due to the imagery not containing true three-dimensional (3D) information. Traditionally stereo pairs of 2D imagery have been used to measure building heights, but these stereo pairs are seldom available “off the shelf” and must often be tasked specifically for a job, which can lead to increased costs.

DETAILED DESCRIPTION

Aspects of the present disclosure are directed to methods and systems for calculating building heights from mono imagery. Light detection and ranging (LiDAR) scanning and stereo photogrammetry are used (e.g., in digital surface models (DSMs)) to measure building heights. LiDAR can be cost prohibitive and challenging to use in many regions. Stereo photogrammetry relies on having stereo pairs of images which are not always available and more expensive than mono imagery. Satellites, planes, and aerial vehicles can capture an image of a building, and a building height calculation system can determine the height of the building in the image. The disclosed method utilizes techniques to identify the footprint and rooftop of a building (e.g., rooftop vectors and footprint vectors) in an image, determine the horizontal offset separating the footprint and rooftop, and calculate the height of the building based on the horizontal offset and the angle the image was captured from. “Building” as used herein can refer to any structure or object in terrain. Note that while this description will refer to a single building, the singular implicates the plural and multiple buildings could be involved.

Several implementations are discussed below in more detail in reference to the figures.FIG.1is a flow diagram illustrating a process100used in some implementations for calculating building heights. In an embodiment, process100is triggered by a camera capturing an image (e.g., of buildings), a user inputting a command, or, when receiving images of terrain. The camera(s) capturing the images can be mounted on a satellite, plane, drone, or any aerial vehicle. Image300(e.g., a 2D satellite image) inFIG.3illustrates an aerial view of terrain with multiple buildings. In some implementations, digital elevation models (DEMs) represent the “bare earth” elevation, from which surface features such as trees and buildings have been removed. Either a digital surface model (DSM) or a digital terrain model (DTM) may be referred to as a DEM. Since no remote sensing mapping system provides direct measurements of the bare earth, a DTM is mostly produced by editing of the DSM. Process100can use an image and DTM as input data and process the data to determine the heights of the buildings in the image. In some embodiments, a DTM is a method to determine a ground elevation for the eventual determination of a building elevation.

At step102, process100performs orthorectification of the image against the DTM to remove effects of terrain distortion from the image. Orthorectification to a DTM is a process in earth imaging, in which the map (geographic or projected) coordinates of each pixel in an image of the earth are identified with the aid of a source DTM. This allows the formation of a new image. As a result of applying orthorectification, one or more pixels in the new image can be placed in its correct position(s) with respect to map coordinates.

At step104, process100can execute an edge detection algorithm on the image to identify the edges (e.g., contrast changes) of the building in the image.FIG.4is an example of a Sobel-Feldman operator, however any common edge detection algorithm can be used. The edges can provide a rooftop vector on the building. Process100can export the edges to a raster file. The edge detection algorithm can produce outlines of the rooftops of buildings. Image400ofFIG.4illustrates a raster edge mask created from image300.

At step106, process100calculates a building lean direction based on image meta data of the angle the camera was facing when the image was captured. At step108, process100executes a machine learning algorithm to determine the footprint and rooftop vectors of the building in the image. Process100can convert the image into an input for a machine learning model, apply the input to the machine learning model, and in response obtaining the rooftop vector and the footprint vector of the building based on output from the machine learning model. Additional details on machine leaning are described in FIG.10. At step110, process100receives the footprint vector and rooftop vector of the building from a third party (e.g., company, organization, documents, etc.). Example500ofFIG.5illustrates footprint vector502of building508, footprint vector504of building510, and footprint vector506of building512.

At step112, process100executes an algorithm with the building footprint vector and scans the edge raster in the direction along a search corridor (e.g., 2 or 3 pixels wide). Process100can identify a connection between the building footprint and the edge raster. For example, the connection is expected to occur at the rooftop of the building. Process100can search for the highest correlation between the rooftop and footprint vectors rather than an exact match, so inaccuracies (e.g., small inaccuracies, such as 95% accuracy) in the building footprint vectors do not affect the detection of the rooftop vector. Process100can perform a correlation calculation between the rooftop and footprint vectors. For example, process100uses the building footprint vector as a template and the edge raster as the comparison image. Process100calculates the mean of all the template pixels (MT), the mean of all the image pixels (MI), and loops through each template pixel and image pixel and calculates equation 1 as the correlation (where T and I are the template and image pixel values). Process100can perform the calculation at each pixel step along the search corridor and use the highest correlation for the roof position.
Sum((T−MT)*(I−MI))/Sqrt(Sum((T−TT)2)*Sum((I−TI)2))  (Equation 1)

Example600ofFIG.6illustrates footprint vector502and rooftop vector602of building508, footprint vector504and rooftop vector604of building510, and footprint vector506and rooftop vector606of building512. At step114, process100calculates the building height based on a camera angle, a distance from the camera to the building, and a pixel offset from the footprint vector to the rooftop vector. Process100can use a rational polynomial coefficients (RPC) algorithm to calculate the building height. For example, the geometric sensor model describing the physical relationship between image coordinates and ground coordinate is known as a Rigorous Projection Model. A Rigorous Projection Model expresses the mapping of the image space coordinates of rows and columns (r,c) onto the object space reference surface geodetic coordinates (φ, λ, h). RPCs support a generic description of the Rigorous Projection Models. The approximation used by RPCs is a set of rational polynomials expressing the normalized row and column values, (rn, cn), as a function of normalized geodetic latitude, longitude, and height, given a set of normalized polynomial coefficients. Below are a few steps to calculate building heights based on RPCs and the displacement observed between a building footprint and its corresponding building. Step 1: Backproject building footprint's centroid (X,Y,Z) in decimal degrees to input image space via RPCs→(r,c), in which “Z” is from NM6 DTM, averaged from all vertices, “r” is the row index, and “c” is the column index; Step 2: Backproject building rooftop's centroid (X′,Y′,Z′) to input image space via RPCs→(r′,c′), in which “Z” is from NM6 DTM, averaged from all vertices; Step 3: Backproject a series of points {X,Y,Z″} with Z″ starting from Z, increased by 1.0 m at a time, to input image space via RPCs→{r″ c″}, in which the calculation finds the closest (r″,c″) with respect to (r′,c′) and the corresponding Z″ is the estimated building rooftop height; and Step 4: calculate building height: BH=Z″−Z. Example700ofFIG.7illustrates the pixel offset702between the footprint vector502and rooftop vector602of building508, the pixel offset704between the footprint vector504and rooftop vector604of building510, and the pixel offset706between the footprint vector506and rooftop vector606of building512.

FIG.2is a flow diagram illustrating a process200used in some implementations for verifying calculated building heights. In an embodiment, process200is triggered by a camera capturing an image (e.g., buildings), a user inputting a command, receiving images of terrain, or when a building height verification is requested.

At step202, process200creates a shadow mask for the building using the footprint vector as a reference and the image. The shadow mask is created by setting a threshold value on the pixel intensity of the panchromatic image. If the pixel intensity is below the threshold, that pixel is flagged as a shadow. The shadow length is measured by counting the number of flagged shadow pixels in the opposite direction of the sun azimuth, starting from the building footprint.

At step204, process200measures a building height based on the camera angle, the distance from the camera to the building, and the angle of the sun (e.g., from latitude, longitude, and time of day). The relationship between shadow length and building height is as follows: BH=L×tan(θ) where BH is the building height, L is the shadow length to the building footprint, and θ is the sun elevation. Both the sun elevation and azimuth are provided in the image metadata.

At step206, process200compares the shadow building height (from step204) to the RPC calculated building height (from step114ofFIG.1). At step208, process200determines whether the comparison result is within a threshold (e.g., any threshold, such as 1%, 5%, or 10% difference in calculation) amount. For example, the threshold can be a selectable parameter based on the accuracy requirements of the job a user is working on When the comparison results are not within the allowed threshold, at step210, process200flags the building height calculation results for manual review by a user. When the comparison results are within the allowed threshold, at step212, process200approves the building height calculation results and sends a notification (e.g., message, email, alert, etc.) to a user.

FIG.8is a block diagram illustrating an overview of devices on which some implementations of the disclosed technology can operate. The devices can comprise hardware components of a device800that manage entitlements within a real-time telemetry system. Device800can include one or more input devices820that provide input to the processor(s)810(e.g. CPU(s), GPU(s), HPU(s), etc.), notifying it of actions. The actions can be mediated by a hardware controller that interprets the signals received from the input device and communicates the information to the processors810using a communication protocol. Input devices820include, for example, a mouse, a keyboard, a touchscreen, an infrared sensor, a touchpad, a wearable input device, a camera- or image-based input device, a microphone, or other user input devices.

In some implementations, the device800also includes a communication device capable of communicating wirelessly or wire-based with a network node. The communication device can communicate with another device or a server through a network using, for example, TCP/IP protocols. Device800can utilize the communication device to distribute operations across multiple network devices.

The processors810can have access to a memory850in a device or distributed across multiple devices. A memory includes one or more of various hardware devices for volatile and non-volatile storage, and can include both read-only and writable memory. For example, a memory can comprise random access memory (RAM), various caches, CPU registers, read-only memory (ROM), and writable non-volatile memory, such as flash memory, hard drives, floppy disks, CDs, DVDs, magnetic storage devices, tape drives, and so forth. A memory is not a propagating signal divorced from underlying hardware; a memory is thus non-transitory. Memory850can include program memory860that stores programs and software, such as an operating system862, building height calculation system864, and other application programs866. Memory850can also include data memory870, LIDAR data, structure data, image data, footprint data, rooftop data, device data, satellite data, machine learning data, vector data, shadow mask data, edge detection data, terrain data, camera data, retrieval data, management data, notification data, configuration data, settings, user options or preferences, etc., which can be provided to the program memory860or any element of the device800.

FIG.9is a block diagram illustrating an overview of an environment900in which some implementations of the disclosed technology can operate. Environment900can include one or more client computing devices905A-D, examples of which can include device800. Client computing devices905can operate in a networked environment using logical connections through network930to one or more remote computers, such as a server computing device910.

In some implementations, server910can be an edge server which receives client requests and coordinates fulfillment of those requests through other servers, such as servers920A-C. Server computing devices910and920can comprise computing systems, such as device800. Though each server computing device910and920is displayed logically as a single server, server computing devices can each be a distributed computing environment encompassing multiple computing devices located at the same or at geographically disparate physical locations. In some implementations, each server920corresponds to a group of servers.

Client computing devices905and server computing devices910and920can each act as a server or client to other server/client devices. Server910can connect to a database915. Servers920A-C can each connect to a corresponding database925A-C. As discussed above, each server920can correspond to a group of servers, and each of these servers can share a database or can have their own database. Databases915and925can warehouse (e.g. store) information such as implement data, LIDAR data, structure data, image data, footprint data, rooftop data, device data, satellite data, camera data, machine learning data, vector data, shadow mask data, edge detection data, and terrain data. Though databases915and925are displayed logically as single units, databases915and925can each be a distributed computing environment encompassing multiple computing devices, can be located within their corresponding server, or can be located at the same or at geographically disparate physical locations.

Network930can be a local area network (LAN) or a wide area network (WAN), but can also be other wired or wireless networks. Network930may be the Internet or some other public or private network. Client computing devices905can be connected to network930through a network interface, such as by wired or wireless communication. While the connections between server910and servers920are shown as separate connections, these connections can be any kind of local, wide area, wired, or wireless network, including network930or a separate public or private network.

FIG.10is a block diagram illustrating components1000which, in some implementations, can be used in a system employing the disclosed technology. The components1000include hardware1002, general software1020, and specialized components1040. As discussed above, a system implementing the disclosed technology can use various hardware including processing units1004(e.g. CPUs, GPUs, APUs, etc.), working memory1006, storage memory1008(local storage or as an interface to remote storage, such as storage915or925), and input and output devices1010. In various implementations, storage memory1008can be one or more of: local devices, interfaces to remote storage devices, or combinations thereof. For example, storage memory1008can be a set of one or more hard drives (e.g. a redundant array of independent disks (RAID)) accessible through a system bus or can be a cloud storage provider or other network storage accessible via one or more communications networks (e.g. a network accessible storage (NAS) device, such as storage915or storage provided through another server920). Components1000can be implemented in a client computing device such as client computing devices905or on a server computing device, such as server computing device910or920.

General software1020can include various applications including an operating system1022, local programs1024, and a basic input output system (BIOS)1026. Specialized components1040can be subcomponents of a general software application1020, such as local programs1024. Specialized components1040can include edge detection module1044, shadow mask module1046, height calculation module1048, machine learning module1050, and components which can be used for providing user interfaces, transferring data, and controlling the specialized components, such as interfaces1042. In some implementations, components1000can be in a computing system that is distributed across multiple computing devices or can be an interface to a server-based application executing one or more of specialized components1040. Although depicted as separate components, specialized components1040may be logical or other nonphysical differentiations of functions and/or may be submodules or code-blocks of one or more applications.

In some embodiments, the edge detection module1044is configured to execute an edge detection algorithm on the image to identify the edges (e.g., contrast changes) of the building in the image. In some cases, the edges can provide a rooftop vector (e.g., outline of the rooftop of the building) on the building. The edge detection module1044can export the edges to a raster file. In some embodiments, the shadow mask module1046is configured to create a shadow mask for buildings using the footprint as a reference and the satellite/aerial imagery. The shadow mask module1046measures the building heights based on the camera angle, distance from camera to building and the angle of the sun (e.g., using the latitude, longitude and time of day). In some embodiments, the height calculation module1048is configured to calculate the building height based on a camera angle, a distance from the camera to the one or more structures, and a pixel offset from the footprint vector to the rooftop vector.

In some embodiments, the machine learning module1050is configured to analyze the input data (e.g., image meta data) from the cameras and determine the rooftop vector and the footprint vector of the building. The machine learning module105may be configured to determine a footprint vector and/or a rooftop vector based on at least one machine-learning algorithm trained on at least one dataset of rooftop and/or footprint vectors. The at least one machine-learning algorithms (and models) may be stored locally at databases and/or externally at databases. Height calculation devices may be equipped to access these machine learning algorithms and intelligently determine rooftop or footprint vectors based on at least one machine-learning model that is trained on a dataset of building footprint vectors and rooftop vectors. As described herein, a machine-learning (ML) model may refer to a predictive or statistical utility or program that may be used to determine a probability distribution over one or more-character sequences, classes, objects, result sets or events, and/or to predict a response value from one or more predictors. A model may be based on, or incorporate, one or more rule sets, machine learning, a neural network, or the like. In examples, the ML models may be located on the client device, service device, a network appliance (e.g., a firewall, a router, etc.), or some combination thereof. The ML models may process building height databases and other data stores to determine a building footprint or rooftop vector.

Based on building height data and image data from building height databases and platforms and other user data stores, at least one ML model may be trained and subsequently deployed to automatically determine rooftop vectors and footprint vectors and calculate a building height. The trained ML model may be deployed to one or more devices. As a specific example, an instance of a trained ML model may be deployed to a server device and to a client device which communicate with a camera. The ML model deployed to a server device may be configured to be used by the client device when, for example, the client device is connected to the Internet. Conversely, the ML model deployed to a client device may be configured to be used by the client device when, for example, the client device is not connected to the Internet. In some instances, a client device may not be connected to the Internet but still configured to receive satellite signals with item information, such as specific image or building information. In such examples, the ML model may be locally cached by the client device.

Unless explicitly excluded, the use of the singular to describe a component, structure, or operation does not exclude the use of plural such components, structures, or operations. As used herein, the word “or” refers to any possible permutation of a set of items. For example, the phrase “A, B, or C” refers to at least one of A, B, C, or any combination thereof, such as any of: A; B; C; A and B; A and C; B and C; A, B, and C; or multiple of any item such as A and A; B, B, and C; A, A, B, C, and C; etc.

As used herein, the expression “at least one of A, B, and C” is intended to cover all permutations of A, B and C. For example, that expression covers the presentation of at least one A, the presentation of at least one B, the presentation of at least one C, the presentation of at least one A and at least one B, the presentation of at least one A and at least one C, the presentation of at least one B and at least one C, and the presentation of at least one A and at least one B and at least one C.