Patent Description:
One challenge of high-elevation digital imagery is that <NUM>-<NUM>% of such images tend to be covered by clouds, shadows, haze and/or snow (for simplicity, these will all be referred to herein as "transient obstructions"), depending on the location and time. Transient obstructions such as clouds make high-elevation digital images less useful, reduce their business and scientific value, and/or decrease the user experience with applications that rely on high-elevation digital images. For example, clouds introduce gaps in observations made during agricultural monitoring, potentially leaving key information such as germination or harvest dates unavailable for use.

<CIT> describes systems and methods for image processing including a memory having stored multi-angled view images of a scene. Each multi-angled view image includes pixels, and at least one multi-angled view image includes a clouded area in the scene, resulting in missing pixels. A processor configured to align three multi-angled view images in the multi-angled view images to a target view angle of the scene, to form a set of aligned multi-angled view images representing a target point of view of the scene, at least one aligned multi-angled view image of the at least three multi-angled view images, has missing pixels due to the clouded area. Form a matrix using vectorized aligned multi-angled view images, wherein the matrix is incomplete due to the missing pixels. Complete the matrix using a matrix completion to combine the aligned multi-angled view images to produce a fused image of the scene without the clouded area.

The present disclosure is generally directed to methods, apparatus, and computer-readable media (transitory and non-transitory) for detecting and removing noise caused by transient obstructions such as clouds from high-elevation digital images of geographic areas, and for predicting/estimating ground features and/or terrain obstructed by transient obstructions. In various implementations, high-elevation digital image pixel data acquired across multiple domains, such as the temporal domain (i.e., multiple digital images captured over time), the spatial domain, and/or the spectral domain (e.g., RGB, infrared, etc.), may be used to predict or estimate data that can be used to replace pixels that depict transient obstructions such as clouds.

In some implementations, pixels of high-elevation digital images may correspond or align spatially with atomic geographic units. For example, if a vision sensor used to capture high-elevation digital images has a resolution of N meters squared per pixel, then each geographic unit may be defined as an area of N meters squared. In other implementations in which fidelity is less critical, multiple pixels may be used to define an atomic geographic unit. Thus, in multiple high-elevation digital images captured of a geographic area over time, a given pixel may align spatially with a given geographic unit in each image.

In some implementations, patterns or "fingerprints" within individual domains and/or across multiple domains may be established for individual geographic units. These fingerprints may be used to identify other geographic units that would most likely have similar appearances in a digital image, assuming those other geographic units are not obscured by clouds. As an example, suppose a particular geographic unit is obscured by clouds in a given high-elevation digital image. An unobscured geographic unit in the same digital image or another related digital image (e.g., taken nearby, taken within the same time interval, taken of the same type of crop, etc.) that matches one or more domain fingerprints of the particular geographic unit may be used to predict/estimate data that is usable to replace the obscured pixel.

As used herein, a domain fingerprint of one geographic unit "matches" a domain fingerprint of another geographic unit when, for example, a similarity measure between the two fingerprints satisfies some threshold, and/or when a Euclidian distance between latent embeddings of the fingerprints satisfies some criterion. Thus, it should be understood that the term "match" as used herein with regard to domain fingerprints is not limited to exact equality. As another example, two domain fingerprints may match when, for instance, they both can be fitted statistically with the same curve, or when they are used to generate curves that fit each other statistically.

High-elevation digital images are often taken of a geographic area over time. For example, many satellites capture multiple temporally-distinct digital images of the same underlying geographic area as they repeatedly travel along their orbital trajectories. Due to the transient nature of clouds and other transient obstructions, each digital image of the particular geographic region may have different areas that are obscured or unobscured by natural obstructions such as clouds. Some implementations described herein leverage these multi-temporal images to predict values of obscured pixels in individual digital images.

For example, in some implementations, a three-dimensional ("3D") array structure may be assembled in memory for a geographic area based on multiple digital images captured of the geographic area. Each row of the 3D array may represent a particular pixel (and spatially corresponding geographic unit). Each column of the array may correspond to, for instance, a different digital image captured at a different time (e.g., during each orbit of a satellite). And in some implementations, each unit in the third dimension of the 3D array-which can be referred to alternatively as "layers," "pages," and/or "aisles"-may correspond to a different spectral frequency, such as red, green, blue, near infrared ("IR"), mid-IR, far-IR, thermal IR, and/or radar.

Once this 3D array is assembled for a particular geographic area, it can be used to remove transient obstructions such as clouds from individual digital images. For example, in some implementations, clouds or other noise may leave "gaps" in one or more "cells" of the 3D array. The 3D array can be applied as input to one or more statistical models so that the gaps can be filled with replacement data, e.g., from other cells of the 3D array. In some such implementations, the 3D array structure may continue to grow as more digital images are captured over time. Consequently, the ability to fill in the gaps and replace clouds with predicted terrain data may be enhanced over time.

Various approaches can be applied to determine whether cells of the 3D array structure represent transient obstructions (or other noise). In some implementations, existing/known masks associated with clouds, haze, snow, shadows, etc. may be packaged with high-elevation digital images. Additionally or alternatively, in some implementations, an artificial intelligence model may be trained to detect clouds--in some cases it may be tuned to be high recall so that, for instance, it is unlikely to miss potential clouds. Additionally or alternatively, in some implementations, one or more models (e.g., two models) may be trained to perform both cloud detection and removal. For example, a model may be trained to, based on an input high-elevation digital image, predict a cloud mask and remove the clouds.

In some implementations, recurrent neural networks or other memory networks (e.g., long short-term memory, or "LSTM") that are able to account for multi-temporal input may be used to fill in the gaps in the 3D array structure. For example, in some implementations, each spatio-spectral "slice" of the 3D array structure (i.e., data extracted from each digital image of multiple digital images captured over time) may be applied as input across a recurrent neural network to generate output. This output may be combined (e.g., concatenated) with a "next" slice of the 3D array structure and applied as input across the same recurrent neural network to generate additional output. This may continue across a whole temporal sequence of digital images captured of a geographic area. At each turn, the output may "predict" what the next slice will look like. When the next slice in actuality includes transient obstruction(s) such as clouds, the predicted output can be used to generate replacement data for the pixels that portray the transient obstruction(s).

In some implementations, the domain fingerprints described previously may be used to classify individual geographic units into particular terrain classifications. These terrain classifications may include, for instance, roadways, buildings, water, vegetation, etc. In some implementations, e.g., in which disclosed techniques are used for agricultural monitoring, terrain classifications may include ground features such as different types of crops (e.g., "corn," "soybeans," etc.), and may be as granular as desired. For example, in some implementations, the Cropland Data Layer ("CDL") released by the United States Department of Agriculture ("USDA") may be used to establish terrain classifications associated with different crop types. In some such implementations, geographic units may be classified into terrain classifications using a trained machine learning model, such as various flavors of artificial neural networks (e.g., convolutional, recurrent, etc.). In some implementations, two or more geographic units may "match" if they share a particular terrain classification.

A variety of different machine learning model types may be trained and used for a variety of different purposes in the present disclosure. In some implementations, one or more generative adversarial networks ("GANs") may be used to facilitate unsupervised machine learning for various aspects of the present disclosure. For example, in some implementations, synthetic transient obstructions such as clouds (and shadows they cast on the ground) may be added to otherwise obstruction-free ground truth high-elevation digital images. These "synthetic images may then be used, e.g., along with the original unaltered high-elevation images, as training data for a GAN that includes a generator model and a discriminator model. The generator model may be used to generate synthetic images (i.e., with synthetic transient obstructions), which are then applied as input across the discriminator model to generate output comprising a best "guess" as to whether the input digital image(s) are "real" or "synthetic. " The input for the discriminator model may be labeled as "real" or "synthetic," so that these labels may be compared to its output to determine error. This error may then be used to train both the discriminator model and the generator model, so that over time the generator model generates synthetic images that are more likely to "fool" the discriminator model, while the discriminator model improves at accurately guessing whether an image is "real" or "synthetic.

Similarly, transient obstructions such as clouds may be removed/replaced in high-elevation digital imagery using another GAN that also includes a generator model and a discriminator model. High-elevation digital image(s) with transient obstruction(s) may be applied as input across the generator model to generate output in the form of synthetic, obstruction-free digital image(s). The synthetic, obstruction-free digital image(s) may then be applied as input across the discriminator model, along with obstruction-free ground truth high-elevation digital images, to generate output comprising a best "guess" as to whether the digital image(s) are "real" or "synthetic. " As described previously, the inputs for the discriminator model may be labeled as "real" or "synthetic," and these labels may be compared to its output to determine error. This error may then be used to train both the discriminator model and the generator model, so that over time the generator model generates synthetic, obstruction-free images that are more likely to "fool" the discriminator model, while the discriminator model improves at accurately guessing whether an image is "real" or "synthetic.

High elevation digital image may be captured in a variety of ways. As noted previously, in various implementations, one or more satellites may traverse an orbit and continuously/periodically capture high elevation images of the same sequence of geographic areas. Additionally or alternatively, in some implementations, high elevation digital images may be captured by unmanned aerial vehicles, such as remote controlled drones. Additionally or alternatively, in some implementations, high elevation digital images may be captured using other mechanisms, including but not limited to manned aircraft, balloons, spacecraft, and so forth.

In some implementations, a computer implemented method may be provided that includes the following operations: obtaining a digital image of a geographic area captured from an elevated vantage point, wherein the digital image comprises a plurality of pixels that align spatially with a respective plurality of geographic units of the geographic area; identifying one or more obscured pixels of the digital image that align spatially with one or more obscured geographic units of the geographic area that are obscured in the digital image by one or more transient obstructions; determining, across pixels of a corpus of digital images that align spatially with the one or more obscured geographic units, one or more spectral-temporal data fingerprints of the one or more obscured geographic units; identifying one or more unobscured pixels of the same digital image or a different digital image that align spatially with one or more unobscured geographic units that are unobscured by transient obstructions, wherein the unobscured geographic units have one or more spectral-temporal data fingerprints that match the one or more spectral-temporal data fingerprints of the one or more obscured geographic units; calculating replacement pixel data based on the one or more unobscured pixels; and generating a transient-obstruction-free version of the digital image in which data associated with the one or more obscured pixels is replaced with the replacement pixel data.

This method and other implementations of technology disclosed herein may each optionally include one or more of the following features.

In various implementations, the digital image may be captured by a satellite, and the elevated vantage point lies on a trajectory of the satellite. In various implementations, the digital image may be captured by an unmanned aerial drone or manned aircraft, and the elevated vantage point may lie on a trajectory of the unmanned aerial drone or manned aircraft. In various implementations, the one or more spectral-temporal data fingerprints of the one or more obscured geographic units may include infrared, panchromatic, RGB, etc..

In various implementations, the method may further include applying the one or more spectral-temporal data fingerprints of the one or more obscured geographic units across a trained machine learning model to determine a terrain classification of the one or more obscured geographic units. In some such implementations, identifying one or more unobscured pixels may include determining that the terrain classification of the one or more obscured geographic units matches a terrain classification of the one or more unobscured geographic units.

In various implementations, the operations of determining, identifying, and calculating collectively may include sequentially applying data indicative of each digital image of the corpus of digital images as input across a recurrent neural network. In various implementations, the calculating may be based at least in part on application of a generator model, and wherein the generator model is part of a generative adversarial network that also includes a discriminator model that is trained in conjunction with the generator model. In various implementations, the generator model and the discriminator model may be trained using at least one training example that takes the form of a high elevation digital image with synthetic transient obstructions added, wherein the synthetic transient obstructions are added by a transient obstruction generation model that is trained as part of another generative adversarial network.

Other implementations may include a non-transitory computer readable storage medium storing instructions executable by a processor to perform a method such as one or more of the methods described above. Yet another implementation may include a system including memory and one or more processors operable to execute instructions, stored in the memory, to implement one or more modules or engines that, alone or collectively, perform a method such as one or more of the methods described above.

<FIG> illustrates an environment in which one or more selected aspects of the present disclosure may be implemented, in accordance with various implementations. The example environment includes a plurality of client devices <NUM><NUM>-N and a high elevation image processing system <NUM>. High elevation image processing system <NUM> may be implemented in one or more computers that communicate, for example, through a network. High elevation image processing system <NUM> is an example of an information retrieval system in which the systems, components, and techniques described herein may be implemented and/or with which systems, components, and techniques described herein may interface.

A user may interact with high elevation image processing system <NUM> via a client device <NUM>. Each client device <NUM> may be a computer coupled to the high elevation image processing system <NUM> through one or more networks <NUM> such as a local area network (LAN) or wide area network (WAN) such as the Internet. Each client device <NUM> may be, for example, a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device of a vehicle of the participant (e.g., an in-vehicle communications system, an in-vehicle entertainment system, an in-vehicle navigation system), a standalone interactive speaker (with or without a display), or a wearable apparatus of the participant that includes a computing device (e.g., a watch of the participant having a computing device, glasses of the participant having a computing device). Additional and/or alternative client devices may be provided.

Each of client device <NUM> and high elevation image processing system <NUM> may include one or more memories for storage of data and software applications, one or more processors for accessing data and executing applications, and other components that facilitate communication over a network. The operations performed by client device <NUM> and/or high elevation image processing system <NUM> may be distributed across multiple computer systems. High elevation image processing system <NUM> may be implemented as, for example, computer programs running on one or more computers in one or more locations that are coupled to each other through a network.

Each client device <NUM> may operate a variety of different applications that may be used, for instance, to view high-elevation digital images that are processed using techniques described herein to remove transient obstructions such as clouds, shadows (e.g., cast by clouds), snow, manmade items (e.g., tarps draped over crops), etc. For example, a first client device <NUM><NUM> operates an image viewing client <NUM> (e.g., which may be standalone or part of another application, such as part of a web browser). Another client device <NUM>N may operate a crop prediction application <NUM> that utilizes high-elevation digital images processed using techniques described herein to make various agricultural predictions and/or recommendations.

In various implementations, high elevation image processing system <NUM> may include a transient obstruction detection engine <NUM>, a terrain classification engine <NUM>, an obstruction replacement engine <NUM>, and a transient obstruction generation engine <NUM>. In some implementations one or more of engines <NUM>, <NUM>, <NUM>, and/or <NUM> may be omitted. In some implementations all or aspects of one or more of engines <NUM>, <NUM>, <NUM>, and/or <NUM> may be combined. In some implementations, one or more of engines <NUM>, <NUM>, <NUM>, and/or <NUM> may be implemented in a component that is separate from high elevation image processing system <NUM>. In some implementations, one or more of engines <NUM>, <NUM>, <NUM>, and/or <NUM>, or any operative portion thereof, may be implemented in a component that is executed by client device <NUM>.

Transient obstruction detection engine <NUM> may be configured to detect, in high-elevation digital images, transient obstructions such as clouds, shadows cast by clouds, rain, haze, snow, flooding, and/or manmade obstructions such as tarps, etc. Transient obstruction detection engine <NUM> may employ a variety of different techniques to detect transient obstructions. For example, to detect clouds (e.g., create a cloud mask), transient obstruction detection engine <NUM> may use spectral and/or spatial techniques. In some implementations, one or more machine learning models may be trained and stored, e.g., in index <NUM>, and used to identify transient obstructions. For example, in some implementations, one or more deep convolutional neural networks known as "U-nets" may be employed. U-nets are trained to segment images in various ways, and in the context of the present disclosure may be used to segment high elevation images into segments that include transient obstructions such as clouds. Additionally or alternatively, in various implementations, other known spectral and/or spatial cloud detection techniques may be employed, including techniques that either use, or don't use, thermal infrared spectral bands.

In some implementations, terrain classification engine <NUM> may be configured to classify individual pixels, or individual geographic units that correspond spatially with the individual pixels, into one or more "terrain classifications. " Terrain classifications may be used to label pixels by what they depict. Non-limiting examples of terrain classifications include but are not limited to "buildings," "roads," "water," "forest," "crops," "vegetation," "sand," "ice," "mountain," "tilled soil," and so forth. Terrain classifications may be as coarse or granular as desired for a particular application. For example, for agricultural monitoring it may be desirable to have numerous different terrain classifications for different types of crops. For city planning it may be desirable to have numerous different terrain classifications for different types of buildings, roofs, streets, parking lots, parks, etc..

Terrain classification engine <NUM> may employ a variety of different known techniques to classify individual geographic units into various terrain classifications. Some techniques may utilize supervised or unsupervised machine learning that includes trained machine learning models stored, for instance, in index <NUM>. These techniques may include but are not limited to application of multivariate statistics to local relief gradients, fuzzy k-means, morphometric parameterization and artificial neural networks, and so forth. Other techniques may not utilize machine learning.

In some implementations, terrain classification engine <NUM> may classify individual geographic units with terrain classifications based on trajectories or "fingerprints" of various domain values over time. For example, in some implementations, terrain classification engine <NUM> may determine, across pixels of a corpus of digital images captured over time, spectral-temporal data fingerprints of the individual geographic units corresponding to each individual pixel. Each fingerprint may include, for instance, a sequence of values within a particular spectral domain across a temporal sequence of digital images (e.g., a feature vector of spectral values).

As an example, suppose a particular geographic unit includes at least a portion of a deciduous tree. In a temporal sequence of satellite images of the geographic area that depict this tree, the pixel(s) associated with the particular geographic unit in the visible spectrum (e.g., RGB) will sequentially have different values as time progresses, with spring and summertime values being more green, autumn values possibly being orange or yellow, and winter values being gray, brown, etc. Other geographic units that also include similar deciduous trees may also exhibit similar domain trajectories, or fingerprints. Accordingly, in various implementations, the particular geographic unit and/or other similar geographic units may be classified, e.g., by terrain classification engine <NUM>, as having a terrain classification such as "deciduous," "vegetation," etc., based on their matching spectral-temporal data fingerprints.

Obstruction replacement engine <NUM> may be configured to generate obstruction-free versions of digital images in which those pixels that depict clouds, snow, or other transient obstructions are replaced with replacement data that estimates/predicts the actual terrain that underlies these pixels. Obstruction replacement engine <NUM> may use a variety of different techniques to generate transient-obstruction-free versions of digital images.

For example, in some implementations, obstruction replacement engine <NUM> may be configured to determine, e.g., based on output provided by transient obstruction detection engine <NUM>, one or more obscured pixels of a high-elevation digital image that align spatially with one or more obscured geographic units of the geographic area that are obscured in the digital image by one or more transient obstructions. Obstruction replacement engine <NUM> may then determine, e.g., across pixels of a corpus of digital images that align spatially with the one or more obscured geographic units, one or more spectral-temporal data fingerprints of the one or more obscured geographic units. For example, in some implementations, terrain classification engine <NUM> may classify two or more geographic units having matching spectral-temporal fingerprints into the same terrain classification.

Obstruction replacement engine <NUM> may then identify one or more unobscured pixels of the same high-elevation digital image, or of a different high elevation digital image that align spatially with one or more unobscured geographic units that are unobscured by transient obstructions. In various implementations, the unobscured geographic units may be identified because they have spectral-temporal data fingerprints that match the one or more spectral-temporal data fingerprints of the one or more obscured geographic units. For example, obstruction replacement engine <NUM> may seek out other pixels of the same digital image or another digital image that correspond to geographic units having the same (or sufficiently similar) terrain classifications.

In various implementations, obstruction replacement engine <NUM> may calculate or "harvest" replacement pixel data based on the one or more unobscured pixels. For example, obstruction replacement engine may take an average of all values of the one or more unobscured pixels in a particular spectrum and use that value in the obscured pixel. By performing similar operations on each obscured pixel in the high-elevation digital, obstruction replacement engine <NUM> may be able to generate a transient-obstruction-free version of the digital image in which data associated with obscured pixels is replaced with replacement pixel data calculated based on other, unobscured pixels that depict similar terrain (e.g., same terrain classification, matching spectral-temporal fingerprints, etc.).

In some implementations, obstruction replacement engine <NUM> may employ one or more trained machine learning models that are stored in one or more indexes <NUM> to generate obstruction-free versions of digital images. A variety of different types of machine learning models may be employed. For example, in some implementations, collaborative filtering and/or matrix factorization may be employed, e.g., to replace pixels depicting transient obstructions with pixel data generated from other similar-yet-unobscured pixels, similar to what was described previously. In some implementations, matrix factorization techniques such as the following equation may be employed: <MAT> wherein r represents the value of a pixel in a particular band if it were not covered by clouds, µ represents global average value in the same band, b represents the systematic bias, i and u represent the pixel's id and timestamp, T represents matrix transpose, and q and p represent the low-dimension semantic vectors (or sometimes called "embeddings"). In some implementations, temporal dynamics may be employed, e.g., using an equation such as the following: <MAT> wherein t represents a non-zero integer corresponding to a unit of time. Additionally or alternatively, in some implementations, generative adversarial networks, or "GANs," may be employed, e.g., by obstruction replacement engine <NUM>, in order to train one or more models stored in index <NUM>. A more detailed description of how GANs may be used in this manner is provided with regard to <FIG>.

In some implementations, a transient obstruction generation engine <NUM> may be provided that is configured to generate synthetic obstructions such as clouds, snow, etc. that may be incorporated into digital images (e.g., used to augment, alter, and/or replace pixel values in one or more spectrums) for a variety of different purposes. In some implementations, digital images with baked-in synthetic transient obstructions may be used as training data to train one or more machine learning models used by other components of high elevation image processing system <NUM>.

For example, in some implementations, a machine learning model employed by obstruction replacement engine <NUM> and stored in index <NUM> may be trained as follows. An obstruction-free (e.g., cloudless) high-elevation digital image of a geographic area may be retrieved. Based on the obstruction-free digital image, transient obstruction generation engine <NUM> may generate, e.g., using one or trained more machine learning models described below, a training example that includes the obstruction-free image with baked in synthetic transient obstructions such as clouds. This training example may be applied, e.g., by obstruction replacement engine <NUM>, as input across one or more machine learning models stored in index <NUM> to generate output. The output may be compared to the original obstruction-free digital image to determine a difference or error. This error may be used to perform operations such as back propagation and/or gradient descent to train the machine learning model to remove transient obstructions such as clouds and replace them with predicted terrain data.

As another example, in some implementations, a machine learning model employed by transient obstruction detection engine <NUM> and stored in index <NUM> may be trained as follows. An obstruction-free (e.g., cloudless) high-elevation digital image of a geographic area may be retrieved. Based on the obstruction-free digital image, transient obstruction generation engine <NUM> may generate, e.g., using one or trained more machine learning models described below, a training example that includes the obstruction-free image with baked-in synthetic transient obstructions such as clouds. The location of the synthetic transient obstruction will be known because it is synthetic, and thus is available, e.g., from transient obstruction generation engine <NUM>. Accordingly, in various implementations, the training example may be labeled with the known location(s) (e.g., pixels) of the synthetic transient obstruction. The training example may then be applied, e.g., by transient obstruction detection engine <NUM>, as input across one or more machine learning models stored in index <NUM> to generate output indicative of, for instance, a cloud mask. The output may be compared to the known synthetic transient obstruction location(s) to determine a difference or error. This error may be used to perform operations such as back propagation and/or gradient descent to train the machine learning model to generate more accurate cloud masks.

Transient obstruction generation engine <NUM> may use a variety of different techniques to generate synthetic transient obstructions such as clouds. For example, in various implementations, transient obstruction generation engine <NUM> may use particle systems, voxel models, procedural solid noise techniques, density models (e.g., low albedo, single scattering approximation for illumination in a uniform medium), ray trace volume data, textured ellipsoids, isotropic single scattering approximation, Perlin noise with alpha blending, and so forth. In some implementations, transient obstruction generation engine <NUM> may use GANs to generate synthetic clouds, or at least to improve generation of synthetic clouds. More details about such an implementation are provided with regard to <FIG>. Transient obstruction generation engine <NUM> may be configured to add synthetic transient obstructions to one or more multiple different spectral bands of a high-elevation digital image. For example, in some implementations transient obstruction generation engine <NUM> may add clouds not only to RGB spectral band(s), but also to NIR spectral band(s).

In this specification, the term "database" and "index" will be used broadly to refer to any collection of data. The data of the database and/or the index does not need to be structured in any particular way and it can be stored on storage devices in one or more geographic locations. Thus, for example, the indices <NUM>, <NUM>, <NUM>, and <NUM> may include multiple collections of data, each of which may be organized and accessed differently.

<FIG> depicts an example of how a ground truth high-elevation digital image (top) may be processed to classify the constituent geographic units that correspond to its pixels. In the top image, which schematically represents a high elevation digital image capturing a geographic area, a T-shaped road is visible that divides two plots of land at bottom left and bottom right. The bottom left plot of land includes a cluster of vegetation, and so does the bottom right plot. The bottom right plot also features a building represented by the rectangle with cross hatching.

The middle image demonstrates how the digital image at top may be classified, e.g., by terrain classification engine <NUM>, into discrete terrain classifications, e.g., based on geographic units that share spectral-temporal fingerprints. The middle image is subdivided into squares that each represent a pixel that aligns spatially with a geographic unit of the top digital image. Pixels that depict roadway have been classified accordingly and are shown in a first shading. Pixels that depict the building have also been classified accordingly and are shown in black. Pixels that represent the vegetation in the bottom left and bottom right plots of land are also classified accordingly in a second shading that is slightly darker than the first shading.

The bottom image demonstrates how techniques described herein, particularly those relating to terrain classification and/or spectral-temporal fingerprint similarity, may be employed to generate replacement data that predicts/estimates terrain underlying a transient obstruction in a high elevation digital image. In the bottom images of <FIG>, a cloud has been depicted schematically primarily over the bottom left plot of land. As indicated by the arrows, two of the vegetation pixels (five columns from the left, three and four rows from bottom, respectively) that are obscured by the cloud can be replaced with data harvested from other, unobscured pixels. For example, data associated with the obscured pixel five columns from the left and three rows from bottom is replaced with replacement data that is generated from two other unobscured pixels: the pixel four columns from left and four rows from top, and the pixel in the bottom right plot of land that is five rows from bottom, seven columns from the right. Data associated with the obscured pixel five columns from the left and four rows from bottom is replaced with replacement data that is generated from two other unobscured pixels: the pixel five columns from left and three rows from top, and the pixel in the bottom right plot of land that is five rows from top and nine columns from the right.

Of course these are just examples. More or less unobscured pixels may be used to generate replacement data for obscured pixels. Moreover, it is not necessary that the unobscured pixels that are harvested for replacement data be in the same digital image as the obscured pixels. It is often (but not always) the case that the unobscured pixels may be contained in another high elevation digital image that is captured nearby, for instance, with some predetermined distance (e.g., within <NUM> kilometers). Or, if geographic units that are far away from each other nonetheless have domain fingerprints that are sufficiently similar, those faraway geographic units may be used to harvest replacement data.

<FIG> depicts an example of how GANs may be used to train a generator model <NUM> employed by obstruction replacement engine <NUM>, in accordance with various implementations. In various implementations, obstruction replacement engine <NUM> may retrieve one or more high elevation digital images <NUM> and apply them as input across generator model <NUM>. Generator model <NUM> may take various forms, such as an artificial neural network. In some implementations, generator model <NUM> may take the form of a convolutional neural network.

Generator model <NUM> may generate output in the form of synthetically cloud-free (or more generally, transient obstruction-free) images. These images may then be applied as input across a discriminator model <NUM>. Discriminator model <NUM> typically will take the same form as generator model <NUM>, and thus can take the form of, for instance, a convolutional neural network. In some implementations, discriminator model <NUM> may generate binary output that comprises a "best guess" of whether the input was "synthetic" or "natural" (i.e., ground truth). At the same time, one or more natural, cloud-free (or more generally, transient obstruction-free) images (i.e., ground truth images) may also be applied as input across discriminator model <NUM> to generate similar output. Thus, discriminator model <NUM> is configured to analyze input images and make a best "guess" as to whether the input image contains synthetic data (e.g., synthetically-added clouds) or represents authentic ground truth data.

In various implementations, discriminator model <NUM> and generator model <NUM> may be trained in tandem, e.g., in an unsupervised manner. Output from discriminator model <NUM> may be compared to a truth about the input image (e.g., a label that indicates whether the input image was synthesized by generator <NUM> or is ground truth data). Any difference between the label and the output of discriminator model <NUM> may be used to perform various training techniques across both discriminator model <NUM> and generator model <NUM>, such as back propagation and/or gradient descent, to train the models.

In other implementations, one or more recurrent neural networks or other memory networks (e.g., long short-term memory, or "LSTM") that are able to account for multi-temporal input may be used, e.g., by obstruction replacement engine <NUM>, to generate replacement data that "fills in the gaps" as described in the summary. For example, in some implementations, each spatio-spectral "slice" of the 3D array structure described elsewhere herein (i.e., data extracted from each digital image of multiple digital images captured over time) may be applied as input across a recurrent neural network to generate output. This output may be combined (e.g., concatenated) with a "next" slice of the 3D array structure and applied, e.g., by obstruction replacement engine <NUM>, as input across the same recurrent neural network to generate additional output. This may continue across a whole temporal sequence of digital images captured of a geographic area. At each turn, the output may "predict" what the next slice will look like. When the next slice in actuality includes transient obstruction(s) such as clouds, the predicted output can be used to generate replacement data for the obscured pixels.

<FIG> schematically depicts an example of how GANs may be used to train one or more machine learning models employed by transient obstruction generation engine <NUM>, in accordance with various implementations. Similar to <FIG>, transient obstruction generation engine <NUM> may utilize a generator model <NUM> and a discriminator model <NUM>, which may or may not take similar forms as models <NUM>-<NUM>. In this example, transient obstruction generation engine <NUM> may retrieve one or more obstruction-free ground truth high-elevation digital images <NUM> and apply them as input across generator model <NUM> to generate synthetic images that include baked-in synthetic obstructions such as clouds. These synthetic images may then be applied as input across discriminator model <NUM>, along with natural, ground truth images that also include obstructions. Similar to before, discriminator model <NUM> may be configured to generate output that constitutes a "guess" as to whether an input digital image is "synthetic" (e.g., generated by generator model <NUM>) or "natural. " These models <NUM>-<NUM> may be trained in a manner similar to that described above with regard to models <NUM>-<NUM>.

Referring now to <FIG>, one example method <NUM> of performing selected aspects of the present disclosure is described. For convenience, the operations of the flow chart are described with reference to a system that performs the operations. This system may include various components of various computer systems, including various engines described herein. Moreover, while operations of method <NUM> are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted or added.

At block <NUM>, the system may obtain a digital image of a geographic area captured from an elevated vantage point. In various implementations, the digital image may include a plurality of pixels that align spatially with a respective plurality of geographic units of the geographic area.

At block <NUM>, the system, e.g., by way of transient obstruction detection engine <NUM>, may identify one or more obscured pixels of the digital image that align spatially with one or more obscured geographic units of the geographic area that are obscured in the digital image by one or more transient obstructions. Put another way, pixels that depict a portion of a cloud or other transient obstruction are identified, e.g., by transient obstruction detection engine <NUM> using one or more techniques described previously.

At block <NUM>, the system, e.g., by way of terrain classification engine <NUM>, may determine, across pixels of a corpus of digital images that align spatially with the one or more obscured geographic units, one or more spectral-temporal data fingerprints of the one or more obscured geographic units. For example, in some implementations, a ("3D") array structure may have been assembled previously for the geographic area, e.g., based on multiple digital images captured of the geographic area. Each row of the 3D array may represent a particular pixel (and spatially corresponding geographic unit). Each column of the array may correspond to, for instance, a different digital image captured at a different time. Each unit in the third dimension of the 3D array may correspond to different spectral frequencies that are available in the digital images, such as red, green, blue, near infrared ("IR"), mid-IR, far-IR, thermal IR, and/or radar. In various implementations, this 3D array structure may be used at block <NUM> to determine domain fingerprints, such as spectral-temporal fingerprints, of individual geographic units.

At block <NUM>, the system, e.g., by way of obstruction replacement engine <NUM>, may identify one or more unobscured pixels of the same digital image or a different digital image that align spatially with one or more unobscured geographic units of the same or different geographic area that are unobscured by transient obstructions. In various implementations, the unobscured geographic units may have one or more spectral-temporal data fingerprints that match the one or more spectral-temporal data fingerprints of the one or more obscured geographic units that were determined at block <NUM>.

At block <NUM>, the system may calculate replacement pixel data based on the one or more unobscured pixels. For example, an average of values across the unobscured pixels within a particular spectrum, or across multiple spectra, may be used. Additionally or alternatively, in some implementations, a single pixel that is "closest" (e.g., has a most similar domain fingerprint) to the unobscured pixel may simply be cloned into the obscured pixel. At block <NUM>, the system may generate a transient-obstruction-free version of the digital image in which data associated with the one or more obscured pixels is replaced with the replacement pixel data.

<FIG> depicts an example of how techniques described herein may be used to generate a transient-obstruction-free version of a high-elevation digital image. On the left, a digital image captured from a high elevation (e.g., a satellite) depicts a geographic area. It also can be seen that a number of clouds are positioned between the ground surface and the vantage point of the satellite, and therefore obstruct portions of the geographic area from view. In addition it can be seen the shadows cast by the clouds also obstruct additional portions of the geographic area.

In the middle image a cloud mask has been detected, e.g., by transient obstruction detection engine <NUM>. The cloud mask has been used to remove obscured pixels that correspond to the clouds or their respective shadows. Put another way, the obscured pixels that align spatially with the geographic units that are obscured by the clouds or their respective shadows have been removed (e.g., values set to black, zeroed out, etc.). In the right image, the removed pixels have been replaced with replacement data generated using techniques described herein. As explained herein, this replacement data estimates the terrain underlying the obscured pixels.

Storage subsystem <NUM> stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem <NUM> may include the logic to perform selected aspects of method <NUM>, and/or to implement one or more components depicted in prior figures.

Claim 1:
A method implemented using one or more processors, comprising:
obtaining (<NUM>) a first digital image of a geographic area captured from an elevated vantage point, wherein the first digital image comprises a plurality of pixels that correspond spatially to a respective plurality of geographic units of the geographic area;
identifying (<NUM>) one or more obscured pixels of the first digital image that correspond spatially to one or more obscured geographic units of the geographic area that are obscured in the first digital image by one or more transient obstructions;
determining (<NUM>), across pixels of a corpus of digital images that correspond spatially to the one or more obscured geographic units, one or more spectral-temporal data fingerprints of the one or more obscured geographic units;
identifying (<NUM>) one or more unobscured pixels of the first digital image or a second digital image that correspond spatially to one or more unobscured geographic units that are unobscured by transient obstructions, wherein the unobscured geographic units are spatially different than the one or more obscured geographic units of the geographic area and have one or more spectral-temporal data fingerprints that match the one or more spectral-temporal data fingerprints of the one or more obscured geographic units;
calculating (<NUM>) replacement pixel data based on the one or more unobscured pixels; and
generating (<NUM>) a transient-obstruction-free version of the first digital image in which data associated with the one or more obscured pixels is replaced with the replacement pixel data.