ACCELERATED DATA COLLECTION USING TRANSFORMER ENCODING LAYERS FOR DATA SEPARATION

Techniques are disclosed herein that are directed towards using satellite image data to narrow down the search space for statistically significant and/or meaningful ground truth data. Various implementations include techniques for labeling agricultural image data using unsupervised clustering and/or active learning techniques. Additional or alternative implementations include collecting more detailed crop information from locations on the ground with a higher quality (e.g., especially representative of a particular crop) ground truth.

BACKGROUND

As agricultural data mining and planning becomes more commonplace, the amount of data analyzed, and the number of sources providing that data, is increasing rapidly. Agricultural data can be used in a variety of ways including crop yield prediction and/or diagnoses. For instance, image data can be processed using one or more machine learning models to generate agricultural predictions. More accurate agricultural predictions can be made by processing higher quality image data. However, as image quality increases, computational resources necessary to store and/or process the image data also increases. Consequently, processing agricultural data for agricultural predictions (e.g., for crop yield predictions) often requires significant data storage and data processing resources.

Collecting ground truth data that can be used as labels to train various agricultural machine learning models, such as remote sensing models that make inferences based on satellite imagery, may be difficult to scale. Deploying human agronomists to comprehensively collect data about (e.g., capture digital images of) crops and/or label them may be prohibitively costly and/or time consuming. Deploying human agronomists to randomly sample data about crops and label them may be less costly and time consuming, but may yield less statistically significant or meaningful ground truth data.

SUMMARY

Implementations described herein are directed towards using satellite data to narrow down the search space for statistically significant and/or meaningful ground truth data. More particularly, but not exclusively, techniques are described herein for labeling agricultural image data using unsupervised clustering and/or active learning techniques, so that more detailed crop information can be collected from locations on the ground with higher quality (e.g., especially representative of a particular crop) ground truth.

In some implementations, a label can indicate one or more crops captured in an instance of agricultural image data (e.g., one or more crops captured in a pixel of satellite image data that represents, for instance, a ten meter by ten meter plot of land). Once generated, the labeled instances of agricultural image data can be used for various types of agricultural processing, such as using the labeled instances of agricultural image data to train a crop classification model. The crop classification model can be trained, for instance, to process instances of image data (e.g., satellite imagery) to generate output predicting one or more crops captured in the instances of image data. Other types of agricultural machine learning models may be trained as well, such as various types of phenotyping models, crop yield prediction models, and so forth.

In some implementations, instances of agricultural satellite image data can be processed using an encoder model to generate an encoded representation of each instance of agricultural image data. In some implementations, the encoder model can be the encoder portion of a pre-trained crop identification model (e.g., an encoder portion of a pre-trained crop identification transformer model). In some of those implementations, the pre-trained crop identification model can be trained using supervised learning (e.g., trained using training image data where a given training instance includes image data and one or more labels identifying one or more crops captured in the corresponding image data). In other implementations, the encoder model may be an encoder portion of a model, such as a transformer model, that is not necessarily pre-trained. In this latter case, homogenous pixels may be clustered together to provide suitable starting points (e.g., centroids) for collecting ground truth labels for the pixels.

The encoded instances of agricultural image data are an intermediate representation of the agricultural image data (e.g., an embedding space representation of the agricultural image data). In some implementations, the intermediate representations of the image data are easier to linearly separate into different classes of crops. In other words, the encoded instances of agricultural image data are easier to linearly separate compared to the same (unencoded) instances of agricultural image data.

In some implementations, the agricultural image data can be separated using one or more clustering techniques. Clustering includes grouping a set of objects in such a way that the objects in the same group (i.e., a cluster) are more similar to each other than to those in other groups. For example, one or more agricultural satellite images can include a first group of pixels capturing wheat and a second group of pixels capturing barley. In some implementations, clustering techniques can be used to separate the encoded representations of the first group of pixels into a wheat cluster and the encoded representations of the second group of pixels into a barley cluster.

Various clustering techniques can use different definitions of what constitutes a cluster of objects and/or different techniques to find those clusters of objects within a data set. For example, hierarchical clustering can build clusters based on distance(s) between the objects; k-means clustering can represent each cluster as a single mean vector; distribution model clustering can represent each cluster using statistical distributions (e.g., multivariate normal distributions); etc.

In some implementations, the system can cluster a set of encoded instances of agricultural image data using k-means clustering, where the system partitions the n instances of image data into k clusters, and where each instance of image data corresponds to the cluster with the nearest mean (e.g., nearest cluster centroid). Additionally or alternatively, k-means clustering can minimize the within-cluster variance between instances of image data.

For example, the system can use k-means clustering to generate a plurality of clusters based on processing the encoded instances of agricultural image data and identify a centroid of each cluster in the plurality of clusters. For each cluster centroid, the system can identify a label indicating the type of crops captured in the corresponding pixel of agricultural satellite image data. In some implementations, the system can identify the centroid as a statistically significant and/or meaningful data instance. For example, the system can identify the location of a pixel of satellite image data corresponding to a given cluster centroid. In some implementations, the system can deploy a ground truth collection entity, such as a human reviewer, to the location captured in the pixel of satellite image data (e.g., the location captured in the pixel of satellite data corresponding to the cluster centroid) to collect a ground truth label (e.g., capture additional image data) of the location. Additionally or alternatively, the system can deploy an aerial vehicle (e.g., a helicopter, an airplane, a balloon, an unmanned aerial vehicle (UAV), a drone, one or more additional aerial vehicles, and/or combinations thereof) to the location captured in the pixel of satellite image data (e.g., the location captured in the pixel of satellite data corresponding to the cluster centroid) to capture additional image data of the location.

In some implementations, the system can generate a label corresponding to the satellite image pixel based on the additional image data captured at the corresponding location. For example, the system can deploy a UVA to capture additional image data at the location captured in the satellite image pixel corresponding to a given cluster centroid. The system can process the additional image data to determine the one or more crops captured in the additional image data. In some of those implementations, a ground truth collection entity such as a human reviewer can generate the label.

In some implementations, the labeled instances of agricultural image data corresponding to the cluster centroids can be used to train the crop classification model. For example, a given instance of agricultural image data can be processed using the crop classification model to generate predicted output indicating the crop captured in the given instance of agricultural image data (e.g., the crop captured in a given pixel of agricultural satellite imagery). The system can compare the predicted output with the label corresponding to the given instance of agricultural image data and can update one or more portions of the crop classification model based on the comparing.

In some implementations, the system can process the unlabeled instances of agricultural image data using the crop classification model to generate output. Additionally or alternatively, the system can identify one or more additional instances of agricultural image data to label based on the generated output. In some of those implementations, the system can use active learning techniques to identify the one or more additional instances of agricultural image data to label.

For example, the system can process the generated output and/or one or more of the remaining unlabeled instances of agricultural image data using active learning techniques to identify the one or more additional instances of agricultural image data that are statistically significant and/or meaningful to label. A variety of metrics can be used to determine the one or more additional instances of agricultural image data to label such as focusing on the confidence of the crop prediction model, focusing on the uncertainty of some instances of agricultural image data, focusing on identifying the boundaries between two classes and labeling instances of agricultural image data at the boundaries, one or more additional or alternative metrics, and/or combinations thereof.

In some implementations, the system can deploy a human reviewer and/or an aerial vehicle to collect ground truth (e.g., capture additional image data) at the location corresponding to the additional agricultural image data identified as statistically significant and/or meaningful via active learning. For example, the system can deploy the human reviewer and/or the aerial vehicle to the location captured in satellite image data corresponding to the additional intermediate instances of agricultural image data identified as statistically significant and/or meaningful, to capture additional image data of the location. The system can process the additional image data to determine the one or more crops captured in the additional image data. In some of those implementations, a human reviewer can generate the label indicating the crop(s) in the additional image data.

Additionally or alternatively, the system can update one or more portions of the crop prediction model based on the additional instances of agricultural image data and the corresponding new labels. For example, the system can process a given additional instance of agricultural image data using the crop classification model to generate additional predicted output (e.g., the additional predicted output predicting the one or more crops captured in the instance of agricultural image data). One or more portions of the crop classification model can be updated based on comparing the additional predicted output and the label corresponding to the given additional instance of agricultural image data.

In some implementations, the system can repeat the process of adding new labels to one or more additional instances of agricultural image data, using the one or more additional instances of agricultural image data and the corresponding new labels to update one or more portions of the crop classification model, processing the unlabeled instances in the agricultural image data set, identifying one or more further instances of agricultural image data to label using active learning (e.g., one or more further instances of agricultural image data which are statistically significant and/or meaningful), and generating labels identifying one or more types of crops captured in the one or more further instances of agricultural image data.

Accordingly, various techniques set forth herein are directed towards identifying statistically significant and/or meaningful instances of agricultural image data (e.g., particular pixels of satellite image data) to label. Compared to traditional approaches (e.g., having a human reviewer generate a label for each pixel of agricultural satellite image data, randomly selecting pixels of the agricultural satellite image data to label, etc.), implementations described herein reduce the total number of instances of agricultural image data necessary to label to predict highly accurate labels for the instances of agricultural image data. The system can conserve resources such as computing resources (processor cycles, memory, power, one or more additional resources, and/or combinations thereof), time, manpower, one or more additional or alternative resources and/or combinations thereof by reducing the total number of instances of agricultural image data to label.

The above description is provided only as an overview of some implementations disclosed herein. These and other implementations of the technology are disclosed in additional detail below.

DETAILED DESCRIPTION

Turning now to the figures,FIG.1illustrates an example of identifying statistically significant and/or meaningful instances of agricultural satellite image data based on clustering intermediate representations of the agricultural satellite image data in accordance with various implementations disclosed herein. The example100includes a set of agricultural satellite image data102. In some implementations, each instance of agricultural satellite image data captures at least a portion of an agricultural plot. In some of those implementations, each instance of agricultural satellite image data captures one or more crops planted in the corresponding agricultural plot.

The set of agricultural satellite image data102can be processed using encoder engine104to generate a set of intermediate representations of the agricultural satellite image data106. The encoder engine104can process the set of agricultural satellite image data102using an encoder model to generate a set of intermediate representations of the agricultural satellite image data106. In some implementations, the encoder model can be an encoder portion of a pre-trained model which has been pre-trained to identify one or more crops captured in image data. In some of those implementations, the pre-trained model can be a recurrent neural network transformer (RNN-T) model, and the encoder can be the encoder portion of the RNN-T.

In some implementations, the intermediate representations of the agricultural satellite image data106are encoded representations of the agricultural satellite image data. In some of those representations, the encoded representations of the agricultural satellite image data are an embedding space representation of the agricultural image data. Additionally or alternatively, the intermediate representations of the image data is easier to linearly separate into different classes of crops. In other words, the encoded instances of agricultural image data are easier to linearly separate compared to the same (unencoded) instances of agricultural image data.

However, this is not meant to be limiting. One or more portions of additional and/or alternative pre-trained models can be used to generate the intermediate representation of the agricultural satellite image data such as a feedforward artificial neural network, multilayer perceptron network, a radial basis network, a long short-term memory network (LSTM), a convolutional neural network (CNN), one or more additional or alternative neural network models, and/or combinations thereof. For example, the agricultural satellite image data102can be processed using one or more portions of a convolutional neural network to generate filtered representations of the agricultural satellite image data102, where the filtered representations can be used as the intermediate representations of the agricultural satellite image data106.

The intermediate representations of the agricultural satellite image data106can be processed using clustering engine108to generate a plurality of clusters110. In some implementations, each cluster, in the plurality of clusters110, can correspond to a predicted crop captured in the instances of intermediate representations of the agricultural satellite image data106. In other words, the agricultural satellite image data can be separated by crop type based on clustering the intermediate representations of the agricultural satellite image data. Clustering includes grouping a set of objects in such a way that the objects in the same group (i.e., a cluster) are more similar to each other than to those in other groups. For example, a set of agricultural image data can include a first group of images capturing wheat and a second group of images capturing barley. In some implementations, clustering techniques can be used to separate the encoded representations (e.g., the intermediate representations) of the first group of images into a wheat cluster and the encoded representations (e.g., the intermediate representations) of the second group of images into a barley cluster.

Various clustering techniques can use different definitions of what constitutes a cluster of objects and/or different techniques to find those clusters of objects within a data set. For example, hierarchical clustering can build clusters based on distance(s) between the objects; k-means clustering can represent each cluster as a single mean vector; distribution model clustering can represent each cluster using statistical distributions (e.g., multivariate normal distributions); etc. In some implementations, the system can cluster a set of encoded instances of agricultural image data using k-means clustering, where the system partitions the n instances of image data into k clusters, and where each instance of image data corresponds to the cluster with the nearest mean (e.g., nearest cluster centroid). Additionally or alternatively, k-means clustering can minimize the within-cluster variance between instances of image data.

The plurality of clusters can be processed using a centroid engine112to identify one or more cluster centroids114corresponding to each of the clusters. In some implementations, the system can identify the centroid as a statistically significant and/or meaningful data instance. The cluster centroids114can be processed by location engine116to generate the locations of the agricultural satellite image data corresponding to the cluster centroids118. For example, the system can identify the location of a pixel of satellite image data corresponding to a given cluster centroid.

Additionally or alternatively, the locations of agricultural satellite images corresponding to the centroids118can be processed using additional image data deployment engine120to collect additional instances of image data122. In some implementations, the additional image data deployment engine120can deploy a ground truth collection entity, such as a human reviewer, to the location captured in the pixel of satellite image data (e.g., the location captured in the pixel of satellite data corresponding to the cluster centroid) to collect a ground truth label (e.g., capture additional image data) of the location. Additionally or alternatively, the additional image data deployment engine120can deploy an aerial vehicle (e.g., a helicopter, an airplane, a balloon, an unmanned aerial vehicle (UAV), a drone, one or more additional aerial vehicles, and/or combinations thereof) to the location captured in the pixel of satellite image data (e.g., the location captured in the pixel of satellite data corresponding to the cluster centroid) to capture additional image data124of the location.

The additional image data122can be processed using label engine124to generate labeled instances of agricultural satellite images which correspond to the cluster centroids126. In some implementations, the label engine124can generate a label corresponding to the satellite image pixel based on the additional image data captured at the corresponding location. For example, the additional image data deployment engine120can deploy a UVA to capture additional image data122at the location captured in the satellite image pixel corresponding to a given cluster centroid118. In some implementations, the label engine124can process the additional image data124to determine the one or more crops captured in the additional image data. In some of those implementations, a ground truth collection entity such as a human reviewer can generate the label.

The labeled instances of agricultural satellite image data126can be processed using a crop classification model training engine128to generate an updated crop classification model130. In some implementations, the labeled instances of agricultural image data corresponding to the cluster centroids126can be used to train the crop classification model. For example, a given labeled instance of agricultural image data126can be processed using the crop classification model to generate predicted output indicating the crop captured in the given instance of agricultural image data (e.g., the crop captured in a given pixel of agricultural satellite imagery). The crop classification model training engine128can compare the predicted output with the label corresponding to the given instance of agricultural image data and can update one or more portions of the crop classification model based on the comparing.

FIG.2illustrates an example of identifying statistically significant and/or meaningful instances of agricultural satellite image data using active learning in accordance with various implementations described herein. The example150includes an updated crop classification model130, a set of unlabeled instances of agricultural satellite image data154, and intermediate representations of the agricultural satellite image data156. In some implementations, the updated crop classification model130is updated using crop classification model training engine128as described herein with respect toFIG.1. In some implementations, the set of unlabeled instances of agricultural satellite image data154can include the set of agricultural satellite image data102. In some of those implementations, the set unlabeled instances of agricultural satellite image data154can include the set of agricultural satellite image data102without the instances of agricultural satellite image data corresponding to cluster centroids126as described herein with respect toFIG.1.

Active learning engine152can process the updated crop classification model130, the set of unlabeled instances of agricultural satellite image data154, and/or the intermediate representations of the agricultural satellite image data156to identify one or more statistically significant instances of agricultural satellite image data158. Additionally or alternatively, location engine116can process the one or more statistically significant instances of agricultural satellite image data158to generate a set of locations of the agricultural satellite images corresponding to the statistically significant instances of agricultural satellite images158. In some of those implementations, the system can process the one or more statistically significant instances of agricultural satellite image data158to generate the corresponding locations160using location engine116described herein with respect toFIG.1.

In some implementations, additional image data deployment engine120can deploy one or more reviewers (e.g., a human reviewer, a UAV, etc.) to one or more of the locations160corresponding to statistically significant instances of image data to capture further additional image data162. Label engine124can process the further additional image data162to generate one or more labeled instances of agricultural satellite images164corresponding to the statistically significant instances of satellite image data. Additionally or alternatively, crop classification model training engine128can process the labeled instances of agricultural satellite image data164to generate a further updated crop classification model130.

FIG.3illustrates a block diagram of an example environment300in which implementations disclosed herein may be implemented. The example environment300includes a computing system302which can include encoder engine104, clustering engine108, location engine116, additional image data deployment engine120, labeling engine124, crop classification model training engine128, active learning engine152, agricultural image data engine304, and/or one or more additional or alternative engines (not depicted). Additionally or alternatively, computing system302may be associated with agricultural image data102, additional image data122, one or more labels322, encoder model324, crop classification model326, and/or one or more additional or alternative components (not depicted).

In some implementations, computing system302may include may include user interface input/output devices (not depicted), which may include, for example, a physical keyboard, a touch screen (e.g., implementing a virtual keyboard or other textual input mechanisms), a microphone, a camera, a display screen, and/or speaker(s). One or more user interface input/output devices (not depicted) may be incorporated with one or more computing systems302of a user. For example, a mobile phone of the user may include the user interface input output devices; a standalone digital assistant hardware device may include the user interface input/output device; a first computing device may include the user interface input device(s) and a separate computing device may include the user interface output device(s); etc. In some implementations, all or aspects of computing system402may be implemented on a computing system that also contains the user interface input/output devices.

Some non-limiting examples of computing system302include one or more of: a desktop computing device, a laptop computing device, a standalone hardware device at least in part dedicated to an automated assistant, a tablet computing device, a mobile phone computing device, a computing device of a vehicle (e.g., an in-vehicle communications system, and in-vehicle entertainment system, an in-vehicle navigation system, an in-vehicle navigation system), or a wearable apparatus of the user that includes a computing device (e.g., a watch of the user having a computing device, glasses of the user having a computing device, a virtual or augmented reality computing device). Additional and/or alternative computing systems may be provided. Computing system302may include one or more memories for storage of data and software applications, one or more processors for accessing data and executing applications, and other components that facilitate communication over a network. The operations performed by computing system302may be distributed across multiple computing devices. For example, computing programs running on one or more computers in one or more locations can be coupled to each other through a network.

In some implementations, agricultural image data engine304can identify one or more instances of agricultural satellite image data for processing. In some of those implementations, the agricultural image data engine304can identify one or more instances of agricultural image data102, which can include satellite agricultural image data capturing one or more crops. For example, one or more crops can be captured in each pixel of the satellite image data that can represent, for instance, a ten meter by ten meter plot of land. Pixel(s) in the agricultural satellite image data can represent additional and/or alternative portion(s) of a plot of land (e.g., a one meter by one meter plot, a 100 meter by 100 meter plot, a one foot by one foot plot, etc.).

Encoder engine104can process the one or more instances of agricultural image data102using the encoder model314to generate one or more intermediate representations of the agricultural image data. In some implementations, the encoder model314can be the encoder portion of a pre-trained crop identification model (e.g., an encoder portion of a pre-trained crop identification transformer model). In some of those implementations, the pre-trained crop identification model can be trained using supervised learning (e.g., trained using training image data where a given training instance includes image data and one or more labels identifying one or more crops captured in the corresponding image data). In other implementations, the encoder model314may be an encoder portion of a model, such as a transformer model, that is not necessarily pre-trained. In this latter case, homogenous pixels may be clustered together to provide suitable starting points (e.g., centroids) for collecting ground truth labels for the pixels.

The encoded instances of agricultural image data are an intermediate representation of the agricultural image data102(e.g., an embedding space representation of the agricultural image data). In some implementations, the intermediate representations of the image data are easier to linearly separate into different classes of crops. In other words, the encoded instances of agricultural image data are easier to linearly separate compared to the same (unencoded) instances of agricultural image data.

In some implementations, clustering engine108can process the one or more intermediate representations of the image data to separate the one or more intermediate representations of the image data into one or more clusters. Clustering includes grouping a set of objects in such a way that the objects in the same group (i.e., a cluster) are more similar to each other than to those in other groups. For example, one or more agricultural satellite images can include a first group of pixels capturing wheat and a second group of pixels capturing barley.

Clustering engine108can use a variety of clustering techniques, where various clustering techniques can use different definitions of what constitutes a cluster of objects and/or different techniques to find those clusters of objects within a data set. For example, hierarchical clustering can build clusters based on distance(s) between the objects; k-means clustering can represent each cluster as a single mean vector; distribution model clustering can represent each cluster using statistical distributions (e.g., multivariate normal distributions); etc.

In some implementations, the clustering engine108can cluster the set of encoded instances of agricultural image data using k-means clustering, where the clustering engine108partitions the n instances of image data into k clusters, and where each instance of image data corresponds to the cluster with the nearest mean (e.g., nearest cluster centroid). Additionally or alternatively, k-means clustering can minimize the within-cluster variance between instances of image data.

In some implementations, a centroid engine (e.g., centroid engine112ofFIG.1) can identify one or more cluster centroids corresponding to each of the clusters. In some implementations, the system can identify the one or more centroid as statistically significant and/or meaningful data. For example, a label for the cluster centroid can correspond to additional data points in the same cluster.

The cluster centroids (e.g., the statistically significant and/or meaningful data points) can be processed using the location engine116to identify the location of the instance agricultural satellite image data102corresponding to a given cluster centroid. For example, the location engine116can identify a physical location corresponding to a given centroid (e.g., a latitude value and a longitude value representing the physical location of the given centroid).

In some implementations, an additional image data deployment engine (such as additional data deployment engine120can deploy a ground truth collection entity to capture one or more instances of additional image data122corresponding to the location of a given centroid. The additional data deployment engine120can deploy the ground truth collection entity, such as a human reviewer, to the location captured in the pixel of satellite image data (e.g., the location captured in the pixel of satellite data corresponding to the cluster centroid) to collect a ground truth label (e.g., capture additional image data) of the location. Additionally or alternatively, the system can deploy an aerial vehicle (e.g., a helicopter, an airplane, a balloon, an unmanned aerial vehicle (UAV), a drone, one or more additional aerial vehicles, and/or combinations thereof) to the location captured in the pixel of satellite image data (e.g., the location captured in the pixel of satellite data corresponding to the cluster centroid) to capture additional image data of the location.

In some implementations, the label engine124can generate a label322corresponding to the satellite image pixel based on the additional image data captured at the corresponding location. For example, the system can deploy a UVA to capture additional image data at the location captured in the satellite image pixel corresponding to a given cluster centroid. The system can process the additional image data to determine the one or more crops captured in the additional image data, and generate a label322based on the one or more crops captured in the additional instance of image data122. In some of those implementations, a ground truth collection entity such as a human reviewer can generate the label.

In some implementations, crop classification model training engine128can process the labeled instances of agricultural satellite image data corresponding to the cluster centroids to generate output used in training the crop classification model326. For example, a given instance of agricultural image data102can be processed using the crop classification model326to generate predicted output indicating the crop captured in the given instance of agricultural image data (e.g., the crop captured in a given pixel of agricultural satellite imagery). The system can compare the predicted output with the label322corresponding to the given instance of agricultural image data and can update one or more portions of the crop classification model326based on the comparing.

In some implementations, the system can process the unlabeled instances of agricultural image data102using the crop classification model326to generate output. For example, the system can process the generated output and/or one or more of the remaining unlabeled instances of agricultural image data102using the active learning engine152one or more additional instances of agricultural image data that are statistically significant and/or meaningful to label. A variety of metrics can be used to determine the one or more additional instances of agricultural image data to label such as focusing on the confidence of the crop prediction model, focusing on the uncertainty of some instances of agricultural image data, focusing on identifying the boundaries between two classes and labeling instances of agricultural image data at the boundaries, one or more additional or alternative metrics, and/or combinations thereof.

In some implementations, location engine116can be used to identify the location of the one or more additional instances of statistically significant and/or meaningful instances of agricultural image data102. Similarly, the system can use additional image data deployment engine120to capture one or more corresponding additional instances of additional image data122. The labeling engine124can process the one or more corresponding instances of additional image data to generate one or more labels322corresponding to the one or more additional instances of statistically significant and/or meaningful instances of agricultural image data102. The labeled additional statistically significant and/or meaningful instances of agricultural image data can be processed using crop classification training engine128to generate output, where the output can be used to update one or more portions of crop classification model326. Additionally or alternatively, active learning engine152can identify one or more further instances of statistically significant and/or meaningful instances of agricultural image data. Additional or alternative iterations of this process can be completed by the system to continue training the crop classification model326.

FIG.4is a flowchart illustrating an example process400updating a crop classification model based on one or more statistically significant and/or meaningful unlabeled instances of agricultural satellite image data in accordance with implementations disclosed herein. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. This system may include various components of various computer systems, such as one or more components of computing system302, and/or computing system710. Moreover, while operations of process400are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, and/or added.

At block402, the system processes a set of agricultural satellite image data using an encoder model to generate a set of intermediate representations of the agricultural satellite image data. In some implementations, an instance of agricultural satellite image data, in the set of agricultural satellite image data, can capture one or more crops planted in an agricultural plot. In some of those implementations, each pixel in the instance of image data can represent a portion of the plot, for example, each pixel can represent a ten meter by ten meter plot of land. In some implementations, the encoder model can be an encoder portion of a pre-trained agricultural crop identification model (e.g., the encoder portion of a pre-trained crop identification transformer model). However, the encoder model is not necessarily pre-trained.

A given instance of the agricultural satellite image data can be processed using the encoder model to generate a corresponding intermediate representation of the given instance of agricultural satellite image data. In some implementations, the intermediate representation of an instance of agricultural satellite image data can be an embedding space representation of the instance of agricultural satellite image data. The intermediate representations of the image data are easier to linearly separate into different classes of crops.

At block404, the system identifies one or more statistically significant and/or meaningful instances of the agricultural satellite image data based on clustering the intermediate representations of the agricultural satellite image data. In some implementations, the system can identify the one or more statistically significant and/or meaningful instances of the agricultural satellite image data based on clustering the intermediate representations of the agricultural satellite image data in accordance with block504and/or block506ofFIG.5described herein.

At block406, the system deploys a ground truth collection entity to the location of each of the statistically significant and/or meaningful instances of agricultural satellite image data to capture additional image data. In some implementations, the ground truth detection entity can include an aerial vehicle (e.g., a helicopter, an airplane, a balloon, an unmanned aerial vehicle (UAV), a drone, one or more additional aerial vehicles, and/or combinations thereof), where the aerial vehicle can be deployed to the physical location captured in the instance of agricultural satellite image data corresponding to the centroid of the cluster. Additionally or alternatively, the ground truth detection entity can include a human reviewer, where the human reviewer can be deployed to the physical location captured in the instance of agricultural satellite image data corresponding to the location of each of the statistically significant and/or meaningful instances of agricultural satellite image data (e.g., the centroid of the cluster).

In some implementations, one or more vision sensors of the ground truth detection entity can capture one or more additional instances of image data of the identified physical location. In some implementations, the one or more additional instances of image data captured by one or more vision sensors of the ground truth detection entity, such as one or more cameras, one or more Light Detection and Ranging (LIDAR) sensors, one or more satellite cameras, one or more additional or alternative vision sensors, and/or combinations thereof.

In some implementations, the one or more crops captured in the one or more additional instances of image data can be more easily identified (compared to the corresponding instance of agricultural satellite image data). For instance, the additional image data can be captured at a higher resolution compared to the agricultural satellite image data.

In some implementations, the additional instances of image data can include additional instances of agricultural satellite image data. For example, an additional instance of agricultural satellite image data can be captured by the ground trough detection entity using a higher resolution camera compared to the corresponding instance of agricultural satellite image data. Additionally or alternatively, an additional instance of agricultural satellite image data can be captured by the ground truth detection entity at a lower altitude (e.g., closer to the ground) compared to the corresponding instance of agricultural satellite image data.

At block408, the system generates labels for the statistically significant instances of the agricultural satellite image data based on the additional image data. In some implementations the labels are generated based on the corresponding additional image data. For example, the one or more additional instances of image data can be processed using a crop classification model to identify one or more crops captured in the one or more additional instances of image data. Additionally or alternatively, a human can review the additional instances of image data to identify the one or more crops captured in the one or more additional instances of image data.

At block410, the system updates a crop classification model based on the labeled instances of agricultural satellite image data. In some implementations, the system can process the one or more instances of agricultural image data (e.g., labeled at block408) using the crop classification model to generate predicted output. Additionally or alternatively, one or more portions of the crop classification model can be updated based on comparing the predicted output and the generated label (e.g., the labeled generated at block408)

At block412, the system identifies one or more additional statistically significant and/or meaningful instances of the agricultural satellite image data using active learning. In some implementations, the system can identify one or more statistically significant and/or meaningful unlabeled instances of the agricultural satellite image data active learning techniques. A variety of metrics can be used to determine the one or more additional instances of agricultural image data to label such as focusing on the confidence of the crop prediction model, focusing on the uncertainty of some instances of agricultural image data, focusing on identifying the boundaries between two classes and labeling instances of agricultural image data at the boundaries, one or more additional or alternative metrics, and/or combinations thereof.

At block414, the system deploys a ground truth collection entity to the location of each of the additional statistically significant and/or meaningful instances of the agricultural satellite image data. In some implementations, the system can deploy the ground truth collection entity as described herein with respect to block406.

At block416, the system generates labels for the additional statistically significant and/or meaningful additional instances of the agricultural satellite image data based on the additional image data. In some implementations the labels are generated based on the corresponding additional image data. For example, the one or more additional instances of image data can be processed using a crop classification model to identify one or more crops captured in the one or more additional instances of image data. Additionally or alternatively, a human can review the additional instances of image data to identify the one or more crops captured in the one or more additional instances of image data.

At block418, the system updates the crop classification model based on the additional labeled instances of the agricultural satellite image data. In some implementations, the system can process the one or more labeled instances of agricultural image data (e.g., instances of image data labeled at block416) using the crop classification model to generate predicted output. Additionally or alternatively, one or more portions of the crop classification model can be updated based on comparing the predicted output and the generated label (e.g., the labeled generated at block416).

At block420, the system determines whether to identify any more additional statistically significant and/or meaningful instances of the agricultural satellite image data. If so, the system proceeds back to block412, identifies further instances of statistically significant and/or meaningful instances of the agricultural satellite image data using active learning, before proceeding to blocks414,416,418, and420based on the further instances of statistically significant and/or meaningful instances of the agricultural satellite image data. If not, the process ends.

FIG.5is a flowchart illustrating an example process500of updating a crop classification model based on labeled instances of agricultural satellite image data in accordance with implementations disclosed herein. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. This system may include various components of various computer systems, such as one or more components of computing system302, and/or computing system710. Moreover, while operations of process500are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, and/or added.

At block502, the system processes a set of agricultural satellite image data using an encoder model to generate a set of intermediate representations of the agricultural satellite image data. In some implementations, an instance of agricultural satellite image data, in the set of agricultural satellite image data, can capture one or more crops planted in an agricultural plot. In some of those implementations, each pixel in the instance of image data can represent a portion of the plot, for example, each pixel can represent a ten meter by ten meter plot of land. In some implementations, the encoder model can be an encoder portion of a pre-trained agricultural crop identification model (e.g., the encoder portion of a pre-trained crop identification transformer model). However, the encoder model is not necessarily pre-trained.

A given instance of the agricultural satellite image data can be processed using the encoder model to generate a corresponding intermediate representation of the given instance of agricultural satellite image data. In some implementations, the intermediate representation of an instance of agricultural satellite image data can be an embedding space representation of the instance of agricultural satellite image data. The intermediate representations of the image data are easier to linearly separate into different classes of crops.

At block504, the system generates a plurality of clusters based on processing the set of intermediate representations of the agricultural satellite image data. Clustering includes grouping a set of objects in such a way that the objects in the same group (i.e., cluster) are more similar to each other than to those in other groups. For example, the system can generate a plurality of clusters of the intermediate representations of the agricultural satellite image data where each cluster corresponds to one or more distinct crops. However, one or more of the initial clusters, in the plurality of clusters, may contain one or more intermediate representations of the agricultural satellite image data that do not correctly correspond with the crop type of the majority of the intermediate representations in the given cluster. The system can use a variety of clustering techniques including (but not limited to) hierarchical clustering, k-means clustering, distribution model clustering, one or more additional or alternative clustering techniques, and/or combinations thereof.

At block506, the system identifies a centroid of each cluster, where each of the centroids correspond to a statistically significant and/or meaningful instance of the agricultural satellite image data. In some implementations, the system can predict the crop(s) captured in a given cluster based on the crop(s) corresponding to the centroid of the given cluster. In some implementations, the centroid of a cluster can correspond to a single intermediate representation of an instance of the agricultural satellite image data. In some other implementations, the centroid of a cluster can correspond to one or more intermediate representations of instances of the agricultural satellite image data.

At block508, the system deploys a ground truth detection entity to the location of each of the centroids to capture additional image data. In some implementations, the ground truth detection entity can include an aerial vehicle (e.g., a helicopter, an airplane, a balloon, an unmanned aerial vehicle (UAV), a drone, one or more additional aerial vehicles, and/or combinations thereof), where the aerial vehicle can be deployed to the physical location captured in the instance of agricultural satellite image data corresponding to the centroid of the cluster. Additionally or alternatively, the ground truth detection entity can include a human reviewer, where the human reviewer can be deployed to the physical location captured in the instance of agricultural satellite image data corresponding to the centroid of the cluster.

In some implementations, one or more vision sensors of the ground truth detection entity can capture one or more additional instances of image data of the identified physical location. In some implementations, the one or more additional instances of image data captured by one or more vision sensors of the ground truth detection entity, such as one or more cameras, one or more Light Detection and Ranging (LIDAR) sensors, one or more satellite cameras, one or more additional or alternative vision sensors, and/or combinations thereof.

In some implementations, the one or more crops captured in the one or more additional instances of image data can be more easily identified (compared to the corresponding instance of agricultural satellite image data). For instance, the additional image data can be captured at a higher resolution compared to the agricultural satellite image data.

In some implementations, the additional instances of image data can include additional instances of agricultural satellite image data. For example, an additional instance of agricultural satellite image data can be captured by the ground trough detection entity using a higher resolution camera compared to the corresponding instance of agricultural satellite image data. Additionally or alternatively, an additional instance of agricultural satellite image data can be captured by the ground truth detection entity at a lower altitude (e.g., closer to the ground) compared to the corresponding instance of agricultural satellite image data.

At block510, the system generates a label for each of the instances of the agricultural satellite image data corresponding to the centroids. In some implementations the labels are generated based on the corresponding additional image data. For example, the one or more additional instances of image data can be processed using a crop classification model to identify one or more crops captured in the one or more additional instances of image data. Additionally or alternatively, a human can review the additional instances of image data to identify the one or more crops captured in the one or more additional instances of image data.

At block512, the system updates a crop classification model based on the labeled instances of agricultural satellite image data. In some implementations, the system can process the one or more labeled instances of agricultural image data (e.g., instances of agricultural image data labeled at block510) using the crop classification model to generate predicted output. Additionally or alternatively, one or more portions of the crop classification model can be updated based on comparing the predicted output and the generated label (e.g., the labeled generated at block510).

FIG.6is a flowchart illustrating an example process600of identifying one or more statistically significant instances of unlabeled agricultural satellite image data using one or more active learning techniques and updating a crop classification model based on those identified statistically significant instances in accordance with implementations disclosed herein. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. This system may include various components of various computer systems, such as one or more components of computing system302, and/or computing system710. Moreover, while operations of process600are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, and/or added.

At block602, the system processes (1) an updated crop classification model, (2) a set of unlabeled instances of agricultural satellite image data, and/or (3) a set of intermediate representations of the unlabeled instances of the agricultural satellite image data, using active learning, to identify one or more statistically significant and/or meaningful instances of the agricultural satellite image data. In some implementations, the updated crop classification model can be generated in accordance with processes400and/or500ofFIG.4and/orFIG.5described herein. In some implementations, the set of unlabeled instances of agricultural satellite image data can be determined on comparing an original set of agricultural satellite image data and the instances of satellite image data previously labeled, such as instance(s) labeled at block510ofFIG.5, instance(s) labeled at block606below, etc.

In some implementations, the system can identify one or more statistically significant and/or meaningful unlabeled instances of the agricultural satellite image data active learning techniques. A variety of metrics can be used to determine the one or more additional instances of agricultural image data to label such as focusing on the confidence of the crop prediction model, focusing on the uncertainty of some instances of agricultural image data, focusing on identifying the boundaries between two classes and labeling instances of agricultural image data at the boundaries, one or more additional or alternative metrics, and/or combinations thereof.

At block604, the system deploys a ground truth collection entity to the location of each of the statistically significant and/or meaningful instances of agricultural satellite image data to capture additional image data. In some implementations, the ground truth detection entity can include an aerial vehicle (e.g., a helicopter, an airplane, a balloon, an unmanned aerial vehicle (UAV), a drone, one or more additional aerial vehicles, and/or combinations thereof), where the aerial vehicle can be deployed to the physical location captured in the instance of agricultural satellite image data corresponding to the centroid of the cluster. Additionally or alternatively, the ground truth detection entity can include a human reviewer, where the human reviewer can be deployed to the physical location captured in the instance of agricultural satellite image data corresponding to the centroid of the cluster. In some implementations, the system can deploy the ground truth detection entities in accordance with block508ofFIG.5described herein.

At block606, the system generates labels for the statistically significant and/or meaningful instances of agricultural satellite image data based on the additional image data. In some implementations the labels are generated based on the corresponding additional image data. For example, the one or more additional instances of image data can be processed using a crop classification model to identify one or more crops captured in the one or more additional instances of image data. Additionally or alternatively, a human can review the additional instances of image data to identify the one or more crops captured in the one or more additional instances of image data.

At block608, the system updates a crop classification model based on the labeled instances of agricultural satellite image data. In some implementations, the system can process the one or more labeled instances of agricultural image data (e.g., instances of agricultural image data labeled at block606) using the crop classification model to generate predicted output. Additionally or alternatively, one or more portions of the crop classification model can be updated based on comparing the predicted output and the generated label (e.g., the labeled generated at block606).

At block610, the system determines whether to identify any more additional statistically significant and/or meaningful instances of the agricultural satellite image data. If so, the system proceeds back to block602, identifies further instances of statistically significant and/or meaningful instances of the agricultural satellite image data using active learning, before proceeding to blocks604,606,608, and610based on the further instances of statistically significant and/or meaningful instances of the agricultural satellite image data. If not, the process ends. In some implementations, the system can determine whether to identify one or more additional statistically significant and/or meaningful instances of the agricultural satellite image data based on determining whether one or more conditions are satisfied including whether a predicted error rate for the crop classification model satisfies a threshold value, whether a threshold value of agricultural satellite image data have been labeled, whether one or more instances of unlabeled agricultural satellite image data are available, whether a threshold number of active learning iterations has been satisfied, whether one or more additional or alternative conditions are satisfied, and/or combinations thereof.

FIG.7is a block diagram of an example computing device710that may optionally be utilized to perform one or more aspects of techniques described herein. In some implementations, one or more of a client computing device, and/or other component(s) may comprise one or more components of the example computing device710.

Computing device710typically includes at least one processor714which communicates with a number of peripheral devices via bus subsystem712. These peripheral devices may include a storage subsystem724, including, for example, a memory subsystem725and a file storage subsystem726, user interface output devices720, user interface input devices722, and a network interface subsystem716. The input and output devices allow user interaction with computing device710. Network interface subsystem716provides an interface to outside networks and is coupled to corresponding interface devices in other computing devices.

Storage subsystem724stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem724may include the logic to perform selected aspects of one or more of the processes ofFIG.4,FIG.5and/orFIG.6, as well as to implement various components depicted inFIG.3.

These software modules are generally executed by processor714alone or in combination with other processors. Memory725used in the storage subsystem724can include a number of memories including a main random access memory (“RAM”)730for storage of instructions and data during program execution and a read only memory (“ROM”)732in which fixed instructions are stored. A file storage subsystem726can provide persistent storage for program and data files, and may include a hard disk drive, a floppy disk drive along with associated removable media, a CD-ROM drive, an optical drive, or removable media cartridges. The modules implementing the functionality of certain implementations may be stored by file storage subsystem726in the storage subsystem724, or in other machines accessible by the processor(s)714.

Bus subsystem712provides a mechanism for letting the various components and subsystems of computing device710communicate with each other as intended. Although bus subsystem712is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple buses.

Computing device710can be of varying types including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computing device710depicted inFIG.7is intended only as a specific example for purposes of illustrating some implementations. Many other configurations of computing device710are possible having more or fewer components than the computing device depicted inFIG.7.

In some implementations, a method implemented by one or more processors is provided, the method includes identifying a set of agricultural satellite image data, where each instance of agricultural satellite image data includes image data capturing at least a portion of an agricultural plot. In some implementations, for each instance of agricultural satellite image data, in the set of agricultural satellite image data, the method includes processing the image data portion using an encoder model to generate an intermediate representation of the instance of agricultural satellite image data. In some implementations, the method further includes processing each of the intermediate representations of the agricultural satellite image data to generate a plurality of clusters of the agricultural satellite image data. In some implementations, for each cluster, in the plurality of clusters, the method includes identifying a centroid of the cluster, wherein the centroid corresponds to one or more of the intermediate representations of the agricultural satellite image data in the cluster. In some implementations, the method includes generating output indicating a location of the agricultural plot captured in the agricultural satellite image data corresponding to the centroid of the cluster.

In some implementations, for each cluster, in the plurality of clusters, the method further includes deploying a ground truth collection entity, to the location of the agricultural plot captured in the agricultural satellite image data corresponding to the centroid of the cluster, to collect additional image data of the agricultural plot. In some implementations, the method further includes generating a label for the instance of agricultural satellite image data based on the additional image data collected at the agricultural plot, wherein the label indicates the one or more crops captured in the instance of agricultural satellite image data. In some versions of those implementations, the ground truth collection entity is a human reviewer. In some versions of those implementations the ground truth collection entity is an unmanned aerial vehicle. In some versions of those implementations, for each of the labels generated based on additional image data collected at the locations of the agricultural plots captured in the agricultural satellite image data corresponding to the centroid of the clusters, the method further includes processing the corresponding instance of agricultural satellite image data using a crop classification model to generate predicted crop output, wherein the predicted crop output indicates one or more crops captured in the instance of agricultural satellite image data. In some versions of those implementations, the method further includes comparing the predicted crop output and the label generated based on the additional image data collected at the location corresponding to the instance of agricultural satellite image data. In some versions of those implementations, the method further includes updating one or more portions of the crop classification model based on comparing the predicted crop output and the label generated based on the additional image data collected at the location corresponding to the instance of agricultural satellite image data.

In some implementations, the method further includes processing, using active learning, the unlabeled instances of agricultural satellite image data and/or the corresponding intermediate representations of the agricultural satellite image data, in the set of agricultural satellite image data, to select one or more additional instances of agricultural satellite image data to label. In some implementations, for each of the additional instances of agricultural satellite image data selected to label, the method further includes generating additional output indicating the location of an additional agricultural plot captured in the additional instance of agricultural satellite image data. In some implementations, the method further includes deploying an additional ground truth collection entity, to the location of the additional agricultural plot captured in the additional instance of agricultural satellite image, to collect further image data of the additional agricultural plot. In some implementations, the method further includes generating an additional label for the additional instance of agricultural satellite image data based on the further image data collected at the additional agricultural plot, wherein the additional label indicates the one or more crops captured in the additional instance of agricultural satellite image data. In some implementations, the method further includes processing the additional instance of agricultural satellite image data using the crop classification model to generate additional crop prediction output, wherein the additional crop prediction output indicates the one or more crops captured in the additional agricultural plot captured in the additional instance of agricultural image data. In some implementations, the method further includes updating one or more portions of the crop classification model based on comparing the additional label and the additional crop prediction output.

In some versions of those implementations, the method further includes processing, using active learning, the unlabeled instances of agricultural satellite image data and/or the corresponding intermediate representations of the agricultural satellite image data, in the set of agricultural satellite image data, to select one or more further instances of agricultural satellite image data to label.

In some versions of those implementations, processing, using active learning, the unlabeled instances of agricultural satellite image data and/or the corresponding intermediate representations of the agricultural satellite image data, in the set of agricultural satellite image data, to select one or more additional instances of agricultural satellite image data to label further includes, for each unlabeled instance of agricultural image data, processing the unlabeled instance of agricultural satellite image data using the crop prediction model to generate candidate output, wherein the candidate output includes a confidence measure indicating the probability one or more crops are captured in the corresponding unlabeled instance of agricultural satellite image data. In some implementations, the method further includes selecting one or more of the additional instances of agricultural satellite image data based on the corresponding confidence measures.

In some versions of those implementations, processing, using active learning, the unlabeled instances of agricultural satellite image data and/or the corresponding intermediate representations of the agricultural satellite image data, in the set of agricultural satellite image data, to select one or more additional instances of agricultural satellite image data to label further includes identifying one or more of the unlabeled instances of agricultural satellite image data at the border of two or more clusters. In some versions of those implementations, the method further includes selecting the one or more additional instances of agricultural satellite image data to label based on the identified one or more of the unlabeled instances of agricultural satellite image data at the border of two or more clusters.

In some implementations, processing each of the intermediate representations of the agricultural satellite image data to generate the plurality of clusters of the agricultural satellite image data includes processing each of the intermediate representations of the agricultural satellite image data using k-means clustering to generate the plurality of clusters of the agricultural satellite image data. In some versions of those implementations, processing each of the intermediate representations of the agricultural satellite image data using k-means clustering to generate the plurality of clusters of the agricultural satellite image data is unsupervised clustering.

In some implementations, the encoder model is an encoder portion of a trained recurrent neural network transformer (RNN-T) model, wherein the RNN-T model is trained for crop classification using supervised learning.

In addition, some implementations include one or more processors (e.g., central processing unit(s) (CPU(s)), graphics processing unit(s) (GPU(s), and/or tensor processing unit(s) (TPU(s)) of one or more computing devices, where the one or more processors are operable to execute instructions stored in associated memory, and where the instructions are configured to cause performance of any of the methods described herein. Some implementations also include one or more transitory or non-transitory computer readable storage media storing computer instructions executable by one or more processors to perform any of the methods described herein.