Patent Description:
This disclosure relates to using a plant identification model to identify and treat plants in a field and, more specifically, using a model that identifies plants in the presence of airborne particulates.

Historically, farming machines that spray crops with treatment fluid have relied on highly non-specific spray techniques such as broadcast spraying. Non-specific spray techniques apply more herbicide than necessary in some cases, a practice which is inefficient and potentially costly for farmers. More recently, non-specific spraying has been supplemented with target-specific spray techniques that utilize detection devices and plant identification models to identify plants in the field <CIT> is an example for such a target specific technique for treating plants based on image analysis to identify plants. However, even these improved identification techniques can inaccurately identify plants in non-ideal operating conditions (e.g., dusty). Accordingly, a farming machine used for targeted spraying that uses an algorithm to rapidly identify plants in non-ideal operating conditions without sacrificing accuracy, specificity, and resolution would be beneficial.

A farming machine can include any number of treatment mechanisms to treat plants as the farming machine moves through the field. Each treatment mechanism may be controlled by a control system that actuates the treatment mechanisms at the appropriate time to treat plants as the farming machine travels through the field. The farming machine may also include multiple detection and verification systems to capture images of plants in the field to facilitate treating plants.

The control system employs a plant identification model configured to identify plants in the field. The plant identification model also identifies plants in the field when the farming machine is operating in non-ideal operating conditions (e.g., windy). For example, the plant identification model can identify pixels representing both plant matter and airborne particulates picked up by the wind. Some of the particulate pixels may be identified as obscuring pixels. Obscuring pixels are particulate pixels that obscure a plant in an accessed image. The plant identification model may identify a plant based on both the obscuring pixels and pixels representing the plant in the image.

The plant identification model may also be configured to identify a particulate level based on the image. The particulate level is a quantification of a measure of particulates present in one or more portions of an image. There are several methods to identify a particulate level, some of which are described herein. The control system can generate a notification for the determined particulate level and transmit that notification to an operator of the farming system.

The control system can generate a treatment map using the results of the plant identification model. The treatment map is a data structure that includes information regarding which treatment mechanisms to actuate such that identified plants are treated. To generate a treatment map, the farming machine captures an image of plants in the field. The control system accesses the image and inputs the image to the plant identification model which processes the image to identify plants and/or particulates in the image. After identification, the control system generates a treatment map which maps treatment areas of the treatment mechanisms to areas in the image including identified plants. The control system converts the treatment map into control signals for treatment mechanisms and actuates treatment mechanisms at the appropriate time to treat identified plants. Farming systems that employ plant identification models configured for non-ideal operating conditions identify and treat plants with higher accuracy and specificity than models configured for ideal operating conditions.

Farming machines that treat plants in a field have continued to improve over time. For example, a crop sprayer can include many independently actuated spray nozzles to spray treatment fluid on specific plants in a field. The farming machine can further include detection mechanisms that can detect both plants in the field and treatments made to plants in the field. Recently, farming machines have included control systems executing algorithms to automatically detect and treat plants using the detection mechanisms. Traditionally, the algorithms are wasteful because they treat areas in the field that do not include identified plants, as, often, the algorithms sacrifice accuracy for processing speed.

Described herein is a farming machine that employs a machine learning model that automatically determines, in real-time, plants in a field and treats the identified plants using a treatment mechanism. In an example, the machine learning model is a semantic segmentation model, but could be other machine learning models. The model is trained to function in non-ideal operating conditions. In example, a semantic segmentation model encodes an image of the field (e.g., an image crops including airborne particulates) using a convolutional neural network. The network is configured to reduce the encoded image to a latent representation space and trained to identify plants in that representation space. Rather than decoding the identified plants back to an image, the model decodes the identified plants to a treatment map. The farming machine uses the treatment map to generate machine instructions for treating identified plants in the field. The dimensionality of the treatment map is, generally, much less than the dimensionality of the image and, therefore, the processing time is reduced. The semantic segmentation model has higher accuracy, specificity, and provides better resolution for the treatment mechanisms than other traditional plant identification models.

<FIG> is a side view illustration of a system for applying a treatment fluid to plants in a field and <FIG> is a front view illustration of the same system, according to one example embodiment. The farming machine <NUM> for plant treatment includes a detection mechanism <NUM>, a treatment mechanism <NUM>, and a control system <NUM>. The farming machine <NUM> can additionally include a mounting mechanism <NUM>, a verification mechanism <NUM>, a power source, digital memory, communication apparatus, or any other suitable component.

The farming machine <NUM> functions to apply a treatment to one or multiple plants <NUM> within a geographic area <NUM>. Often, treatments function to regulate plant growth. The treatment is directly applied to a single plant <NUM> (e.g., hygroscopic material), but can alternatively be directly applied to multiple plants, indirectly applied to one or more plants, applied to the environment associated with the plant (e.g., soil, atmosphere, or other suitable portion of the plant environment adjacent to or connected by an environmental factor, such as wind), or otherwise applied to the plants. Treatments that can be applied include necroing the plant, necrosing a portion of the plant (e.g., pruning), regulating plant growth, or any other suitable plant treatment. Necrosing the plant can include dislodging the plant from the supporting substrate <NUM>, incinerating a portion of the plant, applying a treatment concentration of working fluid (e.g., fertilizer, hormone, water, insecticide, fungicide, etc.) to the plant, or treating the plant in any other suitable manner. Regulating plant <NUM> growth can include promoting plant growth, promoting growth of a plant portion, hindering (e.g., retarding) plant or plant portion growth, or otherwise controlling plant growth. Examples of regulating plant <NUM> growth includes applying growth hormone to the plant, applying fertilizer to the plant or substrate <NUM>, applying a disease treatment or insect treatment to the plant, electrically stimulating the plant, watering the plant, pruning the plant, or otherwise treating the plant. Plant growth can additionally be regulated by pruning, necrosing, or otherwise treating the plants adjacent the plant.

The plants <NUM> can be crops, but can alternatively be weeds or any other suitable plant. The crop may be cotton, but can alternatively be lettuce, soy beans, rice, carrots, tomatoes, corn, broccoli, cabbage, potatoes, wheat or any other suitable commercial crop. The plant field in which the system is used is an outdoor plant field, but can alternatively be plants within a greenhouse, a laboratory, a grow house, a set of containers, a machine, or any other suitable environment. The plants are grown in one or more plant rows (e.g., plant beds), wherein the plant rows are parallel, but can alternatively be grown in a set of plant pots, wherein the plant pots can be ordered into rows or matrices or be randomly distributed, or be grown in any other suitable configuration. The crop rows are generally spaced between <NUM> inches and <NUM> inches apart (e.g. as determined from the longitudinal row axis), but can alternatively be spaced any suitable distance apart, or have variable spacing between multiple rows.

The plants <NUM> within each plant field, plant row, or plant field subdivision generally includes the same type of crop (e.g. same genus, same species, etc.), but can alternatively include multiple crops (e.g., a first and a second crop), both of which are to be treated. Each plant <NUM> can include a stem, arranged superior (e.g., above) the substrate <NUM>, which supports the branches, leaves, and fruits of the plant. Each plant can additionally include a root system joined to the stem, located inferior the substrate plane (e.g., below ground), that supports the plant position and absorbs nutrients and water from the substrate <NUM>. The plant can be a vascular plant, non-vascular plant, ligneous plant, herbaceous plant, or be any suitable type of plant. The plant can have a single stem, multiple stems, or any number of stems. The plant can have a tap root system or a fibrous root system. The substrate <NUM> is soil, but can alternatively be a sponge or any other suitable substrate.

The treatment mechanism <NUM> of the farming machine <NUM> functions to apply a treatment to the identified plant <NUM>. The treatment mechanism <NUM> includes a treatment area <NUM> to which the treatment mechanism <NUM> applies the treatment. The effect of the treatment can include plant necrosis, plant growth stimulation, plant portion necrosis or removal, plant portion growth stimulation, or any other suitable treatment effect. The treatment can include plant <NUM> dislodgement from the substrate <NUM>, severing the plant (e.g., cutting), plant incineration, electrical stimulation of the plant, fertilizer or growth hormone application to the plant, watering the plant, light or other radiation application to the plant, injecting one or more working fluids into the substrate <NUM> adjacent the plant (e.g., within a threshold distance from the plant), or otherwise treating the plant. The treatment mechanism <NUM> is operable between a standby mode, wherein the treatment mechanism <NUM> does not apply a treatment, and a treatment mode, wherein the treatment mechanism <NUM> is controlled by the control system <NUM> to apply the treatment. However, the treatment mechanism <NUM> can be operable in any other suitable number of operation modes.

The farming machine <NUM> can include a single treatment mechanism <NUM>, or can include multiple treatment mechanisms. The multiple treatment mechanisms can be the same type of treatment mechanism, or be different types of treatment mechanisms. The treatment mechanism <NUM> can be fixed (e.g., statically coupled) to the mounting mechanism <NUM> or relative to the detection mechanism <NUM>, or actuate relative to the mounting mechanism <NUM> or detection mechanism <NUM>. For example, the treatment mechanism <NUM> can rotate or translate relative to the detection mechanism <NUM> and/or mounting mechanism <NUM>. In one variation, the farming machine <NUM> includes an assembly of treatment mechanisms, wherein a treatment mechanism <NUM> (or subcomponent of the treatment mechanism <NUM>) of the assembly is selected to apply the treatment to the identified plant <NUM> or portion of a plant in response to identification of the plant and the plant position relative to the assembly. In a second variation, the farming machine <NUM> includes a single treatment mechanism, wherein the treatment mechanism is actuated or the farming machine <NUM> moved to align the treatment mechanism <NUM> active area <NUM> with the targeted plant <NUM>. In a third variation, the farming machine <NUM> includes an array of treatment mechanisms <NUM>, wherein the treatment mechanisms <NUM> are actuated or the farming machine <NUM> is moved to align the treatment mechanism <NUM> active areas <NUM> with the targeted plant <NUM> or plant segment.

In one example configuration, the farming machine <NUM> can additionally include a mounting mechanism <NUM> that functions to provide a mounting point for the system components. In one example, as shown in <FIG>, the mounting mechanism <NUM> statically retains and mechanically supports the positions of the detection mechanism <NUM>, the treatment mechanism <NUM>, and the verification mechanism <NUM> relative to a longitudinal axis of the mounting mechanism <NUM>. The mounting mechanism <NUM> is a chassis or frame, but can alternatively be any other suitable mounting mechanism. In some configurations, there may be no mounting mechanism <NUM>, or the mounting mechanism can be incorporated into any other component of the farming machine <NUM>.

In one example farming machine <NUM>, the system may also include a first set of coaxial wheels, each wheel of the set arranged along an opposing side of the mounting mechanism <NUM>, and can additionally include a second set of coaxial wheels, wherein the rotational axis of the second set of wheels is parallel the rotational axis of the first set of wheels. However, the system can include any suitable number of wheels in any suitable configuration. The farming machine <NUM> may also include a coupling mechanism <NUM>, such as a hitch, that functions to removably or statically couple to a drive mechanism, such as a tractor, more to the rear of the drive mechanism (such that the farming machine <NUM> is dragged behind the drive mechanism), but alternatively the front of the drive mechanism or to the side of the drive mechanism. Alternatively, the farming machine <NUM> can include the drive mechanism (e.g., a motor and drive train coupled to the first and/or second set of wheels). In other example systems, the system may have any other means of traversing through the field.

In some example systems, the detection mechanism <NUM> can be mounted to the mounting mechanism <NUM>, such that the detection mechanism <NUM> traverses over a geographic location before the treatment mechanism <NUM> traverses over the geographic location. In one variation of the farming machine <NUM>, the detection mechanism <NUM> is statically mounted to the mounting mechanism <NUM> proximal the treatment mechanism <NUM>. In variants including a verification mechanism <NUM>, the verification mechanism <NUM> is arranged distal the detection mechanism <NUM>, with the treatment mechanism <NUM> arranged there between, such that the verification mechanism <NUM> traverses over the geographic location after treatment mechanism <NUM> traversal. However, the mounting mechanism <NUM> can retain the relative positions of the system components in any other suitable configuration. In other systems, the detection mechanism <NUM> can be incorporated into any other component of the farming machine <NUM>.

In some configurations, the farming machine <NUM> can additionally include a verification mechanism <NUM> that functions to record a measurement of the ambient environment of the farming machine <NUM>, which is used to verify or determine the extent of plant treatment. The verification mechanism <NUM> records a measurement of the geographic area previously measured by the detection mechanism <NUM>. The verification mechanism <NUM> records a measurement of the geographic region encompassing the plant treated by the treatment mechanism <NUM>. The verification mechanism measurement can additionally be used to empirically determine (e.g., calibrate) treatment mechanism operation parameters to obtain the desired treatment effect. The verification mechanism <NUM> can be substantially similar (e.g., be the same type of mechanism as) the detection mechanism <NUM>, or be different from the detection mechanism <NUM>. The verification mechanism <NUM> can be a multispectral camera, a stereo camera, a CCD camera, a single lens camera, a CMOS sensor, a hyperspectral imaging system, LIDAR system (light detection and ranging system), dynamometer, IR camera, thermal camera, humidity sensor, light sensor, temperature sensor, or any other suitable sensor. In other configurations of the farming machine <NUM>, the verification mechanism <NUM> can be included in other components of the system.

In some configurations, the farming machine <NUM> can additionally include a power source, which functions to power the system components, including the detection mechanism <NUM>, control system <NUM>, and treatment mechanism <NUM>. The power source can be mounted to the mounting mechanism <NUM>, can be removably coupled to the mounting mechanism <NUM>, or can be separate from the system (e.g., located on the drive mechanism). The power source can be a rechargeable power source (e.g., a set of rechargeable batteries), an energy harvesting power source (e.g., a solar system), a fuel consuming power source (e.g., a set of fuel cells or an internal combustion system), or any other suitable power source. In other configurations, the power source can be incorporated into any other component of the farming machine <NUM>.

In some configurations, the farming machine <NUM> can additionally include a communication apparatus, which functions to communicate (e.g., send and/or receive) data between the control system <NUM> and a set of remote devices. The communication apparatus can be a Wi-Fi communication system, a cellular communication system, a short-range communication system (e.g., Bluetooth, NFC, etc.), or any other suitable communication system.

A farming machine <NUM> obtains images of a field using a detection mechanism <NUM> as the farming machine <NUM> travels through the field. Each obtained image includes information that represents various features and objects in the field. For example, an image can include information representing a plant, a plant treatment, soil, field conditions, dust, particulates, etc. The information can include color, shapes, sizes, metadata of the image, detection mechanism characteristics, pixel information, etc. The control system <NUM> can access and process the image to determine the features and objects in the field using the information included in the image. Based on the determined features and objects, the farming machine <NUM> can execute various actions (e.g., a plant treatment) as described herein.

<FIG> is an illustration of an image accessed by the control system (i.e., accessed image), according to one example embodiment. The accessed image <NUM> is obtained by a detection mechanism <NUM> coupled to a farming machine <NUM> as the farming machine <NUM> travels through a field. The accessed image <NUM> is obtained in ideal operating conditions (e.g., a bright, sunny day). The accessed image <NUM> includes information representing a single plant of a first type <NUM>, three plants a second type <NUM>, and soil <NUM> in the field.

<FIG> is an illustration of a labelled image, according to one example embodiment. A labelled image is an accessed image after it has been processed by the control system to label features and objects. In this example, the labelled image <NUM> is the accessed image <NUM> of <FIG> after the control system <NUM> labels its pixels using a machine learning model. As illustrated, the labelled image <NUM> includes pixels the control system <NUM> identifies as representing the first type of plant <NUM>, the second type of plant <NUM>, and the soil <NUM>. Pixels labelled as the first type of plant <NUM> are illustrated with a dotted fill, pixels labelled as the second type of plant <NUM> are illustrated with a hatched fill, and pixels labelled as soil <NUM> are illustrated with no fill. In some examples, the control system <NUM> may not label pixels representing soil and only label pixels representing plant matter ("plant pixels").

Together, <FIG> illustrate an example process of a control system <NUM> labelling pixels of an accessed image <NUM> obtained in ideal operating conditions. However, in some instances, a farming machine <NUM> operates in operating conditions that are non-ideal for correctly identifying and treating plants. For example, some non-ideal operating conditions may include: inclement weather, low light conditions (e.g., sunrise and/or sunset), night conditions where external illumination is required, dusty conditions, windy conditions, etc. More generally, non-ideal operating conditions are operating conditions that affect the precision and accuracy of the plant identification model implemented by the control system <NUM>. For example, consider a control system <NUM> executing a pixelwise semantic segmentation model configured to identify various types of plants in the field using accessed images ("plant identification model"). In this case, the plant identification model is trained using previously labelled images obtained by a farming machine operating in a field during ideal operating conditions. Accordingly, the plant identification model is more accurate and precise at identifying plants when accessed images are obtained during similar ideal operating conditions, and, conversely, less accurate and precise when identifying plants when the accessed images are obtained during non-ideal operating conditions (e.g., windy).

To demonstrate, <FIG> illustrate an example process of a control system <NUM> labelling pixels of an accessed image obtained in non-ideal operating conditions. <FIG> is an illustration of an accessed image, according to one example embodiment. The accessed image <NUM> is obtained by a detection mechanism <NUM> coupled to a farming machine <NUM> as the farming machine <NUM> travels through a field. The accessed image <NUM> includes the same area of field as the accessed image <NUM> and, under ideal conditions, should include the same information. However, in this example, the farming machine <NUM> is operating in non-ideal operating conditions (e.g., windy) and the accessed image <NUM> includes information representing airborne particulates <NUM>. The airborne particulates <NUM> are illustrated as small cloud like shapes, however, in other accessed images, airborne particulates may not take a defined shape. The accessed image <NUM> also includes information representing a single plant of a first type <NUM>, two plants a second type <NUM>, and soil <NUM> in the field. Notably, portions of the plant of the first type <NUM>, a portion of a plant of the second type <NUM>, and an entire plant of the second type (referring to accessed image <NUM>) are obscured from view by the airborne particulates <NUM>. In other examples, the airborne particulates may cause objects (e.g., plants) in an accessed image to appear fuzzy, out of focus, darker, striated, etc..

<FIG> is an illustration of a labelled image, according to one example embodiment. In this example, the labelled image <NUM> is the accessed image <NUM> of <FIG> after the control system <NUM> labels its pixels using a plant identification model. Here, because the plant identification model is configured to identify plants in ideal operating conditions, many of the pixels in the labelled image <NUM> are mislabeled or unlabeled. As illustrated, the labelled image <NUM> includes pixels the control system <NUM> identifies as representing the second type of plant <NUM> (again illustrated with a dotted fill), but for only one of the second type of plants. The labelled image <NUM> does not include pixels labelled as the first type of plant, nor all the pixels that should be labelled as the second type. The control system <NUM> mislabels many of the pixels because the airborne particulates <NUM> are obscuring some portion (or all) of the plant matter in the accessed image <NUM>. In other words, a pixel, or group of pixels, representing plant matter in the accessed image <NUM> is obscured by a pixel, or group of pixels, representing airborne particulates. For convenience, herein, the pixels are defined as "obscuring pixels. " Obscuring pixels obscure objects of interest (e.g., plants) in an accessed image and, thereby, increase the difficulty of accurately and precisely identifying plants in the image.

A control system <NUM> uses labelled images to treat plants in the field. To do so, the control system <NUM> generates a mapped image from the labelled image, and creates a treatment map based on the mapped image. A mapped image maps a labelled image to a real-world area of a field where the image was obtained. A treatment map is a mapped image in which regions (i.e., pixel groups) in the mapped image correspond to treatment areas <NUM> of treatment mechanisms <NUM> of a farming machine <NUM>. The farming machine <NUM> actuates treatment mechanisms <NUM> when its corresponding treatment area <NUM> in a treatment map includes an appropriate object (e.g., a plant).

<FIG> illustrate a mapped image and a treatment map. For convenience, the mapped image and treatment map are generated by a farming machine <NUM> operating in ideal operating conditions. However, the process of generating a mapped image and treatment map are similar in non-ideal operating conditions.

<FIG> is an illustration of a mapped image, according to one example embodiment. The control system <NUM> identifies and labels groups of pixels as the first type of plant <NUM> and the second type of plant <NUM> using a plant identification model. The control system <NUM> then generates the mapped image <NUM> by mapping the labelled image to a real-world area of the field. In the illustrated image, each treatment area <NUM> of a treatment mechanism <NUM> corresponds to a region of pixels in the mapped image <NUM>. The treatment areas <NUM> are represented in the mapped image as small rectangles, but could be other shapes and sizes according to the configuration of the treatment mechanisms <NUM>.

<FIG> is an illustration of a treatment map, according to one example embodiment. In this example, the treatment map <NUM> is generated from the mapped image <NUM> in <FIG>. Here, the control system <NUM> is configured to generate a treatment map that treats plants of the first type <NUM> as the farming machine <NUM> travels past the identified plants in the field. The treatment map <NUM> includes several map elements <NUM>, with each map element corresponding to one or more treatment areas in the mapped image <NUM>. Each map element is also associated with the treatment mechanism <NUM> corresponding to the treatment areas <NUM> of the map elements <NUM>. For example, the center two map elements <NUM> of the treatment map <NUM> correspond to the center two treatment areas <NUM> in the mapped image <NUM>. Two of the map elements <NUM> in the treatment map <NUM> are selected map elements <NUM>. Selected map elements <NUM> are map elements <NUM> corresponding to treatment areas <NUM> in the mapped image <NUM> including pixels identified, for example, as the first type of plant <NUM>. The farming machine <NUM> applies a treatment to treatment areas <NUM> corresponding to the selected map elements <NUM> with their associated treatment mechanism <NUM>. For example, the farming machine applies a treatment to the first plant type <NUM> using the treatment mechanisms <NUM> associated with the selected map elements <NUM>.

A more detailed description of identifying plants in an accessed image, creating a mapped image, and generating a treatment map may be found in <CIT>.

There are several methods to identify plants in an accessed image. Pixelwise semantic segmentation is a general identification method that, when applied to the problem of identifying plants, may be faster and more accurate than other plant identification methods. Pixelwise semantic segmentation models are a subset of deep learning methods that operate on fully convolutional encoder-decoder network. Additionally, a pixelwise semantic segmentation model may be configured to identify plants in both ideal and non-ideal operating conditions. For example, a pixelwise semantic segmentation model can be trained to identify plants using accessed images which include pixels representing both plant matter and airborne particulates. Further, the pixelwise semantic segmentation model may be trained to identify a particulate level in an accessed image.

Semantic segmentation may be implemented by a control system <NUM> using a plant identification model. A farming machine <NUM> can execute the plant identification model to identify features (e.g., plants, particulates, soil, etc.) in an accessed image (e.g., accessed image <NUM>) and quickly generate an accurate treatment map. <FIG> is a representation of a plant identification model based on accessed images and previously identified plants, according to one example embodiment. As described in greater detail below, the plant identification model can identify plants in both ideal and non-ideal operating conditions. The previously identified plants may have been identified by another plant identification model or a human identifier.

In the illustrated embodiment, referred to throughout the remainder of the specification, the plant identification model <NUM> is a convolutional neural network model with layers of nodes, in which values at nodes of a current layer are a transformation of values at nodes of a previous layer. A transformation in the model <NUM> is determined through a set of weights and parameters connecting the current layer and the previous layer. For example, as shown in <FIG>, the example model <NUM> includes five layers of nodes: layers <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>. The control system <NUM> applies the function W<NUM> to transform from layer <NUM> to layer <NUM>, applies the function W<NUM> to transform from layer <NUM> to layer <NUM>, applies the function W<NUM> to transform from layer <NUM> to layer <NUM>, and applies the function W<NUM> to transform from layer <NUM> to layer <NUM>. In some examples, the transformation can also be determined through a set of weights and parameters used to transform between previous layers in the model. For example, the transformation W<NUM> from layer <NUM> to layer <NUM> can be based on parameters used to accomplish the transformation W<NUM> from layer <NUM> to <NUM>.

The control system <NUM> inputs an accessed image <NUM> (or a mapped image <NUM>) to the model <NUM> and encodes the image onto the convolutional layer <NUM>. After processing by the control system <NUM>, the model <NUM> outputs a treatment map <NUM> decoded from the output layer <NUM>. The control system <NUM> employs the model <NUM> to identify latent information in the accessed image <NUM> representing plants and obscuring pixels in the identification layer <NUM>. The control system <NUM> reduces of the dimensionality of the convolutional layer <NUM> to that of the identification layer <NUM> to identify the plants and particulates, and then increases the dimensionality of the identification layer <NUM> to generate a treatment map <NUM>.

As previously described, the control system <NUM> encodes the accessed image <NUM> to a convolutional layer <NUM>. In one example, accessed image is directly encoded to the convolutional layer <NUM> because the dimensionality of the convolutional layer <NUM> is the same as a pixel dimensionality (e.g., number of pixels) of the accessed image <NUM>. In other examples, the accessed image <NUM> can be adjusted such that the pixel dimensionality of the accessed image <NUM> is the same as the dimensionality of the convolutional layer <NUM>. For example, the accessed image may be cropped, reduced, scaled, etc..

The control system <NUM> applies the model <NUM> to relate accessed images <NUM> in the convolutional layer <NUM> to plant and particulate identification information in the identification layer <NUM>. The control system <NUM> retrieves relevance information between these elements can by applying a set of transformations (e.g., Wi, W<NUM>, etc.) between the corresponding layers. Continuing with the example from <FIG>, the convolutional layer <NUM> of the model <NUM> represents an encoded accessed image <NUM>, and identification layer <NUM> of the model <NUM> represents plant and particulate identification information. The control system <NUM> identifies plants and particulates in an accessed image <NUM> by applying the transformations W<NUM> and W<NUM> to the pixel values of the accessed image <NUM> in the space of convolutional layer <NUM>. The weights and parameters for the transformations may indicate relationships between information contained in the accessed image and the identification of a plant and/or particulates. For example, the weights and parameters can be a quantization of shapes, colors, obscuration, etc. included in information representing plants and/or particulates included in an accessed image <NUM>. The control system <NUM> may learn the weights and parameters using historical user interaction data. For example, the control system can access plant identification submitted by users and particulate information created by a dust generation model.

In one example, the weights and parameters for plant and particulate identification can be collected and trained using data collected from previously accessed images <NUM> and a labelling process. The labelling process reduces the amount of time required by the control system <NUM> employing the model <NUM> to identify pixels representing plants while also increasing the accuracy of identified pixels. The labelling process can include: identifying a bounding box including pixels likely to represent a plant, identifying green pixels within the bounding boxes, identifying a contour around the identified green pixels, using the contours to create a label set for the accessed image, and sending the labelled image to users for verification. Users verify labels by identifying which pixels are green near the contours in an accessed image rather than an entire image. In effect, this "pre-identification" by model <NUM> reduces the number of pixels needed to be verified by a human and reduces the costs of training the semantic segmentation model <NUM>.

In another example, the weights and parameters for plant and particulate identification can be collected and trained using data collected from a particulate augmentation model. An example particulate augmentation model is described in more detail below.

Plants in the accessed image <NUM> are identified in the identification layer <NUM>. A particulate level (or airborne particulates) in the accessed image is also identified in the identification layer <NUM>. The identification layer <NUM> is a data structure representing identified plants and particulate levels based on the latent information about the plants and particulates represented in the accessed image <NUM>.

As described previously, identified plants in an accessed image <NUM> can be used to generate a treatment map <NUM>. To generate a treatment map <NUM>, the control system <NUM> employs the model <NUM> and applies the transformations W<NUM> and W<NUM> to the values of the identified plants and particulate level in identification layer <NUM>. The transformations result in a set of nodes in the output layer <NUM>. The weights and parameters for the transformations may indicate relationships between identified plants and a treatment map <NUM>. In some cases, the control system <NUM> directly outputs a plant treatment map <NUM> from the nodes of the output layer <NUM>, while in other cases the control system <NUM> decodes the nodes of the output layer <NUM> into a treatment map <NUM>. That is, model <NUM> can include a conversion layer (not illustrated) that converts the output layer <NUM> to a treatment map <NUM>.

To illustrate conversion, take, for example, a control system <NUM> employing model <NUM> that produces an output image at output layer <NUM>. The output image may include an arbitrary set of pixels marked to indicate the presence of a plant. The control system <NUM> generates a treatment map <NUM> by employing a conversion layer that converts the output image into a set of treatment areas <NUM> corresponding to treatment mechanisms <NUM> of the farming machine <NUM> (i.e., a treatment map). Other methods of a conversion layer generating a treatment map <NUM> are also possible.

Here, the control system <NUM> outputs a treatment map in a manner dissimilar to that of a traditional encoder/decoder scheme. Generally, the output layer <NUM> of a traditional convolutional neural network has the same, or nearly the same, dimensionality as the convolutional layer <NUM>. That is, given the example of an accessed image <NUM> as an input, the convolutional layer <NUM> and the output layer <NUM> both have the pixel dimensionality. This allows a convolutional neural network to identify objects in an accessed image <NUM> and map them back to the original input accessed image <NUM>. Traditionally, decoding objects identified in the identification layer <NUM> to an image having the same dimensionality as the convolutional layer <NUM> is computationally expensive and requires additional processing time. In this example, the dimensionality of the treatment map <NUM> is the treatment dimensionality (e.g., the number of treatment mechanisms) of a farming machine <NUM>. Generally, the treatment dimensionality (e.g., <NUM>× <NUM> treatment mechanisms) is much less than the pixel dimensionality (e.g., <NUM> × <NUM> pixels) and, therefore, decoding the identified plants in the identification layer <NUM> to a treatment map requires fewer transformations and less processing time.

Identified particulates (or particulate level) can be used to generate a particulate level notification. Similarly, to above, the control system employs model <NUM> and applies transformations W<NUM> and W<NUM> to the values of identified particulates and the corresponding particulate level in identification layer <NUM>. In this case, the transformation may result a particulate level for the accessed image <NUM>. For example, the particulate level may be "high" if a threshold proportion (or number) of pixels in the image are identified as obscuring pixels. In another example, the particulate level may be "<NUM>%" if the model <NUM> employs a particulate augmentation image including a particulate level of <NUM>%. In another example, applying transformations W<NUM> (and/or W<NUM>) may not be necessary depending on the desired resolution. To illustrate, it may not be necessary to determine a particulate level for every pixel in an image. In another example, the model may include a convolutional layer (i.e., conv <NUM> × <NUM>) configured to reduce the dimensionality of the image to, for example, N and classify the particulate level of the whole image rather than individual pixels. To illustrate, the convolutional layer reduces the image to a dimensionality of <NUM>. In this case, the convolutional layer determines that the particulate level is <NUM>%, <NUM>%, <NUM>%, etc. Determining particulate levels is described in greater detail below.

Additionally, the model <NUM> can include layers known as intermediate layers. Intermediate layers are those that do not correspond to an accessed image <NUM>, plant identification, or a treatment map <NUM>. For example, as shown in <FIG>, layers <NUM> are intermediate encoder layers between the convolutional layer <NUM> and the identification layer <NUM>. Layer <NUM> is an intermediate decoder layer between the identification layer <NUM> and the output layer <NUM>. Hidden layers are latent representations of different aspects of a plant that are not observed in the data, but may govern the relationships between the elements of an image when identifying a plant. For example, a node in the hidden layer may have strong connections (e.g., large weight values) to input values and identification values that share the commonality of "green leaves. " As another example, another node in the hidden layer may have strong connections to input values and identification values that share the commonality of "airborne particulate in front of a plant. " Specifically, in the example model of <FIG>, nodes of the hidden layers <NUM> and <NUM> can link inherent information in the accessed image that share common characteristics to help determine if that information represents a plant in the accessed image.

Additionally, each intermediate layer is a combination of functions such as, for example, residual blocks, convolutional layers, pooling operations, skip connections, concatenations, etc. Any number of intermediate encoder layers <NUM> can function to reduce the convolutional layer to the identification layer and any number of intermediate decoder layers <NUM> can function to increase the identification layer <NUM> to the output layer <NUM>. Alternatively stated, the encoder intermediate layers reduce the pixel dimensionality to the identification dimensionality, and the decoder intermediate layers increase the identification dimensionality to the treatment dimensionality.

Furthermore, in various embodiments, control system <NUM> employs the functions of model <NUM> to reduce the accessed image <NUM> and identify any number of objects in a field. The identified objects are represented in the identification layer <NUM> as a data structure. In various examples, the model can be configured to identify a location of a plant, a condition of a plant, a type of a plant, a category of a plant (e.g., a weed or a crop), or any other characteristics of a plant in the field. In various other embodiments, the identification layer can identify latent information representing other objects in the accessed image. For example, the identification layer <NUM> can identify a result of a plant treatment, soil, an obstruction, or any other object in the field.

A farming machine <NUM> operating in a field employs a plant identification model (e.g., model <NUM>) to identify plants and/or a particulate level in the field using accessed images of the field obtained by the farming machine. The model <NUM> is configured to identify plants and/or particulate levels in accessed images obtained by the farming machine <NUM> in both ideal and non-ideal operating conditions.

To do identify and treat plants, a control system <NUM> inputs an accessed image <NUM> into the model <NUM> and generates a treatment map (e.g., treatment map <NUM>). The treatment map <NUM> includes selected map elements <NUM> for treatment by treatment mechanisms <NUM> of the farming machine <NUM>. The selected map elements <NUM> are treatment areas <NUM> in the accessed image <NUM> which include an identified plant. The farming machine <NUM> treats the identified plants by actuating treatment mechanisms <NUM> at the appropriate time as the farming machine travels past the plant in the field. Additionally, the control system <NUM> employs the model <NUM> to determine a particulate level in the field where the farming machine is operating. Based on the determined particulate level, the farming machine <NUM> provides feedback to an operator regarding the determined particulate level.

<FIG> and <FIG> illustrate a specific example of a farming machine <NUM> identifying and treating plants using plant treatment mechanisms <NUM> as the farming machine <NUM> travels through the field. The farming machine <NUM> is operating in non-ideal operating conditions and airborne particulates are present in the field. The farming machine <NUM> determines a particulate level for the field and generates a notification based on a determined particulate level. In this example, the farming machine <NUM> is a crop sprayer operating in a field planted with cotton (e.g., first plant type <NUM>). The farming machine is configured to identify weeds (e.g. second plant type <NUM>) in the field and treat the identified weeds by spraying them with an herbicide. The farming machine <NUM> is configured with a single row of eight spray nozzles that serve as treatment mechanisms <NUM>. That is, the spray nozzles spray an herbicide when actuated by the farming machine <NUM>. The farming machine <NUM> includes a detection mechanism <NUM> that captures images of plants and airborne particulates in the field as the farming machine <NUM> travels down the cotton crop rows. Further, the farming machine <NUM> includes a control system <NUM> that identifies plants and particulates in the field and controls the spray nozzles.

<FIG> is a flow diagram illustrating a method for the identification and treatment of weeds in a cotton field despite the presence of airborne particulates in the cotton field, according to one example embodiment. The method <NUM> can include additional or fewer steps and the illustrated steps may be accomplished in any order. In some cases, steps may be repeated any number of times before progressing to a subsequent step.

The farming machine <NUM> images an area of the field using the detection mechanism <NUM>. The image includes information representing cotton plants, weed plants, airborne particulates, and soil in the field. In this example, the detection mechanism <NUM> is mounted to the front of the farming machine <NUM> such that the area of the field is imaged before the front end of the farming machine <NUM> passes over the area. The detection mechanism <NUM> transmits the image to the control system <NUM> of the farming machine <NUM>.

The control system <NUM> accesses <NUM> the image, and the information included therein, from the detection mechanism <NUM>. <FIG> is an illustration of an accessed image, according to one example embodiment. The accessed image <NUM> includes a cotton plant <NUM> near the center of the image and several weeds <NUM> spaced about the cotton plant <NUM>. The accessed image <NUM> also includes airborne particulates <NUM> which are illustrated as, for example, small clouds. For convenience, the treatment areas (e.g., treatment area <NUM>) of the treatment mechanisms for the farming machine <NUM> are overlaid on the accessed image <NUM>. In this case the treatment mechanisms include an array of spray nozzles <NUM> and the treatment areas for the spray nozzles are spray areas <NUM>. Spray nozzles <NUM> labeled <NUM> to <NUM> are pictured above the accessed image <NUM> and correspond to the spray areas <NUM> in the columns below them.

In this example, the spray areas <NUM> are shown for eight different treatments of the spray nozzles <NUM> from time ti through ts. Thus, each row of illustrated spray areas <NUM> corresponds to one treatment of the spray areas <NUM> in that row by the farming machine <NUM> at a time. In other words, over time, as the farming machine <NUM> moves through the field, the spray nozzles <NUM> and their corresponding spray areas <NUM> move down the accessed image <NUM> and the farming machine <NUM> treat weeds <NUM> when appropriate.

Returning to <FIG>, the control system <NUM> applies <NUM> a model (e.g., model <NUM>) to identify pixels in the accessed image <NUM> representing plants (e.g., cotton plant <NUM>, and/or weeds <NUM>) and/or airborne particulates <NUM>. The control system <NUM> encodes the accessed image <NUM> onto a convolutional layer <NUM> of the model <NUM>. Each pixel of the accessed image <NUM> corresponds to a single node of the convolutional layer <NUM>. The control system <NUM> applies a set of transformations to the convolutional layer <NUM> to identify plants (e.g., cotton plant <NUM>, and weeds <NUM>) and/or airborne particulates <NUM> in the accessed image <NUM>. Each transformation identifies latent information in the pixels that represents the plants and/or particulates.

In some configurations, the control system <NUM> identifies one or more of the pixels as occluding pixels, i.e., a pixel representing an airborne particulate <NUM> that occludes a plant in the accessed image <NUM>. In an embodiment, the control system <NUM> may identify the occluding pixel as a pixel representing the occluded plant. For example, referring to <FIG>, a group of pixels representing the cotton plant <NUM> may be at least partially occluded. Some occluded pixels <NUM> are highlighted near the bottom right of the cotton plant <NUM> where some airborne particulates <NUM> are illustrated as overlapping a portion of the cotton plant <NUM>.

While identifying plants and/or airborne particulates, control system <NUM> applies various functions and transformations that reduce the dimensionality of convolutional layer <NUM>. The transformations reduce the convolutional layer <NUM> to the identification layer <NUM>. Generally, the dimensionality of the identification is much smaller than the convolutional layer <NUM>. In this example, the identification layer <NUM> is configured as a data structure including identified weeds, cotton, and airborne particulates.

The control system <NUM> calculates <NUM> a particulate level in the accessed image <NUM>. In this example, the control system determines that the particulate level is high because a ratio of a number pixels representing airborne particulates <NUM> to the total pixels in an accessed image <NUM> is greater than a threshold. In another example, the control system <NUM> determines a particulate level based on a particulate threshold pixel level. For example, if a threshold number of pixels have a particulate level above a particulate threshold level, the control system determines the particulate level for the accessed image is a ratio many pixels representing airborne particulates <NUM> to a total number of pixels in an accessed image <NUM>. In another example, the control system determines a particulate level for an accessed image by averaging a particulate value (e.g., alpha-blend level) for each pixel in the accessed image. Other methods to determine a particulate level are also possible, several of which are described herein.

The control system <NUM> generates a notification for the operator of the farming machine <NUM> indicating the particulate level in the field. For some particulate levels, the control system <NUM> can provide a recommendation to the operator to reduce the particulate level. For example, the control system <NUM> may recommend that the operator reduce the speed of the farming machine to decrease the particulate level.

The control system <NUM> generates <NUM> a treatment map (e.g., treatment map <NUM>). To generate the treatment map, the control system applies a set of transformations to the identification layer <NUM>. Each transformation increases the dimensionality of the identification layer <NUM> and decodes the identified plants (and/or airborne particulates) to a new dimensionality. The set of transformations increases the dimensionality of the identification layer to that of the output layer <NUM>.

In this example, the output layer <NUM> has the same dimensionality as the treatment dimensionality (e.g., an output layer of <NUM> × <NUM> nodes corresponding to a <NUM> × <NUM> arrays of treatment mechanisms) such that the output layer <NUM> can represent a treatment map. That is, each node of the output layer <NUM> corresponds to a map element (e.g., map element <NUM>) of a treatment map and, thereby, a spray area <NUM> of a spray nozzle <NUM> of the farming machine <NUM>. Further, each node also includes information regarding objects identified in that spray area <NUM> (e.g., a weed <NUM>, or cotton <NUM>). As such, the control system <NUM> decodes the nodes of the output layer into a treatment map. In this example, the control system <NUM> generates a treatment map indicating that for each map element of a treatment map including an identified weed, the farming machine <NUM> actuates the corresponding spray nozzle <NUM> to spray herbicide in the appropriate spray area <NUM>.

<FIG> is an illustration of a treatment map, according to one example embodiment. The treatment map <NUM> includes map elements <NUM> corresponding to similarly positioned spray areas <NUM> of <FIG>. The treatment map <NUM> also includes selected map elements <NUM> which correspond to spray areas <NUM> that included information representing a weed <NUM> in the accessed image <NUM>. Selected map elements <NUM> will be treated by the farming machine <NUM> at the appropriate time as it travels through the field. Notably, the control system <NUM> generates an accurate treatment map for the weeds <NUM> despite the presence of airborne particulates <NUM> in the accessed image <NUM>.

Returning to <FIG>, the farming machine <NUM> actuates <NUM> spray nozzles <NUM> to spray the spray areas <NUM> and treat the weeds <NUM> as the farming machine <NUM> travels through the field. To do so, the control system <NUM> generates control signals for the spray nozzles <NUM> and actuates the spray nozzles <NUM> corresponding to selected map elements <NUM> areas in the treatment map <NUM> at the appropriate time.

For example, turning to <FIG>, at time ti the treatment map <NUM> indicates a selected map element <NUM> in the <NUM>th column because the corresponding spray area <NUM> for the <NUM>th spray nozzle <NUM> at time ti included pixels representing a weed <NUM>. As such, the farming machine <NUM> actuates the <NUM>th spray nozzle <NUM> at time ti to treat the weed <NUM>in the corresponding spray area <NUM> as the farming machine <NUM> travels through the field.

The process continues as the farming machine <NUM> travels through the field. As the farming machine <NUM> moves, the model <NUM> generates the appropriate treatment maps <NUM> to spray herbicide on weeds <NUM> and not cotton <NUM>. For example, continuing from time t<NUM>, the farming machine <NUM> actuates the <NUM>th treatment mechanism <NUM> at time t<NUM>, the <NUM>rd treatment mechanism <NUM> at time t<NUM>, the <NUM>st and <NUM>nd treatment mechanism <NUM> at time t<NUM>, and the <NUM>st and <NUM>nd treatment mechanism at time t<NUM>.

As described above, in non-ideal operating conditions a control system <NUM> may misidentify plants in an accessed image. For example, airborne particulates may obscure a plant such that their corresponding pixels in an accessed image are obscuring pixels. If a control system <NUM> employs a plant identification model configured to identify plants in ideal operating conditions, obscuring pixels may affect the accuracy and/or precision of the control system <NUM> identifying plants. Therefore, a control system <NUM> employing a plant identification model configured to identify a plant in non-ideal operating conditions is beneficial to establish more reliable plant identification in a wider range of operating conditions. For example, a plant identification model configured to identify plants in the presence of airborne particulates would allow a farming machine to operate at higher rates of travel (which generates more airborne particulates), during windy conditions, etc. However, training a plant identification model to identify plants in non-ideal operating conditions is a challenging problem. Herein, for convenience, training the plant identification model is described as occurring on the control system <NUM>. However, in various embodiments, a system other than the control system may train the plant identification model such that it may be implemented on the control system <NUM>.

In an example, a plant identification model can be trained to identify plants in non-ideal operating conditions using accessed images obtained by a farming machine in non-ideal operating conditions that are subsequently labelled by a human. For example, a farming machine may obtain images of plants in a field while it is windy such that there are airborne particulates obscuring plants in the images. A human then labels plants in the image (even if they are obscured) and those labelled images are used to train a plant identification model. However, this approach is problematic for several interrelated reasons: (i) each non-ideal operating condition necessitates additional training for a plant identification model, (ii) human labelling of images for training a plant identification model is expensive, and (iii) humans may mislabel plants in the presence of obscuring pixels, which, in turn, may lead to poor plant classification outputs from a plant identification model. Take, for example, airborne particulates in an image due to windy conditions. The number of airborne particulates in an image may vary greatly depending on the amount of wind (e.g., a breeze, or a gale). In these circumstances, a plant identification model trained using labelled images in slightly windy operating conditions may not be able to accurately identify plants in very windy operating conditions. As such, a wide range of training images are required to train a plant identification for the varied wind conditions a farming machine might experience. Obtaining such a vast number of images and subsequently labelling them to train a plant identification model is incredibly expensive and impractical.

In another example, a plant identification model can be trained to identify plants in non-ideal operating conditions using accessed images obtained by a farming machine in ideal operating conditions, labelled by a human, and subsequently augmented with a simulation of non-ideal operating conditions. For example, a farming machine may obtain images of plants in a field on a bright, sunny day where there are few airborne particulates obscuring the plants in the images. A human then labels the plants in the image. A computer (e.g., control system <NUM>) employs a particulate augmentation model to generate an augmented image. An augmented image is a labelled image in which a non-ideal operating condition is simulated. For example, the control system <NUM> may simulate windy conditions by introducing simulated airborne particulates to a labelled image. The augmented image is used to train a plant identification model to identify plants in non-ideal operating conditions. Beneficially, the control system <NUM> may simulate a variety of windy conditions for the same accessed image. For example, the control system <NUM> may simulate airborne particulates for non-ideal operating conditions from, for example, slightly breezy to gale force winds. This example approach provides a more cost-effective method to train a plant identification model in non-ideal operating conditions.

There are several methods of simulating non-ideal operating conditions in an accessed image obtained by a farming machine <NUM> operating in an ideal operating condition. In one example, a control system <NUM> simulates airborne particulates in a labelled image by employing a particulate augmentation model. The particulate augmentation model simulates airborne particulates in a labelled image by generating a digital representation of airborne particulates ("particulate image") and overlaying the particulate image on a labelled image. In this manner, one or more previously accessed images may be used to train a plant identification model to identify plants in non-ideal conditions. That is, the particulate augmentation model generates an array of particulate images from a previously labelled image such that plant matter generated images are correctly labelled despite being obscured by (simulated) particulates.

To generate a particulate image, in an example, the control system <NUM> generates a randomly sized array with each element in the array assigned a value between <NUM> and <NUM>. The values for each cell may be assigned randomly, pseudo-randomly, or according to other methods. The control system <NUM> resizes the array to a similar dimensionality as a labelled image. For example, if the generated array has a dimensionality of <NUM> × <NUM>, and the labelled image has a dimensionality of <NUM> × <NUM>, the control system <NUM> rescales the array to the dimensionality of <NUM> × <NUM>.

The control system <NUM> may rescale the array using a variety of methods. For example, the control system <NUM> may apply a scaling function to the array to modify its dimensionality. Some example scaling functions include, a bilinear area scalar function, a linear scalar function, a polynomial scaling function, etc. The resulting array ("scaled array") approximates airborne particulates that may occur in accessed image obtained by a farming machine operating in non-ideal operating conditions. <FIG> is an illustration of a scaled array, according to one example embodiment. In this example, the scaled array <NUM> is generated from a <NUM> × <NUM> array that is scaled to <NUM> × <NUM> using a bilinear area scalar function. In the scaled array <NUM>, the color of each cell (i.e., pixel) reflects its corresponding value, with white colored cells having a value of <NUM> and black colored cells having a value of <NUM>, and grey colored cells having a value between <NUM> and <NUM>. That is, there is a total of <NUM> cell colors.

The control system <NUM> determines a particulate color for the particulate image. In one example, the control system <NUM> determines the particulate color as an average color of the accessed image. In other words, the control system determines an average value, for each of the RGB channels for all the pixels in a labelled image and the corresponding group of three averaged channel values is the particulate color. In other examples, the control system <NUM> may determine the particulate color using other methods. For example, the control system may access a previously generated particulate color from a datastore, receive a particulate color from a user of the control system <NUM>, etc. The control system <NUM> applies the particulate color to the scaled array. The control system applies the particulate color such that cells with the value <NUM> are the particulate color and cells with the value <NUM> have no color. Cells with values between <NUM> and <NUM> are scaled in transparency according to their value. For example, a cell value of <NUM> indicates full transparency while a cell value of <NUM> indicates fully opaque.

The control system <NUM> generates a particulate image, or particulate images, using the scaled array, the labelled image, and a particulate level. The particulate level is a quantification of how many airborne particulates to include in the particulate image. In an example, the particulate level is a value between <NUM> and <NUM>, with <NUM> indicating no airborne particulates and <NUM> indicating many airborne particulates. The control system may generate a particulate image PI according to the function: <MAT> where PC is the <NUM>-dimensional particulate color array, p is the particulate level, SA is the scaled array, TC is the total cell color (e.g., <NUM>), and LI is the labelled image. Notably, in this example, the control system <NUM> generates a particulate image similarly to an alpha-blend between the scaled array and labelled image, with the amount of blending corresponding to the particulate level.

<FIG> illustrate a process of generating an array of particulate images for a labelled image according to a particulate level. <FIG> illustrates a labelled image, according to one example embodiment. The labelled image <NUM> includes pixels labelled as a plant of a first type <NUM> and pixels labelled as plants of a second type <NUM>. <FIG> illustrates a representation of a scaled array, according to one example embodiment. In this example, for convenience of illustration, cells in the scaled array <NUM> with a value above a threshold value (e.g., <NUM>) are black, and cells with a value below the threshold are transparent. This allows for a more convenient representation of regions in the scaled array representing airborne particulates (e.g., simulated particulate region <NUM>). Notably, in other examples, a scaled array is more like the scaled array <NUM> shown in <FIG>.

<FIG> illustrate particulate images generated according to various particulate levels. <FIG> illustrate a blend between the labelled image of <FIG> and the scaled array of <FIG>.

<FIG> illustrates a particulate image generated using a first particulate level, according to one example embodiment. In this example, particulate image <NUM> includes a first particulate level of <NUM>. The first particulate level indicates that the particulate image <NUM> is an alpha-blend including none of the scaled array and all the labelled image. Here, the simulated airborne particulates <NUM> are shown as an outline for convenience, despite not actually occurring in the particulate image <NUM>.

<FIG> illustrates a particulate image generated using a second particulate level, according to one example embodiment. In this example, particulate image <NUM> includes a second particulate level of <NUM>. The second particulate level is applied in regions of the particulate image where the scaled array has non-zero values. In areas where the particulate level is applied, the second particulate level indicates that the particulate image <NUM> is an alpha-blend including <NUM> % of the scaled array and <NUM> % of the labelled image. As illustrated, some of the simulated airborne particulates <NUM> begin to occlude plants.

<FIG> illustrates a particulate image generated using a third particulate level, according to one example embodiment. In this example, particulate image <NUM> includes a third particulate level of <NUM>. The third particulate level is applied in regions of the particulate image where the scaled array has non-zero values. In areas where the particulate level is applied, the third particulate level indicates that the particulate image <NUM> is an alpha-blend including <NUM> % of the scaled array and <NUM> % of the labelled image. As illustrated, some of the simulated airborne particulates <NUM> further occlude the plants, though the plants are still visible in the particulate image <NUM>.

<FIG> illustrates a particulate image generated using a fourth particulate level, according to one example embodiment. In this example, particulate image <NUM> includes the fourth particulate level of <NUM>. The fourth particulate level is applied in regions of the particulate image where the scaled array has non-zero values. In areas where the particulate level is applied, the fourth particulate level indicates that the particulate image <NUM> is an alpha-blend including <NUM> % of the scaled array and <NUM> % of the labelled image. As illustrated, of the simulated airborne particulates <NUM> further occlude the plants. In this case, some of the plants are wholly occluded in the particulate image <NUM>.

The approach of modelling various particulate levels for one image, as illustrated in <FIG>, is superior than collecting images with varying levels of obscuration because human labelling of acquired images is costly, time-consuming, and error prone. Of course, while <FIG> only showed <NUM> examples of generated particulate levels (e.g., <NUM>, <NUM>, <NUM>, and <NUM>), many other possible particulate levels are also possible. More generally, the stochastic nature of the technique described herein allows a control system <NUM> (or some other system) to generate a vast number of labelled particulate images. The generated particulate images allow for an inexpensive, non-time consuming, and nearly error-free method of training a particulate augmentation model that more accurately identifies plants in non-ideal operating conditions.

The array of particulate images may be used to train a plant identification model (e.g., model <NUM>) to identify plants in non-ideal operating conditions. Arrays of particulate images are informative because each particulate image corresponds to a previously labelled image. That is, even if the particulate image includes simulated particulates that wholly obscure plant matter (i.e., an obscuring pixel), the obscuring pixel is still labelled as plant matter. In this way, a model can be trained to identify latent information in an image to identify plants when one or more of the pixels representing the plant are obscured pixels. For example, referring to <FIG> and <FIG>, a second type of plant is represented by a group of pixels in the bottom left of the labelled image <NUM>. In the particulate image <NUM>, a portion of the plant is obscured by the simulated particulates. However, the obscuring pixels are still labeled as pixels representing the second type of plant. Because the particulate image is used to train a model, the model identifies latent information in an image representing the plant despite the presence of airborne particulates that obscure all or some portion of the plant.

There are various methods for generating arrays of particulate image for training a plant identification model. In one example, as illustrated above, the control system <NUM> can select a range of particulate levels and generate corresponding particulate images for a labelled image. However, this method may cause the control system <NUM> to generate too many particulate images for training a plant identification model. For example, the control system <NUM> may generate a particulate image for labelled images not including any plants, labelled images already including airborne particulates. Thus, in another example, the control system <NUM> determines a particulate probability for a labelled image and generates particulate images based on the probability. The particulate probability is a quantification of a likelihood that a labelled image includes airborne particulates. The control system can determine a particulate probability based on a variety of factors. For example, the control system <NUM> determines a particulate probability based on characteristics of a labelled image. To provide context, the control system <NUM> determines a particulate probability (e.g., <NUM>%) based on the color distribution of a labelled image (<NUM>% dark brown pixels). The control system <NUM> then selects a particulate level (e.g., <NUM>) based on determined particulate probability (<NUM>%) because the color distribution indicates that the labelled image already includes many airborne particulates and/or includes mainly soil. In another example, the control system <NUM> determines a particulate probability based on a particulate distribution. The particulate distribution is a quantification of previously determined particulate levels in accessed images. For example, the control system <NUM> may determine a particulate level for a number of previously obtained images. The control system then generates a distribution representing the particulate levels in the images. Thus, when the control system generates particulate images, they generated images adhere to previously determined particulate distributions. In these manners, the control system <NUM> generates particulate images that are more informative for training a plant identification model configured to identify plants in non-ideal operating conditions.

A control system <NUM> employing plant identification model trained using particulate images is more precise and accurate at identifying plants in non-ideal operating conditions. For example, <FIG> compare the identification capabilities of a plant identification model that is not trained using particulate images ("normal model") and a plant identification model trained using particulate images ("augmented model").

<FIG> is a recall plot comparing the recall of a normal model and an augmented model in identifying weeds in accessed images, according to one example embodiment. In a recall plot, the y-axis is recall and the x-axis is the particulate level in an accessed image. The recall plot <NUM> includes a line for the normal model <NUM> and a line for the augmented model <NUM>. Each line represents the true positive rate (i.e., recall) of each model when identifying a weed in an accessed image. The normal model <NUM> has a substantially lower recall than the augmented model <NUM> when identifying weeds in accessed images with a particulate level above <NUM>.

<FIG> is a precision plot comparing the precision of a normal model and an augmented model in identifying weeds in accessed images, according to one example embodiment. In a precision plot, the y-axis is precision and the x-axis is the particulate level in an accessed image. The precision plot <NUM> includes a line for the normal model <NUM> and a line for the augmented model <NUM>. Each line represents the positive predictive value (i.e., precision) of each model when identifying a weed in an accessed image. The normal model <NUM> has a comparable precision than the augmented model <NUM> when identifying weeds in accessed images with a particulate level above <NUM>.

<FIG> is a characterization metric plot comparing a characterization metric of a normal model and an augmented model in identifying weeds in accessed images, according to one example embodiment. In a characterization metric plot, the y-axis is characterization metric and the x-axis is the particulate level in an accessed image. The characterization metric plot <NUM> includes a line for the normal model <NUM> and a line for the augmented model <NUM>. Each line represents the harmonic mean (e.g., F1 score) of the positive predictive value and true positive rate of each model when identifying a weed in an accessed image. The normal model has a lower characterization metric than the augmented model when identifying weeds in accessed images with a particulate level above <NUM>.

<FIG> is a recall plot comparing the recall of a normal model and an augmented model in identifying crops in accessed images, according to one example embodiment. The recall plot <NUM> includes a line for the normal model <NUM> and a line <NUM> for the augmented model. The normal model <NUM> has a lower recall than the augmented model <NUM> when identifying crops in accessed images with a particulate level above <NUM>.

<FIG> is a precision plot comparing the precision of a normal model and an augmented model in identifying crops in accessed images, according to one example embodiment. The precision plot <NUM> includes a line <NUM> for the normal model and a line <NUM> for the augmented model. The normal model <NUM> has a comparable precision to the augmented model <NUM> when identifying crops in accessed images for all particulate levels.

<FIG> is a characterization metric plot comparing a characterization metric ability of a normal model and an augmented model in identifying crops in accessed images, according to one example embodiment. The characterization metric plot <NUM> includes a line <NUM> for the normal model and a line <NUM> for the augmented model. The normal model <NUM> has a lower characterization metric than the augmented model <NUM> when identifying crops in accessed images with a particulate level above <NUM>.

A control system <NUM> employing an augmented model is more precise and accurate than a normal model which also leads to improved treatment of identified plants (e.g., less overspray, fewer spray misses, etc.). For example, <FIG> compare the treatment capabilities of a farming machine employing a normal model ("normal machine") vs. a farming machine employing an augmented model ("augmented machine") to treat plants in a field.

<FIG> is an overspray plot comparing the overspray of a normal machine and an augmented machine, according to one example embodiment. Overspray is a quantification of an extra amount of area treated by a farming machine when unnecessary (i.e., the farming machine sprays an area when there is no weed). In the overspray plot <NUM>, the y-axis is the overspray and the x-axis is the particulate level in an accessed image. The overspray plot <NUM> includes a line for the normal machine <NUM> and a line for the augmented machine <NUM>. Each line represents the overspray of each machine when treating identified weeds. The normal machine and the augmented machine have similar overspray for all values of particulate level.

<FIG> is an overspray plot comparing an overspray of a normal machine and an augmented machine, according to one example embodiment. In the overspray plot <NUM>, the overspray is normalized to the number of weeds detected. That is, the overspray plot illustrates an average overspray of identified weeds. In the overspray plot <NUM>, the y-axis is a normalized overspray and the x-axis is the particulate level in an accessed image. The overspray plot <NUM> includes a line for the normal machine <NUM> and a line for the augmented machine <NUM>. Each line represents the overspray of each machine when treating identified weeds.

<FIG> is a missed treatment plot comparing the missed treatments of a normal machine and an augmented machine, according to one example embodiment. Missed treatment is a quantification of an amount of area untreated by a farming machine when necessary (i.e., the farming machine does not spray an area when there is a weed). In the missed treatment plot <NUM>, the y-axis is the missed treatment and the x-axis is the particulate level in an accessed image. The missed treatment plot <NUM> includes a line for the normal machine <NUM> and a line for the augmented machine <NUM>. Each line represents the missed treatment of each machine when treating identified weeds. The augmented machine <NUM> has an appreciably lower missed treatment normal than the normal machine <NUM> for particulate values greater than <NUM>.

<FIG> is a missed treatment plot comparing a missed treatment normal of a normal machine and an augmented machine, according to one example embodiment. In the missed treatment plot <NUM>, the missed treatment is normalized to the number of weeds existed in the field. That is, the missed treatment plot <NUM> illustrates an average missed treatment of identified weeds. In the missed treatment plot <NUM>, the y-axis is a normalized missed treatment and the x-axis is the particulate level in an accessed image. The missed treatment plot <NUM> includes a line for the normal machine <NUM> and a line for the augmented machine <NUM>. Each line represents the missed treatment of each machine when treating identified weeds. The augmented machine <NUM> has an appreciably lower missed treatment than the normal machine <NUM> for particulate values greater than <NUM>.

As described herein, a control system <NUM> may employ a plant identification model configured to determine a particulate level in an accessed image. Throughout the embodiments, the plant identification models are trained using alpha-blended augmented images as described above. In some embodiments, the control system <NUM> may utilize the alpha-blend levels in augmented images when determining a particulate level. For example, the augmented images may include augmented images with different alpha-blend levels (e.g., <NUM>%, <NUM>%, etc.). In this case, when determining a particulate level, the control system <NUM> identifies which of the alpha-blend levels in an augmented image most closely corresponds to the particulate level in an accessed image. In this manner, the alpha-blend level is an estimation as to the number of particulates in the accessed image compared to an image that does not include particulates. Additionally, in some embodiments, the control system <NUM> may utilize other information from the augmented images when determining a particulate level (e.g., particulate color, etc.). In these embodiments, the determined particulate level may be a quantification of that other information in the augmented images.

In various embodiments, the plant identification model may employ one or more approaches to determine a particulate level in the accessed image. Broadly, these methods may be grouped into, for example, two groups: (i) image level identification, and (ii) pixel level identification. Other groups are also possible. Image level identification determines a particulate level for an accessed image based on an aggregate classification of pixels in the image. Pixel level identification determines a particulate level based on a classification of individual pixels in an accessed image and subsequent analysis of the classified pixels. Both types of determination may employ classification and/or regression analysis techniques.

To illustrate, in an example, a control system <NUM> employs a plant identification model (model <NUM>) to determine a particulate level for the image at the image level. In this case, the control system <NUM> employs the model to identify obscuring pixels in an accessed image. The control system may determine the particulate level based on the identified obscuring pixels in the output image. For example, the control system <NUM> identifies the particulate level as 'high' if the number of obscuring pixels in an accessed image is above an upper threshold, Contrarily, the control system <NUM> identifies the particulate level as `low' if the number of obscuring pixels is below a lower threshold. Other ranges of particulate levels are also possible, such as, for example, levels <NUM>-<NUM>, a red, yellow, green classification, a continuous value between <NUM> and <NUM>, etc.).

As another illustration, for example, a control system <NUM> employs a plant identification model (model <NUM>) to determine a particulate level for the image at the pixel level. In this case, the control system <NUM> may determine a particulate level for the image based on the particulate levels for individual pixels in the image. In one example, the control system may compute a ratio of obscuring pixels to total pixels (or plant pixels) and determines the particulate level for the image based on the determined ratio. More specifically, the control system may identify a particulate level as 'high' if the ratio of obscuring pixels to total pixels (or plant pixels) in an accessed image is above a threshold, while the control system may identify a particulate level as `low' if the ratio of obscuring pixels to total pixels (or plant pixels) is below a threshold.

In some cases, the control system <NUM> may determine a particulate level for individual pixels and the particulate level for the individual pixels may influence the determination of the particulate level for the image. For example, the control system may assign each pixel in the image a particulate level, and the determined particulate level for the image is the average of the particulate levels in the image. The control system <NUM> may determine a particulate level for an accessed image based on other metrics and/or statistics calculated from pixel level analysis of the image. For example, the control system may determine a distribution, a shape, an average color, etc., of obscuring pixels, plant pixels, or total pixels in an accessed image to determine a particulate level. In various examples, the control system <NUM> may compare metrics for one type of pixel (e.g., obscuring pixels) to another type of pixel (e.g., plant pixels) or total pixels when determining a particulate level.

The control system <NUM> may employ several methods to present determined particulate levels in an accessed image to a user (e.g., via a display). In one example, the control system <NUM> quantifies the particulate level using any of the metrics described herein and presents the metric to the user. To illustrate, the control system <NUM> may present the ratio of obscuring pixels to plant pixels in an accessed to a user. In another example, the control system <NUM> may employ a binary classification system when presenting determined particulate levels. To illustrate, the control system <NUM> may calculate the total number of obscuring pixels in an accessed image. If the number of pixels in the accessed image is above/below a threshold, the control system presents the particulate level as high/low. In another example, the control system <NUM> may employ a contextual classification system when presenting determined particulate levels. To illustrate, the control system <NUM> may determine the ratio of obscuring pixels to plant pixels. The control system <NUM> then applies, for example, a four-bin classification system to the determined ratio. That is, based on the determined ratio, the control system <NUM> may present the determined particulate level as GREEN (i.e., no particulates), YELLOW (i.e., few particulates), ORANGE (i.e., some particulates), or RED (i.e., many particulates) based on the ratio. Other methodologies are also possible.

The control system <NUM> may generate a notification based on the determined particulate level. For example, the control system <NUM> may determine that the particulate level is between <NUM> and <NUM> and generate a notification in response. The control system <NUM> may transmit the notification to an operator of the farming machine <NUM> and/or a party responsible for agricultural management of the field. The notification may indicate the particulate level. For example, the notification may indicate to the operator that the particulate level in the field is high, that treatments are becoming inaccurate, or similar. Additionally, the notification may indicate an action based on the determined particulate level. For example, the notification may encourage the operator to wait for better operating conditions, travel at a slower speed, etc. In another example, the notification may indicate for the operator to switch between broadcast and selective spraying based on the determined particulate level. To illustrate, if the particulate level is high and the plant identification model is unable to accurately identify plants, the control system may suggest that the operator employ broadcast spraying rather than selective spraying.

<FIG> is a block diagram illustrating components of an example machine for reading and executing instructions from a machine-readable medium. Specifically, <FIG> shows a diagrammatic representation of control system <NUM> in the example form of a computer system <NUM>. The computer system <NUM> can be used to execute instructions <NUM> (e.g., program code or software) for causing the machine to perform any one or more of the methodologies (or processes) described herein. In alternative embodiments, the machine operates as a standalone device or a connected (e.g., networked) device that connects to other machines. In a networked deployment, the machine may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.

The machine may be a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a smartphone, an internet of things (IoT) appliance, a network router, switch or bridge, or any machine capable of executing instructions <NUM> (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute instructions <NUM> to perform any one or more of the methodologies discussed herein.

The example computer system <NUM> includes one or more processing units (generally processor <NUM>). The processor <NUM> is, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a controller, a state machine, one or more application specific integrated circuits (ASICs), one or more radio-frequency integrated circuits (RFICs), or any combination of these. The computer system <NUM> also includes a main memory <NUM>. The computer system may include a storage unit <NUM>. The processor <NUM>, memory <NUM>, and the storage unit <NUM> communicate via a bus <NUM>.

In addition, the computer system <NUM> can include a static memory <NUM>, a graphics display <NUM> (e.g., to drive a plasma display panel (PDP), a liquid crystal display (LCD), or a projector). The computer system <NUM> may also include alphanumeric input device <NUM> (e.g., a keyboard), a cursor control device <NUM> (e.g., a mouse, a trackball, a joystick, a motion sensor, or other pointing instrument), a signal generation device <NUM> (e.g., a speaker), and a network interface device <NUM>, which also are configured to communicate via the bus <NUM>.

The storage unit <NUM> includes a machine-readable medium <NUM> on which is stored instructions <NUM> (e.g., software) embodying any one or more of the methodologies or functions described herein. For example, the instructions <NUM> may include the functionalities of modules of the system <NUM> described in <FIG>. The instructions <NUM> may also reside, completely or at least partially, within the main memory <NUM> or within the processor <NUM> (e.g., within a processor's cache memory) during execution thereof by the computer system <NUM>, the main memory <NUM> and the processor <NUM> also constituting machine-readable media. The instructions <NUM> may be transmitted or received over a network <NUM> via the network interface device <NUM>.

In the description above, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the illustrated system and its operations. It will be apparent, however, to one skilled in the art that the system can be operated without these specific details. In other instances, structures and devices are shown in block diagram form in order to avoid obscuring the system.

Reference in the specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the system.

Some portions of the detailed descriptions are presented in terms of algorithms or models and symbolic representations of operations on data bits within a computer memory. An algorithm is here, and generally, conceived to be steps leading to a desired result. The steps are those requiring physical transformations or manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated.

Unless specifically stated otherwise as apparent from the following discussion, it is appreciated that throughout the description, discussions utilizing terms such as "processing" or "computing" or "calculating" or "determining" or "displaying" or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

Some of the operations described herein are performed by a computer physically mounted within a machine <NUM>. This computer may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, and magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any type of non-transitory computer readable storage medium suitable for storing electronic instructions.

The figures and the description above relate to various embodiments by way of illustration only. It should be noted that from the following discussion, alternative embodiments of the structures and methods disclosed herein will be readily recognized as viable alternatives that may be employed without departing from the principles of what is claimed.

One or more embodiments have been described above, examples of which are illustrated in the accompanying figures.

For example, some embodiments may be described using the term "connected" to indicate that two or more elements are in direct physical or electrical contact with each other. In another example, some embodiments may be described using the term "coupled" to indicate that two or more elements are in direct physical or electrical contact. The term "coupled," however, may also mean that two or more elements are not in direct physical or electrical contact with each other, but still co-operate or interact with each other.

For example, a process, method, article or apparatus that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such process, method, article or apparatus. For example, a condition A or B is satisfied by any one of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or present), and both A and B is true (or present).

In addition, use of the "a" or "an" are employed to describe elements and components of the embodiments herein. This is done merely for convenience and to give a general sense of the system. This description should be read to include one or at least one and the singular also includes the plural unless it is obvious that it is meant otherwise.

Claim 1:
A method for treating a plant in a field by a farming machine that moves through the field:
accessing an image of a plant, the image captured as the farming machine move past the plant in the field, the image comprising one or more pixels representing plant matter of the plant and one or more pixels representing airborne particulates;
identifying pixels in the image representing the plant using a plant identification model by:
classifying pixels in the image that represent plant matter as plant pixels,
classifying pixels in the image that represent the airborne particulates as particulate pixels, and
identifying the plant in a set of representative pixels in the image, the representative pixels including one or more plant pixels and one or more particulate pixels; and
actuating a plurality of plant treatment mechanisms coupled to the farming machine to treat the identified plant as the farming machine moves past the plant in the field.