Patent Description:
In crop farming, the process of developing and testing new plant protection products (e.g., herbicides) typically involves analyzing the response of plant species grown at an agricultural growth location (e.g., a greenhouse or an agricultural field) to different protection products and doses in a controlled way. However, protection products may also cause damages to the plants that they intend to protect. The assessment of the damage on the crop plants provides valuable information to the testing process. Typically, such damage assessment is performed on a daily basis in all test trials. Visual evaluation of every plant by experts involve time consuming process and lack repeatability. This assessment of growth locations (e.g., pots) is often performed in comparison with an untreated reference growth location. In a greenhouse context such reference growth locations are often referred to as control pots. A damage value which is typically determined by human experts to assess the development of a plant after herbicide application by comparing the treated plant with an untreated plant is the so-called phytotoxicity. According to EPPO, phytotoxicity is the capacity of a compound (such as a plant protection product) to cause temporary or long-lasting damage to plants. The method and rules for phytotoxicity assessment are described in detail in <NPL> in section PP <NUM>/<NUM> (<NUM>) Phytotoxicity assessment. The same method can be applied to crop plants as well as to weed plants. When applying the method to weed plants, it is often referred to as herbicide control. Phytotoxicity values can range from <NUM> to <NUM>%. In experiments involving a plurality of experts, inter-rater variability was estimated around <NUM>-<NUM>%, and even higher depending on the growth conditions and seasonal variations. Sometimes the phytotoxicity value indicating the plant damage is also referred to as "Plant Disease Control against Untreated", or - in short - PDCU value.

Nevertheless, when herbicide impact assessment is performed visually by expert people, such manual assessment always depends on the experience of the individual expert which does not allow for an objective and accurate quantitative assessment of the damages caused by herbicides.

<NPL> describes neural networks that classify images according to crop-disease pairs that are depicted on the images. More in detail, the paper explains a two-step approach with removing background information from the images by using color re-coding and classifying the pre-processed images by a convolutional neural network.

There is therefore a need for tools which support an objective and accurate quantitative assessment of the crop damage caused by herbicides.

This technical problem is solved by the embodiments - a computer-implemented method, a computer program product, and a computer system - for estimating damage on plants caused by herbicide application by using a regression deep learning model as claimed in the independent claims.

The regression deep learning model is trained such that different damage types (e.g., necrosis, leaf curling, bleaching, etc.) that are known as a consequence of herbicide application are taken into account for plants of a variety of plant species (e.g., crop species such as GLXMA, TRZAW, or weed species, such as ECHCG, AMARE, etc.) at different numbers of days after the treatment of the respective plants. For example, necrosis is a form of cell injury which results in the premature death of cells in living tissue by autolysis. Necrosis is caused by factors external to the cell or tissue, such as infection, toxins, or trauma which result in the unregulated digestion of cell components. Therefore, necrosis can occur as a consequence of herbicide application. Typically, a distinction is made between the damage subtypes early and late necrosis. Another damage type which can occur as a consequence of herbicide application is leaf curling (a plant disease characterized by distortion). Also, leaf curling is typically distinguished by the damage subtypes early and late leaf curling. A further damage type caused by herbicide application is the colouring of the plant leaves. Colouring can be divided into three damage subtypes: white colouring (often referred to as bleaching), purple colouring (also referred to as bleaching), and yellow colouring (chlorosis).

The following list summarizes damage symptoms (<NUM>) to (<NUM>) observed as a consequence of herbicide treatment and their respective descriptions, by name @ description of plant damage symptoms after herbicide treatment:.

Every herbicide treatment may cause one or more symptoms at a time. A further damage type caused by herbicide application is biomass reduction as a consequence of a reduction of the number of leaves, a reduction in leaf surface or a reduced growth in height. Not all symptoms (damage types) have the same relevance for damage estimation. Typically, the experts consider phytotoxicity as a mixed value where biomass reduction and the detected symptom(s) are considered accordingly.

In one embodiment, a computer system for determining plant damage caused by herbicide application includes an interface to receive a zenithal image representing a real-world situation at an agricultural growth location with one or more non-overlapping plants. Non-overlapping plants as used herein means that there is no overlap of plant elements belonging to different plants. However, for example, overlapping leaves of a single plant can be shown on the image. The one or more plants shown on the image are associated with a corresponding plant species identifier indicating the species of the plant shown on the image. That is, the plants on a single zenithal image for which the damage is to be determined are of the same species. The one or more plants have been subject to an herbicide treatment and the zenithal image was recorded a certain number of days after this treatment. The number of days after treatment associated with the zenithal image is known. For example, this information may be stored in a separate database or it may even be part of the metadata of the zenithal image itself. Dependent on the herbicide and the number of days after treatment the one or more plants on the image may show damage symptoms of one or more of the above described damage types.

For example, the received image may be taken by a camera with a resolution that is high enough to visualize the damage symptoms on the plants. Typically, the image includes representations of multiple plants and soil. In a field situation, crop plants and weed plants may be shown on the image. For example, the image may be recorded by a camera mounted on a drone while the drone is flying over the agricultural growth location. In another example, the image may be taken with a camera of a mobile device (e.g., a smart phone) by a human user (e.g., a farmer) from an elevated position (e.g., from a tractor). In general, it is advantageous to position the camera such that the recorded image shows a two-dimensional zenithal view of a part of the agricultural growth location where the plants of interest are growing.

When images are taken by the camera, the maximum distance between the camera and the plants for providing sufficient details regarding the damage symptoms on the plants depends on the image resolution of the camera. When using a camera with high resolution the distance between the plants and the camera can be chosen bigger than when using a camera with lower resolution (assuming a fixed focal length of the camera (no zoom)). That is, a camera mounted on a drone which, while flying over the field, has a higher distance to the plants than a camera mounted on a tripod in the field, typically needs a higher resolution to provide the same level of accuracy regarding the damage symptoms represented in the image.

An image pre-processing module of the computer system segments the received image into pixels associated with plant elements and into pixels associated with non-plant elements to obtain a masked image with the pixels associated with non- plant elements being removed from the image. This is achieved by using a damage agnostic plant segmentation algorithm. Damage agnostic plant segmentation algorithms are algorithms which can distinguish plant pixels from non-plant pixels even when the plant shows damage symptoms. In particular, most plant segmentation algorithms fail to distinguish plant pixels associated with necrosis symptoms from pixels associated with background soil. The meaning of "damage agnostic" as used herein is comparable to the use of the term "agnostic" in computing in general. In computing, an agnostic software component is unaware or noncommittal regarding the specific nature of the components or input with which it interacts. The term "damage agnostic plant segmentation algorithm" herein stands for a plant segmentation algorithm which correctly classifies image pixels into plant or non-plant pixels also for any pixel which is associated with any damage symptom of a plant caused by herbicide treatment.

In one embodiment, a damage agnostic plant segmentation algorithm may include the following steps: applying a gaussian filter to the received image to obtain a filtered image with less plant details; transforming the filtered image to the HSV color space; obtaining, through thresholding of the H and S planes, a mask for the plant wherein an initial mask obtained from the H plane removes the pixels of the transformed image presenting colors that are not likely to be present in the plant, and wherein an initial mask obtained from the S plane removes the parts of the transformed image that belong to uniform background of the image; and post-processing the initial masks to obtain a segmentation mask by reducing noise in the image, the segmentation mask applied to the received image to obtain the masked image.

In an alternative embodiment, the damage agnostic plant segmentation algorithm, when being executed, performs the following steps: providing at least one channel of the received image - being an RGB image - to a semantic regression neural network, the semantic regression neural network trained to estimate at least a near-infrared channel from the at least one channel of the received image, the semantic regression neural network having a topology based on a convolutional segmentation neural network with its last layer being substituted by a monotonic activation function and its loss function being substituted by a regression loss function, to learn a pixel-wise regression transformation that transforms any RGB channel from an RGB domain to a target domain comprising at least a near-infrared domain; obtaining an estimate of the near-infrared channel by applying the semantic regression neural network to the RGB image; providing a multi-channel image comprising at least one of the R-, G-, B-channels of the RGB image, and the estimated near-infrared channel, as test input to a semantic segmentation neural network, the semantic segmentation neural network trained with a training data set comprising multi-channel images of the test input type to segment the test input into pixels associated with plant elements and pixels not associated with plant elements; and segmenting the test input using the semantic segmentation neural network resulting in the segmentation mask indicating pixels of the test input associated with plant elements and indicating pixels of the test input not associated with plant elements, the segmentation mask applied to the received image to obtain the masked image.

The image pre-processing module may also implement an image resizing function. The masked image is provided as part of the test input to the input layer of the regression deep learning model. If the masked image does not match the size of the input layer the resizing function can be used to resize the masked image to match the size of the input layer while the aspect ratio of the original received image is preserved during the resizing of the masked image. Typically, the image size of high-resolution images taken by the camera is between <NUM> x <NUM> pixels and <NUM> x <NUM> pixels. The image size which can be processed by the deep learning model is limited by the memory of the Graphical Processing Unit (GPU) used for processing the model. A typical memory size of a GPU is <NUM> GB. This memory size is not sufficient to process such high-resolution images with such deep learning models. Therefore, the image pre-processing module can adjust the image size (in pixels) of the masked image to a rescaled masked image matching the size of the input layer of the regression deep learning model while preserving the aspect ratio of the original received image. Preferably, the rescaled image is reduced in size (compared to the received image) but the damage symptoms associated with any type of damage remain still visible (i.e. identifiable) on the rescaled image. For example, early-stage necrosis symptoms are small white spots occurring on the crop leaf surface. The resolution of the rescaled image should still be high enough to clearly visualize such spots.

The (resized) masked image is then provided to the input layer as part of the test input. The test input to the regression deep learning model further includes the number of days after treatment and the respective plant species identifier associated with the originally received image. The regression deep learning model has been trained with a training data set comprising a plurality of plant images for multiple plant species. For each plant species, the training data set includes a first subset of images showing one or more damaged plants of said plant species at an agricultural growth training location after herbicide treatment with at least some of the images of the first subset being recorded at different days after the herbicide treatment. Further, for each plant species, the training data set includes a second subset of images showing one or more healthy plants of said plant species growing in an untreated reference growth location (control location), with at least some of the images of the second subset being recorded at days corresponding to the different days after the herbicide treatment. In one embodiment, the training data set may also show herbicide variability in that different images of the training data set are associated with different applied herbicides where the different herbicides may consist of different chemical compositions or may only differ with regard to the applied dose. The agricultural growth training location can be different from the agricultural growth location used for obtaining the test input images for damage degree prediction. Therefore, such a location where test input images are obtained is not necessarily associated with a control location. However, as explained in more detail in the detailed description, an agricultural growth training location has a corresponding control location in its vicinity to enable the regression deep learning model to also learn the growth stage development of undamaged plants which provides the reference for PDCU estimates by experts for damaged plants.

Thereby, each training image of the training data set associated with a particular plant species has been annotated by an expert with a phytotoxicity value (PDCU*) which is used as ground truth for training the regression deep learning model. The phytotoxicity value reflects the expert's (subjective) assessment of a damage degree of one or more plants on the respective training image in comparison to plants of a control section in the vicinity of the agricultural growth location in accordance with predefined assessment rules. For example, such predefined rules can be the same or similar rules as described in the before mentioned Bulletin. Typically, the PDCU values are not just estimated by a single expert but rather by a group of experts. This adds errors to the process of PCDU value assessment. As mentioned above, the inter-rater variability is estimated around <NUM>-<NUM>%, and even higher depending on the agricultural growth conditions and seasonal variation.

The use of the first and second subsets (damaged treated plants and healthy untreated plants) for training is advantageous for the regression deep learning model to learn the characteristics of the normal growth process of the plants without herbicide application. The normal growth process can be learned with a plurality of images of untreated plants. It is advantageous to run multiple trials to capture the variability. This allows the trained model to take into account also the biomass reduction symptom reflected by the respective ground truth PDCU values.

The training data images may be augmented by applying affine transformations to a subset of the training images while maintaining zoom factor and scale for the images of the subset.

The output of the trained regression deep learning model in response to the test input is a predicted phytotoxicity value (PDCU) for the test input. That is, the trained system allows a farming user to receive immediate feedback about the damage degree of plants shown on the test input image as a consequence of herbicide treatment. This information can be advantageously used by the farmer to adjust further treatments of the plants at the agricultural growth location.

The regression deep learning model may be based on a deep convolutional neural network topology using a regression loss function, and using an optimizer selected from the group of: stochastic gradient descent, RMSProp, and Adam.

The deep convolutional neural network topology can be any one of: VGG, ResNet, ResNeXt, SE-ResNet, SE-ResNeXt, SE-Net, DenseNet, Inception ResNet V2, Inception V3, Xception, NASNet, MobileNet, and MobileNet v2. For example, the topology may be a ResNet34 topology, the regression loss function may be mean-squared-error, and the optimizer may be stochastic gradient descent.

Further aspects of the invention will be realized and attained by means of the elements and combinations particularly depicted in the appended claims. It is to be understood that both, the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention as described.

<FIG> includes a block diagram of a computer system <NUM> for estimating damage on plants caused by herbicide application treatment using a single regression deep learning model RDLM1. <FIG> is a simplified flow chart of a computer-implemented method <NUM> estimating damage on plants caused by herbicide application. The method <NUM> can be executed by the system <NUM>. For this reason, <FIG> is described in view of <FIG> and the following description of <FIG> also refers to reference numbers used in <FIG>.

The system <NUM> is communicatively coupled with an image recording device <NUM> via an interface <NUM> to receive <NUM>, from the image recording device, a zenithal image <NUM>-<NUM>, <NUM>- <NUM> representing a real-world situation at an agricultural growth location AGL1 with one or more non-overlapping plants of a particular plant species <NUM> after herbicide application. Thereby, the image is recorded a certain amount of time after the herbicide treatment (i.e. the application of the herbicide). This amount of time is typically referred to as "days-aftertreatment" DAT. Of course, the amount of time can also be given in other units (e.g., hours, minutes or even weeks) which can always be converted into a number of days value. That is, the use of the generally known term "days-after-treatment" is not meant to be limiting in the sense that the respective amount of time which has elapsed since the herbicide application has to be specified in days. For each recorded image, DAT is known and may either be stored directly as metadata of the respective image, or it may be stored in a separate database with a pointer to said image, or it may be directly entered into the system <NUM> for the recorded image by a user. Further, the plant species <NUM> of the plants on the image is also recorded. For example, the camera device recording the image may already be preset to said plant species when it is mounted at the agricultural growth location AGL1. In this case, a plant species identifier ID may be stored as metadata of the recorded image. The plant species may also be input by a farming user or it may be automatically determined by computer-implemented plant species identifier. For example, plant species identifiers based on classification convolutional neural networks are known in the art.

The angle of view <NUM>-<NUM> of the image recording device <NUM> is illustrated by the dash-dotted lines. The agricultural growth location AGL1 may be a section of an open field or a section in a greenhouse. In the greenhouse situation, the plants are typically grown in pots <NUM>. The dashed lines used for the optional pots <NUM> indicate that in a field situation the plants are grown directly in the soil (ground).

The real-world situation in the agricultural growth location AGL1 is schematically illustrated by a plurality of plants of plant species <NUM> showing damage symptoms on some leaves. Thereby, the leaves of type <NUM>-<NUM> are healthy leaves with no damage symptoms. The leaves of type <NUM>-<NUM> show damage symptoms of a first type (e.g., leaf curling) and the leaves of type <NUM>-<NUM> show damage symptoms of a second type (e.g., necrosis). The plants growing in AGL1 were all treated with an herbicide. For example, an herbicide sprayer <NUM> has applied a particular herbicide to the area indicated by the dotted triangle <NUM>-<NUM>.

If the agricultural growth location is used for obtaining training images with damaged plants, in the vicinity of such an agricultural growth training location AGL1, plants of the same plant species <NUM> are grown in a so-called control location CL1. The control location CL1 is in the vicinity of AGL1 if the plants are close enough to AGL1 so that basically the same growth conditions (humidity, temperature, illumination, etc.) prevail in both locations AGL1, CL1. Typically, the control location CL1 is a section located in the same field or greenhouse as the agricultural growth training location AGL1. However, the plants growing in the control location CL1 were not treated with said herbicide. Therefore, they provide a reference for the assessment of damages caused by the herbicide treatment. It is to be noted that images obtained as a test input for determining the damage degree of plants on that test input are typically derived from an agricultural growth location AGL1 which is not associated with a corresponding control location because the regression deep learning model has already been trained with said training images.

For example, the image recording device <NUM> may be a digital camera device which can provide images at resolutions of at least <NUM> x <NUM> pixels. With such a high resolution camera the field of view (illustrated by the dashed lines <NUM>-<NUM>) of the camera <NUM> can be selected to cover a relatively large area (in the order of 1x1m<NUM>) of the agricultural field <NUM> and still provide sufficient image details to perform the herein disclosed image analysis by positioning the camera at an appropriate distance to the plant(s) (e.g., approximately one meter above the plants). Cameras with lower resolution may still be usable when selecting a smaller field of view and getting closer to the plant when recording the images. It is assumed that the recorded image <NUM>-<NUM>, <NUM>-<NUM> includes a representation of at least one plant which may show damage symptoms of any damage types <NUM>-<NUM>, <NUM>-<NUM> caused by herbicide treatment. For example, the camera may be mounted on a pillar/tripod located in the field/greenhouse, or it may be taken by a camera carried by a drone while flying over the field, or it may be mounted on the roof of a greenhouse. It is also possible that a human user (e.g., the farmer) takes a picture with a smartphone camera device from an elevated position (e.g., from a tractor seat).

The recorded image is sent to an image pre-processor <NUM> of the computer system <NUM> where it is further processed. The pre-processor <NUM> segments <NUM> the received image <NUM>-<NUM>, <NUM>-<NUM> into pixels associated with plant elements and into pixels associated with non-plant elements to obtain a masked image <NUM>-<NUM>. The segmentation is performed by using a damage agnostic plant segmentation algorithm. When applying the masked image <NUM>-<NUM> as a segmentation mask for the further processing of the image, the effect is that all pixels associated with non-plant elements are ignored for the further image processing by the regression deep learning model RDLM1.

When providing <NUM> the masked image <NUM>-<NUM> to RDLM1 the size of the masked image has to match the size of the input layer of the regression deep learning model. The pre-processor <NUM> checks <NUM> if the size of the masked image <NUM>-<NUM> matches the size of the input layer processor <NUM>. If it does not match, the pre-processor resizes <NUM> the masked image <NUM>-<NUM> to match the size of the input layer while preserving the aspect ratio. Preserving the aspect ratio is advantageous for the accuracy of the predictions provided by the deep learning model RDLM1. Because of the limited memory of GPUs used for implementing deep learning models, high resolution images of the size of the original received image can typically not be processed by such deep learning models. Therefore, a reduction in size may be applied to the image for allowing the direct processing of the reduced image by the deep learning model.

The relevant information for PDCU value estimation/prediction is contained in the plant portions of the image. Therefore, it is advantageous to discard as much as possible the non-plant related pixels in the image. This way it is ensured that the regression deep learning model which will further process the pre-processed image focuses its learning and prediction on plant elements. Consequently, plant segmentation is performed in the preprocessing stage where the segmentation algorithm needs to be able to recognize also pixels associated with any one of the previously described damage symptoms as pixels associated with plant elements. This damage agnostic property of the segmentation algorithm is particularly advantageous with regard to early or late necrosis symptoms which can easily be confused with soil.

In one implementation, the damage agnostic plant segmentation algorithm may be implemented by a segmentation algorithm as disclosed in "<NPL>. " The segmentation algorithm disclosed in sections <NUM>. (and its subsections) can be used to divide an image into regions and after that, to detect "focused" regions. In this implementation, necrosis is detected as focused region whereas the soil is detected as unfocused region. That is, the segmentation algorithm is able to distinguish necrosis from soil. Plant element pixels associated with the other disclosed damage symptoms are recognized as belonging to the corresponding plants anyway. Therefore, the segmentation algorithm is damage agnostic according to the above definition.

<FIG> illustrates an improved implementation of the damage agnostic plant segmentation algorithm with a flow chart showing details of the segmenting step <NUM>. In this improved implementation, the following steps are performed when the damage agnostic plant segmentation algorithm is executed by the computer system: applying <NUM> a gaussian filter to the received image to obtain a filtered image with less plant details; transforming <NUM> the filtered image to the HSV color space; obtaining <NUM>, through thresholding of the H and S planes, a first initial mask from the H plane to remove pixels of the transformed image presenting colors that are not likely to be present in the plant, and a second initial mask from the S plane to remove parts of the transformed image that belong to uniform background of the image; and post-processing <NUM> the initial masks to obtain a segmentation mask by reducing noise in the image, the segmentation mask applied to the received image to obtain the masked image <NUM>-<NUM>.

In other words, the improved damage agnostic plant segmentation algorithm uses classical image processing techniques. Firstly, the algorithm applies a gaussian filter to the original image, in order to reduce the influence of small details of the plant(s) for the following steps of the segmentation algorithm. The filtered image is then transformed to the HSV color space. Through thresholding of the H and S planes, an initial mask is obtained covering the plant elements. The initial mask can be seen as an aggregate of two initial (sub-) masks. The first initial (sub-)mask obtained from the H plane removes the pixels of the image presenting colors that are not likely to be present in the plant. The second initial (sub- ) mask obtained from the S plane removes the parts of the image that are plain white, grey or similar. That is, the second initial mask removes the background of the image. Lab reference system for color space transformation is not used although clear discrimination can be made for green color, but brownish necrosis would be removed together with the soil. So, this color space representation has been discarded.

The mask is then postprocessed in order to remove some noise. This postprocessing may be different depending on the type of plant (broad leaf or grass plant species), due to their shape differences. For example, in the case of broad leaf plants, only the largest area detected in the mask may be maintained. In the case of grass plants (i.e. plants with narrow leaves), this would not be possible to do without removing pixels associated with plant elements. Therefore, all the elements above the threshold (of the H and S channels) are considered. The resulting mask (segmentation mask) is finally applied to the original image.

It is to be noted that segmenting necrotic parts of a plant from the soil is very difficult using classical image processing techniques. Therefore, in the case of a greenhouse with plants growing in pots it may be advantageous to not remove the soil pixels associated with the pot. It is advantageous for the regression deep learning model to "see" these necrotic plant parts for correct PDCU prediction.

<FIG> show segmentation results achieved by the improved implementation of the damage agnostic plant segmentation algorithm of <FIG>. <FIG> relate to a greenhouse scenario with the received images showing single pot images with a GLXMA pot <NUM> (broad leaf plant), and with a TRZAW pot <NUM> (grass plant). The received images <NUM>, <NUM> are high resolution images with <NUM> x <NUM> pixels. In the example, the corresponding masked images <NUM>, <NUM> were resized to <NUM> x <NUM> pixels while preserving the aspect ratio. The input image of the regression deep learning model is usually a square image. In an example implementation, it was found that an input image of 1024x1024 pixels size is enough for the model to recognize the relevant information. As the original image is bigger and is not a square image, it has been resized accordingly without changing the aspect ratio, so that the deep learning model can correctly recognize the amount of biomass of the plant. In the greenhouse example, the segmentation algorithm cuts from the original image only the bounding box of the plant. This bounding box was pasted in the center of a square black image of size <NUM> x <NUM> pixels. This side size was chosen as being a value in the middle of the range between the height and the width of the original image (<NUM> x <NUM>). Finally, the resulting square image was resized to <NUM> x <NUM> pixels.

<FIG> relates to an open field scenario showing an image <NUM> with non-overlapping GLXMA plants growing in an agricultural field. The corresponding masked image <NUM> shows the segmented plant portions with the entire soil background turned into black pixels and thus being ignored for the further processing by the regression deep learning model. A damage agnostic plant segmentation algorithm which works well in an agriculturally field scenario is described in detail in the context of <FIG>. <FIG> is a zoom picture <NUM>-z for one of the plant portions which remained in the masked image <NUM>. The zoomed picture <NUM>-z shows, that the masked image shows enough details to distinguish healthy plant elements <NUM>-h from damaged plant elements <NUM>-d (bright spots indicate bleaching damage on the damaged plant elements).

An alternative implementation of the damage agnostic plant segmentation algorithm is illustrated in <FIG> in combination with the flowchart of <FIG>. In this implementation, the zenithal image <NUM> is recorded as an RGB image at an agricultural growth location by an appropriate RGB camera device. The triangle symbols <NUM> in the image <NUM> represent plant elements (e.g., leaves, stems, etc.) of plants (e.g., crop, weeds, trees, etc.) growing in the scenery <NUM>. The camera device can be any digital camera device which provides RGB images. This includes all devices with integrated camera functions (e.g., smart phones or tablet computers). The RGB image <NUM> therefore includes pixels which represent plant elements <NUM> and includes pixels which represent non-plant parts <NUM> (e.g., soil, water, etc.) at the agricultural growth location. It is to be noted that pixels belonging to plant elements may be associated with only one plant species or with different plant species, dependent on the situation at the agricultural growth location.

The image <NUM> is received by the pre-processor <NUM> implementing the damage agnostic plant segmentation algorithm with the following modules. Thereby, an RGB image has three channels - a red channel R, a green channel G, and a blue channel B. The following example uses all RGB channels for NIR estimation and vegetation coverage prediction. However, a person skilled in the art can apply the method also by using only a subset of the RGB channels of the RGB image.

The RGB image <NUM> is then provided <NUM> to a semantic regression neural network (SRN) <NUM>. SRN <NUM> is trained to estimate at least a near-infrared channel NIR from the RGB image <NUM>. SRN <NUM> has a topology based on a convolutional segmentation neural network with its last layer being substituted by a monotonic activation function and its loss function being substituted by a regression loss function. Convolutional segmentation networks are well known in the art and a person skilled in neural networks can perform such substitutions without major efforts. When training SRN <NUM>, the neural network learns a pixel-wise regression transformation that transforms any RGB image from an R+G+B domain to a target domain comprising at least a near-infrared domain.

Examples for possible regression loss functions are: Mean Square Error (also known as Quadratic loss or L2 Loss), Mean Absolute Error (also known as L1 Loss), Huber Loss (also known as Smooth Mean Absolute Error), Log-Cosh Loss, and Quantile Loss. A comprehensive explanation of such regression loss function examples is available in the article "<NUM> Regression Loss Functions All Machine Learners Should Know", Prince Grover, Jun <NUM>, <NUM>, available at https://heartbeat. ml/<NUM>-regression-loss-functions-all-machine-learners-should-know-4fb140e9d4b0. Each regression loss function can be used with any of the monotonic activation functions: linear, sigmoid or hyperbolic tangent.

In an example embodiment, a fully convolutional DenseNet architecture was employed as baseline network. This network combines the descriptive power from traditional segmentation networks based on fully convolutional versions of classification ones such as SegNet where the accuracy on the border detection is provided by the skip connections on the U-Net segmentation network. The last layer of this network has been substituted by a linear activation function and the loss function has been substituted by a mean absolute error in order to learn a pixel-wise regression transformation that translates the image from the source to the target domain. The image passes through a subsequent set of convolutional operations and max-pooling operations that reduce the spatial dimension of the signal while increasing the descriptive part gradually. This ends up to a descriptor image. A second stage is composed by a set of convolutional filters and up-sampling layers that recover the spatial resolution of the input image on the desired domain. To be able to recover the input image high level details, the network makes use of skip connections that transfer the low-level features and spatial information from the source domain into the detailed reconstruction of the target domain. A final N channel layer with a sigmoid layer activation performs the final reconstruction of the NIR.

The network was trained by minimizing a mean absolute error loss function which can be enhanced by an adversarial perceptual loss function following a pix2pix architecture. The addition of the perceptual adversarial loss functions ensures that the generated image is visually plausible and the estimated image cannot be distinguished from a real image by a specifically trained network.

For creating a training data set, <NUM> plots were planted combining the presence of the plant species. Each plot followed a different weed control treatment in order to generate damage on the different species. Two pictures per day were taken per plot by a Micasense RedEdge MX camera during <NUM> days. This camera has five different sensors: blue (<NUM>), green (<NUM>), red (<NUM>), red-edge (<NUM>) and NIR (<NUM>) and delivered <NUM> x <NUM> pixel images. The RGB- and NIR channels of the camera were used to provide the training images of the training dataset. As each image band was taken by a different monochrome sensor, acquired images were co-registered by applying affine transformation that minimized the Mattes mutual information loss minimization between channels. Mattes mutual information loss minimize is explained, for example, in the paper: "<NPL>). International Society for Optics and Photonics. " In other words, the R-, G- and B- channels of the used multispectral camera were not the normal RGB channels as provided by a smart phone or a digital single-lens reflex camera (DLSR) camera but cover a much narrower band of RGB. To enable SRN <NUM> to process RGB images recorded by standard RGB cameras, the RGB channels of the used multispectral camera were transformed into standard DLSR channels (real sRGB values) by multiplying the multispectral channels with a matrix: [R_DLSR, G_DLSR, B_DLSR]' = A*[R_SPECTRAL, G_ SPECTRAL, B_ SPECTRAL]', where A is a 3x3 matrix containing the contribution of each R_SPECTRAL, G_ SPECTRAL, B_ SPECTRAL into the R_DLSR, G_DLSR, B_DLSR channels.

From the registered images, images were randomly chosen and the vegetation coverage was manually segmented on the respective images. To avoid biasing, training, validation and testing datasets were selected plot-wise. <NUM>% of the plots were randomly chosen for training whereas the <NUM>% remaining were distributed into validation sets (<NUM>%) and testing sets (<NUM>%). All pictures were incorporated to the set determined by their plot number.

In more detail, training was performed over <NUM> x <NUM> pixel tiles that were extracted randomly from the full-size images during <NUM> epochs. The tiles are fed into the neural network. The Adam classifier [cf. <NPL>] was employed for training and the learning rate was set to <NUM>-<NUM>. A reduction of the learning rate is performed when the loss function value on the validation data set raises or stagnates. It can be appreciated that the use of the pix2pix based adversarial loss contributes not only to generate more plausible images but also reduces the error (difference) between the predicted and real NIR images, and reduces the regression loss. A plausible image, as used herein, is an imaged that reflects reality. For example, blue leaves of a crop plant would not be considered to be plausible.

The trained SRN <NUM> is then applied to the received RGB image <NUM> to obtain <NUM> an estimate of the NIR channel. It is to be noted that there is no need to use a multispectral camera for providing the RGB images which serve as test inputs to the first neural network. Typically, a simple smartphone camera or DLSR camera may be used to record the RGB images to be analyzed.

The R-, G-, B-channels of the originally received RGB image and the estimated NIR channel are then provided <NUM> as a test input TI1 to a semantic segmentation neural network SSN <NUM>. These four channels form the multi-channel image <NUM>. The SSN <NUM> has been trained before with a training data set comprising such multi-channel images of the test input type to segment the test input TI1 into pixels associated with plant elements and pixels not associated with plant elements.

SSN <NUM> has a similar structure as SRN <NUM>. In an example embodiment, the input layer of SSN <NUM> is of size MxNxK, where M and N represent the width and height of the input image, and K represents the number of channels used. In an embodiment with R-G-, B- and NIR channels the number of input channels is K=<NUM>. Even less channels, such as for example, a R-NIR combination (K=<NUM>) or a R-G-NIR combination (K=<NUM>) or other subcombinations may be used. In more advanced embodiments, the number of channels can reach K=<NUM> as explained further down below. In general, the channels are composed by the R <NUM> , G-, B-channels of the original RGB image and all estimated channels of the multi-channel image. The final layer of SSN <NUM> is composed by two output channels of size M and N (MxNx2) resembling the original size of the image. One output channel contains the estimate for the vegetation coverage segmentation and the other contains the estimate for non-vegetation (non-plant) classes. This is followed by a softmax activation layer that ensures the mutually exclusiveness of the two classes. SSN <NUM> may be minimized over a categorical cross-entropy loss function. Other segmentation loss functions, such as for example, weighted categorical cross-entropy, balanced cross-entropy, Focal loss, Focal loss plus Dice loss, Tversky loss, or Lovász-Softmax, or any appropriate combination thereof may also be used by a skilled person for training the semantic segmentation neural network.

The application of SSN <NUM> to the received test input TI1 segments <NUM> the test input into pixels associated with plant elements <NUM> (vegetation class) and pixels not associated with plant elements <NUM> (non-vegetation class). The result of the segmentation is a so-called vegetation coverage map <NUM> which recognizes any pixel associated with a plant independent of damage symptoms. In particular, pixels which belong to necrotic parts of a plant are identified as plant pixels and can be clearly distinguished from soil pixels. The pixels belonging to the vegetation class then define the masked image for the further processing.

Turning back to <FIG>, the regression deep learning model RDLM1 receives a test input comprising the masked image <NUM>-<NUM> provided <NUM> by the pre-processor <NUM>. Further, the respective number of days after treatment DAT, and the respective plant species identifier ID are provided <NUM> as part of the test input. As described earlier, DAT and ID may be part of the metadata of the received image <NUM>-<NUM> or they may be otherwise entered into the system, e.g. by a user of the system.

The regression deep learning model RDLM1 has been trained with a plurality of plant images for multiple plant species and for multiple days-after-treatment values. In <FIG> the dashed double arrows relate to a situation where images <NUM>-<NUM>, <NUM>-<NUM> would be part of the training data set for RDLM1. Of course, the skilled person will understand that images <NUM>-<NUM>, <NUM>-<NUM> cannot be used as training images and test input simultaneously. Therefore, the dashed double arrows are to be understood as an association with any training image not being used as a test input for the system <NUM>. Each training image of the training data set shows either one or more non-overlapping healthy plants or one or more non-overlapping damaged plants of a particular plant species ID1, ID2 at a particular number of days after treatment DAT1, DAT2. Thereby, the training data set includes images of healthy and damaged plants to allow RDLM1 to also learn about the biomass reduction of the plants in addition to the previously describe damage symptoms.

It is to be noted that, in addition to the variability of the training dataset with regard to plant species and days-after-treatment, a further variability may be introduced with regard to herbicides used for treatment. In other words, the training data set may further show herbicide variability in that different images of the training data set are associated with different applied herbicides. Different applied herbicides, as used herein, can either consist of different chemical compositions or may just differ with regard to the applied dose. That is, an herbicide having the same chemical composition but being applied with a different dose is also considered to be a different applied herbicide for the purpose of introducing herbicide variability into the training dataset. When training the regression deep learning model with herbicide variability, the system becomes more flexible in that it can handle any herbicide treatment. Without herbicide variability, the model would learn to make predictions only for a particular herbicide application which was used for the images of the training dataset. After having been trained, the regression deep learning model predicts phytotoxicity values PDCU for the respective test inputs solely based on the received image <NUM>-<NUM>, the associated plant species identifier ID as well as the days-after-treatment value DAT. The PDCU output may then be received <NUM> from the regression deep learning model by a display function to show the predicted value to a human user <NUM>. Alternatively, the value may be received by a software application to be further processed. For example, the value may be used by a sprayer application to determine future herbicide treatment data based on the detected damage level for controlling the sprayer appliance <NUM>.

Each training image is annotated by an expert <NUM> with a phytotoxicity value PDCU1, PDCU2 as ground truth. The phytotoxicity value reflects the expert's assessment of a damage degree of one or more plants on a respective training image in comparison to plants of the control location CL1 in the vicinity of the agricultural growth location AGL1 in accordance with predefined assessment rules. Every herbicide treatment may cause one or more symptoms at a time. Experts <NUM> consider the PDCU value to be a mixed value where both, the biomass reduction and the detected symptom(s), are considered. Not all the symptoms have the same relevance for the damage estimation. The whole damage in the plant is represented by this value.

<FIG> describes the training process of the regression deep learning model RDLM1 in more detail. The training data set <NUM>-t* includes a plurality of training images <NUM>-t1 to <NUM> tn. The training dataset includes images with healthy (untreated) plants indicated by a white filling. For untreated plants no herbicide was applied which is indicated by the "H-x"-label in the example of image <NUM>-t1. Damaged plant images <NUM>-t2, <NUM>-tn are indicated by the dotted filling. In the example, the training dataset <NUM>-t* also uses herbicide variability which is indicated by the herbicides H-<NUM> and H-<NUM> which were applied to the plants shown on images <NUM>-t2 and <NUM>-tn, respectively. When the test images are provided as training input to RDLM1, each training image <NUM>-t1, <NUM>-t2,. , <NUM>-tn is provided together with: the corresponding plant species identifier PS-<NUM>, PS-<NUM>,. , PS-<NUM>, the corresponding days-aftertreatment value DAT-<NUM>, DAT-<NUM>,. , DAT-<NUM>, and the corresponding ground truth PDCU value PDCU-t1, PDCU-t2,. Optionally, in case of using herbicide variability, also the corresponding herbicide identifiers H-x, H-<NUM>,. , H-<NUM> may be provided. From observations of control plants for GLXMA and AMARE on the one hand, and observations of control plants for TRZAW and ECHCG on the other hand, it can be noticed that the rhythms of growth are different. For instance, AMARE at DAT-<NUM> (i.e., two days after treatment) can show as much biomass as GLXMA at DAT-<NUM> (four days after treatment). This may lead to wrong learning of the expected biomass amount at a specific date. By adding the additional information daysafter- treatment and plant species identifier and as part of the input to RDLM1 together with the image this problem gets solved.

RDLM1 then processes each training input and determines a predicted PDCU value as output PDCU-p1 to PDCU-pn based on the current training state of the model. The predicted outputs are compared with the corresponding ground truth values PDCU-t1 to PDCU-tn and, after the processing of each training input, the weights of RDLM1 are adjusted accordingly.

For collecting images to train a prototype of RDLM1 the following procedure was applied. An acquisition campaign took place in a greenhouse. Two crops were selected: GLXMA and TRZAW, and one associated weed per crop, AMARE and ECHCG, respectively. The proposed method was evaluated for two broad leaf species (GLXMA and AMARE) and two grasses (TRZAW and ECHCG). Different trials were made to have different situations for every crop and weed, according to the herbicide applied in the treatment and the applied dose. Different damages were provoked in the treated plants. The objective was to have a wide range of situations where the algorithm has to estimate the damage percentage of the plant. It was paid attention to the fact that on each training image only non-overlapping plants were present.

Experts assessed the plant damage visually for each training image by looking at and even touching the real plants. That is, the experts were in possession of the 3D information of the plant(s). A camera on top of the plant taking a zenithal view does neither capture the height nor the strength and consistency of the plant. The third dimension is lost. However, for the experts the 3D information can provide valuable information in particular for the assessment of biomass reduction.

Images were acquired in the greenhouse with different natural illumination conditions (cloudy, sunny days) along several months. No artificial illumination was used. Images were taken with a high-resolution camera (<NUM> x <NUM> pixels). Conditions for acquisition were the same in all the images. According to experts, biomass reduction in the plant is a main symptom of the effect of an herbicide. So, the biomass reduction of the plant(s) in relation to respective plants from the control location seriously affects the PDCU value estimated by the experts. For biomass comparison both the treated plant(s) and the control plant(s) must be in the same scale, with the same pixel resolution. For this reason, an acquisition distance has been established.

The acquisition conditions which were present for recording the training dataset for developing the prototype are summarized in the following: There was a zenithal view (90º) and the resolution was <NUM>/pixel. Further acquisition conditions had been:.

General comments about acquisition are the following:.

As mentioned earlier, in the training dataset <NUM>-t* four plant species (two crops and two associated weeds) were used: GLXMA (and its weed AMARE) and TRZAW (and its weed ECHCG). GLXMA and AMARE are broad leaf plants, and TRZAW and ECHCG are grass. As aforementioned, the experts assessed the PDCU values as values representing the degree of damage of the respective plants. This damage degree was assessed through observation of different symptoms after application of an herbicide. These symptoms are not present in an untreated control location. The untreated control location was used as reference to establish the damage in the agricultural growth location to be evaluated.

Turning briefly to <FIG>, the two rows <NUM>, <NUM> show examples of various damage symptoms at two different days after treatment period. In row <NUM>, a healthy (untreated) AMARE plant is shown in comparison to three damaged plants three days after herbicide treatment DAT <NUM>. In row <NUM>, the situation is shown for GLXMA plants eight days after herbicide treatment DAT <NUM>. The observation of different damage symptoms after application of an herbicide allows to assess the degree of damage (PDCU value) caused by the herbicide treatment. These damage symptoms are not present on plants growing in an untreated control pot as shown in the first image of each row <NUM>, <NUM>. The untreated control pot is a reference for the damage degree in the treated pots.

Often, biomass reduction is considered to be the most important damage symptom. The chemical products (herbicides) can inhibit the growth of the plants and therefore, the biomass is reduced. Biomass is the alive tissue in the plant, i.e. all tissue of the plant not affected by necrosis. Biomass is not only green pixels. Some regions with other colors, such as yellowish for chlorosis, white pixels <NUM> for bleaching, or pixels <NUM> with purple coloring need to be considered as biomass as well. Biomass reduction can only be estimated in comparison to a reference control location. The control location indicates how the real growing state of the plant would have to be after the respective days-after-treatment period. In trial experiments used herein, there was always a control location representing this adequate growing stage. The evaluation period for every plant in the trial was about <NUM>- <NUM> days. Every plant was assessed daily unless the plant died before that time, or after scoring a PDCU of <NUM>%, when it is no longer evaluated. In the experiments, only postemergency situation has been considered, this is, the seed had already germinated. The remaining images of row <NUM> show pixels <NUM> of the damage type early leaf curling, and pixels <NUM> of the damage type early necrosis at three days after treatment. The remaining images of row <NUM> show pixels <NUM> of damage type late leaf curling, and pixels <NUM> of the damage type <NUM>.

Turning briefly to <FIG>, table <NUM> shows three examples of PDCU values (column <NUM>) which were assigned by an expert to a particular herbicide treated pot in column <NUM> showing damaged AMARE plants in comparison to a corresponding control pot in column <NUM>. A corresponding control pot is a pot with a plant of the same species which is at the same growth stage as the plant in the pot to assess and which was not treated with herbicides. A PDCU value of <NUM>% indicates a healthy plant and a PDCU value of <NUM>% indicates a dead plant. In column <NUM> the possible damages recognized by the expert are listed. For example, the first image row shows an AMARE plant with relatively small damages (biomass reduction and coloring yellow) compared to the control pot. The expert assigned a PDCU value of <NUM>% to this pot. In the second image row of table <NUM>, a much higher damage (biomass reduction and coloring purple) is recognized by the expert. For this pot a PDCU value of <NUM>% has been assigned. As mentioned above, the assessment by experts is subjective and the inter-rater variability with regard to PDCU values assigned to the same pot by different experts was estimated around <NUM>-<NUM>%. Despite such variability, the trained regression deep learning model outputs predictions for phytotoxicity values (PDCU) in response to test inputs with a high level of accuracy which exceeds the accuracy of the expert's assessments.

The topology used for implementing the regression deep learning model is based on a deep convolutional neural network topology. It can be based on any of the following topologies: VGG, ResNet, ResNeXt, SE-ResNet, SE-ResNeXt, SE-Net, DenseNet, Inception ResNet V2, Inception V3, Xception, NASNet, MobileNet, or MobileNet v2. The regression deep learning model uses a regression loss function, and further uses an optimizer which can be any of the following: stochastic gradient descent, RMSProp, or Adam. The implementation which provided the most accurate PDCU predictions included a topology based on a ResNet34 topology with a mean-squared-error loss function and a stochastic gradient descent optimizer.

PDCU values are typically assigned in <NUM>% steps <NUM>%, <NUM>%, <NUM>%,. , <NUM>%, <NUM>%, and <NUM>%. Additionally, a <NUM>% value is normally distinguished from <NUM>%. The regression output provides a float value ranging from <NUM> to <NUM>. In one implementation, the output is multiplied by <NUM> to obtain the final PDCU value as a percentage value. It may be rounded to the nearest integer, and finally it may be rounded to a value multiple of <NUM>, except for the admitted value of <NUM>. The output which is obtained by this implementation has the same format and expected values as the PDCU values provided by experts. It is therefore easily comparable with the Ground Truth values provided as annotations by the experts. In this implementation, the regression deep learning model output provides the same resolution (<NUM>%) as provided by the manual PDCU assessments by the experts although the regression model can of course provide higher resolution from a mathematical point of view in other implementations. The last layer for the regression branch (PDCU estimation) in this implementation has a 'sigmoid' activation. The learning rate was also varied with <NUM> being the final value.

<FIG> is a diagram that shows an example of a generic computer device <NUM> and a generic mobile computer device <NUM>, which may be used with the techniques described here. Computing device <NUM> is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Ideally, device <NUM> has a GPU adapted to process machine learning algorithms. Generic computer device <NUM> may correspond to the computer system <NUM> of <FIG>. Computing device <NUM> is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, and other similar computing devices. For example, computing device <NUM> may be used as a GUI frontend for a user to capture test input images and provide them to the computer device <NUM>, and in turn, receive from the computer device, the predicted PDCU value.

In other implementations, multiple processing units and/or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices <NUM> may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a processing device).

The high speed controller <NUM> manages bandwidth-intensive operations for the computing device <NUM>, while the low speed controller <NUM> manages lower bandwidth intensive operations.

The processor may be implemented as a chipset of chips that include separate and multiple analog and digital processing units.

Thus, for example, expansion memory <NUM> may act as a security module for device <NUM>, and may be programmed with instructions that permit secure use of device <NUM>. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing the identifying information on the SIMM card in a non-hackable manner.

As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal.

The systems and techniques described here can be implemented in a computing device that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components.

The computing device can include clients and servers.

A number of embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the scope of the claims.

Claim 1:
A computer-implemented method (<NUM>) for estimating damage on plants (<NUM>) caused by herbicide treatment, the method using a regression deep learning model (RDLM1), comprising:
receiving (<NUM>) a zenithal image (<NUM>-<NUM>) representing a real-world situation at an agricultural growth location (AGL1) with one or more non-overlapping plants (<NUM>), the zenithal image associated with a particular plant species identifier (ID1), wherein the one or more plants have been subject to herbicide treatment, with the zenithal image being recorded a certain number of days after the treatment (DAT1);
segmenting (<NUM>), by using a damage agnostic plant segmentation algorithm, the received image (<NUM>-<NUM>) into pixels associated with plant elements and into pixels associated with non-plant elements to obtain a masked image (<NUM>-<NUM>) with the pixels associated with non-plant elements being ignored for further processing;
if the masked image (<NUM>-<NUM>) does not match the size of an input layer of the single regression deep learning model (RDLM1), resizing (<NUM>) the masked image (<NUM>-<NUM>) to match the size of the input layer while preserving the aspect ratio;
providing (<NUM>) to the input layer a test input comprising the masked image (<NUM>- <NUM>), the respective number of days after treatment (DAT1), and the respective plant species identifier (ID1), wherein the regression deep learning model has been trained with a plurality of plant images for multiple plant species and for multiple days-aftertreatment values, with each training image showing either one or more non-overlapping healthy plants at a control location or one or more non-overlapping damaged plants of a particular plant species at an agricultural growth training location, at a particular number of days after treatment, each training image annotated by an expert with a phytotoxicity value (PDCU*) as ground truth, wherein the phytotoxicity value reflects the expert's assessment of a damage degree of one or more plants on the respective training image in comparison to plants of the control location (CL1) in the vicinity of the agricultural growth training location (AGL1) in accordance with predefined assessment rules; and
receiving (<NUM>), from the regression deep learning model, a predicted phytotoxicity value (PDCU) for the test input.