Patent Description:
The presence of weed communities in crop fields has a negative impact (cf. Weed in the context of this document relates to any plant growing in a field which is different from the crop grown in the field. Studies have identified two main reasons: competition and plant health-issues. Certain plant species (e.g., weeds) compete with crops for soil, nutrients and sunlight causing crops to grow slower and lesser. Also, some weeds are hosts for pests and diseases. For that, farmers use herbicides to exterminate or control weed populations.

The following table includes some examples of weeds and their respective EPPO-Codes:.

Thus, nowadays agriculture faces one complex challenge: The necessity of minimizing the impact on the environment assuring the optimization of the available resources to optimize the food yield. Taking weed control as an example, farmers usually apply the same amount of herbicide per surface disregarding the fact that different weeds have distinct density, growth-rate and growing stage. Nevertheless, biological studies show that the use of different types and rates of herbicides optimizes the effectiveness of the product, achieving better crop growth and reducing the chemical deposition to the environment. The early identification of weeds allows an optimization and increased performance in the use of phytosanitary products, leading to a less intensive and more specific herbicide use.

New technologies have brought Site Specific Weed Management (SSWM) (cf. <NPL>), that includes applying the precise quantity of herbicide only on a region where weed is present. SSWM greatly reduces the use of herbicides by spraying optimally. The two critical tasks when applying SSWM are achieving accurate discrimination between weeds and crops, and appropriate weed quantification and staging. The traditional way to tackle that problem is to manually segment the plants on an image, which costs a great amount of time.

More recently, machine learning techniques based on convolutional neural networks (CNN) have been introduced. Although CNNs have many applications in agriculture, weed quantification has not yet been solved at a satisfactory level. Semantic segmentation for identifying weeds in an agricultural field based on pre-trained standardized CNNs does not perform well enough for plant image datasets due to domain differences. Semantic segmentation implies understanding an image at pixel level, i.e., to assign each pixel in the image an object class. In addition, the intrinsic complexity of segmenting plants with very little visual differences prevents a successful application of standardized CNN topologies for solving the weed identification problem with sufficient accuracy for a farmer.

Mortensen et al. presented a work on semantic segmentation of crop and weeds using deep learning (cf. <NPL>) where they obtained pixel accuracy of <NUM>% at semantic segmentation of different crop species. Later on they were able to distinguish corn crops from <NUM> different weed species correctly labeling the pixels as "corn" or "weed" in real cases with a great pixel accuracy of <NUM>% ( <NPL>). Other authors have worked on semantic segmentation of crops and weeds with deep CNNs to find new architectures and methods that could lead to better segmentation. In <NUM> Sa et al. <NPL>) obtained <NUM>% F1-score at segmenting crop and weed with their modified VGG-<NUM> called weedNet, and Milioto et al. <NPL>) achieved a mloU of <NUM>% at pixel-wise classification of crop, weed and soil. Such prior art works focus on crops, taking every weed species as a single class (in terms of classification). The pixel accuracy obtained with such prior art methods is not yet at a satisfactory level to sufficiently support farmers optimizing their activities to protect their fields. Relevant prior art documents in the field are for example <CIT>, <CIT>.

There is therefore a need to provide systems and methods with improved image analysis functions for the identification of plant species. Thereby, plant species identification as used herein relates to the problem of volume quantification of plants belonging to particular plant species, such as for example, weed species competing with crop in an agricultural field. That is, the result of the plant species identification process is the information about which plant species are present in an agricultural field and where exactly plants of a particular species can be found. Further, there is an interest in gaining additional information about the presence and volume of different parts of respective plants, such as for example, stem, leaves, fruits, etc. of a plant. For example, such information with higher granularity regarding plant elements (e.g., fruits) of a particular plant species can provide useful information with regards to the potential crop yield provided by a particular agricultural field, or even the risk that certain weeds may rapidly expand because of the number of seeds to be expected.

The problem of weed volume quantification is solved by the application of semantic segmentation techniques using a CNN topology which results in a higher pixel accuracy in the segmentation of weeds than achievable with previously known segmentation approaches, such as for example, a standard PSPNet.

Embodiments of the invention comprise a computer-implemented method for identifying plant species in a crop field, and a computer program product with computer readable instructions that, when being stored in a memory of a computer system and being executed by one or more processors of the computer system, causes the one or more processors to execute the method. A further embodiment relates to the computer system, which is configured to execute the computer implemented method (e.g., when running said computer program product).

The computer-implemented method for identifying plant species in a crop field uses a particular convolutional neural network which is referred to herein as dual task CNN. The dual task CNN has a topology with is configured to perform two different tasks. Each of the tasks is associated with its associated loss function and the entire dual task CNN is trained by taking into account the two (different) loss functions. With this approach, the first task - a classification task performed by an intermediated module - is guiding the second task - a segmentation task performed by a semantic segmentation module of the dual task CNN leading to an improved overall accuracy of the plant species segmentation results. The semantic segmentation module is also referred to as "segmentation module" herein.

The intermediate module of the dual task CNN executes the first task in determining plant species which are present on a test input image. Thereby, the first task is associated with a first loss function. Determining a plant species corresponds to a classification task. Therefore, the intermediate module can be implemented by a classification neural network or a regression neural network (e.g., based on a Residual Network using a RESNET* backbone, such as for example, a RESNET50 convolutional neural network). When using a classification neural network (i.e. a neural network configured to perform a classification task), the output is the information about which plant species are present on a particular image showing, for example, crop and weed plants. When using a regression neural network, in addition, the information about the ratios of the present plant species is provided. Both CNN types provide the information about the plant species being present on a test input image with crop and weed plants.

In case a classification neural network is used as intermediate module, the first loss function advantageously is "weighted binary cross-entropy" where each sample (pixel) is weighted depending on the class it belongs to. The intermediate module uses "sigmoid" as last activation layer to support the presence of multiple classes simultaneously. For example, an analyzed section of the test input image (i.e., a tile of the image) may simultaneously include pixels belonging to corn plants, weed plants of different weed species and soil. A sigmoid activation layer can deal with such multiple classes simultaneously when making a prediction regarding the presence of the various classes on the text input image.

Binary cross-entropy and categorical cross-entropy and are known by experts in the field. Weighted categorical cross-entropy: <MAT> is similar to categorical cross-entropy but with the addition of a weight wc. yo,c represents if the target class belongs to the pixel, and ŷo,c is the value predicted by the method. The same applies to binary cross-entropy and weighted binary cross-entropy. Selected weight values wc can range between <NUM> to <NUM>. For example, a weight value can be <NUM> for pixels that were not annotated by the expert. For the annotated pixels, an appropriate weight could be the inverse of the percentage of the pixel class on the dataset.

In case the intermediate module is implemented by a regression neural network the first loss function is advantageously "mean squared error" or "mean average error". The intermediate module may use "linear" or "sigmoid" as last activation layer to support the presence of multiple classes simultaneously.

The segmentation module of the dual task CNN performs a second task in segmenting the test input image to determine a class for each pixel of the test input image. The classes include the determined plant species. The second task is associated with a second loss function which differs from the first loss function. Advantageously, the second loss function is "weighted categorical cross-entropy". For example, the segmentation module may be implemented by a pyramid pooling module (e.g., based on a PSPNet, DeepLab or Piecewise topology).

In other words, each task performed by the dual task CNN is optimized based on its own loss function. However, the intermediate module and the segmentation module are being trained together, taking into account the first and second loss functions. This joint training of the two modules with the two different loss functions has the effect that the training of the intermediate module is affected by the training of the segmentation module and vice versa leading to an improved pixel accuracy of the final segmentation results. The training of a classic PSPNet for semantic segmentation (<NPL>") relies on a two stage training process with generating initial results by supervision with an intermediate segmentation loss, and a second step learning the residue afterwards with a final loss. Thus, optimization of the deep learning neural network is decomposed into two optimization tasks with each optimization task being simpler to solve. However, although this approach can lead to good results, learning from the first (intermediate) loss vanishes while training with the network with the second (final) loss. Despite advantages which can be realized when using a classic PSPNet for sematic segmentation, it lacks the ability for extracting classes that are present in only a few percentages of the pixels of the analyzed image. This problem is solved with the disclosed extension of the classic segmentation module (e.g., PSPNet) by adding a second classification or regression task (performed by the intermediate module) being trained simultaneously with the segmentation task (performed by the segmentation module). This provides a guiding to the learning process by the two loss functions simultaneously.

Contrary to the classic PSPNet approach where the neural network is divided into two different problems that are trained sequentially with a single loss function being active at a given point in time as the training strategy, in the herein disclosed approach both tasks (classification and segmentation task) are being trained at the same time (i.e. simultaneously) by a simple weighted addition of the respective loss functions of both tasks.

The herein disclosed dual task CNN topology extends the classic semantic segmentation network into a real dual task network where network weights are optimized simultaneously against the two loss functions, thus, the classification loss guiding the segmentation loss. Loss functions may be weighted cross-entropy functions where each sample (pixel) is associated with a weight. In the following, a training data set for the dual task CNN is described which combines different data subsets where one data subset includes manually annotated images and a further data subset includes automatically annotated images. The sample weight can be related to the data subset the target belongs to. Samples from the data subset with automatically annotated images may have a higher weight than samples from the manually annotated dataset. Typically, the manually annotated data subset includes pixels that have been classified as 'other' or 'unknown' by a human. On such pixels, the weight may be decreased (e.g., by a number in the range of <NUM> to <NUM>) in order to reduce the influence of such pixels on the training of the dual task CNN while having a remaining small weight to allow domain adaptation to real images. Thereby, the reduction of a weight can however not result in a negative number.

Advantageously, the dual task CNN modules are jointly trained based on an image training data set which includes a combination of two training data subsets with one subset including manually annotated training images and the other subset including automatically annotated training images.

For example, a first data subset may include images showing real world situations in an agricultural field with crop plants of a particular crop species and weed plants of one or more weed species wherein the weed plants are spread between the crop plants. The first data subset has manual pixel annotations indicating the plant species to which the pixels of the training images belong. Typically, a human user is looking at each of the images in the first data set and marks certain subsections of the image as belonging to a particular class (e.g., crop species, weed, species, soil). In one implementation, the manual pixel annotations may be at a higher level of granularity in that not only pixels of plants of a particular plant species are annotated with the respective plant species, but, in a hierarchical manner, the particular plant species may also have sub-classes for the annotation of various plant elements, such as stem, leaf, fruit, etc. That is, the annotation can be performed with tags such as corn1, corn1:leaf, corn1: fruit, weed1, weed1:leaf, weed1:fruit, etc. In most cases, such annotations are quite inaccurate at the pixel level because the user simply indicates rectangle shapes (or other shapes including free form shapes) on the image and enters an annotation for the indicated area. In view of the natural distribution of the classes in a training image it is clear that such manual annotations can only be rough approximations.

For this purpose, the first data subset is complemented (enhanced) by a second subset which includes training images with automatically generated annotations which are correct at the pixel level. Obtaining automatically annotated training images may be achieved in different ways.

For example, a second data subset may include images showing a plurality of plants of different plant species originally obtained from single plant images. Thereby, each single plant image shows a single plant of a particular species. A test image can then be synthesized by extracting from the single plant images the image portions belonging to the respective single plants and pasting the extracted image portions into a soil background image. Thereby, multiple single plant images may be associated with various plant species. However, for each single plant image the respective species is known and the extracted image portions which are later pasted into a soil background image are associated with the respective annotation at the pixel level (because it is known that each pixel of the extracted section shows parts of the plant of the respective species). Therefore, the pixels of the second data subset are automatically annotated with the class (species) they belong to as known from the original single plant images.

For example, another data subset with automatically generated annotations can be a third data subset including real world situation images showing a plurality of (weed) plants of a single (weed) species (typically also showing different growth stages of the same plant species in one image). As the third data subset only includes plants of a single species, the pixels can easily and automatically be annotated with the corresponding class annotations corresponding to the respective plant species. For example, well known leaf segmentation algorithms can be used to extract all pixels from an image of the original real-world single-species image and annotate them with the corresponding class information.

The trained dual task CNN is then applied to a test input image in the following way: A test input is received by the computer system running the dual task CNN. The test input includes an image showing plants belonging to different species. For example, the image may show crop plants of a particular crop species in an agricultural field and weed plants of one or more weed species among said crop plants (i.e., being spread between the crop plants).

The computer system has an image tile extractor which extracts tiles from the test input image having the dimensions of the input shape of the intermediate module. Typically, the test input images are expected to be of high resolution. It is assumed that the dual task CNN has also been trained with images of similar resolution. For example, an image with a resolution of 1024x1024 to <NUM> x <NUM> pixels or more is considered to be a high-resolution image. The dimensions of the input shape (first layer) of the intermediate module however are lower (e.g., the input shape of a typical RESNET50 based classification neural network can be (<NUM>, <NUM>, <NUM>). Therefore, the image tile extractor is dividing the test input image into image tiles matching the input shape of the intermediate module.

In the following, each of the extracted tiles is processed separately and at the end of the segmentation task the segmented tiles are reconstructed into the entire segmented image. For each extracted tile, the intermediate module predicts the presence of one or more plant species which are present in the respective tile. The output of this first (classification) task to the segmentation module is an intermediate feature map with all the features classified by the intermediate module.

The segmentation module uses the intermediate feature map in generating a mask image where each pixel on the mask is associated with a "<NUM>-<NUM>" value (i.e. a value in the interval [<NUM>, <NUM>]) representing the probability for said pixel to belong to the associated class. This is achieved by extracting multiscale features and context information from the intermediate feature map and concatenating the extracted information to perform semantic segmentation.

Finally, the generated masks (a mask for each tile) are combined into a final image. The final reconstructed image corresponds to the original test input image with additional information indicating for each pixel if it belongs to a particular plant species, and if so, to which species it belongs. For example, color coding may be used where each plant species is assigned to a unique color and the pixel colors of the pixels in the final image are adjusted with the assigned color.

When the segmentation module is implemented by a pyramid pooling module for performing semantic segmentation, it typically includes four separate filters with different receptive fields which scan the intermediate feature map provided by the intermediate module, and create four arrays for multi-scale feature detection to integrate information of different scales and sizes.

Further, the pyramid pooling module typically includes a plurality of up-sampling layers configured to restore the image size of each array to the size of the intermediated feature map using bilinear interpolation. Further, convolutional layers of the pyramid pooling module extract contextual information from the four separate filters and concatenate the contextual information with the information of different scales and sizes to generate a final feature map of the same size as the intermediate feature map. Further, the pyramid pooling module typically includes fully-connected layers to compute final pixel-wise predictions as the generated masks with a last activation layer "softmax". The "softmax" activation function is advantageous because it turns numbers aka logits into probabilities that sum to one. Logits are the raw scores output by the last layer of a neural network before activation takes place. In general, the "softmax" function outputs a vector that represents the probability distributions of a list of potential outcomes. Applied to the plant species segmentation problem, the pixels are mutually exclusive in that each pixel can only belong to exactly one class (e.g., the pixel is either soil or a plant of a particular species, but not both at the same time). "softmax" therefore predicts the probability for each pixel to belong to a certain class (e.g., plant species or soil).

<FIG> includes a block diagram of a computer system <NUM> for identifying plant species (e.g., crop or weed species) in a crop field using a dual task convolutional neural network <NUM> according to an embodiment. <FIG> is a simplified flow chart of a computer-implemented method <NUM> for identifying plant species in a crop field according to an embodiment. The method <NUM> may be executed by the computer system <NUM>. In the following detailed description, the method <NUM> of <FIG> is disclosed in the context of the system <NUM> of <FIG>. Therefore, the description refers to reference numbers used in both figures. Further, <FIG> illustrates an example topology of a dual task convolutional neural network <NUM> according to an embodiment. The description will therefore also refer to reference numbers of <FIG> in the context of the description of <FIG> when example embodiments are discussed for components or modules of the computer system <NUM>.

The goal of the computer system <NUM> is to support a farmer to identify the species and the location of plants which grow between crop plants in a section <NUM> of an agricultural field (freeland or greenhouse). Such sections are also sometimes referred to as plots in literature. In the figure, different object shapes are used to distinguish between different plant species. In the example, triangles are used to represent crop plants of a particular species grown in the field. All other shapes represent weed plants of different weed species. The dotted background represents the soil parts in section <NUM> (i.e., the parts of the ground which are not hidden by plants). An image recording device <NUM> (e.g., a digital camera capable of recording high resolution pictures with a resolution in the range of <NUM> up-to 10000px) takes an image of section <NUM> and provides the image as a test input image <NUM> to the computer system <NUM> where it is received <NUM> by a corresponding interface <NUM>. The test input image <NUM> schematically shows crop plants of a crop species <NUM> (triangles) in the agricultural field where section <NUM> belongs to. Further, the test input <NUM> shows weed plants of one or more weed species <NUM>, <NUM>, <NUM> among said crop plants. The weed plants are spread between the crop plants (crop species <NUM>). In a natural field situation, weeds of different weed species can be spread quite regularly or they may appear in certain clusters. In the example, there is a cluster of weed plants of species <NUM> (e.g., Digitaria sanguinalis), a cluster of weed plants of species <NUM> (e.g., Setaria verticillata), and two clusters of weed plants of species <NUM> (e.g., Chenopodium albums). As illustrated in the schematic example, plants in the image <NUM> can have overlapping parts. For example, some crop plants overlap other crop plants and overlap some of the weed plants (as schematically shown in <FIG>). Weed plants may also overlap crop plants.

Besides the interface <NUM> for receiving test input images (and also training images), the computer system has an image tile extraction module <NUM> which extracts tiles from the test input for further processing. Further, an image reconstruction module <NUM> is used to reconstruct the processed tiles at the end into a full-blown segmented image <NUM> which is output to the user (e.g. a farmer). The image processing for achieving a semantic segmentation of the text input image is performed by a dual task convolutional neural network <NUM> (DTCNN). DTCNN <NUM> has two submodules:.

A final post-process interprets and combines those masks to reconstruct all tiles into the final segmented image.

The DTCNN model shows a degree of invariance to different illumination conditions (e.g., of plot <NUM>), leaf overlapping, background and multiscale detection which outperforms the models used in prior art approaches.

Before applying DTCNN <NUM> to a test input, the network gets trained with images of a training dataset whereby the intermediate module <NUM> and the segmentation module <NUM> are trained together, taking into account the first and second loss functions LF1, LF2. This is done directly by minimizing against the two loss functions:<MAT> where alpha can be a number in the range of [<NUM>, <NUM>]. Thereby, "Loss_segmentation" is associated with LF2 and "Loss_classification" is associated with LF1. For example, one may select alpha=<NUM> and consider the weighted_categorical_cross_entropy loss function LF2 for the segmentation task and the weighted_binary_cross_entropy loss function LF1 for the classification task. That is, the training of both modules occurs concurrently with an optimization for two associated loss functions at the same time. As discussed earlier, prior art segmentation networks, such as the PSPNet topology, have two subsequent training stages where the training of the first stage gets pretty much lost when performing the training of the second stage. In contrast, the joint training approach with a separate loss function for each task allows a separated measurement of the performance of each task through the respective loss function while, at the same time, setting the weights for the entire topology of DTCNN <NUM> including the intermediate module <NUM> (for classification tasks) and the segmentation module <NUM> (for segmentation task).

The disclosed network architecture <NUM> was selected by analyzing the intrinsic characteristics that describe the kind of images to be segmented. As color does not provide additional information (weed and crop plants are typically all green), the decision-making is rather to be based on the analysis of shapes and borders of the plants. The DTCNN has three main properties:.

<NPL>et al. (see above) specialized in semantic segmentation for scene understanding. This includes to classify each pixel of an image as part of an object, taking into account the color, shape and location of each element in the image. PSPNet is a standard semantic segmentation network that aggregates two main features: multi-scale information (the pyramidal module) and contextual information. At the <NUM> PASCAL VOC dataset (cf. <NPL>) the PSPNet performed better than other models such as DeepLab (<NPL>) or Piecewise (<NPL>). Further, PSPNet appears to fit to the parameters needed to solving the weed identification problem, as it has a pyramid pooling layer (for multi-scale detection), it specializes in semantic segmentation (high resolution) and scene parsing (contextual information). Nevertheless, a skilled person may also use any of the other semantic segmentation modules known in the art as a basis for the segmentation module <NUM>.

However, the results when applying a classic PSPNet topology to a real field situation image are not satisfying. A problem is that usually semantic segmentation datasets for training present very different classes. Discrepancies can be found in color, shape and textures and thus the different models specialize in gathering all this information to predict each pixel. On the other hand, the classes present on the images with crop and different weed species are very similar classes in shape and color. Differences are primarily found, in small borders and edges of plant leaves (or other characteristic plant elements such as plant fruits). Further, real field images typically show leaf overlapping, changing illumination, as well as different multi-scale and growing stage morphologies. For such reasons pre-trained networks perform poorly for plant image datasets.

The ability of PSPNet to extract contextual information can even be counterproductive. In other detection scenarios, for example, detecting sky as background, can help classifying a plane or aircraft. However, in plant image datasets the background and neighbor objects often look almost the same as the target to be identified. Using that information can actually mislead the classification. Further, all leaves have almost the same color. Usually a single object presents common pixel intensities (color) that distinguish the object from other objects. In this case all leaves look similar on that part, so that color does not provide additional information. Therefore, the training of the DTCNN <NUM> is focused on edges and borders.

To benefit from the advantages of a semantic segmentation module in the application to plant image datasets, a classification task is added to the topology of the DTCNN <NUM>. The model is trained to classify small portions of the image at the same time so that it learns pixel-wise classification. With this modification of a classic semantic segmentation net, such as the PSPNet, improves pixel grouping (with the classification task) without losing focus on detecting minor differences (with the segmentation task). It is thereby critical that the classification task is associated with its own loss function and the segmentation task is also associated with its own loss function, and that both tasks are trained together simultaneously taking into account both loss functions at the same time.

Once the computer system <NUM> has received the test input image <NUM>, the extraction module <NUM> extracts <NUM> tiles from the test input image having the dimensions of the input shape of the intermediate module <NUM>. The input to a CNN is always a 4D array. So, input data has a shape of (batch_size, height, width, depth), where the first dimension represents the number of images processed each time and the other three dimensions represent the dimensions of the image which are height, width and depth. The depth of the image is the number of color channels. For example, RGB image would have a depth of <NUM> and the greyscale image would have a depth of <NUM>. For example, the intermediate module may be implemented as a classification CNN <NUM>-<NUM> or a regression CNN <NUM>-<NUM> based on a RESNET architecture, such as for example, a RESNET50 topology or any other appropriate member of the RESNET family of topologies. The dimensions of the first layer of the intermediate module determine the dimensions for the tiles into which the image is partitioned by the extractor <NUM> for further tile-wise processing. For example, in case of using a RESNET50 CNN topology the dimensions of a tile are adapted to meet a (<NUM>, <NUM>, <NUM>) input shape.

For each tile the intermediate module <NUM> predicts <NUM> the presence of one or more plant species which are present in the respective tile. The output of the intermediate module includes a classification result 121o2 providing (as classes) the plant species which are present on the test input image (of course, besides the classes for weed species, the classification result also includes classes for the crop species and the soil), and further includes a corresponding intermediate feature map with the extracted features associated with the identified classes. Only the intermediate feature map 121o1 is output <NUM> to the segmentation module <NUM> for further processing. The size of the intermediate feature map is a fraction (e.g., <NUM>/<NUM>) of the size of input image (which corresponds to the size of a tile).

The example embodiment in <FIG> illustrates the segmentation module <NUM> being implemented with a pyramid pooling module based on a PSPNet topology. It is to be noted that the PSPNet outperformed other semantic segmentation topologies in test runs of the system. However, a person skilled in the art may also use other segmentation topologies to implement the segmentation module <NUM>. In the PSPNet implementation, the intermediate feature map 121o1 is typically processed by a pooling layer <NUM>-<NUM> performing an initial filter function in selecting from the intermediate feature map the features with the highest activations (i.e. features with a maximum of a local neighborhood of the activations).

The selected features are then forwarded to a filtering layer <NUM>-<NUM> implementing four separate filters with different receptive fields which scan the selected features of the intermediate feature map 121o1 and create four arrays for multi-scale feature detection to integrate information of different scales and sizes.

The filter to the right of the filtering layer <NUM>-<NUM> is the coarsest level which performs global average pooling over each feature map, to generate a single bin output. The filter following to the left is the second level which divides the feature map into <NUM>×<NUM> sub-regions and then performs average pooling for each sub-region. The next filter to the left is the third level which divides the feature map into <NUM>×<NUM> sub-regions and then performs average pooling for each sub-region. The filter to the left is the finest level which divides the feature map into <NUM>×<NUM> sub-regions and then perform pooling for each sub-region. In the example with N=<NUM> filter levels and a number of input feature maps of M=<NUM>, the output feature map is (<NUM>/<NUM>)×<NUM> = <NUM>, i.e. <NUM> number of output feature maps.

The next stage of the pyramid pooling module includes a plurality of up-sampling layers <NUM>-<NUM> configured to restore the image size of each array to the size of the intermediate feature map 121o1 using bilinear interpolation. In general, bilinear interpolation is performed to up-sample each low-dimension feature map to have the same size as the original feature map.

The following convolutional layers <NUM>-<NUM> are configured to extract contextual information from the four separate filters and to concatenate <NUM> the contextual information with the information of different scales and sizes to generate a final feature map <NUM>-4o of the same size as the intermediate feature map <NUM>-o1. In other words, all different levels of up-sampled feature maps are concatenated with the original feature map. These feature maps are fused as global prior. Sometimes in literature, the convolutional layers <NUM>-<NUM> providing the final feature map <NUM>-4o are seen as the end of the pyramid pooling module. However, in the context of this document, the pixel-wise prediction layer <NUM>-<NUM> is also considered to be a layer of the pyramid pooling module.

The pixel-wise prediction layer <NUM>-<NUM> is a convolution layer which uses the final feature map to generate a final prediction map. For example, it may be implemented by fully-connected layers <NUM>-<NUM> to compute the final pixel-wise predictions as generated masks with a last activation layer "softmax" (i.e., normalized exponential function). The advantage of a softmax activation has already been explained earlier. The final prediction result is a pixel-wise segmentation 122o of the currently processed image tile.

Once all extracted tiles have been processed by the DTCNN <NUM>, the image reconstruction module <NUM> reconstructs a completely segmented image <NUM> which corresponds to the size of the original image and includes for each pixel the class to which it belongs. For example, the reconstructed image <NUM> can use a color code to indicate the class of the respective pixel. In the schematic illustration of <FIG>, the segmented image <NUM> uses different textures to differentiate between the classes of the various pixels. For example, surfaces with pixels which are classified as belonging to class <NUM> (crop) are shown with a brick texture. Of course, textures cannot be used to mark a single pixel. However, distinct colors with a particular color value for each class can be used. Therefore, the textures are merely used as a simplified marking in the schematic view to illustrate the marking concept behind. For example, pixels belonging to weed plants of class <NUM> (first weed species) are marked by a grey shading texture, pixels belonging to weed plants of class <NUM> (second weed species) are marked by a chess board like texture, pixels belonging to weed plants of class <NUM> (third weed species) are marked by a stripe pattern texture. Finally, pixels belonging to the soil background class in the image <NUM> are marked by the dotted texture <NUM>.

<FIG> illustrate different methods for the creation of training data subsets which can be used for training the dual task CNN. An image training data set used for training the intermediate module together with the segmentation module of the DTCNN includes at least a first data subset generated using manual annotation of images as disclosed in <FIG> and a further subset generated with automatic annotation as disclosed in any of the <FIG>.

<FIG> illustrated the creation of manually annotated training image of a first data subset of the training data. An original image <NUM> showing a real-world situation in an agricultural field with crop plants of a crop species and weed plants of one or more weed species amongst the crop plants is provided to a human user for manual annotation. The user tries to assign the different elements in the image to the corresponding classes (e.g., crop species, weed species, soil). The image <NUM> and its elements in the example of <FIG> correspond to the image <NUM> and its elements in <FIG>. The result of the manual annotation task is for each training image belonging to the first data subset that the manual pixel annotations <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM> indicate the species to which the pixels of the respective training image belong. The textures used in the annotated image <NUM>-a correspond to the textures explained for image <NUM> in <FIG>. In the example of <FIG>, the result of the manual annotation is schematically shown only the upper right corner of the original image <NUM>. Although the schematic view implies that the annotation is correct at the pixel level this is not the case in reality for a manually annotated image. Typically, there are many pixels in a manually annotated image which are either assigned to a wrong class or to no class at all because the user was not able to recognize a certain plant. In other words, the manual annotations are noisy in the sense that many pixels are not correctly annotated.

Typically, a user is just selecting rectangles in the image and assigns such rectangles to a class. Rectangle R1 may be used to classify the pixels inside the rectangle as the crop species <NUM>-<NUM>. However, R1 also contains pixels which relate to weed species <NUM>-<NUM> and <NUM>-<NUM>. The user may indicate further rectangles R2, R3 within R1 or overlapping with R1 to assign them to the respective weed species classes. It is clear that such an annotation method cannot lead to a correct annotation at the pixel level. To support the user in the manual annotation task, the computer system may provide some classification support functions to the user.

For example, the system may provide for automated soil segmentation: A robust and simple color-based segmentation algorithm can be used automatically remove the presence of ground (soil) and automatically subtract it from the manual segmentation. An example algorithm is based on simple thresholding over the Lab color channel (of the L*a*b* Color space) where the pixels with positive values of channel a are removed from the segmentation to obtain a refined segmentation.

Further, the system may provide support for overlapping plant parts: Especially on later phenological stages, plant overlapping makes the annotation more complicated to precisely segment all classes. To alleviate this, the manual annotation function of the system allows marking an annotation inside of another annotation (e.g. R2 inside of R1). In this case, the inner annotation (assigned to R2) is removed from the segmentation belonging to the outer annotation (assigned to R1). This simplifies the annotation process as there is no need to precisely annotate all species. It is sufficient to annotate only the species overlapping with the "enclosing" annotation or any other annotation indicating overlap.

To generate the first image data subset, the following conditions prevailed in test runs for the system. An extensive image acquisition campaign was carried out in two different locations in Germany and Spain in the year <NUM>. A set of <NUM> plots with each of <NUM>. <NUM> were planted. On these plots, two rows of corn (Zea mays) were planted along with <NUM> different weed species, three "grass leaf" weeds (Setaria verticillata, Digitaria sanguinalis, Echinochloa crus-galli) and three "broad leaf" weeds (Abutilon theophrasti, Chenopodium album, Amaranthus retroflexus). Each plot was imaged with a top view and perspective view using two different devices: a Canon EOS 700D SLR camera and Samsung A8 mobile phone. To facilitate image acquisition, a metallic structure was created to hold two mobile phones and two SLR cameras to acquire a top image (<NUM> meters height, <NUM> focal length) and a perspective image (<NUM> meters height, 30º angle, <NUM> focal length). Such four images may be taken simultaneously to save time but this has not impact on the quality of the training data.

Images were taken twice a day, three times a week over a period of <NUM> weeks in order to gather different phenological stages of corn and weeds. Trials started in May <NUM> and ended in June <NUM>. After removing overexposed and/or blurred images a total number of <NUM> images were manually segmented into the <NUM> targeted classes that are named according to their corresponding EPPO codes (ZEAMX, SETVE, DIGSA, ECHCG, ABUTH, CHEAL, AMARE).

Although the targeted weeds were planted at specific positions, wild growing of unknown weeds on the experimental plots made this task more complex. In order to cope with this issue, two new classes (generic broad leaf weed and generic grass leaf weed) were added allowing the annotation of unknown or not targeted weeds. The DTCNN topology was adapted to ignore these noisy annotations.

For training purposes, and to avoid any biasing, the experimental plots were separated into train, test and validation plots. <NUM> plots were used for training, <NUM> for validation and another <NUM> for testing.

The first data subset was then combined into the training image dataset with at least one further subset which can be either the second or the third data subset described in the following:.

The generation of the second data subset is described in <FIG>. The second set is composed of synthetically generated images which can be automatically annotated in a correct manner at the pixel level. A final synthetically generated image <NUM>-a of the second subset shows a plurality of weed plants of different weed species obtained from original single plant images <NUM> with each single plant image showing a single plant <NUM> of a single species. The single plant elements are extracted from the single plant images (e.g. with a leave segmentation algorithm) and pasted into a soil background image <NUM>-b. Therefore, as the extracted single plant elements belong to known plant species (indicated by different textures in image <NUM>-s), the second data subset can be completely automatically annotated at the pixel level with annotations <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM> of the pixels belonging to the respective weed species. Such automatic annotations include far less noise than the manual annotations as the pixels extracted by the leaf segmentation algorithm include substantially only the pixels which really belong to the single plant and which can then be automatically annotated with the known species of the plant of the respective single plant image.

The combination of the first data subset with the second data subset overcomes several drawbacks of the first data subset where the annotation is difficult and prone to error due to the dataset's substantial complexity. As a consequence, the amount of annotated images for training and testing is limited and noisy. This can be overcome by using the synthetic images of the second subset containing image communities generated by single plant images in combination with the first subset. An additional acquisition campaign of single plants was performed for this purpose.

The synthetic dataset featured three new weed species: Chenopodium, Datura stramonium and Fallopia convolvulus. It consists of images with each image showing a single plant on a greenhouse 80x80cm plot. There were two greenhouses from Spain. In each of them different species were sowed: AMARE, DIGSA, ECHCG and SETVE in Greenhouse <NUM>; ABUTH, CHESS, DATST, POLCO and ZEAMX in Greenhouse <NUM>. There was a total of <NUM> weeds and <NUM> crop. Out of each species <NUM>-<NUM> single plants were sowed. A single image was taken every labour day (M-F) for each of the individual plants, from day <NUM> to day <NUM>. Not all of them made it to the last day so the final (second) data subset contained <NUM> images of single plants of <NUM> different species and at different growing stages.

Since only one plant appears in each image, all images in the subset are labeled. Using a deep learning model for leaf segmentation allowed to automatically annotate the entire dataset. A synthetic plant community generator algorithm can take real leaf segmented images and paste them on a real background image. Using the single plant dataset allowed to automatically segment leaves and/or plants and store them into a candidate repository. After discriminating which candidates were viable the final folder contains <NUM> images unevenly divided in <NUM> species. The community generator algorithm takes the candidates from the repository and pastes them in a specific way onto a soil image.

To generate these images, several random regions associated to three parameters describing a respective region are created. The model parameters are: plant species, growing stage and density. The plant species are grown following a Monte-Carlo approach accordingly to the region's parameters. The pipeline of this algorithm is the following:.

By this method images were generated where several plant species are present at varying growing stages with in-homogeneous densities. The second data subset was created with <NUM> synthetic images. Out of the <NUM> generated plot images, <NUM>% were reserved for training, <NUM>% for validation and another <NUM>% for testing.

<FIG> illustrates a third data subset which can be alternatively used in combination with the first data subset to form the training image data set. Of course, all three data subsets may be combined as well into the training dataset. The third data set includes images <NUM>-a showing a plurality of weed plants of a single weed species. The images of the third data subset are also automatically annotated with the annotations <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM> of the pixels belonging to the single weed species.

The synthetic second data subset may have some issues for appropriately mimicking real plant communities growing with overlapping plant elements while the first data subset presents unbalanced classes and noisy annotations. An example of a situation with unbalanced classes is to have one class which is associated with <NUM>% of the pixels, and another class which is associated with only <NUM>% of the pixels. Besides this, there can be pixels that are unknown which can be of any of the classes. The third data subset contains images of plants growing in a controlled environment having a single species on each plot. The plot fields were checked daily and any time a plant of another species grew, it was manually removed. Having a single species per plot implies that all the images are already labeled and hence automatic segmentation can be achieved. There were plots of three densities (number of crop plants per area): high, medium and sparse. Images were taken in two campaigns, one in Spain with <NUM> images and the other one in Germany with <NUM> images. There were substantial differences between Spanish and German images, especially in the soil/background, though the concept is the same.

Using a leaf segmentation algorithm (e.g. the leaf segmentation algorithm described earlier), automatically generated labelled masks for each image are obtained which serve as semantic segmentation ground-truth labels. Although this segmentation method still makes a few mistakes at the pixel level the third data subset can be considered as precisely annotated.

The second and third data subsets are similar but complementary in their differences: the second data subset is more realistic in terms of plant community growing as it presents several species in the same image, whereas the third data subset presents better textures, overlapping, shadows and shapes (i.e., more information) of real field images although only one species is present.

Different training experiments were performed by including combinations of the three data subsets. All the experiments were evaluated against the validating and testing of the first data subset. In some of the experiments that more than one image training dataset was used for training. Because the data subsets had different numbers of images a generator was used to fetch images from the different data subsets in an equal manner. The generator takes one image from each data subset each time. When a data subset runs out of images (i.e. the generator retrieves the last image of the respect subset) it starts over again with the respective subset while incrementing the images in the other subset(s).

In order to avoid bias, as already mentioned each data subset was divided into <NUM>% of the images for training, another <NUM>% for validation and a final <NUM>% for testing.

Data augmentation was applied every time a new image was fetched by the generator. Transformations applied for data augmentation included: rotation, height and/or width shift, zoom, vertical and/or horizontal flip, pixel-intensity shift (color change) and Gaussian blur. Shear is not recommended as the herein disclosed semantic segmentation method extracts tiles from the image and it is important to keep coherence.

The program code was implemented with the Keras Deep Learning library using TensorFlow as background. Stochastic Gradient Descent was used as optimizer for both tasks, using a learning rate of Ir = <NUM> with a decay = <NUM>-<NUM> per epoch, momentum = <NUM> and Nesterov's acceleration. Balanced Accuracy (BAC) was selected as the most suitable algorithm performance metric, in order to account for the class imbalance present in the data subsets (in such cases, the use of "regular" accuracy is discouraged).

For training the DTCNN a NVIDIA Tesla V100 GPU with 16GB of memory was used. Considering the size of the input images the batch size was set to <NUM>. Following the same methodology described by <NPL>, and by <NPL>, the validation subset of the first data subset and the computed values of balanced accuracy (BAC) and Dice-Sorensen Coefficient were used to calculate the threshold values that maximize the validation set for the different weed species.

Various experiments were tested using the images for testing from the first data subset as they represent real field conditions. To measure influence of the use different datasets trained several models were trained combining different data subsets. Two sets of experiments were used. One set focused on validating the performance of the proposed dual task CNN based on a PSPNet topology for the segmentation module, and another set for measuring the influence on the different data subset combinations.

Two experiments focused on validating that dual task PSPNet implementation has better performance than the normal single task PSPNet (experiments are named by the used topology and the number of the used data subsets):.

The obtained results show that the use of a dual task (classification and segmentation) CNN obtained an average Dice-Sorensen Coefficient (DSC) of ~ <NUM>% against the ~ <NUM>% obtained when using the classical architecture. Further, balanced accuracy is improved slightly. Both models show a peak performance for images recording during the second week after sowing (mid-stage). Further, Dual task PSPNet <NUM>st + <NUM>rd provides better scores than PSPNet, especially at early stages. Although its performance decreases faster than PSPNet as time passes. The worst DTCNN predictions (for images recorded during the fourth week after sowing) attain similar values than the classic PSPNet predictions.

The influence of the various data subsets on the performance of the dual task CNN was validated by the following experiments:.

DTCNN <NUM>st:: in this experiment training was performed over the first data subset only. This dataset had several issues: a scarce number of images, high complexity, inaccurate annotation and high class unbalance.

DTCNN <NUM>nd:: In this experiment, the synthetic second data subset was used for training. A performance decrease was expected due to domain shift as the synthetic images present differences in spatial distribution, illumination, background and scales. The information about shapes and edges of the proper leaves is appropriate for training with almost perfect ground-truth annotation because the automatic annotation of the pixels ensures that each pixel is annotated with the correct class.

DTCNN <NUM>rd: In this experiment, the single species (third) data subset is used for training. Although the plant images are obtained under real conditions, plant communities interaction cannot be obtained from this dataset.

DTCNN <NUM>st+ <NUM>nd: On this experiment, images from the <NUM>st and <NUM>nd data subsets are combined for training. The second data subset allows reducing the effect of class unbalancing and bad quality annotation from the first data subset by incorporating synthetic images.

DTCNN <NUM>st + <NUM>rd: On this experiment, images <NUM>st and <NUM>rd data subsets are combined for training. The third data subset allows to reduce the effect of class unbalancing and bad quality annotation from the first data subset by including the single species images from the third data subset.

DTCNN <NUM>st + <NUM>nd + <NUM>rd: the last model complements all data subsets.

To conclude, when the targeted first data subset is combined with any of the supporting datasets (<NUM>nd, <NUM>rd), domain shift is reduced obtaining more accurate results. The best results were obtained when using the first and third data subsets in combination for training the DTCNN.

<FIG> illustrates a scenario where a single plant image <NUM> (as used for automatic annotation in <FIG>) shows further elements of a plant. Besides the leave <NUM> a stem <NUM> and fruits 12f of the plant are visible on this image <NUM>. For many plants, the fruits have a color which is different from the color of the leaves of the stem. In such cases, existing segmentation methods can be used to segment pixels belonging to the fruits 12f and pixels belonging to the leaves <NUM> of the plant (or other elements of the plant having the same color as the leaves). Then, in a similar way as explained for <FIG>, not only the leaves of the plant but also its fruits can be pasted into a synthetically generated image <NUM>-s. In combination with the background image <NUM>-b the more realistic annotated image <NUM>-a is generated which now also includes annotated objects <NUM>-<NUM> representing the fruits of plant <NUM>* (besides the objects <NUM>-<NUM>, <NUM>-<NUM> and <NUM>-<NUM> as known from <FIG>).

It is to be mentioned that a person skilled in the art can also used color differences between fruits and leaves of a plant to modify the method explained in <FIG> to generate automatically annotated images including objects representing leaves and fruits of the respective plants. When it comes to the annotation of other plant elements (e.g., the stem) which are of similar colors as the leaves, manual annotation of such elements may be used.

When now using automatically annotated images which also include representations of plant fruits, the DTCNN can be trained to not only distinguish between different plant species but also to segment the image into pixels which belong to the respective fruits of a plant (e.g., crop). Normally, only one crop species is grown in an agricultural field. In such case, it is sufficient to train the DTCNN with automatically annotated images which include leaves and fruits of this crop species and the images of other plant species (weeds) as described earlier. <FIG> is a diagram that shows an example of a generic computer device <NUM> and a generic mobile computer device <NUM>, which may be used with the techniques described here. Computing device <NUM> is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Ideally, device <NUM> has a GPU adapted to process machine learning algorithms. Generic computer device <NUM> may correspond to the computer system <NUM> of <FIG>. Computing device <NUM> is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, and other similar computing devices. For example, computing device <NUM> may be used as a GUI frontend for a user to capture test input images and provide them to the computer device <NUM>, and in turn, receive from the computer device, a segmented image indicating the location(s) of various weed plant and the respective species of the weed plants on the image. Thereby computing device <NUM> may also include the output device <NUM> of <FIG>.

In other implementations, multiple processing units and/or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices <NUM> may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a processing device).

The processor may be implemented as a chipset of chips that include separate and multiple analog and digital processing units.

Thus, for example, expansion memory <NUM> may act as a security module for device <NUM>, and may be programmed with instructions that permit secure use of device <NUM>. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing the identifying information on the SIMM card in a non-hackable manner.

As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal.

The systems and techniques described here can be implemented in a computing device that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components.

The computing device can include clients and servers.

In addition, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. In addition, other steps may be provided, or steps may be eliminated, from the described flows, and other components may be added to, or removed from, the described systems. Accordingly, other embodiments are within the scope of the following claims.

Furthermore, the embodiments of the present invention, especially the methods of the present invention, may be used for interacting with, operating, controlling, and/or monitoring farming machinery. As a preferred embodiment of the present invention, the methods of the present invention further comprise a step to output a signal, such as a control signal or an on-off signal, for operating, controlling, and/or monitoring farming machinery. As an advantageous embodiment of the present invention, the methods of the present invention further comprise a step to output a signal, such as a control signal or an on-off signal, for operating, controlling, and/or monitoring farming machinery, depending on the outcome of the weed identification or plant identification steps in the method of the present invention. More preferably, if a specific weed is identified, a control or on-off signal for operating farming machinery in a way targeting this specific weed is outputted, for example a control signal for operating farming machinery in order to spray or apply or in order to prepare for spraying or applying a herbicide or another crop protection agent targeting this specific weed is outputted. Advantageously, if a specific weed is identified and if a certain predefined threshold value related to this specific weed - for example regarding the weed quantity, or the weed volume quantity, or the area (e.g. hectares) or number of geographic locations where this weed has been identified - is exceeded, a control or on-off signal for operating farming machinery in a way targeting this specific weed is outputted. For example, a control signal for operating farming machinery in order to spray or apply or in order to prepare for spraying or applying an herbicide or another crop protection agent targeting this specific weed is outputted. Farming machinery may include one or more treatment mechanisms to treat plants in a field. Treatment mechanisms include chemical, mechanical, electrical treatment mechanisms or a combination of such treatment mechanisms to treat weeds, diseases or insects. The farming machinery may further include a detection and a control system. The detection system may be configured to detect in field conditions as the smart machinery moves through the field. The control system may be configured to control treatment mechanism(s) based on the detected field conditions.

In one embodiment, the treatment mechanism is a chemical treatment mechanism. The farming machinery in such embodiment includes a sprayer with one or more nozzle(s) to release chemical agent or a crop protection agent to the field.

In one embodiment, the detection system comprises one or more detection component(s) to detect field conditions as the farming machinery traverses through the field. The detection component may be an optical detection component such as a camera taking images of the field. The optical detection component may be for example the image recording device <NUM> (cf.

In a further embodiment, the farming machinery includes one or more treatment element(s) associated with one or more detection component(s). In such embodiment the detection components may be arranged in front of the treatment element(s) when seen in drive direction. This way the detection component can sense the field condition, the system can analyze the sensed field condition and the treatment element can be controlled based on such analysis. This allows for targeted treatment based on the real-time field condition as present at the time of treatment while the farming machinery traverses in the field.

In a further embodiment, the sprayer includes multiple nozzles associated with multiple optical detection components. In such embodiment the optical detection components are arranged in front of the nozzles when seen in drive direction. Furthermore, each of the optical detection components is associated with a nozzle, such that the field of view of the optical component and the spray profile of the associated nozzle at least partly overlap as the sprayer moves through the field.

In a further embodiment, the control system is configured to analyze the sensed field condition as provided by the detection system. Based on such analysis the control system is further configured to generate control signals to actuate the treatment mechanism once the position of the treatment mechanism reached the field position that was analyzed.

<FIG> illustrates smart farming machinery <NUM> as part of a distributed computing environment.

The smart farming machinery <NUM> may be a smart sprayer and includes a connectivity system <NUM>. The connectivity system <NUM> is configured to communicatively couple the smart farming machinery <NUM> to the distributed computing environment. It may be configured to provide data collected on the smart farming machinery <NUM> to one or more remote computing resources <NUM>, <NUM>, <NUM> of the distributed computing environment. One computing resource <NUM>, <NUM>, <NUM> may be a data management system <NUM> that may be configured to send data to the smart farming machinery <NUM> or to receive data from the smart farming machinery <NUM>. For instance, as detected maps or as applied maps comprising data recorded during application may be sent from the smart farming machinery <NUM> to the data management system <NUM>. A further computing resource <NUM>, <NUM>, <NUM> may be a field management system <NUM> that may be configured to provide a control protocol, an activation code or a decision logic to the smart farming machinery <NUM> or to receive data from the smart farming machinery <NUM>. Such data may also be received through the data management system <NUM>. Yet a further computing resource <NUM>, <NUM>, <NUM> may be a client computer <NUM> that may be configured to receive client data from the field management system <NUM> and/or the smart farming machinery <NUM>. Such client data includes for instance application schedule to be conducted on certain fields with the smart farming machinery <NUM> or field analysis data to provide insights into the health state of certain fields.

<FIG> illustrates an example of a smart sprayer system.

The system comprises a tractor with a sprayer <NUM> for applying a pesticide such as an herbicide, a fungicide or an insecticide. The sprayer <NUM> may be releasably attached or directly mounted to the tractor. The sprayer <NUM> comprises a boom with multiple nozzles <NUM> arranged along the boom of the sprayer <NUM>. The nozzles <NUM> may be arranged fixed or movable along the boom in regular or irregular intervals. Each nozzle <NUM> includes a controllable valve to regulate fluid release from the nozzles <NUM> to the field.

One or more tank(s) <NUM> are in fluid connection with the nozzles <NUM> through pipes <NUM>. Each tank <NUM> holds one or more component(s) of the fluid mixture to be distributed on the field. This may include chemically active or inactive components like an herbicide mixture, components of an herbicide mixture, a selective herbicide for specific weeds, a fungicide, a fungicide mixture, a fungicide and plant growth regulator mixture, a plant growth regulator, water, oil, or the like. Each tank <NUM> may further comprise a controllable valve to regulate fluid release from the tank <NUM> to the pipes <NUM>. Such arrangement allows to control the mixture released to the field.

Additionally, the smart sprayer system includes a detection system <NUM> with multiple detection components <NUM> arranged along the boom. The detection components <NUM> may be arranged fixed or movable along the boom in regular or irregular intervals. The detection components <NUM> are configured to sense one or more field conditions. The detection component <NUM> may be an optical detection component <NUM> providing an image of the field. Suitable optical detection components <NUM> are multispectral cameras, stereo cameras, IR cameras, CCD cameras, hyperspectral cameras, ultrasonic or LIDAR (light detection and ranging system) cameras. Alternatively, or additionally, the detection components <NUM> may include sensors to measure humidity, light, temperature, wind or any other suitable field condition.

The detection components <NUM> are arranged in front of the nozzles <NUM> (seen from drive direction). In the embodiment shown in <FIG>, the detection components <NUM> are optical detection components and each detection component <NUM> is associated with a single nozzle <NUM> such that the field of view comprises or at least overlaps with the spray profile of the respective nozzle <NUM> on the field once the nozzle reach the respective position. In other arrangements each detection component <NUM> may be associated with more than one nozzle <NUM> or more than one detection component <NUM> may be associated with each nozzle <NUM>.

The detection components <NUM>, the tank valves and the nozzle valves are communicatively coupled to a control system <NUM>. In the embodiment shown in <FIG>, the control system <NUM> is located in the main sprayer housing and wired to the respective components. In another embodiment, detection components <NUM>, the tank valves or the nozzle valves may be wirelessly connected to the control system <NUM>. In yet another embodiment, more than one control system <NUM> may be distributed in the sprayer housing or the tractor and communicatively coupled to detection components <NUM>, the tank valves or the nozzle valves.

The control system <NUM> is configured to control and/or monitor the detection components, the tank valves or the nozzle valves following a control protocol. In this respect the control system <NUM> may comprise multiple modules. One module for instance controls the detection components to collect data such as an image of the field. A further module analyses the collected data such as the image to derive parameters for the tank or nozzle valve control. Yet further module(s) control(s) the tank and/or nozzle valves based on such derived parameters.

<FIG> illustrates the control protocol for the smart sprayer system to control weeds, diseases or insects via a chemical control mechanism.

The control protocol of the smart sprayer system may be triggered once the smart sprayer activates application operation on the field. In a first step <NUM>, the optical detection components are triggered to provide data such as an image of the field. In a second step <NUM>, the provided data such as the images provided by each optical detection components are analyzed with respect to weeds, diseases or insects depending on the target of the chemical control mechanism. In the context of the present invention, such images are analyzed using the method of the present invention. In a third step <NUM>, parameters are derived from such analysis to derive and/or output control signals for the tank and nozzle valves. For example, if specific weeds are identified using the method of the present invention, control signals for the tank and nozzle valves in order to spray or apply or to prepare for spraying or applying specific herbicides or crop protection agents targeting the identified weeds are derived and/or outputted. In a fourth step <NUM>, such control signals are provided to the respective tank and/or nozzle valves.

Owing to the system set up each tank and nozzle valve can be controlled individually. Hence, if only one image shows the presence of a weed only the respective nozzle associated with that optical detection component having the spray profile covering the field of view of that optical detection component will be triggered. Similarly, if multiple images show the presence of a weed - after an image analysis using the method of the present invention has been conducted - the respective nozzles associated with those optical detection components having the spray profile covering the fields of view of those optical detection components will be triggered.

In addition to such targeted treatment, the control of tank valves allows to adjust the treatment composition in dependence on the conditions sensed by the optical detection components in the field. For instance, first tank may include a first herbicide comprising a first active ingredients composition and a second tank may include a second herbicide comprising a second active ingredients composition. Depending on the outcome of the image analysis using the method of the present invention, the valve of the first or the second or both tanks may be triggered to provide respective herbicides for application on the field.

In another advantageous embodiment, a variable rate application (VRA) map for applying crop protection agents may be generated on the basis of the image analysis using the methods of the present invention, wherein the to-be-analyzed images are obtained, for example, through image recording device <NUM> which may be mounted on an agricultural machine, an unmanned aerial vehicle (e.g. a drone), or any movable equipment. This variable rate application (VRA) map may be used later by another agricultural machine, unmanned aerial vehicle, movable equipment for applying herbicides or crop protection agents.

Claim 1:
A computer-implemented method (<NUM>) for identifying plant species in an agricultural field using a dual task convolutional neural network (<NUM>) having a topology with:
an intermediate module (<NUM>) configured for executing a first task in determining plant species (<NUM>, <NUM>, <NUM>) which are present on a test input image (<NUM>), the first task being associated with a first loss function (LF1), and
a semantic segmentation module (<NUM>) configured for executing a second task in segmenting the test input image (<NUM>) to determine a class for each pixel of the test input image (<NUM>), the classes comprising the determined plant species, the second task associated with a second different loss function (LF2),
wherein the intermediate module and the segmentation module being trained together, taking into account the first and second loss functions (LF1, LF2);
the method comprising:
receiving (<NUM>) a test input (<NUM>) comprising an image showing plants of a plurality of plant species in an agricultural field;
extracting (<NUM>) tiles from the test input image, the tiles having the dimensions of the input shape of the intermediate module;
for each extracted tile:
the intermediate module (<NUM>) predicting (<NUM>) the presence of one or more plant species which are present in the respective tile;
the intermediate module (<NUM>) outputting (<NUM>) a corresponding intermediate feature map (<NUM>-o1) to the segmentation module (<NUM>) as output of the first task;
the segmentation module generating (<NUM>) a mask for each plant species class as segmentation output of the second task by extracting (<NUM>) multiscale features and context information from the intermediate feature map and concatenating (<NUM>) the extracted information to perform semantic segmentation, the mask being an image having the same size as a tile where each pixel on the mask is associated with a value representing the probability for said pixel to belong to the associated class; and
combining (<NUM>) the generated masks into a final image indicating for each pixel if it belongs to a particular plant species, and if so, to which plant species it belongs.