Patent Description:
It is well known that agricultural plants - such as crops - grow in environments in that they co-exist with biological objects. These objects tend to be located at the plant, usually by being attached to the plants (at least temporarily); and the objects interact with the plant.

Different objects tend to attach themselves to different parts of the plants. For example, some insects may sit on the leaves, some others may stick to the stem or to a branch, and so one.

From the perspective of the farmer, the object-and-plant interaction has two directions or aspects. In a first direction, there are biological objects with a detrimental effect on the plants. For example, animal pests destroy crops, causing large economic loss to the food supply and to property. To give an illustrative example, a pest animal may eat from the leaves or may eat from the fruits of the crop.

In the other direction, there are also animals to the benefit of the plant. For example, the plant may have a direct benefit when a butterfly visit its flower or blossom, or the plant may have an indirect benefit if a ladybug eats aphids (or other pest).

The farmer may control the presence of these objects: to favor the absence of some objects (such as pest animals) and the presence of beneficiary objects.

The control measures are usually adapted to the quantity of the objects on the plants. In principle it does not matter where the object is located on the plant (leaf, stem, fruit or wherever) and what direction the interaction has.

To illustrate some of these aspects by way of example, insects of many species live on plant leaves. For example, whiteflies live on the leaves of eggplants.

In the broadest sense, the insects interact with the plant (for example by consuming part of the leaves). The insects can cause diseases or other abnormal conditions of the plant. Eventually, the plant does not survive the presence of the insects. But in agriculture, the plants should become food (i.e. crop for humans or animals), and insects being present on leaves are not desired at all. Food security is of vital importance.

Usual terms for such phenomena are "infestation" and "pest". Farmers apply countermeasures (e.g., treatment by applying insecticides) in order to remove the insects.

However, applying countermeasures may cause further problems or challenges. Countermeasures must be specific to particular insects, for example to remove the whiteflies but to keep the bees and others. Countermeasures should also take the quantity of the insects into account.

Quantifying the infestations, such as by counting insects (on plant leaves) is therefore an important task for pest management.

In theory, farmers could visually inspect the plant and could count the insects (taking the insect development stages into account). As different people have different knowledge (regarding insects) and have different eyes, different people would arrive at different numbers.

Using computer vision techniques appears as an improvement. A well-known (classical or traditional) approach is the extraction of image features with subsequent classification. However, there are many constraints arising. The constraints have many aspects, such as limitations of the computers and cameras, non-ideal conditions in the field and constraints related to the insects themselves.

<CIT> explains an approach to selectively sterilizing insects. Insects are being reared, placed on a surface, photographed and selectively manipulated by robots. A computer processes the image and identifies location data for particular inspects, and thereby differentiates male insects from female insects. With the location data, the robot can then perform actions with respect to insects, such as removing particular insects.

In this article, Wu et al explain a large-scale database for insect pest recognition.

The constraints are addressed by a computer system, a computer-implemented method and a computer program product for quantifying biological objects on plant parts (in the example: quantifying plant infestation, estimating the number of insects on leaves of a plant).

The computer program product - when loaded into a memory of a computer and being executed by at least one processor of the computer - performs the steps of the computer-implemented method.

In a production phase, the computer applies convolutional neural networks that had been trained previously in a training phase. The production phase is summarized first:.

The computer receives a plant-image taken from a particular plant. The plant-image shows at least one of the leaves of the particular plant, the so-called main leaf (or main part).

The computer uses a first convolutional neural network to process the plant-image to derive a leaf-image being a contiguous set of pixels that show a main part of the particular plant completely (i.e. as a whole). The first convolutional neural network has been trained by a plurality of leaf-annotated plant-images, wherein the plant-images had been annotated to identify main parts.

The computer splits the leaf-image into a plurality of tiles. The tiles are segments or portions of the plant-image having pre-defined tile dimensions.

The computer uses a second convolutional neural network to separately process the plurality of tiles to obtain a plurality of density maps having map dimensions that correspond to the tile dimensions. The network having been trained by processing object-annotated plant-images, and the training comprised the calculation of convolutions for each pixel based on a kernel function, leading to density maps. The density maps have different integral values for tiles showing biological objects and tiles not showing biological objects.

The computer combines the plurality of density maps to a combined density map in the dimension of the leaf-image, and integrates the pixel values of the combined density map to an estimated number of biological objects for the part of the plant.

The first convolutional neural network - that is the network to identify the main leaf - can be of the DenseNet type. The second convolutional neural network - that is the network to estimate the number of biological objects on the part - can be of the convolutional neural network (FCRN) type.

The second convolutional neural network (FCRN type) can be a modified network that uses global sum pooling instead of global average pooling.

The convolutional neural network can be also be modified by using dropout.

The second convolutional neural network can also be modified by implementing an input layer as a pixel value filter for individual pixels or for pixel pluralities, the so-called tile segments.

For the pixel pluralities, the network uses a layer to convolute the pixel pluralities, to encode the convoluted segments into segment values so that a further layer applies filtering to the segment values. The second convolutional neural network can also been implemented with a layer to subsequently decode the segment values to the convoluted segments. This approach lets the network layers between the encoder and the decoder operate with numerical values for the segments and not with numerical values for the pixels. Since the number of segments is less than the number of pixels, the approach uses less computation resources (compared to pixel processing).

The second convolutional neural network can be trained by processing object-annotated plant-images for different classes. Classes are identified by object species (e.g., insect species) and by growing stages of the objects. Processing is separated for the classes by branches and output channels.

This approach can be advantageous - in the scenario that the biological objects are pest because it quantifies the plant infestation (by pest) with a granularity that allows fine-tuning countermeasures to particular pest classes.

In the production phase, receiving the plant-image can be performed by receiving the plant-image from the camera of a mobile device. This can be advantageous because mobile devices are readily available to farmers and because the mobile device allows the immediate communication of the plant-image to the computer that quantifies the plant infestation.

Receiving the plant-image can comprise evaluating the class (e.g., the pest class) and the pixel resolution of the camera of the mobile device according to pre-defined rules, wherein for some classes and resolutions, the mobile device is caused to take the image with a magnifying lens. This measure can improve the accuracy of the estimation. The farmer can be instructed via the user interface of the mobile device to apply the lens.

The second convolutional neural network can have been trained by using a loss-function being the mean absolute error or being the mean square error.

The description starts by explaining some writing conventions.

The term "image" stands for the data-structure of a digital photograph (i.e., a data-structure using a file format such as JPEG, TIFF, BMP, RAW or the like). The phrase "take an image" stands for the action of directing a camera to an object (such as a plant, or a part of a plant) and letting the camera store the image.

The description uses the term "show" when it explains the content of images (i.e., the semantics), for example in phrases such as "the image shows a plant". There is however no need that a human user looks at the image. Such computer-user interactions are expressed with the term "display", such as in "the computer displays the plant-image to an expert", where an expert user looks at the screen to see the plant on the image.

The term "annotation" stands for meta-data that a computer receives when an expert user looks at the display of an image and interacts with the computer. The term "annotated image" indicates the availability of such meta-data for an image (or for a subregion of that image), but there is no need to store the meta-data and the image in the same data-structure. Occasionally, the drawings illustrate annotations as part of an image, such as by polygon and/or dots, but again: the annotations are meta-data and there is no need to embed them into the data-structure of the image.

The images will only show plants with their above-ground (air) components, but not the root. Therefore, the term "plant part" (or "part" in short) refers to any of the following: stem, branch, leaf, flower (or blossom), fruit, bud, seed, fruit, node, and internode. The same principle applies to the plural form "parts": stems, branches, leaves, flowers and so on. Of course, not every plant will have parts in each category, so a young plant may not yet have fruits. As used herein, the description writes "leaf" as pars per toto for "part".

This convention also applies to phrase such as "leaf-annotated" standing for "part-annotated".

In general, the term "insect" stands for animalia in the phylum "Arthropoda" or 1ARTHP (EPPO-code by the European and Mediterranean Plant Protection Organization). In implementations, the insects are of the subphylum "Hexapoda" (1HEXAQ). The description uses the term "insect" for simplicity and for convenience. It is noted that "insect" is a noun (in usual language) that most readers can easily apply for counting. The skilled person reading "one insect" or "two insects" immediately understands.

The term "insect" is also used to represent biological objects that are located on parts of the plant be counted.

A biological object (to be counted) has a physical size that is relatively smaller than the part on that it is located. It is also noted that the objects are located on one part. Since the plant images are processed to images showing one part (by segmentation), the size relation also transfers to the image.

To illustrates that: an insect sitting with some legs on a first leaf, and sitting with the other legs on a second leaf is not counted because the image would be segmented to one of the leaves. Or, a relatively large insect that shows up on the image covering a leaf and covering a branch could not be counted.

In terms of biological taxonomy, the biological objects can be insects or can be arachnids (i.e., being arthropoda), or the biological objects can be mollusca (not arthropod). The internal structure of the biological objects does not matter, as long as it fits the size criterion. On other words, it does not matter is the object has an exoskeleton, a segmented body, and paired jointed appendages (as arthropods have) or not.

Further, for the computer, the different number of legs (e.g., insects <NUM> legs, arachnids <NUM> legs, or even no legs as with snails) does not matter for the computer.

The objects can also be spots on the surface of the plant parts (spots that are the result of biological processes, such as fungi interacting with the plant or the like, animal excrements, etc.). The person of skill in the art can identify suitable measures (e.g., countermeasures).

The use of the term "insects" is applicable to phrases such as "insect-annotated" or the like to the meaning "object -annotated".

Further, the interaction of the biological object with the plant (or with the stem branch etc. part) does not matter. The biological object can be pest or beneficial.

The term "stage" (also "development stage", "growing stage") identifies differences in the life-cycle (or metamorphosis) of insects (i.e., of the biological objects in general), wherein an insect in a first stage has a different visual appearance than an insect in a second, subsequent stage. Biologists can differentiate the stages (or "stadia") by terms such as egg, larva, pupa, and imago. Other conventions can also be used, such as "adults" and "nymphs", or even standardized numerical identifiers such as "nln2", "n3n4" and so on. Development stages of the plants are not differentiated.

The term "count" is short for "estimating a number", such as for estimating the number of insects on a leaf (i.e., number of biological objects on plant parts).

The description uses the term "train" as a label for a first process - the "training process" - that enables CNNs to count insects, and for the particular task to train a particular CNN by using annotated images.

For convenience, the description refers to hardware components (such as computers, cameras, mobile devices, communication networks) in singular terms. However, implementations can use multiple components. For example, "the camera taking a plurality of images" comprises scenarios in that multiple cameras participate so that some images are taken from a first camera, some image are taken from a second camera and so on.

In the figures, the suffixes "-<NUM>, -<NUM>. " and so on distinguish like items; and suffixes "(<NUM>), (<NUM>). " distinguish different items.

The term "class" is used in the general meaning to differentiate sets or the like, in case that the term "class" refers to a taxonomic rank in biology, this will be explained if needed.

Referring to <FIG>, the description provides an overview to the application of CNNs in two phases, and thereby introduces pre-processing activities as well as computer-implemented methods. Referring to <FIG>, the description introduces insects in different development stages to be counted. Referring to <FIG>, the description investigates details (regarding plants, images, image portions to be processed). With <FIG>, the description discusses accuracy.

<FIG> illustrate overviews to computer-implemented approaches.

Throughout this description, references noted as **<NUM>/**<NUM> stand for elements that are similar but that have different use in both phases.

From left to right, <FIG> illustrate plants <NUM>/<NUM> (with leaves and insects), cameras <NUM>/<NUM> to take plant-images <NUM>/<NUM>, and computers <NUM>/<NUM> with CNNs to perform computer-implemented methods 601B/602B/701B/702B. The figures also illustrate human users <NUM>/<NUM>.

<FIG> illustrate computers <NUM>/<NUM> by rectangles with bold frames. Computers <NUM>/<NUM> implement methods 601B, 602B, 701B and 702B (<FIG>) by techniques that are based on Machine Learning (ML). <FIG> also illustrate computer <NUM> and mobile device <NUM>, performing auxiliary activities (or participating therein), such as.

Some of the auxiliary activities are pre-processing activities that prepare method executions. In <FIG>, the pre-processing activities are illustrated by references 601A, 602A, 701A and 702A.

Computers <NUM>/<NUM> use CNNs and other modules to be explained below (such as user interfaces, databases, splitter and combiner modules etc.). While <FIG> just introduce the CNNs, the other figures provide details for pre-processing images and for setting parameters to the CNNs. CNNs <NUM> and <NUM> are being trained in the training phase **<NUM> to become trained CNNs <NUM> and <NUM>, respectively. In other words, the difference between untrained and trained CNNs is the availability of parameters obtained through training.

<FIG> illustrates an overview to computer-implemented methods 601B, 602B, 701B and 702B. The methods are illustrated in a matrix. In general, counting insects on leaves is divided into a sequence, with simplified:.

<FIG> differentiates pre-processing activities 601A, 602A, 701A, and 702A (such as taking images and annotating images) from computer-implemented methods 601B, 602B, 701B, and 702B with machine-learning techniques.

Methods 601B and 602B are performed with CNNs <NUM>/<NUM>, and methods 701B and 702B are performed with CNNs <NUM>/<NUM>. CNNs <NUM>/<NUM> and CNNs <NUM>/<NUM> differ from each other by parameters (explained below).

The CNNs use density map estimation techniques, where - simplified - the integral of the pixel values leads to the estimated insect numbers. In other words, counting is performed by calculating an integral. The estimated numbers NEST can be non-integer numbers. For the above-mentioned purpose (to identify appropriate countermeasures against the infestation), the accuracy of NEST is sufficient.

Using density maps to count objects is explained by "<NPL>.

Training phase **<NUM> is illustrated in the first row of <FIG> in reference to <FIG>.

As illustrated by pre-processing 601A, camera <NUM> takes a plurality of plant-images <NUM> (in an image acquisition campaign). Computer <NUM> interacts with expert user <NUM> to obtain leaf-annotations and to obtain insect-annotations. User <NUM> can have different roles (details in <FIG>). Combinations of images and annotations are provided as annotated images <NUM>, <NUM>. For convenience, the description differentiates leaf-annotated plant-images <NUM> and insect-annotated leaf-images <NUM>. It is however noted that a particular image can have both leaf- annotations and insect-annotations.

Computer <NUM> forwards annotated images <NUM>, <NUM> to computer <NUM>.

In performing computer-implemented method 601B, computer <NUM> (<FIG>) receives the plurality of plant-images in combination with the leaf-annotations (collectively "leaf-annotated plant-images"). Computer <NUM> then uses a sub-set of the plurality and trains CNN <NUM> to identify a particular leaf in a plant-image (that is not annotated). Thereby, computer <NUM> converts un-trained CNN <NUM> into trained CNN <NUM>. In other words, CNN <NUM> is the output of computer <NUM>.

In performing method 701B, computer <NUM> receives the plurality of leaf-annotated plant-images in combination with insect-annotations (collectively "insect-annotated leaf-images"). Computer <NUM> then trains CNN <NUM> to count insects on particular leaves. Thereby, computer <NUM> turns un-trained CNN <NUM> into trained CNN <NUM>. In other words, CNN <NUM> is output of computer <NUM> as well.

It is noted that the description assumes the annotations to be made for the same plurality of plant-images <NUM>. This is convenient, but not required. The pluralities can be different. For example, the plurality of plant-images <NUM> to be leaf-annotated can show non-infested plants. Using leaf-annotated plant-images <NUM> (from such healthy plants) to further provide insect-annotations would fail because there would be no insects to annotate. Providing insect-annotations could be performed for images that are not segmented to leaves.

Production phase **<NUM> is illustrated in the second row of <FIG>, in reference to <FIG>.

As illustrated by pre-processing 602A, camera <NUM> of device <NUM> takes plant-image <NUM> and forwards it to computer <NUM>.

In performing method 602B, computer <NUM> (<FIG>) uses CNN <NUM> to identify a particular leaf (and thereby creates a leaf-image). Subsequently, in performing method 702B, computer <NUM> uses CNN <NUM> and processes leaf-image <NUM>. Thereby, computer <NUM> counts insects (and potentially other objects if trained accordingly) on that particular leaf. Thereby, computer <NUM> obtains the estimated number of insects per leaf NEST as the result.

In scientific literature, using trained CNNs to obtain results is occasionally called "testing".

The description now explains further aspects and implementation details, again in view of <FIG>.

Returning to <FIG>, it illustrates plant <NUM>. Plant <NUM> has leaves <NUM>, and leaves <NUM> are occupied by insects <NUM> and non-insect objects <NUM>. Plant <NUM> can be a land-plant (not a water-plant). Camera <NUM> takes (a plurality of) plant-images <NUM> that are processed in computer <NUM> during training phase **<NUM>.

Training phase **<NUM> has two sub-phases.

Although computer <NUM> is illustrated by a single box, it can be implemented by separate physical computers. The same principle applies for plant <NUM> and for camera <NUM>. The plant and the camera do not have to be the same for all images. It is rather expected to have plant-images <NUM> from cameras <NUM> with different properties. Also, the plurality of images <NUM> represents a plurality of plants <NUM>. There is no need for a one-to-one relation, so one particular plant may be represented by multiple images.

Training the CNNs can be seen as the transition from the training phase to the production phase. As in <FIG>, CNNs <NUM> and <NUM> are being trained to become trained-CNNs <NUM> and <NUM>.

There is no need to copy the CNNs (quasi from figure to figure). The person of skill in the art can take over parameters from one network to another, such as from CNN <NUM> (of <FIG>) to CNN <NUM> (of <FIG>) and from CNN <NUM> (of <FIG>) to CNN <NUM> of (<FIG>).

As in <FIG>, a progress of time can be seen from left to right. It does not matter if during training phase **<NUM>, plant <NUM> remains alive; data communication goes into one direction only. However, in the production phase **<NUM> to be explained with <FIG>, timing is of vital importance for plant <NUM>, in the real sense of the word, because the output from trained-CNNs <NUM>, <NUM> is information used to treat plant <NUM> (or its neighbors on the field).

Training phase **<NUM> is usually performed once, in supervised learning with expert user <NUM>. The setting for the training phase with camera <NUM> taking plant-images <NUM> (as reference images), with expert user <NUM> annotating plant-images <NUM> (or derivatives thereof) and with computer-implemented processing will be explained. The description assumes that training phase **<NUM> has been completed before production phase **<NUM>. It is however possible to perform training phase **<NUM> continuously and in parallel to production phase **<NUM>.

Returning to <FIG>, it illustrates an overview of a computer-implemented approach by that a computer - illustrated as computer <NUM> - counts insects <NUM> on leaves <NUM> of plants <NUM> in an exemplary application in an agricultural field. It does not matter if the field is located in open air or located in a green-house.

Simplified, computer <NUM> processes plant-image <NUM> received from mobile device <NUM> through communication network <NUM>. In difference to training phase **<NUM> of <FIG>, one image <NUM> is theoretically enough.

Leaves <NUM> are so-called "infested leaves" because insects are located on them. Counting can be differentiated for insects of particular class <NUM>(<NUM>) (illustrated by plain ovals) and - optionally - of particular class <NUM>(<NUM>) (bold ovals). Optionally, counting can consider further classes (cf. <FIG> and <FIG>).

Non-insect objects <NUM> are not necessarily to be counted. Such objects <NUM> can be located within the leaf and can be structural elements of leaves <NUM>, such as damages on the leaf, shining effects due to light reflection or the like. It is noted that many insects camouflage themselves. Therefore, for the computer it might be difficult to differentiate insects <NUM> and non-insect objects <NUM>.

Insect classes (<NUM>) and (<NUM>) are defined.

A more fine-tuned granularity with more classes is given in <FIG>, and an adaptation of the CNNs to such granularity is given in <FIG>. As used herein, the term "insect" is used synonymous to "bug".

Plant <NUM> has a plurality of leaves <NUM>. For simplicity, only two leaves <NUM>-<NUM> and <NUM>-<NUM> are illustrated. Leaves <NUM> are occupied by insects <NUM> (there is no difference to the training phase **<NUM>). For convenience, <FIG> is not scaled, with the size of the insects being out of proportion. Field user <NUM> - for example the farmer who is growing plant <NUM> - uses camera <NUM> of mobile device <NUM> to take an image of plant <NUM> (plant-image <NUM>). For mobile devices having two (or more) cameras, camera <NUM> is conveniently the camera with the highest resolution.

Mobile device <NUM> can be seen as a combination of an image device (i.e. camera <NUM>), processor and memory. Mobile device <NUM> is readily available to the farmers, for example as a so-called "smartphone" or as a "tablet". Of course, mobile device <NUM> can be regarded as a "computer". It is noted that mobile device <NUM> participate in auxiliary activities (cf. <FIG>, 602A, 702A) only.

Field user <NUM> tries to catch at least one complete leaf (here leaf <NUM>-<NUM>) into (at least one) plant-image <NUM>. In other words, field user <NUM> just makes a photo of the plant. Thereby, field user <NUM> may look at user interface <NUM> (i.e. at the visual user-interface of device <NUM>) that displays the plant that is located in front of camera <NUM>.

Mobile device <NUM> then forwards plant-image <NUM> via communication network <NUM> to computer <NUM>. As the illustration of communication network <NUM> suggests, computer <NUM> can be implemented remotely from mobile device <NUM>.

Computer <NUM> returns a result that is displayed to user interface <NUM> (of mobile device <NUM>). In the much simplified example of this figure, there are N(<NUM>) = <NUM> insects of class (<NUM>) (i.e., insects <NUM>(<NUM>)) and N(<NUM>) = <NUM> insects of class (<NUM>) (i.e., insects <NUM>(<NUM>)) counted. The numbers N(<NUM>), N(<NUM>) (or N in general) are numbers-per-leaf, not numbers per plant (here in the example for main leaf <NUM>-<NUM>). The numbers correspond to NEST (with NEST being rounded to the nearest integer N).

Optionally, by proving infestation data that is separated by classes (such as (<NUM>) and (<NUM>)), field user <NUM> can identify countermeasures to combat infestation with better expectation of success.

The term "main leaf" does not imply any hierarchy with the leaves on the plant, but simply stands for that particular leaf for that the insects are counted. Adjacent leaf <NUM>-<NUM> is an example of a leaf that is located close to main leaf <NUM>-<NUM>, but for that insects are not to be counted. Although illustrated in singular, plant <NUM> has one main leaf but multiple adjacent leaves. It is assumed that plant-image <NUM> represents the main leaf completely, and represents the adjacent leaves only partially. This is convenient for explanation, but not required.

It is usual that main leaf <NUM>-<NUM> is on top of adjacent leaf <NUM>-<NUM> (or leaves). They overlap each other and it is difficult to identify the edges between one from the other.

The numbers N are derived from estimated numbers NEST, the description describes an approach to accurately determine N.

Ideally, only insects <NUM> located on main leaf <NUM>-<NUM> are counted. Insects <NUM> that are not counted but that are located on main leaf <NUM>-<NUM> would be considered to be "false negatives", and insects that are counted but that are located on an adjacent leaf would be considered to be "false positives". As it will be explained with more detail below, counting comprises two major sub-processes:.

In other words, identifying the main leaf prior to counting keeps the number of "false negatives" and "false positives" negligible.

The communication between mobile device <NUM> and computer <NUM> via communication network <NUM> can be implemented by techniques that are available and that are commercially offered by communication providers.

Computer <NUM> has CNN <NUM>/<NUM> that performs computer-implemented method 602B and 602B (details in connections with <FIG>). CNNs <NUM> and <NUM> have been trained before (methods 601B and 701B).

In an embodiment, computers <NUM>/<NUM> use operating system (OS) Linux, and the module that executes methods 601B/602B, 701B/702B was implemented by software in the Python programming language. It is convenient to implement the modules by a virtualization with containers. Appropriate software is commercially available, for example, from Docker Inc. (San Francisco, California, US). In a software-as-a-service (SaaS) implementation, mobile device <NUM> acts as the client, and computer <NUM> acts as the server.

Besides CNNs <NUM>/<NUM>, computer <NUM> has other modules, for example, a well-known REST API (Representational State Transfer, Application Programming Interface) to implement the communication between mobile device <NUM> and computer <NUM> can use. Computer <NUM> appears to mobile device <NUM> as a web-service. The person of skill in the art can apply other settings.

The time it takes computer <NUM> with CNNs <NUM>/<NUM> (performing the method) to obtain NEST depends on the resolution of plant-image <NUM>. Performing methods 602B and 702B may take a couple of seconds. The processing time rises with the resolution of the image. (The processing time has been measured in test runs. For plant-image <NUM> with <NUM> x <NUM> pixels, the processing time was approximately <NUM> seconds.

It is convenient, to transmit plant-image <NUM> in its original pixel resolution, otherwise the accuracy to count insects will deteriorate. In other words, there are many techniques to transmit images in reduced resolutions, but for this application, such techniques should be ignored here. However, transmitting a compressed image (in a lossless format) can be possible. In modern communication networks, the bandwidth consumption (for transmitting the image in original resolution) is no problem any longer.

It is noted that for field user <NUM>, the conditions for catching images are not always ideal. For example, there are variations in.

In the following, the description shortly investigates the objects to be counted: insects <NUM>/<NUM> (cf. <FIG>), but then turns to a discussion of problems with existing technology and of solution approach that is adapted to count insects.

The description uses two examples of plant/insect combinations.

It is noted that the person of skill in the art can differentiate such (and other combinations) without further explanations herein. Taking images, annotating images, training the CNNs, counting insects (cf. pre-processing and method execution in <FIG>) are usually made for one of the combinations.

Exceptions from the general rule are available. Having different plant species in the training phase **<NUM> and the production phase **<NUM> can be possible if the plants are similar in appearance. In that case pre-processing 601A and executing method 601B (i.e. to train CNN <NUM>/<NUM> to segment leaves) would be performed with a first plant species (e.g., eggplant) and pre-processing 602A and executing method 602B would be performed with a second plant species. The second plant species can belong to other crops such as for example cotton, soy bean, cabbage, maize (i.e., corn).

Since the infestation is made by the insects, the description focuses on the insects. It is a constraint that insects change appearance in the so-called metamorphosis with a sequence of development stages.

As the accuracy in obtaining data regarding infestation is related to the efforts to obtain the data, the description now introduces granularity aspects.

As illustrated by arrows (from left to right), the development stages occur in a predefined sequence with state transitions: from stage (A) to (B), from (B) to (C), from (C) to (D). The arrows are dashed, just to illustrate that other transitions (such as from (B) to (D)) are possible. Biologists can associate the stages with semantics relating to the age of the insects, such as "egg", "nymph", "adult", "empty pupae" (an insect has left the pupa and only the skin of the pupa is left), with semantics relating to life and death. As particular way to express stages is the "nln2"/ "n3n4" nomenclature, well known in the art.

Details for the appearance in each stage are well-known. Just to mention one point, insects can develop wings. For example, the presence or absence of wings can indicate particular development stage for thrips.

Below the stages, <FIG> illustrates a stage-to-species matrix, with.

In the example there are two species: (i) "whitefly" and (ii) "thrips". Insects of both species develop through the (A) to (D) stages (of course separately: (i) do not turn into (ii) or vice versa). The black dots at the column/row crossings indicate that insects of particular stage/species combinations should be counted. This is a compromise between accuracy (e.g. infestation critical for black dotted situations, but countermeasures available) and efforts (annotations, calculations, training etc.).

Rectangles group the particular stage/species combinations into classes (<NUM>) to (<NUM>) and thereby differentiate use cases <NUM> to <NUM>.

For each particular use case, the following assumptions applies:.

The description explains use cases by example:.

In use case <NUM>, CNN <NUM>/<NUM> is trained to provide NEST as the number of species (i) insects in stages (B) and (C), without differentiating (B) and (C), that is.

In use case <NUM>, CNN <NUM>/<NUM> is trained to provide NEST in <NUM> separate numbers (cf. the introduction in <FIG>):.

In use case <NUM>, CNN <NUM>/<NUM> is trained to provide NEST in <NUM> separate numbers:.

The rectangles are illustrated with class numbers (<NUM>) to (<NUM>), wherein the classes are just alternative notations. The description will explain adaptations to the CNNs for multi-class use cases (use cases <NUM> and <NUM>) in connection with <FIG>.

The description now refers to some challenges, but in combination with solution approaches.

The impact of the insects to the plant (as well as the appropriate countermeasures) can be different for each development stage. For example, it may be important to determine the number of nymphs (per leaf), the number of empty pupae and so on. Differentiating between young and old nymphs can indicate the time interval that has passed since the arrival of the insects, with the opportunity to fine-tune the countermeasure. For example, adults may lay eggs (and that should be prevented).

<FIG> illustrates plant-image <NUM>/<NUM> (dashed frame, cf. Plant-image <NUM>/<NUM> shows a particular plant with leaves <NUM>/<NUM> and with insects <NUM>/<NUM>. <FIG> is simplified and uses symbols for the leaves (without illustrating the characteristic leaf shape). Leaf <NUM>-<NUM> corresponds to leaf <NUM>-<NUM> (of <FIG>) and leaf <NUM>-<NUM> corresponds to leaf <NUM>-<NUM> (of <FIG>), illustrated partly. Leaf <NUM>-<NUM> corresponds to leaf <NUM>-<NUM> (of <FIG>) and the leaf <NUM>-<NUM> corresponds to leaf <NUM>-<NUM> (of <FIG>), illustrated partly as well.

<FIG> illustrates insects <NUM>/<NUM> by small squares, there are some of them on leaf <NUM>-<NUM>/<NUM>-<NUM> and some of them on leaf <NUM>-<NUM>/<NUM>-<NUM>. Non-insect object <NUM>/<NUM> is symbolized by a small square with round corners. Although not illustrated here by the symbols in this figure, the insects can belong to different classes (cf.

It is noted that the spatial arrangement (i.e. pixel coordinates (X, Y)) of the leaves, the insects and the non-insect objects is different from image to image. The reason is simple: the images show different physical plants (even taken at different time points).

The description uses terms such as "insect <NUM>/<NUM>" and "non-insect object <NUM>/<NUM>" for convenience of explanation. It is however noted that <FIG> illustrates an image (being a data-structure) so that "insect <NUM>/<NUM>" actually symbolizes the pixels that show the insect (likewise for <NUM>/<NUM>).

For use in training phase **<NUM>, plant-image <NUM> (cf. <FIG>) can be taken by a high-resolution camera (e.g., <NUM> mega pixel) or by a main camera of a mobile device.

Although illustrated here as a single image, in training phase **<NUM>, images are taken in pluralities. It is noted that the variety of different cameras can be taken into account when taking images for training.

In the production phase, plant-image <NUM> is usually taken by camera <NUM> of mobile device <NUM> (cf. <FIG> is also convenient to explain constraints that arise from the objects (i.e., plants with leaves and insects) and from insufficiencies of mobile device cameras.

<FIG> illustrates image <NUM>/<NUM> in portrait orientation (height larger than width), this is convenient but not required. Image coordinates (X, Y) to identify particular pixels are given for convenience. Image dimensions are discussed in terms of pixels. For example, image <NUM>/<NUM> can have <NUM> pixels in the Y coordinate, and <NUM> pixels in the X coordinate (i.e. <NUM> Mega Pixels, or "<NUM>"). The pixel numbers are the property of the camera sensor and can vary.

Image <NUM>/<NUM> is usually a three-channel color image, with the color usually coded in the RGB color space (i.e., red, green and blue).

It is noted that image <NUM> does not have to be displayed to field user <NUM>. Also, the field scenario will be explained for a single image <NUM>, but in practice it might be advisable for field user <NUM> to take a couple of similar images <NUM>.

Image <NUM> represents reality (i.e. plant <NUM>, leaves <NUM>, insects <NUM>, non-insect objects <NUM>), but with at least the following further constraints.

As mentioned already, plant <NUM>/<NUM> has multiple leaves at separate physical locations. Therefore in image <NUM>/<NUM>, one leaf can overlay other leaves. Or in other words, while in reality (cf. <FIG>), leaves are separate, leaf <NUM>-<NUM> and leaf <NUM>-<NUM> appear as adjacent leaves (<NUM>-<NUM> and <NUM>-<NUM> as well).

However, with the goal to count N as "insect in a particular class per leaf", the overlay must be considered. As multiple leaves have similar color (usually, green color), their representations in plant-image <NUM>/<NUM> appear in the same color (i.e., small or zero color difference in the image).

Further, each insect of a particular class has a particular color. This color could be called text-book color, or standard color. For example, as the name suggests, an adult whitefly is white (at least in most parts).

However, the image would not properly represent the text-book color. There are at least the following reasons for that:.

In the coding of the image (the numerical values that represent color, e.g., in the mentioned RGB space), the numerical values would be different. Therefore, the absolute value (of the color) in the image is therefore NOT particular characterizing.

Due to the mentioned camouflage, it can be complicated to differentiate insect <NUM>/<NUM> from non-insect object <NUM>/<NUM>. This is complicated in nature and even more complicated in images.

Further, insects can be relatively tiny in comparison to the leaves. For example, an insect can be smaller than one millimeter in length. In contrast to the emphasis in <FIG>, a couple of hundred insects may occupy a single leaf easily. The insects are also usually relative tiny things for the human eye to detect. This is in sharp contrast to, for example, a single bee in the petal leaves of a flower.

<FIG> symbolizes the dimensions of insects <NUM>/<NUM>. For example, an insect is shown by approximately Yins = <NUM> pixels and approximately Xins = <NUM> pixels (i.e., approximately <NUM> pixels only).

Further, it is natural behavior of the insects to sit on the leaf close to each other. In other words, insects tend to be present on the leaf in pairs (i.e., two insects), or even in triples (i.e., three insects). So in other words, a <NUM> x <NUM> pixel portion of image <NUM>/<NUM> might represent two or more insects.

The pixel numbers <NUM> x <NUM> are exemplary numbers, but it can be assumed that insects <NUM>/<NUM> are dimensioned with two-digit pixel numbers (i.e. up to <NUM> pixels in each of the two coordinates). The same limitations can be true for non-insect objects <NUM>/<NUM>.

As it will be explained, CNNs <NUM>/<NUM> (to count insects) use density map estimation (instead of the above-mentioned traditional object detection). In density maps, insects would be represented as areas, and the integral of the pixel values of the area would be approximately <NUM> (assuming that the pixel values are real numbers, normalized between <NUM> and <NUM>, and also assuming to have one insect per map). It is noted that for situations in that two insects are located close together and overlapping on the image, there would be a single area, but the sum of the pixel values would be approximately <NUM>.

It is noted that insects of two or more stages can be available on a single leaf at the same time. It is a constraint that the differences between two stages can be subtle. For example, on a leaf in reality, insects in stages (C) and (D) may look similar.

As a consequence, a computer using a conventional computer-vision technique (such as the mentioned technique with feature extraction) may not recognize the differences. However, an expert user can see differences (on images), and training images can be properly annotated (cf. use case <NUM>).

There are also constraints related to mobile device <NUM> (cf.

Further, the farmer (i.e. the user of the mobile device) requires a result shortly after taking the image. To be more accurate: the time interval from taking the image to determining the insect-number-per-leaf must be negligible so that.

The identification and the application of the countermeasures can only start when the insect-number-per-lead has been established. A countermeasure - although properly identified - may be applied too late to be effective. For example, a countermeasure that is specialized to destroy eggs would not have any effect if the insects have already hatched from the eggs (cf. stage specific countermeasures).

The following is taken into account by the solution. The species of the plant is usually known (for example, the farmer knows eggplant) so that the computer has the information (as an attribute of the image). Therefore, the plant species is therefore not further discussed here.

<FIG> illustrates a user interface of computer <NUM> (cf. <FIG> also illustrates expert user <NUM> annotating images. Thereby, <FIG> explains some of the pre-processing activities (cf. <FIG>, 601A and 701A) in training phase **<NUM>. In other words, <FIG> is related to supervised learning.

Those of skill in the art can implement the interaction between computer <NUM> and expert user <NUM> by appropriate user interfaces, for example with a display showing images and with interface elements to identify parts of the image (e.g., touch-screen, mouse, keyboard etc.). Software tools for such and other annotations are known in the art. A convenient tool "LabelMe" is described by <NPL>.

<FIG> gives more details how to obtain annotated images <NUM>, <NUM> introduced above in connection with <FIG>. The coordinate system (X, Y) is given for convenience (cf.

Expert user <NUM> conveys ground truth information to the images, not only regarding the presence or absence of a main leaf (by the leaf-annotations), or the presence or absences of particular insects (by the insect-annotations), but also information regarding the position of the main leaf and of the insect in terms of (X, Y) coordinates. Depending on the selected granularity of the use cases (cf. <FIG>), the annotations can also identify insect species, development stages and so on.

Both annotation processes (leaf annotations, insect annotations) can be performed independently, and even the expertise of user <NUM> can be different. For both stages, user <NUM> rather assumes particular roles:.

Although <FIG> illustrates single image <NUM>, annotating is repeated, for example for <NUM> images (leaf annotation). In view of that number, it is noted that expert user <NUM> is not necessarily always the same person.

The description now explains details for each type of annotation separately:.

As illustrated on the left side of <FIG>, expert user <NUM> annotates plant-image <NUM> to obtain leaf-annotated plant-image <NUM>. The leaf-annotation identifies the leaf border of the main leaf <NUM>-<NUM> in difference to adjacent leaf <NUM>-<NUM>. In implementations, user <NUM> can draw polygon <NUM> (dashed line) around that part of plant-image <NUM> that shows the complete leaf (i.e. the main leaf). In the example, image <NUM> shows leaf <NUM>-<NUM> as the complete leaf, and shows leaf <NUM>-<NUM> only partially, cf. It is convenient to display polygon <NUM> to expert user <NUM>, but this is not required. Computer <NUM> can close polygon <NUM> automatically. Instead of polygons, the person of skill in the art can use other user interfaces, for example picture processing tools to manipulate images, for example, by "erasing" the pixels surrounding the main leaf.

The leaf-annotation allows computer <NUM> (cf. <FIG>) for each pixel of plant-image <NUM> to differentiate if the pixel belongs to the main leaf or not. This differentiation is relevant for performing method 601B (cf. <FIG>, leaf segmentation).

For the leaf-annotation, it does not matter if the leaf shows insects (or non-insect objects).

As illustrated on the right side of <FIG>, user <NUM> also annotates leaf-image <NUM> (cf. <FIG>) to obtain insect-annotated leaf-image <NUM>. The insect annotation identifies insects and - optionally - identifies insect classes (cf. species and/or stages, as explained by the classes in <FIG>). The term "insect-annotated" is simplified: image <NUM> can comprise annotations for the non-insect objects are well.

The insect annotation also identifies the position of the insects (and/or non-insect objects) by coordinates.

It is noted that the annotations can take the use cases (cf. <FIG>) into account. In the example of <FIG> (right side), the annotations are illustrated by dots with references α to ζ, with - for example -.

Expert user <NUM> can actually set the dots next (or above) to the insects. As used herein, a single dot points to a particular single pixel (the "dot pixel" or "annotation pixel"). The coordinate of that single pixel at position coordinate (X', Y') of an insect (or non-insect object) is communicated to computer <NUM>. <FIG> illustrates the position coordinate for annotation β, by way of example. The user interface can display the dot by multiple pixels, but the position is recorded by the coordinates at pixel accuracy.

Computer <NUM> stores the position coordinates as part of the annotation. Coordinates (X', Y') can be regarded as annotation coordinates, and the computer would also store the semantic, such as (i)(C) in annotation <NUM>, as (i)(B) in annotation β and so on.

As it will be explained further, the insect-annotation (for a particular image <NUM>) is used by computer <NUM> in training CNN <NUM>, for example, by letting the computer convolute images (i.e., tiles of images) with kernel functions that are centered at the position coordinate (X', Y'). Also, the insect-annotation comprises ground truth data regarding the number of insects (optionally in the granularity of the use cases of <FIG>).

The annotations can be embedded in an annotated image by dots (in color coding, e.g. red for stage (C), stage (D), or as X, Y coordinates separately.

Using dot annotations is convenient, because the (X, Y) coordinates of the annotations indicate where the insects are shown on the image.

<FIG> illustrates leaf-image <NUM>/<NUM>. Leaf-image <NUM>/<NUM> shows a particular plant with its main leaf and with insects. In difference to plant-image <NUM>/<NUM>, leaf-image <NUM>/<NUM> only shows the main leaf, but not the adjacent leaves. The way to obtain leaf-image <NUM> and <NUM> can differ, as described in the following:
In training phase **<NUM>, computer <NUM> obtains leaf-image <NUM> through interaction with expert user <NUM>, as explained below (cf. <FIG>, left side). Leaf-image <NUM> can be considered as the portion of leaf-annotated plant-image <NUM> that shows the leaf. Leaf-image <NUM> is illustrated here for explanation only. As explained in connection with <FIG>, computer <NUM> processes the plurality of leaf-annotated plant-images <NUM> to obtain CNN <NUM>, this process uses the annotations.

In the production phase, computer <NUM> obtains leaf-image <NUM> through segmenting plant-image <NUM> by using (trained) CNN <NUM> (in method 602B). In the production phase, annotations are not available. It is noted that leaf-image <NUM> (production phase) is not the same as leaf-image <NUM> (training phase).

Reference <NUM> illustrates portions of leaf image <NUM>/<NUM> that do not show the main leaf. The pixels in portions <NUM> can be ignored in subsequence processing steps. For example, a processing step by that an image is split into tiles does not have to be performed for portions <NUM> (because insects are not to be counted according to the insects per leaf definition). In implementations, these portions <NUM> can be represented by pixels having a particular color or otherwise. In illustrations (or optionally in displaying portions <NUM> to users), the portions can be for example displayed in black or white or other single-color (e.g., white as in <FIG>).

<FIG> illustrates image <NUM>/<NUM> being split into tiles <NUM>-k/<NUM>-k (or sub-regions). The image can be.

The number of tiles <NUM>-k/<NUM>-k in image <NUM>/<NUM> is given be reference K. In the example, image <NUM> can have an image dimension of <NUM> x <NUM> pixels (annotations do not change the dimension). The tiles have tile dimensions that are smaller than the image dimensions. For example, the tile dimension is <NUM> x <NUM> pixels. The tile dimensions correspond to the dimension of the input layer of the CNNs (cf. Other tile/input dimensions are also possible (e.g. <NUM> x <NUM> pixels). Taking overlap into account (e.g., overlap by <NUM> pixels, as illustrated), the number of tiles can be up to K = <NUM>.

The figure illustrates particular tiles <NUM>-k/<NUM>-k in a close-up view on the right side, with examples:
In the example alpha, tile <NUM>-k was split out from annotated image <NUM>. Therefore, annotations are applicable for tile <NUM>-k as well. For example, if an annotation indicates the presence of insect <NUM> for a particular (X', Y') coordinate, tile <NUM>-k comprises the pixel with that particular coordinate and tile <NUM>-k takes over this annotation (cf. the dot symbol, with position coordinates (X', Y') cf. The person of skill in the art can consider the different coordinate bases (cf. <FIG> for the complete image, <FIG> for a tile only).

During training, CNN <NUM> would learn parameters to obtain density map <NUM>-k with the integral summing up to <NUM> (corresponding to <NUM> insect, assuming normalization of the pixel values in the density maps). For example, CNN <NUM> would take the position coordinate (X', Y') to be the center for applying a kernel function to all pixels of tile <NUM>-k.

In the example beta, tile <NUM>-k was split from image <NUM> (production phase), it shows insect <NUM>. Of course, the insect is not necessarily at the same position as in "annotated" tile <NUM>-k above in alpha). Using the learned parameters, CNN <NUM> would arrive at density map <NUM>-k with integral <NUM>.

The example gamma is a variation of the example alpha. Tile <NUM>-k was split up, and annotations are de facto available as well. Although expert user <NUM> did not provide annotations (dots or the like), the meta-data indicates the absence of an insect. During training, CNN <NUM> would learn parameters to obtain density map <NUM>-k with the integral summing up to <NUM>.

The example delta is a variation of case beta. A non-insect tile <NUM>-k is processed in the production phase (by CNN <NUM>) and it would arrive at a density map with integral <NUM>.

It is noted that - in the production phase - the density maps are provided for all tiles <NUM>-k (k=<NUM> to K). The person of skill in the art can implement this, for example, by operating CNN <NUM> in K repetitions (i.e. one run per tile), and combiner module <NUM> can reconstruct the density maps of the tiles in the same order as splitter module <NUM> has split them (cf. <FIG> for the modules).

While in <FIG>, the examples alpha to gamma are explained to detect the presence (or absence) of insects (in tiles), the classes (introduced above) can be taken into account as well.

As explained above, expert user <NUM> can annotate images for different insect species (e.g., (i) and (ii)), development stages (for example (A) to (B), at least for the combinations highlighted in <FIG> with the black dots. Therefore, there can be specific tiles <NUM>-k for specific use cases.

Taking use case <NUM> as an example, there can be annotations for (i) (A), (i) (B), (i) (C), and (i) (D) (i.e. whitefly in four stages, classes (<NUM>) to (<NUM>)). In other words, the annotations are class specific.

This leads to different tiles <NUM>-k for these combinations (or classes). Density maps <NUM>-k (training phase) and <NUM>-k (production phase) are different as well, simply because images <NUM>/<NUM> are different. The integrals can be separately calculated for different insect classes, resulting in the NEST (for the complete image, after combination) specific for the classes (cf.

<FIG> illustrates CNN <NUM>/<NUM> with splitter module <NUM>/<NUM> and with combiner module <NUM>/<NUM>.

Splitter module <NUM>/<NUM> receives images <NUM>/ <NUM> (images <NUM> with annotations as images <NUM>, <NUM>) and provides tiles <NUM>-k/<NUM>-k. As it will be explained, the CNNs provide density maps, combiner module <NUM>/<NUM> receives maps <NUM>-k/<NUM>-k and provides combined density map <NUM>.

Since there is an overlap (cf. <FIG>), combiner module <NUM>/<NUM> can compose the image (or combined density map) by overlapping likewise. Pixel values at the same particular coordinate (X, Y coordinates for the <NUM> x <NUM> pixels) are counted only once.

Combiner module <NUM>/<NUM> can also calculate the overall integral of the pixel values (of the combined density map), thus resulting in NEST.

<FIG> illustrates CNNs <NUM>/<NUM>/<NUM>/<NUM> with layers, in a general overview. The CNNs are implemented by collections of program routines being executed by a computer such as by computer <NUM>/<NUM>. <FIG> illustrates the CNNs with the input to an input layer and with the output from an output layer. <FIG> also illustrates (at least symbolically) intermediate layers. CNNs <NUM>/<NUM>/<NUM>/<NUM> are deep networks because they have multiple intermediate layers. The intermediate layers are hidden. In other words, deep learning is applied here.

<FIG> also illustrates some parameters and illustrates intermediate images (being tiles and maps). Since CNNs are well known in the art, the description focuses on the parameters that are applied specially for segmenting by CNNs <NUM>/<NUM> and for counting by CNNs <NUM>/<NUM>.

As already mentioned in connection with <FIG>, the CNNs receive images as input.

The CNNs do not receive the images in the original image dimension (e.g., <NUM> x <NUM> pixels) but in tile/map dimensions (e.g., <NUM> x <NUM> pixels).

During processing, the CNNs obtain intermediate data. For convenience, <FIG> illustrates an example for intermediate data by intermediate images: tile <NUM>/<NUM>-k and density map <NUM>-k/<NUM>-k. Index k is the tile index explained with <FIG>.

There is however no need to display the tiles and the maps to a user. <FIG> illustrates tile <NUM>-k that shows two insects. Tile <NUM>-k is a portion of a plant-image or a portion of a leaf-image and has tile dimensions optimized for processing by the CNN layers. In the example, tile <NUM>-k has <NUM> x <NUM> pixels (or <NUM> x <NUM> pixels in a different example). Tile <NUM>-k is obtained by splitting an image (details in connection with <FIG>) to tiles.

Map <NUM>-k is a density map derived from tile <NUM>-k. Map <NUM>-k has the same dimension as the tile <NUM>-k. In other words, the map dimensions and the tile dimensions are corresponding to each other. The density map can be understood as a collection of single-color pixels in X-Y-coordinates, each having a numerical value V(X, Y). The integral of the values V of all X-Y-coordinates corresponds to the number of objects (i.e. insects). In the example, the integral is <NUM> (in an ideal case), corresponding to the number of insects (e.g., two insects shown in tile <NUM>-k).

In the production phase **<NUM>, map <NUM>-k is obtained by prediction (with the parameters obtained during training). During the training phase **<NUM>, one of the processing steps is the application of a kernel function (e.g., a Gaussian kernel) with the kernel center corresponding to an annotation coordinate (X', Y'), if an annotation (for an insect) is available in the particular tile <NUM>-k. In other words, during training the tiles with annotations are processed to normalized Gaussians. In the absence of annotations, kernel functions are not applied.

Since tile <NUM>-k is only a portion of the (complete) image (at the input of splitter <NUM>), combiner module <NUM> (cf. <FIG>) can sum up the integrals for the plurality of maps <NUM>-k, leading to NEST. Image overlap can be considered.

Converting tile <NUM>-k to map <NUM>-k is based on layer-specific parameters obtained by training (i.e., training CNN <NUM> to become CNN <NUM>). Since the insect-annotations (cf. <FIG>) indicate the presence (or absence) of an insect (or more insects as here), the annotations are also applicable to the (plurality of tiles). There are tiles with annotations (insects are present) and there are tiles without annotations (insects are not present).

In the example, tile <NUM>-k has the annotation "<NUM> insects". It is noted that both insects can belong to different classes (cf. <FIG>), the differentiation between classes (i.e. counting the insects in a class-specific approach) is explained in connection with class-branching (cf.

Networks are publicly available in a variety of implementations, and the networks are configured by configuration parameters.

The description shortly refers to input / output parameters in general as well as to configuration parameter (in connection with <FIG>) and then specifies parameters in view of the approach to count insects. Occasionally, an existing network is being modified.

Exemplary networks comprise the following network types (or "architectures"):.

The following particulars are introduced (or used) by setting parameters accordingly (skilled person):.

The FCRN network (by Xie et al) was modified by the following:.

Also (for all <NUM> network types), the following parameter settings are useful:.

Convenient parameters are also the following:.

Auxiliary parameters can be used to deal with technical limitations of the computers. For example, computer <NUM>/<NUM> that implements the CNNs may use floating point numbers, with a maximum highest number of <NUM>. However, numerical values that the CNN uses to decide for activation (non-activation) could be in the range between <NUM> and <NUM> (e.g., in Gaussian kernel with σ = <NUM>).

It may be problematic that CNN <NUM>/<NUM> is not capable of learning what information has to be learned. This is because the contrast (in a density map) between insect (pixel activation of <NUM>) and "no insect" (pixel activation of <NUM>) is relatively small. Applying a scale factor increases the contrast in the density maps, and eases the density map estimations (i.e., with integrals over images indicating the number of objects). The scale factor can be introduced as auxiliary parameter. For example, all pixel values may be multiplied by the factor <NUM> at the input, and all output values (i.e., insect counts) would be divided by that factor at the output. The factor just shifts the numerical value into a range in that the computer operates more accurately. The mentioned factor is given by way of example, the person of skill in the art can use a different one.

In implementations, CNN <NUM>/<NUM> (to detect leaves) is a CNN of the DenseNet type. For this purpose, the following parameters are convenient:.

In implementations, CNN <NUM>/<NUM> (i.e. the CNN to detect insects) is a CNN of the FCRN type. For this purpose, the following parameters are convenient:.

<FIG> and <FIG> illustrate a pixel value filter that is implemented as part of a layer in CNN <NUM>/<NUM> in implementations that differentiate more than two insect classes (c). As explained above in connection with <FIG>, the classes can be, for example (c) = (<NUM>), (<NUM>) for use case <NUM>, or can be, for example, (c) = (<NUM>), (<NUM>), (<NUM>), (<NUM>) for use case <NUM>.

While <FIG> focuses on a filter that takes individual pixels into account, <FIG> focuses on a filter that takes sets of adjacent pixels (or tile segments) into account. In other words, the filter can be applied to properties of individual pixels (in the example: the color, in <FIG>), and the filter can also be applied to properties of pixel pluralities (in the example: a texture made my multiple pixels, in the pixel-group filter of <FIG>).

On the left side, <FIG> illustrates tile <NUM>-k/<NUM>-k as input tile, and on the right side, <FIG> illustrates <NUM>-k/<NUM>-k as output tiles, differentiated for class (first) and for class (second).

Each tile should have pixels from p=<NUM> to p=P. For tiles with <NUM> x <NUM> pixels, there are <NUM> pixels. The figure illustrates tiles with <NUM> x <NUM> = <NUM> pixels just for simplification.

Each pixel "pix" has a RGB triplet (r, g, b) that indicate the share of the primary colors. There are many notations available, for example each share could also be noted by an integer number (e.g. from <NUM> to <NUM> for each color in case of <NUM> bit coding per color).

The filter condition can be implemented, for example, such that pixels from the input are forwarded to the output if the pixel values comply with color parameters Red R(c), Green G(c) and Blue B(c). The conditions can be AND-related.

The color parameters are obtained by training (the insect classes annotated, as explained above).

Much simplified, in a hypothetical example, there should be insects of a first class (first) and of a second class (second). The insects in (first) should be "red", so that the parameters are R(first) > <NUM>, G(first) > <NUM>, and B(first) < <NUM>. An input pixel that complies with the condition is taken over as an output pixel. <FIG> symbolizes such a (c) = (first) insect at the left edge of the tile (with <NUM> pixel) that are taken over to the output tile (first).

The insects in the (second) class should be "blue", so that the parameters are R(first) < <NUM>, G(first) > <NUM>, and B(first) > <NUM>. Such an insect is illustrated at the lower part of the input tile, again here much simplified with <NUM> pixels.

An input pixel that complies with the condition (for class (second)) is taken over as an output pixel, and <FIG> symbolizes such a (c) = (second) insect with pixels (<NUM>, <NUM>, <NUM>), (<NUM>, <NUM>, <NUM>), (<NUM>, <NUM><NUM>), that are taken over to the corresponding output tile.

It is noted that applying the filter is conveniently implemented as a convolutional layer in CNN <NUM>/<NUM> (the filter filtering tiles, cf. <FIG>), but the filter can also be implemented before the splitter module (cf. <FIG>, the filter for pixels, cf.

The filter can be part of the processing channel of CNN <NUM>/<NUM> before the layer(s) that creates the density maps. Therefore, the density maps are class specific.

The color parameters Red R(c), Green G(c) and Blue B(c) are just examples for parameters that are related to pixels, but the person of skill in the art can use further parameters such as transparency (if coded in images) etc..

<FIG> illustrates a pixel-group filter that takes neighboring (i.e. adjacent pixels) into account. As the pixel-group filter uses convolution, it is implemented within CNN <NUM>/<NUM> that processes tiles <NUM>-k/<NUM>-k (i.e., after splitter module <NUM>/<NUM>).

<FIG> repeats the much simplified "<NUM>-pixel-insect" from the left edge of <FIG>. The <NUM> x <NUM> = <NUM> pixel tile is separated into <NUM> x <NUM> = <NUM> segments (from #<NUM> to #<NUM>). Each segment has <NUM> x <NUM> pixels, being a square segment.

The figure is simplified, but in implementations, the <NUM> x <NUM> tiles are separated into a different number of pixels, potentially having more pixels per segment.

Segment #<NUM> is being convoluted (with a particular convolution variable, e.g., <NUM> pixels) to modified segment #<NUM>'. Thereby, the pixels values (of the <NUM> pixels) change.

For example, segment #<NUM> can have the pixel values (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>) and segment #<NUM>' can have pixel values (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>). As <FIG> illustrates the pixels in "black" or "white" only, the pixels with values over <NUM> are illustrated "black".

Filter criteria can now be applied to the modified segment #<NUM>'. In this respect, <FIG> also illustrates a further implementation detail. The modified segments can be encoded in segment-specific values. The figure illustrates such values (by arbitrary numbers) "<NUM>", "<NUM>" and "<NUM>" for segments #<NUM>', #<NUM>' and #<NUM>', respectively. The filter criteria can then be applied to the segment-specific values.

In other words, CNN <NUM>/<NUM> can then perform subsequent processing steps by using the segment codes (the segment-specific values). This reduces the number of pixels to be processed (simplified, by a factor that corresponds to the number of pixels per segment, with <NUM> in the illustrative example). In one of the last layers, CNN <NUM>/<NUM> can then apply decoding.

<FIG> illustrates CNN <NUM>/<NUM> with branches for particular classes (<NUM>), (<NUM>), (<NUM>) and (<NUM>). As explained in connection with <FIG>, the insect-annotations can specify the class (species and growing stage). Training the CNN is performed for channels (or branches) separately. In the example of <FIG> there are <NUM> channels corresponding to <NUM> classes.

The separation into class-specific layers (by a filter, such as explained in the example of <FIG>) is illustrated here between internal layer <NUM> and internal layer <NUM> (that is provided in a plurality corresponding to the number of classes).

On the right side, <FIG> illustrates combined density maps that combiner <NUM> obtains by combining maps <NUM>-<NUM> to <NUM>-K, separately for each branch to density maps <NUM>(<NUM>), <NUM>(<NUM>), <NUM>(<NUM>) and <NUM>(<NUM>). In the example, there are K = <NUM> tiles combined (into one map <NUM>), this number is just selected for simplicity.

Density maps <NUM> that indicate the presence of an insect (in the particular class) are illustrated with a dot. As in the example, density map <NUM>-<NUM> (in the example illustrated as the map with k = <NUM>) indicates an insect of class (<NUM>) and an insect of class (<NUM>).

In the simplified overview, there the overall integral (k=<NUM> to K) for the combined density maps <NUM> leads to different estimations: NEST (<NUM>) = <NUM>, NEST (<NUM>) = <NUM>, NEST (<NUM>) = <NUM>, NEST (<NUM>) = <NUM>, and NEST (<NUM>) = <NUM> (map <NUM>-<NUM> reflects <NUM> insects). The overall number of insect is NEST (<NUM>)(<NUM>)(<NUM>)(<NUM>) = <NUM>. The illustration of <FIG> is simplified in that individual pixels have binary values ("<NUM>" or "<NUM>"). The calculation by the CNN layers provides pixels with real values (i.e., "grayscale" values). Nevertheless, a cluster of pixels with, for example, <NUM>, <NUM>, <NUM>, <NUM> and <NUM> sum up to an integral <NUM>.

Providing infestation data separated for species and growing stage can be advantageous for the farmer to identify the appropriate countermeasures.

As training is separated for the classes, training is performed separately as well.

It is noted that the illustration as separate branches (i.e., in parallel) is convenient for explanation, but not required. Parallel processing is possible, but the channels can be implemented by serial processing as well. In other words, CNN <NUM> would be trained for insects in class (<NUM>), then for insects in class (<NUM>) and so on. In the production phase, CNN <NUM> would provide density maps for inspects of class (<NUM>), then of class (<NUM>) and so on.

The description now explains further details regarding the training phase **<NUM> by that CNN <NUM> is enabled to segment leaves (by becoming CNN <NUM>, method 601B) and enabled to count insects (by becoming CNN <NUM>, method 701B).

As explained above, in production phase **<NUM>, CNN <NUM> provides NEST (the estimated number of insects per leaf for particular plant-image <NUM>) as the output. In an ideal situation, the combination of CNN <NUM> and CNN <NUM> would calculate NEST to be exactly the so-called ground truth number NGT: here the number of insects sitting on the particular main leaf of the plant (from that farmer <NUM> has taken image <NUM>). The difference between NEST and NGT would indicate how accurate camera <NUM> and CNNs <NUM>/<NUM> are performing.

However, farmer <NUM> would not manually count the insects (NGT). The description now explains how the accuracy of CNN <NUM>/<NUM> is validated. As insect-annotations identify insects (as leaf-annotations identify leaves), the ground truth numbers NGT are known for annotated images <NUM> already. That data is used as explained in the following:.

<FIG> illustrates a set of insect-annotated images <NUM> (i.e., resulting from insect-annotations as in <FIG>), with sub-sets:.

The sub-sets have cardinalities S<NUM>, S<NUM> and S<NUM>, respectively. The number of insect-annotated images S is the sum of the subsets: S = S<NUM> + S<NUM> + S<NUM>.

Differentiating images into sub-sets in known in the art. Therefore, <FIG> takes the insect-annotated leaf-images <NUM> as an example only. The person of skill in the art can fine-tune training CNN <NUM> for the leaf segmentation accordingly.

The cardinalities are specific to the cases (cf. <FIG>), in the example of the case <NUM> (<NUM> class), there are S = <NUM> annotated images in total (i.e., insect-annotated leaf-images <NUM>), with.

CNN <NUM>/<NUM> have been trained with the S<NUM> images of the training sub-set to become trained-CNN <NUM>, <NUM> Trained-CNN <NUM>, <NUM> have been used to estimate NEST for the Sz images of the validation sub-set. The S<NUM> values NGT are known from the insect-annotations. If for a particular image, NEST is higher than NGT CNNs <NUM>/<NUM> have counted more insects that present in reality.

Below the testing sub-set, <FIG> illustrates a simplified graph <NUM> showing NEST on the ordinate versus ground truth NGT on the abscissa, with a dot identifying an (NEST, NGT) pair. Graph <NUM> is simplified in illustrating <NUM> dots only (instead of, for example, <NUM> or <NUM>).

Most of the dots are located approximately along regression line <NUM>. The (graphical) distance of a dot from line <NUM> indicates the quality of the estimation. Dot <NUM> stands for an outlier, with much more insects estimated than present.

A metric can be defined as Mean Absolute Error (MAE), or MAE = NEST s - NGT s (The formula given here is simplified, MAE is actually calculated as the sum of the MAEs for s=<NUM> to S<NUM> divided by S<NUM>).

Since this is a mean value, NEST and NGT are obtained as the average of the S<NUM> images.

A further metric can be defined as Mean Square Error (MSE), or MSE = ROOT [(NEST - NGT)<NUM>]. Again, NEST AND NGT for all S<NUM> has to be taken into account (i.e. [ ] being the sum of ( )<NUM> for all S<NUM>).

For case <NUM> (whitefly single class (<NUM>)), the set S<NUM> provided values MAE = <NUM> and MSE = <NUM>. In comparison to a traditional approach (candidate selection with subsequent classification, MAE = <NUM> and MSE = <NUM>), the error values are smaller. In other words, the error by the new approach is less than half of the error of the traditional error.

Differentiating the main leaf from its adjacent leaves (or neighbor leaves) can be implemented by known methods as well (among them feature extraction). For leaves that are green over a non-green ground, color can be used as a differentiator. However, such an approach would eventually fail for "green" over "green" situations, for example, when one leaf overlaps another leaf.

In alternative implementations, counting insects can be implemented by other known approaches, such as by the above-mentioned candidate selection with subsequent classification.

However, both for leaf differentiation and for insect counting (at the main leaf), the above-described deep learning techniques provide accuracy (i.e., the terms of false positives, false negatives).

For enhance understanding, the description has described a scenario with field user <NUM> operating mobile device <NUM>. This however not required, image <NUM> could be taken otherwise, for example by aircraft flying over the field. The example of an unmanned aerial vehicle (UAV) is noted.

In such a scenario, the mobility to catch images on the field would be implemented by the UAV. User interface <NUM> (cf. <FIG>) of a smartphone or the like (to communicate the results to field user <NUM>) would potentially implemented otherwise.

As explained above - for example, in connection with <FIG> - the tiles have smaller dimensions than the images, and the tile dimensions correspond to the input layer dimension of CNN <NUM>/<NUM>. Since the biological objects (e.g., the insects) are represented in the tiles (both in the training phase as annotations and in the production phase), the physical sizes of the biological object are limited to a maximal physical size. In the extreme case (maximum), the biological object of the largest allowable physical size would correspond to the representation of that object on a single tile.

In other words, the the relation of the physical size of the biological objects (<NUM>) to the physical size of the parts (<NUM>) is such that the representation of the biological objects on the part-images (<NUM>) are such that the representation is smaller than the tile dimension.

Persons of skill in the art can estimate the image resolution (i.e., the number of pixels per physical dimension).

There is also a limitation to the minimum. In the extreme case (minimum), the biological object of the smallest allowable size would be represented (in theory) by one pixel. More practical sizes have been explained above (cf. <FIG>, <NUM> pixels times <NUM> pixels). This translates to absolute minimum size of biological objects that can be recognized. Usually, the biological objects have at least <NUM> in diameter, preferably at least <NUM>, more preferably at least <NUM>, most preferably at least <NUM> in diameter.

From a high level perspective, the biological objects (<NUM>) are (or were) living organisms that are located on the parts (<NUM>) (of the plant), or the biological objects are traces by that organisms. (Optionally, the organism may be considered as no longer living, cf. the example with the pupa). More in detail, the biological objects (<NUM>) on the parts (<NUM>) (of the plant) are selected from the following: insects, arachnids, and mollusca. In an alternative, the biological objects are selected from spots or stripes on the surface of the plant parts. In that alternative, it does not matter if the objects are considered to be organisms or not, spots or stripes can be disease symptoms. For example, brown spots or brown stripes on a plant part indicates that the plant is potentially damaged. For example, some fungi cause brown stripes like yellow rust.

In view of the size (min/max) limitations, not all insects, arachnids or mollusca (or spots and stripes) would fit. For example, a large butterfly insect would potentially cover a single leaf (not countable), but small whiteflies (or a thrips) would be countable, as explained with much detail.

The description now shortly returns to the left side of <FIG> that illustrates an overview to segmentation, with training the CNN on the basis of leaf-annotated plant images <NUM> and operating the CNN to provide leaf images.

So far, the description has used two examples of plant/insect combinations (cf. the above section "Insects, plants and use cases"). In such embodiments with the special focus to estimate the numbers of insects (such as whitefly or thrips) on plants leaves, the segmentation is performed to segment out the main leaf from the rest of the image. The resulting image is the leaf-image.

However, the embodiments are based on a more general approach: It does not matter if the plant image <NUM> shows plant parts <NUM> such the stem, one or more branches, one or more leaves, flowers (usually with petals) or fruits (or other parts mentioned above). These parts - when represented by plant images <NUM>/<NUM> - have borders that the CNN can recognize (if appropriately being training before). Looking at <FIG>, expert user <NUM> can provide part-specific annotations, such as annotations that identify the stem, that identify a branch, the main leaf (cf. the detailed explanations above), a flower or a fruit (or other parts, as mentioned). More in detail, expert user <NUM> could provide annotations with more granularity, for example, by annotating the petals that belong to the flowers.

Expert user <NUM> does not have to identify all parts in one image, but he/she can makes the annotations for parts of the plant should be segmented out by the CNN. For example, if a plant image shows a one or more leaves and a fruit, expert uses <NUM> could provide a first annotation for the leaf, and a second annotations for the fruit. The annotations would be used separately: in a CNN to identify the main leaf (as described in much detail, trained on the basis of the first annotations) and a CNN to identify the fruit (the skilled person can apply the existing description accordingly, trained on the basis of the second annotations).

In other words, computer <NUM> interacts with expert user <NUM> to obtain part-annotations (in general) and/or to obtain leaf-annotations (in particular). Thereby, expert user <NUM> conveys ground truth information to the images, not only regarding the presence or absence of a main leaf (by the leaf-annotations), but also to the parts in general (by part-annotations).

More in general, there is a computer-implemented method 602B for providing part-segmented images (showing parts of plants). Optionally, the method belongs to an overall process to estimate the number (NEST) of objects <NUM> on parts <NUM> of a plant <NUM>, wherein method 602B identifies images for the parts.

In a method step, the computer uses first convolutional neural network <NUM> to process plant-image <NUM> to derive part-image <NUM> being a contiguous set of pixels that show a part <NUM>-<NUM> of particular plant <NUM> completely. First convolutional neural network <NUM> has been trained (before that method step) by a plurality of part-annotated plant-images <NUM>, wherein the plant-images <NUM> are annotated to identify parts <NUM>-<NUM>.

In one embodiment, there is provided a computer-implemented method for providing a segmentation of plant-images to part-segmented images. A computer program product - when loaded into a memory of a computer and being executed by at least one processor of the computer - performs the steps of this computer-implemented method. The same principle applies to a computer system for providing part-segmented images (showing parts of plants), with the system adapted to perform the method.

As mentioned above, for the leaf-annotation, it does not matter if the leaf shows insects or not. The same principle applies to the parts in general: if the parts (no the images) show biological objects or not does not matter. There is an assumption that the biological objects are located above the parts (e.g., the leaf or the fruit), not at the borders.

<FIG> illustrates an example of a generic computer device <NUM> and a generic mobile computer device <NUM>, which may be used with the techniques described here. Computing device <NUM> is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Generic computer device may <NUM> correspond to computers <NUM>/<NUM> of <FIG>. Computing device <NUM> is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, and other similar computing devices. For example, computing device <NUM> may include the data storage components and/or processing components of devices as shown in <FIG>.

Thus, for example, expansion memory <NUM> may act as a security module for device <NUM>, and may be programmed with instructions that permit secure use of device <NUM>. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing the identifying information on the SIMM card in a non-hackable manner.

As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal.

The systems and techniques described here can be implemented in a computing device that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components.

The computing device can include clients and servers.

Claim 1:
Computer-implemented method (602B/702B) for quantifying biological objects (<NUM>) on parts of plants, by estimating the number (NEST) of the objects (<NUM>) on parts (<NUM>) of a plant (<NUM>), the method (602B/702B) comprising:
receiving a plant-image (<NUM>) taken from a particular plant (<NUM>), the plant-image (<NUM>) showing at least one of the parts (<NUM>) of the particular plant (<NUM>);
using a first convolutional neural network (<NUM>) to process the plant-image (<NUM>) to derive a part-image (<NUM>) being a contiguous set of pixels that show a part (<NUM>-<NUM>) of the particular plant (<NUM>) completely, the first convolutional neural network (<NUM>) having been trained by a plurality of part-annotated plant-images (<NUM>), wherein the plant-images (<NUM>) are annotated to identify parts (<NUM>-<NUM>);
splitting the part-image (<NUM>) into a plurality of tiles (<NUM>-k), the tiles being segments of the plant-image (<NUM>) having pre-defined tile dimensions;
using a second convolutional neural network (<NUM>) to separately process the plurality of tiles (<NUM>) to obtain a plurality of density maps (<NUM>-k) having map dimensions that correspond to the tile dimensions, the second convolutional neural network (<NUM>) having been trained by processing object-annotated plant-images (<NUM>), the processing comprising the calculation of convolutions for each pixel based on a kernel function leading to density maps (<NUM>) with different integral values for tiles showing biological objects and tiles not showing biological objects; and
combining the plurality of density maps (<NUM>) to a combined density map (<NUM>) in the dimension of the part-image (<NUM>), and integrating the pixel values of the combined density map (<NUM>) to an estimated number of biological objects (NEST) for the main part.