Patent Description:
Embodiments of the present disclosure relate generally to a control system and method for an object detection system, and in particular an object detection system for an agricultural machine.

It is advantageous to be able to detect objects within an agricultural working environment. This can include detection of static or dynamic objects, which may provide an obstacle to the motion of a machine operating in the environment, or can include other machines working in the environment. To date, operator assistance systems have been provided which may detect and alert an operator of a machine to the presence of objects within the environment. In extensions of such systems, this could be incorporated into into machines with autonomous capabilities, where preemptive action may be taken automatically on the detection of one or more obstacles, or for controlling motion of the machine with respect to other detected machines within the environment, for example.

Background information can be found in the following documents: <NPL>), <NPL>), <NPL>) and <NPL>).

Some known systems may utilise object detection algorithms. It will be appreciated that such systems require intensive training prior to being released to be able to detect and classify a pre-defined set of obstacles, and can require a significant number of image examples from each class of objects. Therefore such training is usually limited to the most common classes of objects that may be encountered in the working environment of the agricultural machine (e.g. other machines, humans, implements, poles, bales, etc.).

However, agricultural machines are not only used in agricultural environments, and are regularly used on-road, e.g. during a transport operation or when moving between fields, farms etc. It is also possible that non-agricultural equipment may be present within a working environment, for example cars may be parked at field boundaries, trucks may be used in the unloading of grain from harvesters, etc. By not accounting for all possible working scenarios, such object detection algorithms may potentially result in obstacles and other objects not being identified or being misclassified. For operator assistance systems this may increase the workload for the operator and lead to a lower trust by the operator in the effectiveness of the system.

One solution may be to train the object detection algorithms on a wider dataset to include all possible working scenarios for an agricultural machine. However, in doing so the level of training required may become unworkable, and could reduce the effectiveness of a particular trained model in positively identifying certain objects.

It would be advantageous to provide a system (and associated method) which overcomes or at least partly mitigates one or more problems associated with the prior art systems.

In an aspect of the invention there is provided a control system for an object detection system for an agricultural machine, as recited in claim <NUM>.

Advantageously, the present invention utilises a pair of detection models to improve identification of objects within the environment of the agricultural machine. This can provide added redundancy to the object detection system, in turn improving the positive identification rate. Prioritising the first model over the second model in dependence on an overlap - e.g. where outputs from the models relate to the same object - can additionally provide contextual awareness for the system. As described herein, the first model is trained on a dataset which relates at least in part to an agricultural context and as such it prioritises the output of this model for an object detection system for an agricultural machine, in manner of the present invention.

The one or more controllers may collectively comprise an input (e.g. an electronic input) for receiving one or more input signals. The one or more input signals may comprise image data from the one or more imaging sensors. The one or more controllers may collectively comprise one or more processors (e.g. electronic processors) operable to execute computer readable instructions for controlling operational of the control system, for example, to analyse the image data and/or to determine the classification metric. The one or more processors may be operable to generate one or more control signals for controlling operation of the one or more operable components. The one or more controllers may collectively comprise an output (e.g. an electronic output) for outputting the one or more control signals.

The first and second detection models comprise machine-learned models. The machine-learned models are trained on training datasets with known objects with respective classifications. The machine-learned model(s) may comprise a deep learning model utilising an object detection algorithm. The deep learning model may include a YOLO detection algorithm, such as a YOLOv5 detection model, for example.

The training datasets for first and second models are different. For example, the training dataset for the first model may comprise an "off-road" dataset, and the training dataset for the second model may comprise an "on-road" dataset. The training dataset for the first model comprises an agricultural dataset, comprising training images including agricultural-specific objects. The training dataset for the second model comprises a generic dataset, which may for example, include training images including generic objects, including on-road and the like. The generic dataset may, for example, comprise the widely available Microsoft Common Objects in COntext dataset (MS COCO), although other datasets may be used, including datasets assembled specifically for training the second model. Advantageously, utilising two models trained on different datasets may advantageously provide an object detection system operable in multiple working environments, This makes it particularly beneficial for agricultural machines which may operate across multiple environments, including both off-road (e.g. working in a field) and on-road during transport tasks or when moving between working environments, for example. Using two different models and two different datasets prevents dilution of a single model trained on datasets across multiple working contexts providing a better positive identification rate for each model individually. As discussed herein, the present invention relates to combining the outputs of the two models.

Classification by the first and/or second detection models may comprise assignment of a class to the object. The class may be one of a plurality of classes for the respective model, as determined during the learning process through assignment of suitable labels to known objects. The plurality of classes may be grouped by category, and optionally by subcategory. For example, the plurality of classes may include 'tractor', 'combine', 'car', 'truck', 'trailer', 'baler', 'combine header', 'square bale', 'round bale', 'person', and 'animal', for example. The classes may be grouped into categories, which may include, for example, 'Vehicle', 'Implement', 'Static Object', and 'Dynamic Object'. The 'Vehicle' category may be further divided into sub-categories, including 'Agricultural' - for example including classes 'tractor' and 'combine' - and 'Road' - including classes 'car', and 'truck', for example. The 'Implement' category may be further divided into subcategories, including "Tractor' - for example including classes 'trailer' and 'baler' - and 'Combine' - including the 'combine header' class, for example. The 'Static Object' category may include a sub-category 'Bale', for example including classes 'round bale' and 'square bale'. The 'Dynamic Object' category may include a sub-category 'Person', for example including the 'person' class.

The control system may be configured to determine the identity for the object in dependence on the class, sub-category and/or category of the classification for the first and/or second model. For example, where the first and second models output a classification in the same category or subcategory, priority may be assigned to the first model. This may be of particular use where the first model outputs a class of 'tractor' and the second model outputs a class of 'truck'. Assigning priority to the first model may provide a final identity for the object as 'tractor' which has a higher probability of being the correct identity given the operational context of the agricultural machine. As discussed herein, this is also determined in dependence on the classification metric.

The classification output by the first and second models may comprise a bounding box overlaid onto the image data at the location of the object as determined by the respective model. In embodiments, the classification metric may comprise a measure of an overlap of the bounding boxes determined for the first and second models. The measure of overlap may comprise a calculation of an Intersection over Union, or 'IOU', associated with the bounding boxes output from the first and second models.

Where the models have classified a common object it is envisaged that the bounding boxes will overlap. For distinct objects there should be minimal or no overlap. Accordingly, the control system may be configured to utilise the overlap to determine whether the outputs of the models relate to a common object or to distinct objects.

The control system is therefore configured to compare the classification metric with a threshold value. In this way, thresholding can be used to identify where the models have classified a common object. In embodiments, the control system is configured to discount the classification from the second model in dependence on the classification metric exceeding the threshold. In other words, the control system is configured to determine the identity for the object in dependence on the classification for the first model only in dependence on the classification metric exceeding the threshold.

The threshold may be predetermined. The threshold may be variable, and may be dependent on a context parameter. For example, the threshold may be set at a comparatively lower value where it is known that the agricultural machine is operating in an agricultural environment. Accordingly, by lowering the threshold the priority assigned to the first model is effectively increased by discounting more classifications determined by the second model. In some embodiments the threshold may be set at a comparatively higher value where it is known that the agricultural machine is operating in a non-agricultural environment, such as on a road, where non-agricultural objects may be encountered more often. In effect, raising the threshold assigns more weight to the second model thereby utilising the classifications determined for the second model more often.

In some embodiments, the threshold may be dependent on the classification assigned by the first model. For example, it may be known (e.g. through calibration, experimentation etc.) that the second model misclassifies a certain object type more often than others. Accordingly, the threshold may be set at a comparatively lower value where the first model classifies the object in a class or classes corresponding to that object type. In embodiments, the threshold may be set at a comparatively lower value where the output from the first model includes classification of an agricultural vehicle and a corresponding implement - e.g. two classifications (and optionally associated bounding boxes) adjacent and possibly overlapping one another. This may in turn effectively reduce the weight assigned to the output of the second model, which may for example misclassify the vehicle-implement combination as a 'truck'. In embodiments, the threshold may be set at a comparatively higher value where the output from the first model includes classification of a separate vehicle or implement - e.g. a single classification. This may in turn effectively increase the weight assigned to the output of the second model where there is a lower probability of the second model misclassifying the object.

A 'lower' value for the classification metric threshold may comprise an IOU value of between <NUM> and <NUM>, inclusive, for example. In embodiments, the lower threshold may be set at an IOU value of <NUM>, and the control system may be configured to discount the output from the second model in dependence on the classification metric corresponding to an IOU value equal to or exceeding an IOU value of <NUM>. A 'higher' value for the classification metric threshold may comprise an IOU value of between <NUM> and <NUM>, inclusive, for example. In embodiments, the higher threshold may be set at an IOU value of <NUM>, and the control system may be configured to discount the output from the second model in dependence eon the classification metric corresponding to an IOU value equal to or exceeding an IOU value of <NUM>.

The one or more imaging sensors may include a camera. The one or more imaging sensors may comprise a LIDAR sensor. Where used, a LIDAR sensor may be used to obtain information relating to the distance between the object and the sensor, and as such can be used in conjunction with the image data from another sensor, e.g. a camera, for classification by the first and/or second models, for example, for analysing the expected size of a given object within the camera data using the depth information from the LIDAR sensor.

In embodiments, the one or more operable components may include a user interface. The user interface may comprise a display terminal of the machine, for example, a display terminal provided within an operator cab of the agricultural machine, or a remote user device, which may be operable in combination with the agricultural machine.

The control system may be operable to output, e.g. via the user interface, an indicator indicative of the determined identity to an operator of the agricultural machine. This may provide increased situational awareness to the operator of the working environment.

The one or more operable components may include a guidance system for the agricultural machine, which may include a steering and/or propulsion system of the machine. Advantageously, the control system may be configured to control operation of the guidance system for controlling operation thereof for controlling motion of the machine in dependence on the determined identity for the object. This may include reducing a speed of the machine, including bringing the machine to a stop where an object is detected and identified in the path of the machine. This may include controlling a speed and or steering of the machine for controlling movement of the machine along an operational path which is dependent on the identity of the object. For instance, in embodiments the control system may be configured to control an operational speed and/or path for the machine in order to follow an identified vehicle ahead of the machine for performing a fully or at least semi-autonomous operation. In an example, the machine may comprise a tractor with a trailer and the control system may be configured to identify a further agricultural machine within the environment, e.g. a combine harvester performing a harvesting operation. The control system may be configured to control movement of the tractor along an operational path to align the trailer with an unloading mechanism of the combine for receiving crop material from the combine in the trailer of the tractor-trailer combination. The present invention is advantageously used here for positively identifying the combine.

The one or more operable components may comprise a data server associated with the agricultural machine. This may comprise a local server or a remote server, for example, accessible via a wireless data communication channel such as a data network, cellular network or the like. The control system may be operable to store on the data server information indicative of the determined identity for the object. This may be advantageous, for example, where the present invention is utilised to detect and track the positions of multiple agricultural machines within a working environment. The control system can store this information on the data server which in turn may allow, e.g. a remote operator, to remotely track a farming operation in the working environment utilising multiple vehicles. Using an image based system on a machine with an appropriate data communication capability may allow for tracking of multiple machines and implements within an environment without such systems, or without onboard positioning systems such as a GNSS system, which may be the case for example for individual implements.

A further aspect of the invention comprises an object detection system for an agricultural machine, comprising one or more imaging sensors; wherein the system further comprises and/or is controllable by the control system of any preceding aspect.

Another aspect of the invention provides an agricultural machine comprising a control system and/or an object detection system as described herein.

The agricultural machine may comprise a harvesting machine, such as a combine harvester or forage harvester, for example. The agricultural machine may comprise a tractor.

A further aspect of the invention provides a method of object detection, comprising: receiving image data from one or more imaging sensors associated with an agricultural machine; analysing the image data utilising first and second detection models to classify, for one or both models, an object within the environment of the agricultural machine; determining a classification metric for the object indicative of an overlap associated with the classification obtained for each of the models for the object; determining an identity for the object in dependence on the classification metric and the classification obtained for the first and/or second models; and controlling operation of one or more operable components associated with the agricultural machine in dependence on the determined identity.

The method of the present aspect of the invention may comprise performance of any one or more of the functional features of the control system of a preceding aspect discussed herein.

A further aspect of the invention comprises computer software which, when executed by one or more processors, causes performance of the method of the preceding aspect of the invention.

An aspect of the invention provides a computer readable storage medium comprising the computer software of the preceding aspect of the invention.

Within the scope of this application it should be understood that the various aspects, embodiments, examples and alternatives set out herein, and individual features thereof may be taken independently or in any possible and compatible combination, provided that they fall within the scope of the appended set of claims.

Where features are described with reference to a single aspect or embodiment, it should be understood that such features are applicable to all aspects and embodiments unless otherwise stated or where such features are incompatible, provided that they fall within the scope of the appended set of claims.

One or more embodiments of the disclosure will now be described, by way of example only, with reference to the accompanying drawings, in which:.

Systems and methods are provided for controlling operation of an object detection system for an agricultural machine (e.g. a harvester <NUM>). Image data from one or more imaging sensors (e.g. cameras <NUM>, LIDAR units, etc.) mounted, coupled or otherwise associated with the harvester <NUM> is obtained and used to detect objects (e.g. object 'O') within the environment of the harvester. Specifically, the image data is analysed utilising first and second detection models to classify, for both models, object(s) within the environment. This classification is analysed to determine a classification metric for the object which indicative of an overlap associated with the classification obtained for each of the models for the object O. An identity for the object O is determined in dependence on the classification metric along with the classification obtained for the first and/or second models. Operation of one or more operable components associated with the harvester <NUM> can then be controlled in dependence on the determined identity. As discussed herein, the operable component(s) may include functional components of the harvester <NUM>, including a guidance system of the harvester <NUM>, for example, for controlling motion of the harvester <NUM>, e.g. through control over a propulsion, braking and/or steering system of the harvester <NUM>. Additional operable components may include a user interface, e.g. display terminal <NUM> provided within an operator cab <NUM> of the harvester <NUM> for displaying information indicative of the determined identity to an operator of the harvester <NUM>.

<FIG> illustrates an agricultural harvester <NUM>, which includes a chassis <NUM> supported and propelled along the ground by ground-engaging wheels <NUM>. Although the harvester <NUM> is illustrated as being supported and propelled on ground-engaging wheels <NUM>, the harvester <NUM> can also be supported and propelled by full tracks or half tracks, as will be appreciated. A harvesting header <NUM> carried by the chassis <NUM> is used to gather crop and to conduct the crop material to a feederhouse <NUM> and then to a beater in the harvester <NUM>. The beater guides the crop upward to a threshing and separating system <NUM>, which is configured to separate grain from material other than grain (MOG), and deliver the grain to a grain tank <NUM> carried by the harvester <NUM>. The operation of the harvester <NUM> is controlled from an operator cab <NUM>. A user interface <NUM> is provided within the operator cab <NUM> for displaying or otherwise providing information to an operator of the harvester <NUM> data indicative of the operation of the harvester <NUM> or other associated components, e.g. the header <NUM> or the object detection system as discussed herein.

The harvester <NUM> is additionally provided with an imaging sensor in the form of camera <NUM> mounted thereto and configured to obtain image data representative of the working environment of the harvester <NUM>, and analysed in the manner discussed herein. In an extension of the present disclosure, the harvester <NUM> (or more generally the agricultural machine) may additionally or alternatively be provided with a different type of imaging sensor. For example, a transceiver-type sensor, such as a LIDAR sensor may be provided for imaging the environment of the machine. Advantageously, transceiver-type sensors may provide depth information for the environment, including a measure of a distance between the machine and the object. This can be utilised by the present disclosure, and in particular by the detection models for classifying objects within the environment of the machine.

Whilst discussed herein with reference to harvester <NUM>, it will be apparent to the skilled reader that the systems and methods discussed herein may be applicable to any agricultural machine, including forage harvesters, self-propelled balers, tractors and tractor/implement combinations, including others.

As discussed herein, a control system <NUM> is provided and configured to control operation of one or more operable components (e.g. a guidance system <NUM>, data server <NUM>, and/or user interface <NUM>) associated with the harvester <NUM>. In the illustrated embodiment the controllable components include a guidance system <NUM> of the harvester for controlling operational parameters of the harvester <NUM>, e.g. a forward speed or path, a data server <NUM> for storing information representative of the classification outputs of the first and/or second detection models; and the user interface <NUM>, here provided as a display terminal <NUM> provided within the operator cab <NUM> of the harvester <NUM> as discussed, and utilised to provide an indication of the operation of the control system <NUM>, e.g. a determined identity for an object within the working environment of the harvester <NUM>.

<FIG> illustrates the control system <NUM> further. As shown, control system <NUM> comprises a controller <NUM> having an electronic processor <NUM>, electronic input <NUM> electronic outputs <NUM>, <NUM> and electronic input/output <NUM>. The processor <NUM> is operable to access a memory <NUM> of the controller <NUM> and execute instructions stored therein to perform the steps and functionality of the present disclosure discussed herein, e.g. by controlling operation of operable components associated with the harvester <NUM> in dependence on a determined identity for one or more objects in the environment of the harvester <NUM>, as determined by the control system <NUM>, optionally including controlling the user interface <NUM> to display information indicative of determined identities and/or to receive an operator input for initiating control actions based on the determined identities, through generation and output of one or more control signals.

The processor <NUM> is operable to receive via input <NUM> which, in the illustrated embodiment, takes the form of input signals <NUM> received from a control unit associated with camera <NUM> associated with the harvester <NUM>, image data representative of the working environment of the harvester <NUM>. The processor <NUM> is configured to analyse the image data and determine therefrom a classification for one or more objects in the environment in the manner discussed herein, i.e. using first and second trained object detection models for identifying the objects. The processor <NUM> is further configured to utilise the determined classifications to determine a classification metric for the or each object. Here, the classification metric is a measure of an overlap associated with the classification obtained for each of the models for the object, and effectively quantifies a likelihood of the models classifying a common object, as discussed herein.

As described above, the controller <NUM> includes an electronic output <NUM> configured to output control signals <NUM> generated by the processor <NUM> for controlling operation of one or more operable components associated with the harvester <NUM>. Specifically, in the illustrated embodiment, the processor <NUM> is operable to generate, and the controller <NUM> operable then to output via output <NUM>, control signals <NUM> to local control unit of a guidance system <NUM> of the harvester <NUM> for controlling motion of the harvester <NUM> in dependence on the identity of the objects as determined in the manner described herein. In practice, this may include control over a forward speed and/or operational path for the harvester <NUM>. For certain objects such as obstacles, animals, humans, etc. this may include slowing the forward speed of the harvester <NUM>, upto and including bringing the harvester <NUM> to a stop to reduce the likelihood of a collision. For other objects, such as other working machines in the environment, the guidance system may be utilised to control movement of the agricultural machine for performing a cooperative task. The guidance system <NUM> may be operable to control operation of one or more of a steering system, braking system and/or propulsion system of the harvester <NUM>, for example, for controlling motion of the harvester <NUM> in the manner described herein.

Input/output <NUM> is operably connected to user interface <NUM>. The control system <NUM> is operable to control operation of the user interface <NUM>, e.g. through output of control signals <NUM> in order to display data to an operator of the harvester <NUM> indicative of the identity of the object(s), as determined by processor <NUM>. This can include simply providing an indication to the operator of the determined identity(ies). This may include providing a graphical representation, optionally including image data from the camera <NUM>, of the working environment, object(s) located therein and any identities determined for those objects as per the present disclosure. The input/output <NUM> is additionally configured to receive input signals <NUM> from the user interface <NUM> indicative of an operator input at the user interface <NUM>, for example to provide the operator an ability to interact with the data.

Output <NUM> is operably connected to a data server, preferably a remote data server <NUM> for storing information indicative of the determined identities. This may be used, for example, by a remote user monitoring operation of multiple machines within the working environment. Here, the image data from the camera <NUM> can therefore be used to provide tracking data for multiple machines to a remote server <NUM>.

<FIG> illustrates a method <NUM> in accordance with the present disclosure. Method <NUM> comprises, at step <NUM>, receiving image data from one or more image sensors (here camera <NUM>) mounted to the harvester <NUM> and operable, in use, to capture image data representative of the working environment of the harvester <NUM>. As shown herein, the camera <NUM> is mounted on the operator cab <NUM> of the harvester <NUM> and is directed such that the field of view R of the camera <NUM> encompasses a region ahead of the harvester <NUM>, in use. The camera <NUM> or indeed any imaging sensor employed may be mounted with a field of view about any direction of the working machine.

At step <NUM>, the image data received, here from camera <NUM>, is analysed utilising first and second detection models to classify, for both models, an object within the environment of the harvester <NUM>. Each of the first and second detection models comprise trained object detection algorithms, trained using respective training datasets having a plurality of images and known objects in those images. In an embodiment, the first model is trained on an agriculture-specific dataset, and the second model is trained on a generic dataset, e.g. as discussed in detail hereinbelow.

At step <NUM>, a classification metric for the object is determined. Here, the classification metric quantifies an overlap associated with the classification obtained for each of the models for the object. Specifically, the classification output by the first and second models can include a bounding box overlaid onto the image data at the location of the object as determined by the respective model. Here, the classification metric then comprises a measure of an overlap of the bounding boxes determined for the first and second models. In a preferred embodiment, the measure of overlap includes a calculation of an Intersection over Union, or 'IOU', associated with the bounding boxes output from the first and second models. The bounding boxes for a common object are likely to overlap. For distinct objects there should be minimal or no overlap. Accordingly, the control system <NUM> and method <NUM> utilises this overlap to determine whether the outputs of the models relate to a common object or to distinct objects and acts accordingly. A threshold is defined for the classification metric for the purpose of differentiating between common and distinct objects, e.g. in the manner described in detail below. This may be pre-defined, or may be variable. In some instances it may depend on the classification output from the first and/or second detection models.

At step <NUM>, an identity for the object is determined in dependence on the classification metric and the classification obtained for the first and/or second models. Where both the first and second detection models have classified a common object, a decision is taken as to which classification represents the likely identity of the object, and this is done using the classification metric. For example, for a high level of overlap, priority may be assigned to the first or second model as appropriate. When operating in an agricultural context, a higher weight may be applied to the output of a model trained on agriculture-specific training data, For other use cases, e.g. on-road use, the model trained on generic dataset may be more applicable. Where only one of the models has classified a particular object, the classification metric will be zero, and the identity is determined as has been classified by the detection model which has classified the object.

At step <NUM>, the operable component(s) associated with the machine are then controlled based on the identity. In an example, a forward speed for the harvester <NUM> may be adjusted e.g. reduced, in dependence on an identification of an object or more specifically an obstacle ahead of the harvester <NUM>. In practice, adjusting a forward speed of the harvester <NUM> includes control over a braking and/or propulsion system of the harvester <NUM>, e.g. via guidance system <NUM>. This may extend to controlling a steering system of the harvester <NUM>, e.g. for controlling the harvester <NUM> to operate along an operational path, e.g. to avoid the location of an identified object. In some instances this may extend to controlling movement of the harvester <NUM> with respect to other identified machines in the environment.

Additionally or alternatively, this may include output of information relating to the determined identity(ies) to the remote data server <NUM>, serving as a tracking system for tracking the position of objects, including other agricultural machines, within the working environment.

Further, operable component(s) can, in embodiments, include control over the user interface <NUM> in the manner discussed herein. For example, controlling the user interface <NUM> to display or otherwise present to the operator an indication of determined identity(ies), and/or a representation of image data obtained by the camera <NUM> or other applicable imaging sensors.

<FIG> illustrate example representations that may be provided by the user interface <NUM> upon successful identification of a given object. For example, <FIG> represents a representation of image data obtained by a camera <NUM>' provided on an agricultural machine in the form of a tractor <NUM>'. Here an object 'O' has been identified in the manner discussed herein, and an appropriate bounding box B and label L has been provided to highlight the position and identity of the object - here another tractor - to an operator of the tractor <NUM>'. <FIG> represents a representation of image data obtained by a LIDAR unit also provided on the tractor <NUM>'. Again, the object 'O' is identified and highlighted in the image data from the sensor - LIDAR unit. The LIDAR unit may be used to provide depth information which is used in conjunction with the image data from camera <NUM>' for identifying the object O.

Further discussion of the experimental setup for proving the operational benefit of the present disclosure is discussed below, and with reference to <FIG>. This additionally provides further detail on the combination of classification output from the first and second detection models and the advantages realised by the present disclosure.

This present disclosure is focused on object detection for agricultural machines operating during summer harvest, although it will be appreciated that the disclosure is applicable to other agricultural scenarios.

The present disclosure utilises the benefits of pre-trained models for general classes of objects (the second detection model) and a model trained on a smaller agriculture specific dataset (the first detection model) by proposing an ensemble module that combines their predictions. The second detection model is trained on MS COCO dataset and three classes 'person', 'car' and 'truck' are selected as relevant for this scenario. The model for detection of agriculture specific classes is trained to detect <NUM> classes: 'tractor', 'combine', 'trailer', 'combine header', 'baler', 'square bale', and 'round bale'. Both models have YOLOv5 architectures. The performance of models is evaluated on a test set that contains <NUM> classes of objects that the two models are able to detect together. The evaluation of this approach is discussed below.

Since multiple classes from both models refer to different types of vehicles, a single vehicle object may be detected by both models and assigned different classes. To addresses this challenge the present disclosure introduces an ensemble step (determining the identity of the object based on the classification metric) that removes redundant detections. Ensemble methods that combine the output of detection models by eliminating redundant bounding boxes, and the present disclosure utilises an ensemble module based on hierarchical information of class structure within test dataset. It has been shown that applying the proposed ensemble module improves mean average precision (mAP) on the test set. Moreover, it is also shown that translating the model predictions to a higher level in the hierarchical structure of classes improves mAP even further at the cost of class granularity.

The second detection model utilises an object detection algorithm trained on an Microsoft COCO (MS COCO) dataset. The dataset contains <NUM> training and validation images annotated for <NUM> categories of objects. The images are taken from everyday scenes containing common objects. Annotated classes of objects are grouped into <NUM> supercategories. Amongst the supercategories and classes, those relevant to an agricultural context are selected. Specifically, in the case of object recognition for agricultural vehicles, supercategories persons, vehicles and animals are of interest. For this paper, the classes 'person', 'car' and 'truck' are selected. During field operation, it is very common that farmers and workers are present in the field and especially around the vehicles during servicing. Vehicles such as cars are often left parked at the boundaries of the field. Trucks are often used for unloading of harvested grain.

The dataset used for training of the first detection model (the agriculture specific dataset) consists of <NUM> images annotated for <NUM> agriculture specific classes. The annotated classes are: 'tractor', 'combine', 'trailer', 'combine header', 'baler', 'square bale', and 'round bale'.

For testing of the first and second detection models, dataset annotated for boxbased object detection is used. The dataset consists of <NUM> images collected by two agricultural vehicles during <NUM> days. The annotated classes are: 'tractor', 'combine', 'trailer', 'combine header', 'baler', 'square bale', 'round bale', 'person', 'car' and 'truck'. Adopting the hierarchical approach, the base classes are grouped into categories at two levels of granularity. The first level, 'subcategory' represents coarse labels for the base classes. At the top level, labelled 'supercategory', the subcategories are grouped even further to very general categories such as 'Vehicle', 'Implement', 'Static object' and 'Dynamic object'. This is shown in Table <NUM>, below.

As discussed, two object detection models are used, here based on YOLOv5. The first model is trained to detect <NUM> agriculture specific classes using transfer learning. The second model is a trained on a COCO dataset of the type discussed above.

The YOLOv5 models are a single-stage object detectors and consist of a backbone network, neck and detection head. The cross stage partial connections (CSP) backbone network is used to extract features from the input image. The layers in the neck of the network are inserted to collect feature maps from different stages. This enables the model to perform better when detecting objects of various sizes and scales. In YOLOv5, PANet is used as the neck network to obtain feature pyramids. The YOLO head applies anchor boxes on features and generates final output vectors with class probabilities, objectness scores and bounding boxes.

The model for detection of agricultural classes is YOLOv5s model trained on the agricultural dataset as discussed above. The dataset has been randomly split to training dataset and validation with <NUM>:<NUM> ratio. The images are resized to <NUM> x <NUM> and the model is trained for <NUM> epochs using default hyperparameters.

The model used for detection of classes 'person', 'car' and 'truck' is YOLOv5l model trained on COCO dataset to detect <NUM> classes of common objects. During non-max suppression stage the three classes are selected from the predictions.

Since the two models are trained for the classes that belong to the same supercategories, it is expected that the same objects will be detected by both models. However, since the base classes that the models detect are distinct the models assign different classes to the object. Therefore, it is necessary to address the ensemble of detection results from the first and second models to a combined output. The present disclosure achieves this utilising the described classification metric.

For the better overview of linkage between different base classes and their corresponding subcategories and supercategories, the hierarchy of classes is shown in <FIG>. While operating in the field, the agricultural machines often have an implement attached to them. In our test set, this is a very common case and in the hierarchy this dependence is shown as a link between the base classes 'tractor' and 'combine', and subcategories 'Tractor' and 'Combine' for the supercategory 'Implement'. In general sense, an agricultural vehicle with an implement is considered to be part of 'Agricultural vehicle' subcategory and the high level supercategory 'Vehicle'.

As discussed, the present invention utilises a classification metric in the step of determining an identity for an object. Here, this is demonstrated by removing detections from the second model in certain scenarios, specifically utilising a classification metric as a measure of an overlap between the classifications from the two models. In this experimental setup, 'Truck' detections are discounted based on an IoU value for the classifications from the first and second models in the following cases, for respective classifications from the first model:.

Similarly, 'Car' detections are discounted based on IoU value in the following cases:.

This section analyses performance of the models in three cases. First, the performance of the individual models on the test dataset is evaluated. Then, the performance of both models combined with ensemble module is evaluated. Finally, the performance of the models at subcategory level is evaluated.

First, the performance of the individual models on the test set is evaluated. As expected, the models are not able to detect the classes that they were not trained on. This results in mAP equal to zero for these classes affecting the average mAP on the test set. The results are shown in table <NUM>, below.

The first detection model trained on the internal dataset has good mAP for all the classes it has been trained on, except 'baler'. However, because the model is not able to detect classes 'person', 'car' and 'truck', the average mAP@. <NUM> for all classes is <NUM> and mAP@. <NUM> is <NUM>. The second model trained on the generic COCO dataset is able to detect classes 'person' and 'car' very well, while class 'truck' has low mAP. Similar to internal model, the mAP is affected by the classes the model cannot detect resulting in mAP@. <NUM> equal to <NUM> and mAP@. <NUM> equal to <NUM>. It can be concluded from the presented results that the individual models, for themselves, are not performing well adequately on the targeted test set.

The performance of combined models is evaluated with ensemble module and compared to performance without the ensemble module. The results are reported using multiclassification confusion matrix and mAP at loU threshold equal to <NUM> and as well as averaged over <NUM> IoU thresholds [<NUM>:<NUM>] - <FIG>.

First, the detections of the two models were concatenated into combined output without the ensemble module and the performance is evaluated. Compared to the performance of the individual models, the combined model is able to detect all classes in the test set. Therefore, the overall performance computed as mAP@. <NUM> increased from <NUM> and <NUM> to combined mAP@. <NUM> of <NUM> - Table <NUM>.

Then, the ensemble module is added and applied to combined output and the performance is evaluated with emphasis on misclassification of similar classes detected by the two models.

A confusion matrix comprehensively illustrates classification accuracy of object detector as well as misclassifiaction rate for pairs of classes. The confusion matrices for combined model without the ensemble module and with the ensemble module are shown in <FIG> respectively. The rows in the matrix correspond to the instances predicted to belong to a class, and the columns correspond to the instances in the actual class. In the calculation of confusion matrix, only boxes with confidence score greater then <NUM> are considered. The threshold for IoU between ground-truth and detected bounding box is set to <NUM> and columns in the confusion matrices are normalized.

It can be seen that the ensemble module has significantly better performance, and the confusion between road vehicle classes and classes for agricultural vehicles and implements is lower. Initially the combined model the classes 'trailer' and 'baler' are most easily confused with class 'truck'. After applying the ensemble module the values of corresponding elements in the confusion matrix are lower and values in diagonal elements for classes 'trailer' and 'baler' improved. The similar can be observed for classes 'tractor' and 'combine', while values for 'combine header' remain the same. When it comes to class 'car', there is a small decrease in noticeable in misclassifiaction of classes 'tractor' and 'baler'.

Looking at Table <NUM>, below, it can be seen that the overall mAP@. <NUM> increased from <NUM> to <NUM> when ensemble module is applied. The same trend can be observed with mAP@. <NUM> which increased from <NUM> to <NUM>. The increase is due to the increase in mAP for class 'truck'. This resulted from removal of misclassification of agricultural vehicles and implements as 'truck' as these are considered as false positives for this class.

The qualitative performance of ensemble module is presented in <FIG>. The first example shows 'truck' detection removed because internal model detected 'combine' and 'combine header' pair. The second example shows 'truck' detection removed because internal model detected 'tractor' and 'trailer' pair. The fourth example shows 'baler' attached to 'tractor' and two 'square bales'. The fifth example, shows again 'baler' attached to 'tractor' and one 'square bale'. In this case there are two redundant detection, one 'truck' detection corresponding to 'baler' and one 'car' detection corresponding to 'tractor'. The sixth example shows 'trailer' attached to 'tractor' and two redundant 'truck' detections. A first redundant detection corresponds to 'tractor' and 'trailer' pair and the other one to 'tractor' only.

Any process descriptions or blocks in flow diagrams should be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps in the process, and alternate implementations are included within the scope of the embodiments in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present disclosure.

Claim 1:
A control system for an object detection system for an agricultural machine, the control system comprising one or more controllers, and being configured to:
receive image data from one or more imaging sensors associated with the agricultural machine;
analyse the image data utilising first and second detection models to classify, using
both models, an object within the environment of the agricultural machine;
determine a classification metric for the object indicative of an overlap associated with the classification obtained for each of the models for the object; and
determine an identity for the object in dependence on the classification metric and the classification obtained for the first and/or second models; and
control operation of one or more operable components associated with the agricultural machine in dependence on the determined identity;
wherein the first and second detection models comprise machine-learned models trained on training datasets with known objects with respective classifications; the training dataset for the first model comprising an agricultural dataset and the training dataset for the second model comprising a generic dataset; and
wherein the control system is further configured to:
compare the classification metric with a threshold value; and
determine the identity for the object in dependence on the classification for the first model only, when the classification metric exceeds the threshold value.