Patent Description:
There are many different types of agricultural machines. One such machine is an agricultural sprayer (<CIT>). An agricultural spraying system often includes a tank or reservoir that holds a substance to be sprayed on an agricultural field. Such systems typically include a fluid line or conduit mounted on a foldable, hinged, or retractable and extendible boom. The fluid line is coupled to one or more spray nozzles mounted along the boom. Each spray nozzle is configured to receive the fluid and direct atomized fluid to a crop or field during application. As the sprayer travels through the field, the boom is moved to a deployed position and the substance is pumped from the tank or reservoir, through the nozzles, so that it is sprayed or applied to the field over which the sprayer is traveling.

In accordance with an aspect of the present invention, there is provided a computing system as defined in claim <NUM>, with preferred features as set out in dependent claims <NUM> to <NUM>.

In accordance with another aspect of the present invention, there is provided a method performed by a computing system as defined in claim <NUM>, with preferred features as set out in dependent claim <NUM>.

<FIG> is a block diagram showing one example of a computing environment that can be used in the architectures shown in the previous figures.

The present description generally relates to agricultural machines. More specifically, but not by limitation, the present description relates to plant evaluation and machine control using field images.

As an agricultural spraying machine (or agricultural sprayer) traverses a field, it applies a spray of a liquid (e.g., herbicide, fertilizer, fungicide, or other chemical) using nozzles mounted on a boom. A spraying system, which typically includes a pump that pumps the liquid from a reservoir to the nozzles mounted on the boom, is controlled to deliver a target or prescribed application to the agricultural field. For example, in precision spraying applications, the sprayer is controlled to deliver the liquid to a precise dispersal area, such as directly on a plant (crop or weed), in between plants, or otherwise, at a particular rate so that a target quantity of the liquid is applied to the dispersal area. Accordingly, precise application of the liquid is important in these applications. For example, if an herbicide is unevenly applied or applied to incorrect plants, it is wasted in areas of over-application, and areas of under-application experience reduced weed prevention.

For sake of illustration, image processing is performed in some precision spraying applications by acquiring images of the field to identify the locations of the weeds to be sprayed with an herbicide (or other liquid chemical). In some approaches, a weed classifier is trained with training data to detect the various types of weeds to be sprayed. However, due to a wide range of possible field scenarios, such as a plurality of different possible weed types, different weed conditions, different lighting, etc., a large collection of images are required to train the weed classifier. The computing resources and processing required to both acquire and process these images, and to train the classifier, can be quite burdensome and computational expensive. Even still, the trained weed classifier may perform poorly during varying runtime field applications when trying to classify in different lighting conditions and/or when the weeds are damaged by insects, weather, etc. The inaccurate weed identification results in poor spraying performance.

For sake of the present discussion, a "weed" or "weed plant" refers to any non-crop plant identified in the field. That is, it includes plant types other than crop plants (e.g., corn plants in a corn field) expected to be present in the field under consideration. In the corn field example, weeds or weed plants include any plants other than corn plants.

The present disclosure provides a plant evaluation and control system that acquires images of a field and processes those images to identify portion(s) of the image that represent ground (i.e., the soil or other non-plant areas of the field terrain), and omits those identified portions from the image to identify a remaining image portion. This remaining image portion is processed by applying a crop classifier to detect, from the plants in the remaining image portion, areas of the image that represent crop. Locations of weed plants represented in the image are identified based on the identification of the portions of the image that represent ground and the portions of the image that represent crop, e.g., by omitting the ground and crop portions from the image, and then correlating the remaining weed plant image portion to the geographic field locations. This weed location information can be utilized in any of a number of ways, for example by controlling an agricultural sprayer, generating a weed plant map for the field, to name a few.

<FIG> illustrates an agricultural spraying machine (or agricultural sprayer) <NUM>. Sprayer <NUM> includes a spraying system <NUM> having a tank <NUM> containing a liquid that is to be applied to field <NUM>. Tank <NUM> is fluidically coupled to spray nozzles <NUM> by a delivery system comprising a set of conduits. A fluid pump is configured to pump the liquid from tank <NUM> through the conduits through nozzles <NUM>. Spray nozzles <NUM> are coupled to, and spaced apart along, boom <NUM>. Boom <NUM> includes arms <NUM> and <NUM> which can articulate or pivot relative to a center frame <NUM>. Thus, arms <NUM> and <NUM> are movable between a storage or transport position and an extended or deployed position (shown in <FIG>).

In the example illustrated in <FIG>, sprayer <NUM> comprises a towed implement <NUM> that carries the spraying system, and is towed by a towing or support machine <NUM> (illustratively a tractor) having an operator compartment or cab <NUM>. Sprayer <NUM> includes a set of traction elements, such as wheels <NUM>. The traction elements can also be tracks, or other traction elements as well. It is noted that in other examples, sprayer <NUM> is self-propelled. That is, rather than being towed by a towing machine, the machine that carries the spraying system also includes propulsion and steering systems.

<FIG> illustrates one example of an agricultural sprayer <NUM> that is self-propelled. That is, sprayer <NUM> has an on-board spraying system <NUM>, that is carried on a machine frame <NUM> having an operator compartment <NUM>, a steering system <NUM> (e.g., wheels or other traction elements), and a propulsion system <NUM> (e.g., internal combustion engine).

<FIG> illustrates one example of an architecture <NUM> having an agricultural spraying machine <NUM> configured to perform a spraying operation on an agricultural field. Examples of agricultural spraying machine <NUM> include, but are not limited to, sprayers <NUM> and <NUM> illustrated in <FIG> and <FIG>. Accordingly, machine <NUM> can comprise a towed implement or it can be self-propelled. <FIG> illustrates this with dashed box <NUM> representing a towing machine, such as a tractor that is coupled to machine <NUM> through one or more links <NUM> (electrical, mechanical, pneumatic, etc.).

Machine <NUM> includes a control system <NUM> configured to control other components and systems of machine <NUM>. For instance, control system <NUM> includes a communication controller <NUM> configured to control communication system <NUM> to communicate between components of machine <NUM> and/or with other machines or systems, such as remote computing system <NUM> and/or machine(s) <NUM>, either directly or over a network <NUM>. Network <NUM> can be any of a wide variety of different types of networks such as the Internet, a cellular network, a local area network, a near field communication network, or any of a wide variety of other networks or combinations of networks or communication systems.

A remote user <NUM> is illustrated interacting with remote computing system <NUM>. Remote computing system <NUM> can be a wide variety of different types of systems. For example, remote system <NUM> can be a remote server environment, remote computing system that is used by remote user <NUM>. Further, it can be a remote computing system, such as a mobile device, remote network, or a wide variety of other remote systems. Remote system <NUM> can include one or more processors or servers, a data store, and it can include other items as well.

Communication system <NUM> can include wired and/or wireless communication logic, which can be substantially any communication system that can be used by the systems and components of machine <NUM> to communicate information to other items, such as between control system <NUM>, sensors <NUM>, controllable subsystems <NUM>, image capture system <NUM>, and plant evaluation system <NUM>. In one example, communication system <NUM> communicates over a controller area network (CAN) bus (or another network, such as an Ethernet network, etc.) to communicate information between those items. This information can include the various sensor signals and output signals generated by the sensor variables and/or sensed variables.

Control system <NUM> is configured to control interfaces, such as operator interface(s) <NUM> that include input mechanisms configured to receive input from an operator <NUM> and output mechanisms that render outputs to operator <NUM>. The user input mechanisms can include mechanisms such as hardware buttons, switches, joysticks, keyboards, etc., as well as virtual mechanisms or actuators such as a virtual keyboard or actuators displayed on a touch sensitive screen. The output mechanisms can include display screens, speakers, etc..

Sensor(s) <NUM> can include any of a wide variety of different types of sensors. In the illustrated example, sensors <NUM> include position sensor(s) <NUM>, speed sensor(s) <NUM>, and can include other types of sensors <NUM> as well. Position sensor(s) <NUM> are configured to determine a geographic position of machine <NUM> on the field, and can include, but are not limited to, a Global Navigation Satellite System (GNSS) receiver that receives signals from a GNSS satellite transmitter. It can also include a Real-Time Kinematic (RTK) component that is configured to enhance the precision of position data derived from the GNSS signal. Speed sensor(s) <NUM> are configure to determine a speed at which machine <NUM> is traveling the field during the spraying operation. This can include sensors that sense the movement of ground-engaging elements (e.g., wheels or tracks) and/or can utilize signals received from other sources, such as position sensor(s) <NUM>.

Control system <NUM> includes control logic <NUM>, and can include other items <NUM> as well. As illustrated by the dashed box in <FIG>, control system <NUM> can include some or all of plant evaluation system <NUM>, which is discussed in further detail below. Also, machine <NUM> can include some or all of image capture system <NUM>. Control logic <NUM> is configured to generate control signals to control sensors <NUM>, controllable subsystems <NUM>, communication system <NUM>, or any other items in architecture <NUM>. Controllable subsystems <NUM> include a spraying subsystem <NUM>, machine actuators <NUM>, a propulsion subsystem <NUM>, a steering subsystem <NUM>, and can include other items <NUM> as well. Spraying subsystem <NUM> includes one or more pumps <NUM>, configured to pump material (liquid chemicals) from tank(s) <NUM> through conduits to nozzles <NUM> mounted on a boom, for example. Spraying subsystem <NUM> can include other items <NUM> as well.

Machine <NUM> includes a data store <NUM> configured to store data for use by machine <NUM>, such a field data. Examples include field location data that identifies a location of the field to be operated upon by a machine <NUM>, field shape and topography data that defines a shape and topography of the field, crop location data that is indicative of a location of crops in the field (e.g., the location of crop rows), or any other data.

Machine <NUM> is illustrated as including one or more processors or servers <NUM>, and can include other items <NUM> as well. As also illustrated in <FIG>, where a towing machine <NUM> tows agricultural spraying machine <NUM>, towing machine <NUM> can include some of the components discussed above with respect to machine <NUM>. For instance, towing machine <NUM> can include some or all of sensors <NUM>, component(s) of control system <NUM>, some or all of controllable subsystems <NUM>. Also, towing machine <NUM> can include a communication system <NUM> configured to communicate with communication system <NUM>, one or more processors or servers <NUM>, a data store <NUM>, and it can include other items <NUM> as well. As also illustrated in <FIG>, towing machine <NUM> can include some or all components of image capture system <NUM>, which is discussed in further detail below.

Image capture system <NUM> includes image capture components configured to capture one or more images of the area under consideration (i.e., the portions of the field to be operated upon by spraying machine <NUM>) and image processing components configured to process those images. The captured images represent a spectral response captured by image capture system <NUM> that are provided to plant evaluations system <NUM> and/or stored in data store <NUM>. A spectral imaging system illustratively includes a camera that takes spectral images of the field under analysis. For instance, the camera can be a multispectral camera or a hyperspectral camera, or a wide variety of other devices for capturing spectral images. The camera can detect visible light, infrared radiation, or otherwise.

In one example, the image capture components include a stereo camera configured to capture a still image, a time series of images, and/or a video of the field. An example stereo camera captures high definition video at thirty frames per second (FPS) with one hundred and ten degree wide-angle field of view. Of course, this is for sake of example only.

Illustratively, a stereo camera includes two or more lenses with a separate image sensor for each lens. Stereo images (e.g., stereoscopic photos) captured by a stereo camera allow for computer stereo vision that extracts three-dimensional information from the digital images. In another example, a single lens camera can be utilized to acquire images (referred to as a "mono" image).

Image capture system <NUM> can include one or more of an aerial image capture system <NUM>, an on-board image capture system <NUM>, and/or other image capture system <NUM>. An example of aerial image capture system <NUM> includes a camera or other imaging component carried on an unmanned aerial vehicle (UAV) or drone (e.g., block <NUM>). An example of on-board image capture system <NUM> includes a camera or other imaging component mounted on, or otherwise carried by, machine <NUM> (or <NUM>). An example of image capture system <NUM> includes a satellite imaging system. System <NUM> also includes a location system <NUM>, and can include other items <NUM> as well. Location system <NUM> is configured to generate a signal indicative of geographic location associated with the captured image. For example, location system <NUM> can output GPS coordinates that are associated with the captured image to obtain geo-referenced images <NUM> that are provided to plant evaluation system <NUM>.

Plant evaluation system <NUM> illustratively includes one or more processors <NUM>, a communication system <NUM>, a data store <NUM>, an image analysis system <NUM>, and can include other items <NUM> as well. Data store <NUM> can store the geo-referenced images <NUM> received from image capture system <NUM>, plant evaluation data generated by system <NUM>, or any other data used by system <NUM> or other machines or systems of architecture <NUM>. Communication system <NUM>, in one example, is substantially similar to communication system <NUM> discussed above.

<FIG> illustrates one example of plant evaluation system <NUM>. As shown in <FIG>, system <NUM> includes a user interface component <NUM> configured to generate user interface(s) <NUM> having user input mechanism(s) <NUM> for access by a user <NUM>. User <NUM> interacts with user input mechanisms <NUM> to control and manipulate plant evaluation system <NUM>. For example, user <NUM> can control image analysis system <NUM>, view images <NUM> stored in data store <NUM>, to name a few. Also, user <NUM> can view the image analysis results and evaluate how to treat the field (or various portions within the field) based upon the results. Plant evaluation system <NUM> can also generate recommendations for treating various spots within the field, based upon the analysis data. This can vary widely from things such as applying more herbicide, applying fertilizer, to name a few. A control signal generator logic <NUM> is configured to generate control signals to control items of system <NUM>, or other items in architecture <NUM>.

Image analysis system <NUM> includes image receiving logic <NUM> configured to receive images from image capture system <NUM> and image pre-processing logic <NUM> configured to pre-process those images. For example, logic <NUM> includes a shadow corrector <NUM> configured to perform shadow correction on the images, illumination normalizer <NUM> configured to normalize illumination in the image, image combiner <NUM> configured to combine images, and can include other items <NUM> as well.

Image combiner <NUM>, in one example, is configured to combine a number of images into a larger image of the field under analysis. For instance, image combiner <NUM> can mosaic the images and geo-reference them relative to ground control points. In order to mosaic the images, geographic location information corresponding to each of the images is used to stich them together into a larger image of the field under analysis, which is then analyzed by system <NUM>. Further, the geo-referencing of images can be done automatically against the ground control points, or it can be done manually as well.

Geo-referencing logic <NUM> is configured to geo-reference the images, or combined images, to locations in the field, spatial analysis logic <NUM> is configured to perform spatial analysis on the images, and spectral analysis logic <NUM> is configured to perform spectral analysis on the images. Spatial analysis logic <NUM>, in one example, obtains previously-generated crop location data which provides a geographic location of the rows of crop plants (or the plants themselves). For example, this can be generated during a planting operation using a planting machine. Of course, crop location data can be obtained from other sources as well. In any case, the crop location data can be utilized to identify the crop rows, and thus the areas between the crop rows that are expected to be free of crop plants. This can include identifying a reference line that corresponds to the center of each crop row along with a margin window around that reference line, for each row. As discussed in further detail below, plants identified between two adjacent reference lines (and/or margin window) can be assumed to be a non-crop plant (e.g., a weed plant).

Spectral analysis logic <NUM> performs spectral analysis to evaluate the plants in the images. In one example, this includes identifying areas in the image that have a spectral signature that corresponds to ground versus plants. For instance, this can include a green/brown comparison.

Image segmentation logic <NUM> is configured to perform image segmentation on a received image, to segment or divide the image into different portions for processing. This can be based on ground and/or plant area identifications by ground/plant identification logic <NUM>, and crop classification performed by crop classification logic <NUM>. This is discussed in further detail below. Briefly, however, ground/plant identification logic <NUM> identifies areas of an image that represent ground and areas of an image that represent plants, for example using the spatial and spectral analysis performed by logic <NUM> and <NUM>, respectively.

Crop classification logic <NUM> uses a crop classifier, that can be trained by crop classifier training logic <NUM>. In one example, the crop classifier is trained using crop training data <NUM> stored in data store <NUM>, or obtained otherwise. The crop classifier is configured to identify areas in the image that represent crop plants.

Weed identification logic <NUM> is configured to identify weeds in the image, based on the image segmentation performed by image segmentation logic <NUM>. This is discussed in further detail below. Briefly, however, image segmentation logic <NUM> is configured to identify a weed plant portion of a received image, that omits a first portion of the image that represents ground in the field and a second portion of the image that represents crop plants. This remaining portion of the image is determined to represent weeds. Illustratively, in one example, this process is performed without using a weed classifier or otherwise directly identifying weed plants in the image.

The location of the weeds can be stored as crop location data <NUM>, weed data <NUM> in data store <NUM>, which can store other items <NUM> as well.

Image analysis system <NUM> can also include anomaly detection logic <NUM> and can include other items <NUM> as well. Anomaly detection logic <NUM> is configured to detect anomalies based on the weed plant image (e.g., the portion of the image remaining after the ground image portion and the crop image portion have been omitted). Illustratively, a detected anomaly represents anomalous crop detections. For instance, one example of an anomaly is a crop plant detection in an area that is in between the crop rows. This can represent a false positive detection by the crop classifier, and can be used to re-train the crop classifier to improve its performance.

<FIG> illustrates one example of a flow diagram <NUM> for identifying weed plants from image data and corresponding machine control. For sake of illustration, but not by limitation, <FIG> will be described in the context of plant evaluation system <NUM> in architecture <NUM>.

At block <NUM>, a crop classifier to be used by crop classification logic <NUM> is selected based on a selected crop. For example, the selected crop can be selected by operator <NUM>. In one example, the crop classifier is a classifier that is trained by training logic <NUM> accessing crop training data <NUM> to classify portions of an image representing plants as being crop plants, as opposed to non-crop plants or weeds. The training data can take any of a variety of forms, such as images labeled with crop data identifying areas of the images that are crop plants and/or areas of the image that are non-crop plants. For example, to train a corn plant classifier configured to identify corn plants in a plant image, a set of training images are labeled with identifiers that identify the areas of the image that represent crop plants.

At block <NUM>, image data indicative of an image of a field is received by image receiving logic <NUM>. As noted above, the image can be obtained in any of a wide variety of ways. The image can comprise a single image obtained by a camera, an image within a time series of still images, or an image from a video. Further, the image can comprise a stereo image (represented by block <NUM>), a mono image <NUM>, or other type of image <NUM>. Also, the image can be received from an on-board imaging sensor, such as from on-board image capture system <NUM>. This is represented by block <NUM>. Alternatively, or in addition, the image can be received from a remote source, such as from remote computing system <NUM>. This is represented by block <NUM>. Further yet, the image can be received from another machine <NUM>. For instance, a UAV that flies over the field prior to a spraying operation acquires images of the field. This is represented by block <NUM>.

At block <NUM>, the image is pre-processed. This can include removing or correcting shadows using shadow corrector <NUM>. This is represented by block <NUM>. Also, the image can be processed to normalize illuminations using illumination normalizer <NUM>. This is represented by block <NUM>. Of course, the image can be pre-processed in other ways as well. This is represented by block <NUM>.

At block <NUM>, one or more portions of the image representing ground in the field are identified. This can be done in any of a number of ways. For example, this can be done based on colors identified in the image. For instance, image processing is performed using RGB (red-green-blue) color vectors. RGB color data refers to a color model in which red, green and blue light (or signals or data representative thereof) are combined to represent other colors. Each pixel or group of pixels of the collected image data may be associated with an image parameter level or a corresponding pixel value or aggregate pixel value. Thus, each pixel stands for one discrete location in the image and stores data for three channels (red, green, and blue) to represent a certain color. The image parameter level is an indicator of or a measure of an image parameter that observed, reflected and/or emitted from one or more objects in any other portion of one or more objects within the image. A clustering algorithm can be configured to cluster pixel values to generate a base color vector and/or averaged color vector for the image. Accordingly, at block <NUM>, a clustering algorithm can be utilized with RGB IR segmentation to segment the image based on RGB color vectors.

Referring to <FIG>, which illustrates an example image <NUM> of a portion of field, block <NUM> identifies portions <NUM> as areas of the image representing ground, determined based on differences in the color during color segmentation. Portions <NUM> are, in this example, identified as containing brown, or threshold shades of brown, and the other areas of the image are green or at least beyond a threshold difference from the brown color of portions <NUM>. Thus, block <NUM> identifies portions <NUM> as representing the ground.

Alternatively, or in addition, the portions of the image representing ground can be identified using stereo data, such as a point cloud. This is represented by block <NUM> in <FIG>. For example, as noted above, the stereo data can provide three-dimensional information to distinguish the location of the plant material relative to the ground plane. Thus, the stereo data at block <NUM> can be used to identify areas of the image representing material that is above the ground plane (e.g., above the ground by a threshold distance. Referring again to <FIG>, block <NUM> identifies the areas of the image generally represented by reference numeral <NUM> as representing plants, and these portions of the image are separated from the ground portions <NUM>. In one example, a remaining image portion is obtained that omits the ground portions <NUM>.

The image portions representing the ground can be identified in other ways as well. This is represented by block <NUM> in <FIG>. At block <NUM>, the ground portion(s) identified at block <NUM> are separated to obtain a remaining (non-ground or plant) image portion that represents areas in the field that include plants (both crop and non-crop plants). The remaining image portion obtained at block <NUM> omits the ground portions. This can be done in any of a number of ways. For example, image segmentation logic <NUM> can segment or divide the image, and extract the non-ground portions, to obtain the remaining image portion. In one example, the received image, or a copy thereof, is stored in memory and the remaining image portion is obtained by removing the image data for the non-ground portions from the memory, so that only the image data for the remaining image portion remains. This, of course, is by way of example only.

At block <NUM>, crop portions in the remaining image portion are detected. Illustratively, the detection is performed by applying a crop classifier to the remaining image portion. As noted above, crop classification logic <NUM> can apply a crop classifier by crop classifier training logic <NUM> using crop training data <NUM>.

The detection of the crop portions at block <NUM> can be based on location within the image (block <NUM>) and/or based on color within the image (block <NUM>). Of course, the detection can be performed in other ways as well. This is represented by block <NUM>.

In one example of block <NUM>, logic <NUM> obtains crop location data <NUM>. As discussed above, crop location data <NUM> can be obtained from a prior planting operation, or otherwise, and indicates the locations where crop seeds were planted, which can be indicative of the crop rows, the crop spacings within the rows, etc. Using crop location data <NUM>, logic <NUM> can identify the locations within the image where a crop plant is expected. Also, using crop location data <NUM>, logic <NUM> can determine that plants that appear in between the rows are likely to be non-crop plants.

In one example of block <NUM>, logic <NUM> looks at the RGB color vectors from a pixel clustering algorithm to determine whether an area of the image that represents a plant indicates a crop plant or a non-crop plant.

At block <NUM>, a weed (non-crop plant) image is obtained which represents weed portions in the received image. In the illustrated example, the weed image is obtained based on the identification of a first image portion (i.e., a ground portion identified at block <NUM>) and a second image portion (i.e., a crop portion identified at block <NUM>). That is, the weed image a remaining image portion that omits the ground portions and the crop portions in the image. This can be done by image segmentation logic <NUM> separating the weed image portion from the other portions, or otherwise.

For sake of illustration, with reference again to <FIG>, block <NUM> detects image portion <NUM> as representing crop based on the location of that image portion relative to the rows and based on the color of that image portion relative to a portion <NUM> that represents a weed plant. Block <NUM> obtains a weed image that includes portions <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>, and omits the ground portions <NUM> and crop portions <NUM>.

Referring again to <FIG>, at block <NUM>, the weed image is analyzed to detect anomalies. As noted above, anomalies can be detected for any of a number of reasons. In one example, an anomaly is detected based the location on the crop rows. This is represented by block <NUM>. For example, if a crop portion is detected at block <NUM> and resides in between crop rows, identified based on location at block <NUM>, then an anomaly is detected as the plant in that portion of the image is unlikely to be a crop plant. Anomalies can be detected in other ways as well. This is represented by block <NUM>. At block <NUM>, logic <NUM> dynamically updates or tunes the crop classifier based on the detected anomalies. The updated crop classifier can be re-applied by returning to block <NUM>.

At block <NUM>, the weed portions identified at block <NUM> are correlated to their respective areas of the field. These areas of the field are identified as containing weeds. In one example, weed identification logic <NUM> generates geographic coordinates for each separate field area that has been identified as containing weeds, and stores this data as weed data <NUM>.

At block <NUM>, control signal generator logic <NUM> generates a control signal based on the identified field areas. This control signal controls one or more systems or machines in architecture <NUM> in any of a variety of ways.

For example, control signal generator logic <NUM> can generate a control signal to control spraying subsystem <NUM> to apply a liquid chemical to the identified field areas. This is represented by block <NUM>. In another example, the control signal generator logic <NUM> can control a weed map generator <NUM> to generate a weed map that identifies locations of the weeds on a map of the field.

Alternatively, or in addition, the control signal can control communication system <NUM> to send the weed data or weed map to a remote machine or system. This is represented by block <NUM>. For instance, the weed data can be sent to another spraying machine, remote computing system <NUM>, etc..

In one example, the control signal controls a display device, such as interfaces <NUM> and/or <NUM> for operator <NUM> or user <NUM>. This is represented by block <NUM>. Of course, the machine control signal can be generated to control other items in architecture <NUM>. This is represented by block <NUM>.

It can thus be seen that the present system provides a number of advantages. For example, but not by limitation, performing weed identification using a remaining image portion, remaining after omitting ground and crop portions, increases the speed and efficiency in the processing. Further, applying a crop classifier to identify crops from the plant portions of the image reduces the computational burden and expensive. Further, actions can be taken based upon the weed identification to save chemicals, to save time, and to otherwise improve the agricultural operations.

The present discussion has mentioned processors, processing systems, controllers and/or servers. In one example, these can include computer processors with associated memory and timing circuitry, not separately shown. They are functional parts of the systems or devices to which they belong and are activated by, and facilitate the functionality of the other components or items in those systems.

<FIG> is a block diagram of one example of the architecture shown in <FIG>, where machine <NUM> communicates with elements in a remote server architecture <NUM>. In an example, remote server architecture <NUM> can provide computation, software, data access, and storage services that do not require end-user knowledge of the physical location or configuration of the system that delivers the services. In various examples, remote servers can deliver the services over a wide area network, such as the internet, using appropriate protocols. For instance, remote servers can deliver applications over a wide area network and they can be accessed through a web browser or any other computing component. Software or components shown in <FIG> as well as the corresponding data, can be stored on servers at a remote location. The computing resources in a remote server environment can be consolidated at a remote data center location or they can be dispersed. Remote server infrastructures can deliver services through shared data centers, even though they appear as a single point of access for the user. Thus, the components and functions described herein can be provided from a remote server at a remote location using a remote server architecture. Alternatively, they can be provided from a conventional server, or they can be installed on client devices directly, or in other ways.

In the example shown in <FIG>, some items are similar to those shown in <FIG> and they are similarly numbered. <FIG> specifically shows that system <NUM> and data store <NUM> can be located at a remote server location <NUM>. Therefore, agricultural machine <NUM> accesses those systems through remote server location <NUM>.

<FIG> also depicts another example of a remote server architecture. <FIG> shows that it is also contemplated that some elements of <FIG> are disposed at remote server location <NUM> while others are not. By way of example, data store <NUM> can be disposed at a location separate from location <NUM>, and accessed through the remote server at location <NUM>. Alternatively, or in addition, system <NUM> can be disposed at location(s) separate from location <NUM>, and accessed through the remote server at location <NUM>.

Regardless of where they are located, they can be accessed directly by agricultural machine <NUM>, through a network (either a wide area network or a local area network), they can be hosted at a remote site by a service, or they can be provided as a service, or accessed by a connection service that resides in a remote location. Also, the data can be stored in substantially any location and intermittently accessed by, or forwarded to, interested parties. For instance, physical carriers can be used instead of, or in addition to, electromagnetic wave carriers. In such an example, where cell coverage is poor or nonexistent, another mobile machine (such as a fuel truck) can have an automated information collection system. As the agricultural machine comes close to the fuel truck for fueling, the system automatically collects the information from the machine or transfers information to the machine using any type of ad-hoc wireless connection. The collected information can then be forwarded to the main network as the fuel truck reaches a location where there is cellular coverage (or other wireless coverage). For instance, the fuel truck may enter a covered location when traveling to fuel other machines or when at a main fuel storage location. All of these architectures are contemplated herein. Further, the information can be stored on the agricultural machine until the agricultural machine enters a covered location. The agricultural machine, itself, can then send and receive the information to/from the main network.

It will also be noted that the elements of <FIG>, or portions of them, can be disposed on a wide variety of different devices. Some of those devices include servers, desktop computers, laptop computers, tablet computers, or other mobile devices, such as palm top computers, cell phones, smart phones, multimedia players, personal digital assistants, etc..

<FIG> is a simplified block diagram of one illustrative example of a handheld or mobile computing device that can be used as a user's or client's hand held device <NUM>, in which the present system (or parts of it) can be deployed. For instance, a mobile device can be deployed in the operator compartment of agricultural machine <NUM> (or <NUM>) or as remote computing system <NUM>. <FIG> are examples of handheld or mobile devices.

<FIG> provides a general block diagram of the components of a client device <NUM> that can run some components shown in <FIG>, that interacts with them, or both. In the device <NUM>, a communications link <NUM> is provided that allows the handheld device to communicate with other computing devices and under some embodiments provides a channel for receiving information automatically, such as by scanning. Examples of communications link <NUM> include allowing communication though one or more communication protocols, such as wireless services used to provide cellular access to a network, as well as protocols that provide local wireless connections to networks.

In other examples, applications can be received on a removable Secure Digital (SD) card that is connected to an interface <NUM>. Interface <NUM> and communication links <NUM> communicate with a processor <NUM> (which can also embody processors or servers from previous FIGS. ) along a bus <NUM> that is also connected to memory <NUM> and input/output (I/O) components <NUM>, as well as clock <NUM> and location system <NUM>.

I/O components <NUM>, in one example, are provided to facilitate input and output operations. I/O components <NUM> for various embodiments of the device <NUM> can include input components such as buttons, touch sensors, optical sensors, microphones, touch screens, proximity sensors, accelerometers, orientation sensors and output components such as a display device, a speaker, and or a printer port. Other I/O components <NUM> can be used as well.

Memory <NUM> stores operating system <NUM>, network settings <NUM>, applications <NUM>, application configuration settings <NUM>, data store <NUM>, communication drivers <NUM>, and communication configuration settings <NUM>. Memory <NUM> can include all types of tangible volatile and non-volatile computer-readable memory devices. It can also include computer storage media (described below). Memory <NUM> stores computer readable instructions that, when executed by processor <NUM>, cause the processor to perform computer-implemented steps or functions according to the instructions. Processor <NUM> can be activated by other components to facilitate their functionality as well.

<FIG> shows one example in which device <NUM> is a tablet computer <NUM>. In <FIG>, computer <NUM> is shown with user interface display screen <NUM>. Screen <NUM> can be a touch screen or a pen-enabled interface that receives inputs from a pen or stylus. It can also use an on-screen virtual keyboard. Of course, it might also be attached to a keyboard or other user input device through a suitable attachment mechanism, such as a wireless link or USB port, for instance. Computer <NUM> can also illustratively receive voice inputs as well.

<FIG> is one example of a computing environment in which elements of <FIG>, or parts of it, (for example) can be deployed. With reference to <FIG>, an example system for implementing some embodiments includes a computing device in the form of a computer <NUM>. Components of computer <NUM> may include, but are not limited to, a processing unit <NUM> (which can comprise processors or servers from previous FIGS. ), a system memory <NUM>, and a system bus <NUM> that couples various system components including the system memory to the processing unit <NUM>. The system bus <NUM> may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. Memory and programs described with respect to <FIG> can be deployed in corresponding portions of <FIG>.

The hard disk drive <NUM> is typically connected to the system bus <NUM> through a non-removable memory interface such as interface <NUM>, and optical disk drive <NUM> is typically connected to the system bus <NUM> by a removable memory interface, such as interface <NUM>.

The computer <NUM> is operated in a networked environment using logical connections (such as a local area network - LAN, or wide area network - WAN or a controller area network - CAN) to one or more remote computers, such as a remote computer <NUM>.

Claim 1:
A computing system comprising:
image receiving logic (<NUM>) configured to receive image data indicative of an image of a field;
ground identification logic (<NUM>) configured to identify a first image portion of the image representing ground in the field;
image segmentation logic (<NUM>) configured to identify a remaining image portion representing non-ground in the field that omits the first image portion from the image;
crop classification logic (<NUM>) configured to:
apply a crop classifier to the remaining image portion; and
identify a second image portion of the image that represents locations of crop plants in the field;
the computer system being characterized in that the image segmentation logic (<NUM>) is configured to obtain a third image portion that omits the first and second image portions from the image; weed identification logic (<NUM>) is configured to identify locations of weed plants in the field based on the third image portion; and
control signal generation logic (<NUM>) is configured to generate a machine control signal based on the identified locations of the weed plants.