CALIBRATION ADJUSTMENT FOR AGRICULTURAL SPRAYER WITH REAL-TIME, ON-MACHINE TARGET SENSOR

A plurality of different visual markers are deployed on a field. The markers include a target marker identifying a portion of the field that has a target of material to be applied to the field and a non-target marker identifying a portion of the field that does not have a target. An on-machine target identification system sense targets as an agricultural machine travels over the visual markers to identify targets in the field. An image processing adjustment controller correlates identified targets with the visual markers to determine an accuracy of the target identification system. An action signal is generated based upon the identified accuracy of the target identification system.

FIELD OF THE DESCRIPTION

The present description relates to the application of material to an agricultural field. More specifically, the present description relates to improving the accuracy of an agricultural machine that applies material to a field, using run-time, on-machine, target sensing.

BACKGROUND

Agricultural sprayers and other agricultural applicators apply chemicals and nutrients to agricultural fields. The chemicals and nutrients may be dry or liquid materials, and the materials can be applied for a number of reasons. For instance, the materials that are applied to a field may be pesticides, herbicides, fungicides, growth regulators, fertilizers, among others.

Some current agricultural sprayers and applicators apply product uniformly across the field, regardless of specific, localized needs. This is sometimes referred to as “broadcast” application. Some current systems also generate a prescription, prior to beginning the application process, that indicates where to apply material, which material to apply, and an application rate. The prescription is then loaded onto the agricultural sprayer and the selected product is applied to the locations in the field, based upon the prescription.

The prescription is often generated based on data that is aggregated using manual scouting, or imagery taken by machines, such as drones, aircraft or satellites. The prescriptions may also be generated based on past field history.

SUMMARY

A plurality of different visual markers are deployed on a field. The markers include a target marker identifying a portion of the field that has a target of material to be applied to the field and a non-target marker identifying a portion of the field that does not have a target. An on-machine target identification system senses targets as an agricultural machine travels over the visual markers to identify targets in the field. An image processing adjustment controller correlates identified targets with the visual markers to determine an accuracy of the target identification system. An action signal is generated based upon the identified accuracy of the target identification system.

DETAILED DESCRIPTION

As described above, some current agricultural sprayers and agricultural applicators apply product uniformly across a field, regardless of any specific localized needs. This approach, sometimes referred to as a “broadcast” approach, results in the application of chemical and other materials where it is not required. This increases production costs and may have a potentially negative environmental impact. In some cases where herbicide is applied, for instance, up to 80% of the total product is applied where it is not needed.

Also, as briefly discussed above, some current systems attempt to generate prescription indicating where to apply material to the field. However, the prescription is created ahead of time (prior to the application process by which the agricultural machine applies the material to the field). The prescription is then loaded into the agricultural sprayer, or agricultural applicator, and used in applying the material to the field.

Although this process may reduce the amount of material being applied, it has significant limitations. For instance, because the data used to generate such prescriptions is obtained through manual scouting or through imagery, or through past field history, the data is subject to georeferencing and application errors. Therefore, the locations of the particular targets of the material application are not precisely defined. This, in turn, means that larger application zones around the targets are used in order to ensure that the desired targets are indeed covered by the material being applied.

A problem with data collection from aerial images is that the image quality is often not adequate to identify targets, such as pests or weeds. The image quality issues are normally attributed to the height at which the images were taken or the distance from the targets at which the images were taken, lighting conditions, cloud cover, obscurants, and other atmospheric conditions. Similarly, because these types of images and other data collection processes are performed hours or even days or weeks ahead of the application process, the targets in the field may have changed or additional targets may have appeared, so that the sprayer will not be operating on an accurate prescription.

The present description thus proceeds with respect to a system that provides a real-time, on-board target identification and control system that uses optical sensors, mounted on a sprayer or other agricultural applicator (hereinafter referred to as the agricultural machine). The target identification and control system captures an image of an area ahead of the agricultural machine, in the direction of travel, and processes that image to identify targets in time for applicator functionality on the agricultural machine to apply a material to those targets.

Also, a challenge with such a system is that it is very difficult to tell whether the target identification and control system is operating accurately. For instance, the system may generate a false positive so that material is applied where it should not be applied or a false negative so that no material is applied where it should be applied. By way of example, if the agricultural machine is applying herbicide, the system may generate a false positive by identifying a rock or a piece of residue as a weed and applying herbicide to the rock or piece of residue. The system may generate a false negative by failing to identify a weed as a target so that no herbicide is applied to the weed. The present system thus includes an image processing adjustment controller. Markers are deployed in a portion of a field. The markers include target markers and non-target markers that are visually distinguishable from one another. The target markers are deployed in the field to identify targets and the non-target markers are deployed in the field to identify non-targets. The agricultural machine then travels over the portion of the field with deployed markers. The image processing adjustment controller determines how accurately the target identification system is working based on what the targeting system identified in the areas marked by the markers. The control system can then make adjustments to the agricultural machine and/or the targeting system to improve accuracy.

FIG.1Ashows a pictorial illustration of one example of an agricultural machine100. Agricultural machine100is depicted as an agricultural sprayer that has an operator compartment102, supported by a frame structure104, which also supports ground engaging elements106. In the example shown inFIG.1A, ground engaging elements106are wheels, but they could be tracks or other implementations.FIG.1Aalso shows that agricultural machine100has a spray system generally indicated by108. Spray system108illustratively includes a tank or other material reservoir110that carries material that is to be applied to an agricultural field112. In the example shown inFIG.1A, agricultural field112has row crops planted in rows114and a plurality of weeds116that are growing therein. WhileFIG.1Ashows one material reservoir110, it will be noted that agricultural machine100may have more than one material reservoir110each carrying a different material or different concentration of material. Also, whileFIG.1Ashows machine100in a field with rows114of crops, the present description can also proceed with an example in which machine100is treating an area without crops, such as a field after harvest and before planting, or another area without crops.

Spray system108also illustratively includes a boom structure118that supports a plurality of controllable nozzle bodies120. Nozzle bodies120can include an electronic controller that receives commands over a network, such as a controller area network—CAN, or other data communication protocols. The nozzle body120can also include one or more controllable valves that can be moved between an open position and a closed position. The nozzle body120can also include one or more nozzle spray control tips. Material to be applied by agricultural machine100is pumped by one or more pumps from tank110, through hoses or other conduits, to the nozzle bodies120. The controller in the nozzle bodies120controls the controllable valves to open (or move to the on position) so that the material moves through the nozzle body and out through the nozzle spray control tip where the material is applied to the field112. When the valve is controlled to be in the closed position (or the off position) the material does not pass through the valve. In one example, the valves are variable between the on and off positions, such as proportional values. In other examples, a variable flow rate can be achieved through the valves by controlling the pump or by controlling the valves in a pulse width modulated manner (varying the cycle time) or in other intermittent ways.

FIG.1Aalso shows that agricultural machine100is fitted with a plurality of different optical image sensors122(shown as cameras inFIG.1A). Image sensors122may be optical sensors which capture images by sensing radiation in the optical spectrum which, for purposes of the present discussion, includes ultraviolet, visible, and infrared frequencies. The image sensors122are disposed along the boom so that they have fields of view that cover the length of the ground in front of the boom118. For instance, the image sensors122are disposed across boom118so that their fields of view cover all of the area of field112forward of nozzle bodies120, as agricultural machine100travels through the field.

The image sensors122are illustratively coupled to one or more image processing modules124. The image processing modules124illustratively process the images captured by image sensors122to identify targets (e.g., weeds116or rows114) on field112over which agricultural machine100is traveling. Image sensors122can have an image processing system that performs some preprocessing. For instance, different cameras may be different so the on-camera image processing system may generate color correction matrices that adjust or calibrate the camera so all cameras produce images of the same color. The on-board image processing system can also perform other processing, such as lens shading correction, local tone mapping, demosaic, color correction, and distortion correction. The correction information can be captured in correction matrices or in other ways. Some or all of the pre-processing can be performed on the image processing modules124as well.

It will be noted that, in one example, the position of boom118(and in particular the position of each image sensor122) relative to the surface of field112, may change the field of view of the image sensors122. For example, at a first height above the field, an image sensor122may have a field of view with a first size so the area or region of interest being analyzed for targets takes up most of the field of view. However, when the image sensor122is moved to a greater height (further from the ground), then the width of the region on the ground that is included in the field of view of image sensor122may be larger, but the area being examined for targets remains the same.

Therefore, in one example, boom118has one or more boom sensors126that sense the height (in another implementation, sensor126can also or alternatively sense the angle and/or boom vibrations) of boom118relative to the surface of field112over which it is traveling. The boom height (and boom angle) can be used by image processing modules124to correct the images received from the various image sensors122, based upon their location relative to the ground from which the images are captured. Thus, in one example, the image processing modules124identify weeds116as targets of a herbicide being applied by agricultural machine100and transmits information about the location of the weeds116to a nozzle controller so that the nozzle controller can control the valves in the nozzle bodies120to apply the herbicide to the weeds116. In one example, the nozzle bodies are controlled to apply the material in a treated area128that has a buffer area on either side of weed116to increase the likelihood that the weed116is treated by the herbicide.

Image processing may be affected by ambient light conditions. Therefore,FIG.1Aalso shows that boom118may have one or more supplemental light sources131which can be activated in low light conditions.

Also, in order to process the images in various different types of light conditions (which may change based on whether agricultural machine100is heading into the sun, away from the sun, or otherwise),FIG.1Ashows that agricultural machine100can have a white balance camera or an incidental light sensor (light sensor130). Light sensor130can sense the direction of the sun relative to agricultural machine100, the color of the sun (such as whether the sky is overcast, whether machine100is traveling through a shadow, or other conditions that change the color of the light), and the light intensity among other things. Similarly, light sensors130may be disposed at one or more locations along boom118instead of, or in addition to, light sensor130on the body of the agricultural machine100, as shown inFIG.1A. The ambient lighting conditions are sensed by light sensor(s)130and the information representing the ambient lighting conditions is sent to image processing modules124. The data can be sent using data over power transmission, using a gigabit multimedia serial link (GMSL or GMSL2) or using another communication mechanism.

FIG.1Bshows a pictorial illustration of a rear view of agricultural machine100, and items that are similar to those shown inFIG.1Aare similarly numbered.FIG.1Bshows that boom118can have a central boom section134and one or more boom arms136and138on either side of central boom section134. In one example, central boom section134can be raised and lowered under force of a central boom actuator (not shown inFIG.1B). As shown inFIG.1B, boom arms136and138may rotate about pivot points144and146, respectively. Thus, the image sensors122may not simply be traveling in a vertical direction when boom arms136and138are raised and lowered, but they are moving in an arc about pivot points144and146. This can cause the orientation of the cameras to be focused more inwardly, toward a central axis of agricultural machine100, or outwardly, away from agricultural machine100. Thus, as the boom118moves, the perspectives of the cameras, and thus the fields of view of the image sensors122on the ground, will move as well. Similarly, as agricultural machine100travels through the field, it may encounter bumps, ruts, or other disturbances on the ground. This may cause the boom arms136and138to move upwardly or downwardly in the directions indicated by arrows140and142. Therefore, in one example, the cameras or image sensors122are calibrated at different heights from the ground (e.g., at different boom positions). A calibration transform is generated that can be used to transform the captured images so that the area of interest (or region of interest—ROI) within the image captured by each image sensor122remains at a fixed location on the ground relative to the corresponding image sensor122(e.g., one meter in front of the image sensor in the direction of travel), regardless of the boom position.

FIG.1Cis a block diagram showing some portions of agricultural machine100in more detail. Some of the items shown inFIG.1Care similar to those shown inFIGS.1A and1Band they are similarly numbered.FIG.1Cshows that agricultural machine100can also include one or more processors or servers150, data store151, a communication system152, one or more operator interface mechanisms154that an operator156can interact with in order to control and manipulate agricultural machine100, target identification system158, control system160, controllable subsystems162, and agricultural machine100can include a wide variety of other agricultural machine functionality164. Target identification system158can include optical sensors122, image processing modules124, light sensors130, image processing adjustment controller133, boom height/angle sensors126, double knock processing system165, and it can include other items166. Control system160can include calibration controller168, nozzle/valve controller170, pump controller172, boom position controller174, steering controller176, propulsion controller178, and multi-product controller179. Control system160can also include other items180. Controllable subsystems162can include boom position actuators182, one or more pumps184, nozzle bodies120(which, themselves, can include one or more nozzle tips188, valves190, valve controllers192, and other items194), steering subsystem196, propulsion subsystem198, and a wide variety of other items200.

Before describing the overall operation of agricultural machine100in identifying visual markers and determining the accuracy of target identification system158and control system160in applying material to targets, a description of some of the items shown inFIG.1C, and their operation, will first be provided. Operator interface mechanisms154can include any of a wide variety of mechanisms that can be used to provide information to operator156and receive interactive inputs from operator156. Operator interface mechanisms154can include audio, visual, and haptic mechanisms, among others. Examples of operator interface mechanisms154can include a steering wheel, joysticks, pedals, levers, buttons, microphones and speakers (such as when speech recognition/synthesis functionality is provided), among other things. User interface mechanisms154can include display screens, touch sensitive display screens, lights, audible alert mechanisms, etc. When the user interface mechanisms154include a display screen, operator input mechanisms can be provided on the display screen. Such operator input mechanisms can include buttons, links, icons, or other user actuatable elements that can be actuated using a point and click device, a touch gesture, a voice input, or other interactions.

Communication system152can include a bus controller that controls information on one or more bus structures (such as a CAN bus, a plurality of different CAN subnetworks, or another bus) on agricultural machine100. Communication system152can include wired networking components such as ethernet components that operate according to a known standard (e.g., IEEE 802.3), and other types of network and communication system components. Communication system152can also include other communication systems that allow agricultural machine100to communicate with remote devices or systems. Such communication systems can include a cellular communication system, a local area network communication system, a wide area network communication system, a near field communication system, or a wide variety of other communication systems.

Target identification system158illustratively identifies targets where material is to be applied by agricultural machine100and also identifies the visual markers when they are deployed in a field during calibration. For example, when agricultural machine100is to apply the material to crop plants, then target identification system158identifies crop plants (such as crop rows or other crop plants such as seeded crops). When agricultural machine100is to apply the material to a weed, for instance, then target identification system158identifies weeds so that the material can be applied to them. Therefore, each of the image sensors122captures images of a region of interest within the field of view corresponding to the image sensor122. The captured image can be compensated or corrected based on information detected by light sensor130. Image processing modules124then process the images captured by image sensors122to correct them and to identify targets (e.g., crop rows, weeds, etc.) and markers (visual target markers and non-target markers and possibly other markers, such as April tags, etc.) in the images. The images can then be transformed based on information captured by boom sensors126and mapping coefficients that match pixels in the image (e.g., the pixels corresponding to a target or marker) to actual locations on the ground. The image processing modules124identify which nozzles are to be actuated, and when they are to be actuated, to apply the material to the targets. That information can then be provided to control system160to control the nozzle bodies120.

It may also happen that agricultural machine100makes multiple passes through a field, when the passes are separated by some duration of time. For instance, some weeds may need multiple applications of one or more herbicides, with one to two weeks between applications, in order to kill them. After the first application, the weeds may appear to be dead, but unless they are treated again, they may again begin actively growing. Similarly, the weeds may be resistant to the chemical that is applied during the first pass, so that the weed still appears vibrant during the second pass. Therefore, it may be desirable to have agricultural machine100apply an additional dose of herbicide to the weeds, or to apply a dose of different herbicide, even though they were previously treated.

In such cases, target identification system158stores the location of the targets during the first pass through the field. Then, during the second pass through the field, even though the weeds may appear to be dead so that they are not identified as weed targets by target identification system158, double knock processing system165identifies that particular geographic location (where the weed was treated during the first pass) as a target for a second application of the herbicide. Similarly, double knock processing system165can identify that a vibrant weed still exists where it was treated during the first pass and multi-product controller179can generate an output to apply a different chemical or an increased dose of the original chemical to the weed on the second pass than was applied during the first pass. Double knock processing system165receives the stored map of weed locations that was generated during the first pass and a geographic position sensor senses a geographic position of agricultural machine100. The geographic position sensor may thus be a global navigation satellite system (GNSS) receiver, a dead reckoning system, a cellular triangulation system, or another position sensor. Based upon the current position of agricultural machine100, its speed, and the dimensions of the machine, double knock processing system165can identify which nozzles will be passing over weed locations where another application of herbicide is to be administered. Multi-product controller179can determine whether the same or a different material is to be administered.

Image processing adjustment controller133correlates the locations and types of the visual markers to the locations of the targets (e.g., weeds) to determine the accuracy of target identification system158in identifying targets. Controller133can also identify adjustments that can be made to the operation of machine100to improve image processing accuracy. Thus, target identification system158(whether targets are identified based on inputs from sensors122or double knock processing system165) generates an output indicating which nozzles are to be activated, when they are to be activated and a duration of time for which they are to be activated, based upon the image analysis performed by image processing modules124and the processing performed by double knock target identification system165. Target identification system158also outputs an indication of where visual markers have been identified and characteristics of the visual markers (e.g., whether the identified markers are target markers, non-target markers, visual fiducial markers, etc.). The outputs from target identification system158are provided to control system which generates control signals to control controllable subsystems162.

Calibration controller168can perform calibration operations to calibrate various items on agricultural machine100. Multi-product controller179determines which product is to be applied. Nozzle/valve controller170generates control signals to control nozzle bodies120. The control signals are received by controller192which controls the on and off state of valves190to apply the correct material at the correct location, according to the correct timing. Controller192can also control nozzle tips188(where they are configurable) to change the area of application of the nozzle.

Pump controller172may generate control signals to control pumps184that pump the material to be applied through the conduits on boom118to the nozzle bodies120. Boom position controller174may generate control signals to control boom position actuators182to move the various portions of boom118to different desired positions. Steering controller176may generate control signals to control steering subsystems196to control the heading of agricultural machine100. Propulsion controller178may generate control signals to control propulsion system198(which may be an engine that drives ground engaging mechanisms106through a transmission, individual motors that drive the individual ground engaging mechanisms106, or another power source that drives the propulsion of agricultural machine100) to control the speed and forward/reverse direction of travel of agricultural machine100.

FIG.2is a block diagram showing one example of image processing adjustment controller133in more detail. It will be noted that controller133can be located on image processing modules124or elsewhere. It is shown as a separate item for the sake of example only. In the example shown inFIG.2, controller133includes trigger detector210, marker test diagnostic system212, and other items214. Marker test diagnostic system212can include marker image identifier216, metadata processor218, target recognition/marker location correlation system220, result generation system222, adjustment processor224, output system226, and other items228. Marker image identifier216can include target marker component230, non-target marker component232, visual fiducial marker component234, and other items236. Metadata processor218can include marker locator238, and other metadata processing components240. Output system226can also include operator output generator242, control signal generator244, storage control system246, and other items248.

Trigger detector210detects a trigger indicating that controller133is to perform an image processing adjustment or calibration. In one example, the trigger may be a manually actuated trigger, or may be an automated trigger. For instance, if the operator has deployed visual markers, and image processing modules124detect one of those markers, this may serve as a trigger that is detected by trigger detector210indicating that controller133is to begin to perform the adjustment or calibration operation. In another example, operator156may provide an input initiating an adjustment or calibration process.

Marker test diagnostic system212then processes information to determine the accuracy of agricultural machine100in identifying targets and in applying material to those targets. Marker image identifier216receives an input from image processing modules124indicating that at least one of the image processing modules124has detected a visual marker. Target marker component230identifies whether the detected marker is a target marker that is identifying a target, and non-target marker component232determines whether the detected marker is a non-target marker that is identifying a non-target. In one example, the visual markers have visual indicia that distinguish between whether they are a target marker and a non-target marker. In one example, the markers are rings that may be 12 inches in diameter or that may have another size, and the rings have one color that corresponds to a target marker and a different color that corresponds to a non-target marker. By way of example, a blue ring may be deployed by the operator around a weed (or another target) and a red ring may be deployed by the operator around a rock, around residue, etc., that identifies a non-target. Thus, when image processing modules124identify a red ring, an indication of this is output to marker image identifier216and identified by non-target marker component232as a non-target marker. When image processing modules124identify a blue ring, an indication of that is output to target marker component230which determines that the identified marker is a target marker.

Image processing modules124may also identify other markers, such as visual fiducial markers (e.g., April tags or other visual markers) that may include other metadata, such as a marker identifier, a location identifier, or the identity of a particular non-target or target (the type of weed, the type of non-target—rock, residue, etc.). The information extracted from the visual fiducial marker by visual fiducial marker component234can be output to metadata processor218. Marker locator238may identify the particular geographic location of the marker based on the metadata and other metadata processing components240can process any other metadata to identify information (such as the type of target, the type of non-target, etc.).

Target recognition/marker location correlation system220receives an output from image processing modules124indicative of the location where image processing modules124identifies targets (e.g., weeds). System220then correlates the location of the weeds detected by image processing modules124in target identification system158with the location of the target markers and non-target markers to determine whether target identification system158accurately identified targets in areas marked by target markers, and whether target identification system158erroneously identified targets in areas marked by non-target markers. Result generation system222generates a result indicative of the correlation between the identified targets and the visual markers. For instance, assume that ten target markers are deployed on a portion of the field over which machine100is traveling, and ten non-target markers are deployed as well. Assume further that target identification system158correctly identified targets in locations correlated to eight of the ten target markers and identified a target in one location correlated to one of the ten non-target markers. Result generation system222can provide an output indicative of the accuracy in terms of ratios or percentages (e.g., target identification was accurate 85% of the time), in terms of raw data (e.g., target identification was accurate 17 of 20 times, or target identification produced two false negatives in ten tries and one false positive in ten tries, etc.), and the output can include other processing results. For instance, it may be that target identification system158is identifying certain types of non-targets (such as types of crops, types of residue, certain types of rocks) as targets more often than other types of non-target objects. This information can be output by result generation system222as well.

Adjustment processor224receives the results output by result generation system222and can identify different types of adjustments that can be made to the control of agricultural machine100in an attempt to improve accuracy. For instance, adjustment processor224may identify that changing the boom height may improve accuracy. Similarly, adjustment processor224may determine that slowing down the ground speed of agricultural machine100may improve accuracy. Adjustment processor224may identify a wide variety of other adjustments that may be made in order to improve the accuracy of image processing modules124and/or other items in target identification system158, based upon the results generated by result generation system222.

The various items in marker test diagnostic system212can provide their outputs to output system226which generates an output to other items in agricultural machine100, or to remote systems or elsewhere. Operator output generator242can generate an output to control operator interface mechanisms154in order to show the results generated by result generation system222to operator156. The results can be visual, audible, or other results. Control signal generator244can generate control signals as well. For instance, the operator output generator242may generate an output indicating the results of the marker test to operator156, along with a ranked set of possible adjustments that were identified by adjustment processor224, that may be made in order to improve accuracy. The ranked set of possible adjustments may be selectable so that operator156may select one or more of the set of possible adjustments for implementation. In response to such a selection, control signal generator244may generate a control signal to control the image processing modules124or the various controllable subsystems162to implement the adjustment. For instance, generator244may generate an output to automatically adjust the sensitivity of image processing modules124. In another example, where the operator selects the proposed adjustment to reduce the ground speed of agricultural machine100, then control signal generator244can generate a control signal to control propulsion subsystem198to reduce the speed of machine100. In another example, control signal generator244can generate the control signals to automatically implement one or more of the adjustments identified by adjustment processor224either with or without notifying operator156of the adjustments. In still another example, result generation system222can generate intermediate results (such as when agricultural machine100has traveled through half of the portion of the field that has visual markers deployed) indicating the accuracy of agricultural machine100over the first half of the test. Adjustment processor224may identify possible adjustments to improve accuracy, and control signal generator can generate control signals to automatically implement those adjustments and then determine whether the accuracy of the system improves over the second half of the marker test. By automatically it is meant, for example, that the operation or function is performed without further operator input except, perhaps, to initiate or authorize the operation or function. Output system226can then use operator output generator244to generate an output to operator156indicating the accuracy results from the first half of test, the adjustments that were made, and the accuracy results from the second half of the test after those adjustments were made. This may help inform operator156as to whether the adjustments should be made during subsequent spraying operations.

Storage control system246can interact with data store151or remotely located data stores or other systems or vehicles to store the results and adjustments, and other information corresponding to the marker test, the possible adjustments, etc.

FIG.3is a flow diagram illustrating one example of the operation of agricultural machine100and image processing adjustment controller133in performing a marker test to determine the accuracy of target identification system158. It is assumed that agricultural machine100is a sprayer operating in a field to be sprayed.

It is also assumed that a set of pre-defined markers are deployed to a portion of the field to be sprayed, as indicated by block259. The pre-defined markers may be target markers that mark things to be sprayed as indicated by block261and non-target markers that mark things that are not to be sprayed as indicated by block263. The markers can be visually (e.g., color or shape) coded with visual indicia so that the target markers can be visually distinguished from the non-target markers, as indicated by block265. The markers can be hoops, chalk, paint, etc., as indicated by block267. The markers can also include visual fiducial markers that may include other metadata (such as April tags or other markers) as indicated by block269, or other markers as indicated by block271.

At some point, trigger detector210detects a trigger indicating that the target identification system158is to be evaluated for accuracy, as indicated by block250in the flow diagram ofFIG.3. As discussed above, the trigger may be based on an operator input252. The trigger may be periodic or an otherwise time-based trigger as indicated by block254. The trigger may be an automatic trigger256or another trigger258.

Assume, for instance, that the operator has deployed target markers and non-target markers on a portion of the field.FIG.4illustrates such a field.FIG.4is similar toFIG.1A, and similar items are similarly numbered. However,FIG.4also shows that the operator, or another person or system has deployed a plurality of target markers in the form of colored hoops260,262,264,266,268,270,272,274,276,278, that are colored in a first color and a plurality of non-target markers in the form of colored hoops280,282,284, and286that are colored with a second color that is different form the first color. Target markers260-278are deployed around targets (e.g., weeds) where agricultural machine100is to spray. Non-target markers280-286are deployed around non-target items. For instance, non-target marker280is deployed around crop plants. Non-target markers282and286are deployed around residue, while non-target markers283and284are deployed around rocks.

FIG.4also shows that a plurality of visual fiducial markers (such as April tags)288and290have also been deployed proximate one of the target markers260and one of the non-target markers280, respectively. For purposes of the present discussion, it will be assumed that the target markers260-278are visually distinguished from the non-target markers280-286based on the color of the markers but could be distinguished based on shape or other visual indica. The target markers may, for instance, be blue, while the non-target markers may, for instance, be red, or the target markers may be round while the non-target markers may be square.

After the visual markers are deployed, agricultural machine100is navigated over the portion of the field where the markers have been deployed. As discussed above, the agricultural machine100may be an agricultural sprayer with on-machine, runtime target sensors, as indicated by block300in the flow diagram ofFIG.3. In one example, the marker test may be performed while agricultural machine100is spraying, as indicated by block302. In another example, agricultural machine100may be navigated over the markers without spraying, to assess the accuracy of target identification system158, without spraying any chemical, as indicated by block304. The agricultural machine100may be navigated over the portion of the field with the visual markers deployed in other ways as well, as indicated by block306.

While agricultural machine100is navigated over the visual markers, target identification system158performs target detection and identification, and also identifies the locations of the target and visual markers deployed in the field. Performing target detection and marker detection is indicated by block308in the flow diagram ofFIG.3.

The information generated by image processing modules124is provided to marker image identifier216. Target marker component230identifies target markers based on the visual indicia (e.g., color or shape, etc.) identified by modules124in the image while non-target marker component232identifies non-target markers based on the visual indicia in the image and visual fiducial marker component234identifies the visual fiducial markers288and290. Metadata processor218can use marker locator238to identify the location of the visual fiducial markers, and other metadata processing component240can identify other metadata from the visual fiducial markers.

Target recognition/marker location correlation system220then compares the results of the target identification performed by target identification system158relative to the visual markers deployed in the field, as indicated by block310in the flow diagram ofFIG.3. By way of example, system220determines whether target identification system158has accurately identified targets in the areas marked by the target markers260-278. System220also determines whether target identification system158has erroneously identified any targets in the areas marked by the non-target markers282-286. This correlation can be used to determine the accuracy of target identification system158. For instance, a metric, indicative of how often target identification system158properly identified targets in the areas marked by the target markers and improperly identified targets in the areas marked by the non-target markers can be used by result generation system222to generate an output indicative of the accuracy of target identification system158. Determining the accuracy of target detection is indicated by block312in the flow diagram ofFIG.3. In an example where agricultural machine100is actually spraying during the marker test, then the accuracy of the spraying operation can also be determined as corresponding to the accuracy of the target identification system158.

As discussed above, the marker test diagnostic system212can generate the results at the end of the marker test (after agricultural machine100has traveled over the portion of the field where the visual markers are deployed) as indicated by block314. In another example, the results of the marker test can be output in an intermediate fashion, such as after agricultural machine100has traveled over half of the area where the visual markers are deployed, or in another intermediate fashion, where the results are output during the target identification by target identification system158, as indicated by block316in the flow diagram ofFIG.3. The accuracy of the target detection can be determined in other ways as well, as indicated by block318.

In one example, result generation system222also determines whether the accuracy of the target identification system158is adequate (such as by comparing the accuracy to a threshold which may be manually set, automatically set, static, or dynamically changing,). Determining whether the accuracy of the target identification system158is sufficient (e.g., meets a threshold) is indicated by block320in the flow diagram ofFIG.3.

If the accuracy level does meet the threshold value, then output system226can output the results (e.g., notify operator156, store the results in data store151or elsewhere, etc.), as indicated by block322. However, if, at block320, result generation system222determines that the accuracy does not meet the accuracy threshold value, then result generation system222can generate an action signal to take further action, as indicated by block324. For instance, the action signal can be used to control operator interface mechanisms154to notify operator156that the accuracy of the target identification system158does not meet the threshold value, as indicated by block326. The action signal may also be provided to adjustment processor224which may be a neural network or other classifier, a rules-based processor, or other logic or functionality that identifies and outputs possible changes or adjustments to the operation of agricultural machine100in order to improve accuracy. The possible changes may be output to operator156for operator selection or approval, as indicated by block328. In another example, the action signal may be generated by control signal generator244to automatically make adjustments to the operation of agricultural machine100and then to reevaluate target identification accuracy with or without notice to the operator156that the adjustments are being made and that the accuracy is being reevaluated, as indicated by block330. In another example, control signal generator244can generate the action signal to control communication system152to communicate with other vehicles, remote systems, remote storage mechanisms, etc., as indicated by block332. The action signal can be used to perform other operations as well, as indicated by block334.

Until the spraying operation is complete, as indicated by336, processing reverts to block350where the operation continues until another targeting evaluation trigger is detected.

It can thus be seen that the present description describes a system in which the accuracy of a target identification system on an agricultural sprayer or other agricultural machine with runtime, on-machine target identification, is evaluated. Action signals are generated based upon the evaluation. The evaluation is performed by comparing whether targets are accurately identified within areas marked by visual target markers and in areas marked by visual non-target markers. The action signal can be used to notify the operator, to adjust machine operation, or to perform other operations.

FIG.5is a block diagram of machine100, shown inFIG.1, except that it communicates with elements in a remote server architecture930. In one example, remote server architecture930can provide computation, software, data access, and storage services that do not require end-user knowledge of the physical location or configuration of the system that delivers the services. In various examples, remote servers can deliver the services over a wide area network, such as the internet, using appropriate protocols. For instance, remote servers can deliver applications over a wide area network and they can be accessed through a web browser or any other computing component. Software or components shown in previous FIGS. as well as the corresponding data, can be stored on servers at a remote location. The computing resources in a remote server environment can be consolidated at a remote data center location or they can be dispersed. Remote server infrastructures can deliver services through shared data centers, even though they appear as a single point of access for the user. Thus, the components and functions described herein can be provided from a remote server at a remote location using a remote server architecture. Alternatively, they can be provided from a conventional server, or they can be installed on client devices directly, or in other ways.

In the example shown inFIG.5, some items are similar to those shown in previous FIGS. and they are similarly numbered.FIG.5specifically shows that target identification system158, control system160, and data store151can be located at a remote server location932. Therefore, machine100accesses those systems through remote server location932.

It is also contemplated that some elements of previous FIGS. can be disposed at remote server location932while others are not. By way of example, data store151can be disposed at a location separate from location932, and accessed through the remote server at location932. Regardless of where they are located, they can be accessed directly by machine100, through a network (either a wide area network or a local area network), they can be hosted at a remote site by a service, or they can be provided as a service, or accessed by a connection service that resides in a remote location. Also, the data can be stored in substantially any location and intermittently accessed by, or forwarded to, interested parties. For instance, physical carriers can be used instead of, or in addition to, electromagnetic wave carriers. In such an example, where cell coverage is poor or nonexistent, another mobile machine (such as a fuel truck) can have an automated information collection system. As the machine100comes close to the fuel truck for fueling, the system automatically collects the information from the machine100using any type of ad-hoc wireless connection. The collected information can then be forwarded to the main network as the fuel truck reaches a location where there is cellular coverage (or other wireless coverage). For instance, the fuel truck may enter a covered location when traveling to fuel other machines or when at a main fuel storage location. All of these architectures are contemplated herein. Further, the information can be stored on the machine100until the machine100enters a covered location. The machine100, itself, can then send the information to the main network.

It will also be noted that the elements ofFIG.1, or portions of them, can be disposed on a wide variety of different devices. Some of those devices include servers, desktop computers, laptop computers, tablet computers, or other mobile devices, such as palm top computers, cell phones, smart phones, multimedia players, personal digital assistants, etc.

FIG.6is one example of a computing environment in which elements of previous FIGS., or parts of them, (for example) can be deployed. With reference toFIG.6, an example system for implementing some embodiments includes a computing device in the form of a computer1010programmed to operate as described above. Components of computer1010may include, but are not limited to, a processing unit1020(which can comprise processors from previous FIGS.), a system memory1030, and a system bus1021that couples various system components including the system memory to the processing unit1020. The system bus1021may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. Memory and programs described with respect to previous FIGS. can be deployed in corresponding portions ofFIG.6.

The system memory1030includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM)1031and random access memory (RAM)832. A basic input/output system1033(BIOS), containing the basic routines that help to transfer information between elements within computer1010, such as during start-up, is typically stored in ROM1031. RAM1032typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit1020. By way of example, and not limitation,FIG.6illustrates operating system1034, application programs1035, other program modules1036, and program data1037.

The computer1010may also include other removable/non-removable volatile/nonvolatile computer storage media. By way of example only,FIG.6illustrates a hard disk drive1041that reads from or writes to non-removable, nonvolatile magnetic media, an optical disk drive1055, and nonvolatile optical disk1056. The hard disk drive1041is typically connected to the system bus1021through a non-removable memory interface such as interface1040, and optical disk drive1055are typically connected to the system bus1021by a removable memory interface, such as interface1050.

The drives and their associated computer storage media discussed above and illustrated inFIG.6, provide storage of computer readable instructions, data structures, program modules and other data for the computer1010. InFIG.6, for example, hard disk drive1041is illustrated as storing operating system1044, application programs1045, other program modules1046, and program data1047. Note that these components can either be the same as or different from operating system1044, application programs1035, other program modules1036, and program data1037.

A user may enter commands and information into the computer1010through input devices such as a keyboard1062, a microphone1063, and a pointing device1061, such as a mouse, trackball or touch pad. Other input devices (not shown) may include a joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit through a user input interface1060that is coupled to the system bus, but may be connected by other interface and bus structures. A visual display1091or other type of display device is also connected to the system bus1021via an interface, such as a video interface1090. In addition to the monitor, computers may also include other peripheral output devices such as speakers1097and printer1096, which may be connected through an output peripheral interface1095.

The computer1010is operated in a networked environment using logical connections (such as a controller area network—CAN, local area network—LAN, or wide area network WAN) to one or more remote computers, such as a remote computer1080.

When used in a LAN networking environment, the computer1010is connected to the LAN1071through a network interface or adapter1070. When used in a WAN networking environment, the computer1010typically includes a modem1072or other means for establishing communications over the WAN1073, such as the Internet. In a networked environment, program modules may be stored in a remote memory storage device.FIG.6illustrates, for example, that remote application programs1085can reside on remote computer1080.