Patent Description:
Agricultural harvesters, such as combines or windrowers, travel through fields of agricultural crop harvesting the crop. In one common arrangement, agricultural harvesting heads extend forward from the agricultural harvester to engage the plant stalks, sever them, and carry the severed crop into the body of the agricultural harvester, itself, for processing.

In agricultural harvesters, the throughput (rate of crop moving through the machine) is dependent on the forward ground speed of the harvester and the density of the crop being harvested. Some machine settings can be set, assuming a throughput, and machine speed is then varied, as the operator observes differences in crop density, to maintain the desired throughput.

Some current systems automatically adjust the forward ground speed of the harvester in an attempt to maintain a desired crop throughput. For instance, some systems have attempted to use a priori data (such as aerial imagery of a field) in order to generate a predictive yield map, which predicts yields at different geographic locations in the field being harvested. This can be done by attempting to identify the crop density based on an image classifier classifying images of the field in a pass of the harvester.

As discussed, some current harvester systems attempt to use a priori data (such as aerial images) in order to generate a predictive map that can be used to control the harvester. By way of example, there has been a great deal of work done in attempting to generate a predictive yield map for a field, based upon vegetation index values generated from aerial imagery. Such predictive yield maps attempt to predict a yield at different locations within the field. The systems attempt to control a combine harvester (or other harvester) based upon the predicted yield.

Also, some systems attempt to use forward-looking perception systems, which can involve obtaining optical images of the field, forward of a harvester in the direction of travel. A yield can be predicted for the area just forward of the harvester, based upon those images. This is another source of a priori data that can be used to generate a form of a predictive yield map.

All of these types of systems can present difficulties.

For instance, in systems that utilize image processing to predict yield of the field in a pass of the harvester, it can be difficult to distinguish between standing crop and areas having crop stubble (i.e., areas that have already been harvested). Some systems attempt to address this issue using an image classifier trained to distinguish between such area. However, such image classifiers often require extensive training with large amounts of training data, which requires large amounts of processing bandwidth and time. Even so, such classifiers can be inaccurate.

<CIT> shows a harvesting machine with an electronic control unit controlling operating parameters of the harvesting machine, like threshing parameters and speed, based on a functional system model of the machine using a pre-stored multidimensional relationship between sensed input values for example for crop throughput and output values representing expected operating results. The system model is adapted based on information on the environment in which the machine operates, for example on whether the crop is downed or the crop density.

Control of an agricultural harvesting machine uses a geometric image classifier and harvest map. The system utilizes a geometric classifier to distinguish between field areas having standing crop and field areas having non-standing crop (downed crop or crop stubble). This information is fused with harvest map(s) to determine if the areas having non-standing crop are areas that have already been harvested (and thus are areas of crop stubble) or are areas of downed crop. The geometric classifier is configured to measure the height of the crop relative to the ground, which can be used to predict the yield (or mass flow) in the path of the harvester, to control various subsystems on the harvester (e.g., to achieve a desired throughput).

<FIG> is a partial pictorial, partial schematic, illustration of an agricultural harvesting machine <NUM> (also referred to as a "harvester" or "combine"). It can be seen in <FIG> that machine <NUM> illustratively includes an operator compartment <NUM>, which can have a variety of different operator interface mechanisms, for controlling machine <NUM>, as will be discussed in more detail below. In one example, machine <NUM> is fully autonomous and may not have an operator compartment. Machine <NUM> can include a set of front-end equipment that can include header <NUM>, and a cutter generally indicated at <NUM>. It can also include a feeder house <NUM>, a feed accelerator <NUM>, and a thresher generally indicated at <NUM>. Thresher <NUM> illustratively includes a threshing rotor <NUM> and a set of concaves <NUM>. Further, machine <NUM> can include a separator <NUM> that includes a separator rotor. Machine <NUM> can include a cleaning subsystem (or cleaning shoe) <NUM> that, itself, can include a cleaning fan <NUM>, chaffer <NUM> and sieve <NUM>. The material handling subsystem in machine <NUM> can include (in addition to a feeder house <NUM> and feed accelerator <NUM>) discharge beater <NUM>, tailings elevator <NUM>, clean grain elevator <NUM> (that moves clean grain into clean grain tank <NUM>) as well as unloading auger <NUM> and spout <NUM>. Machine <NUM> can further include a residue subsystem <NUM> that can include chopper <NUM> and spreader <NUM>. Machine <NUM> can also have a propulsion subsystem that includes an engine that drives ground engaging wheels <NUM> or tracks, etc. It will be noted that machine <NUM> may also have more than one of any of the subsystems mentioned above (such as left and right cleaning shoes, separators, etc.).

In operation, and by way of overview, machine <NUM> illustratively moves through a field in the direction indicated by arrow <NUM>. A forward-looking sensor <NUM> is mounted on the front of machine <NUM> and senses characteristics of crop in front of the machine <NUM>. In one example, sensor <NUM> is an image capture sensor that captures images (e.g., video feed, series of still images, etc.) of an area forward of header <NUM>. The video feed or image(s) can be used to show (e.g., on a display device in operator compartment <NUM>) a view forward of operator compartment <NUM>, such as showing header <NUM> and/or the crop in front of header <NUM>. The image(s) can be used to identify a volume of crop to be engaged by header <NUM>. This can be used to automatically increase or decrease the ground speed of machine <NUM> to maintain a desired crop throughput. This is described in greater detail below.

As machine <NUM> moves, header <NUM> engages the crop to be harvested and gathers it toward cutter <NUM>. After it is cut, it is moved through a conveyor in feeder house <NUM> toward feed accelerator <NUM>, which accelerates the crop into thresher <NUM>. The crop is threshed by rotor <NUM> rotating the crop against concave <NUM>. The threshed crop is moved by a separator rotor in separator <NUM> where some of the residue is moved by discharge beater <NUM> toward the residue subsystem <NUM>. It can be chopped by residue chopper <NUM> and spread on the field by spreader <NUM>. In other implementations, the residue is simply dropped in a windrow, instead of being chopped and spread.

Grain falls to cleaning shoe (or cleaning subsystem) <NUM>. Chaffer <NUM> separates some of the larger material from the grain, and sieve <NUM> separates some of the finer material from the clean grain. Clean grain falls to an auger in clean grain elevator <NUM>, which moves the clean grain upward and deposits it in clean grain tank <NUM>. Residue can be removed from the cleaning shoe <NUM> by airflow generated by cleaning fan <NUM>. That residue can also be moved rearwardly in machine <NUM> toward the residue handling subsystem <NUM>.

Tailings can be moved by tailings elevator <NUM> back to thresher <NUM> where they can be re-threshed. Alternatively, the tailings can also be passed to a separate re-threshing mechanism (also using a tailings elevator or another transport mechanism) where they can be re-threshed as well.

<FIG> also shows that, in one example, machine <NUM> can include ground speed sensor <NUM>, one or more separator loss sensors <NUM>, a clean grain camera <NUM>, one or more cleaning shoe loss sensors <NUM>, forward looking camera <NUM>, rearward looking camera <NUM>, a tailings elevator camera <NUM>, and a wide variety of other cameras or image/video capture devices. Ground speed sensor <NUM> illustratively senses the travel speed of machine <NUM> over the ground. This can be done by sensing the speed of rotation of the wheels, the drive shaft, the axel, or other components. The travel speed can also be sensed by a positioning system, such as a global positioning system (GPS), a dead reckoning system, a LORAN system, or a wide variety of other systems or sensors that provide an indication of travel speed. In one example, optical sensor(s) capture images and optical flow is utilized to determine relative movement between two (or more) images taken at a given time spacing.

Cleaning shoe loss sensors <NUM> illustratively provide an output signal indicative of the quantity of grain loss. In one example, this includes signal(s) indicative of the quality of grain loss by both the right and left sides of the cleaning shoe <NUM>. In one example, sensors <NUM> are strike sensors which count grain strikes per unit of time (or per unit of distance traveled) to provide an indication of the cleaning shoe grain loss. The strike sensors for the right and left sides of the cleaning shoe can provide individual signals, or a combined or aggregated signal. In one example, sound-based sensors across an area of the cleaning shoe and/or rotor can be utilized to obtain a count of grain strikes and a spatial distribution associated with the count. It will be noted that sensors <NUM> can comprise only a single sensor as well, instead of separate sensors for each shoe.

Separator loss sensor <NUM> provides a signal indicative of grain loss in the left and right separators. The sensors associated with the left and right separators can provide separate grain loss signals or a combined or aggregate signal. This can be done using a wide variety of different types of sensors as well. It will be noted that separator loss sensors <NUM> may also comprise only a single sensor, instead of separate left and right sensors.

Cameras <NUM>, <NUM> and <NUM> illustratively capture video or still images that can be transmitted to, and displayed on, a display in operator compartment <NUM> or a remote device (shown in more detail below) in near real time. Clean grain camera <NUM>, for instance, generates a video feed showing grain passing into clean grain tank <NUM> (or through clean grain elevator <NUM>). Cameras <NUM> and <NUM> illustratively generate a video feed showing the tailings in the elevator and discharge beater and an area of the field behind machine <NUM>, respectively. Alternatively, or in addition to a video feed, captured images can be augmented and presented to the operator, for example in a manner aimed to reduce cognitive load on the operation. These are examples only, and additional or different cameras can be used and/or they can be devices that capture still images or other visual data.

It will also be appreciated that sensor and measurement mechanisms (in addition to the sensors already described) can include other sensors on machine <NUM> as well. For instance, they can include a residue setting sensor that is configured to sense whether machine <NUM> is configured to chop the residue, drop a windrow, etc. They can include cleaning shoe fan speed sensors that can be configured proximate fan <NUM> to sense the speed of the fan. They can include a threshing clearance sensor that senses clearance between the rotor <NUM> and concaves <NUM>. They include a threshing rotor speed sensor that senses a rotor speed of rotor <NUM>. They can include a chaffer clearance sensor that senses the size of openings in chaffer <NUM>. They can include a sieve clearance sensor that senses the size of openings in sieve <NUM>. They can include a material other than grain (MOG) moisture sensor that can be configured to sense the moisture level of the material other than grain that is passing through machine <NUM>. They can include machine setting sensors that are configured to sense the various configurable settings on machine <NUM>. They can also include a machine orientation sensor that can be any of a wide variety of different types of sensors that sense the orientation or pose of machine <NUM>. Crop property sensors can sense a variety of different types of crop properties, such as crop type, crop moisture, and other crop properties. They can also be configured to sense characteristics of the crop as they are being processed by machine <NUM>. For instance, they can sense grain feed rate, as it travels through clean grain elevator <NUM>. They can sense yield as mass flow rate of grain through elevator <NUM>, correlated to a position from which it was harvested, as indicated by position sensor <NUM>, or provide other output signals indicative of other sensed variables. Some additional examples of the types of sensors that can be used are described below.

In one example, various machine settings can be set and/or controlled to achieve a desired performance. The settings can include such things as concave clearance, rotor speed, sieve and chaffer settings, cleaning fan speed, among others. These settings can illustratively be set or controlled based on expected crop throughput (e.g., the amount of crop processed by machine <NUM> per unit of time). Thus, if the mass of the crop varies spatially in the field, and the ground speed of machine <NUM> remains constant, then the throughput will change with crop mass. As discussed below, forward-looking sensor <NUM> can be utilized to estimate the height of the crop in a given area and to further estimate a volume of the crop that is about to be processed. This volume can be converted into a biomass metric indicative of the biomass of the crop that is about to be engaged. The machine speed can then be controlled based on the estimated biomass to maintain the desired throughput.

<FIG> is a block diagram showing one example of an agricultural architecture <NUM> that includes the agricultural harvesting machine <NUM>, shown in <FIG>, that harvests a crop <NUM>. Some items shown in <FIG> are similar to those shown in <FIG>, and they are similarly numbered.

Machine <NUM> includes a processing and control system <NUM> (also referred to as control system <NUM>), configured to control other components and systems of architecture <NUM>, one or more processors or servers <NUM>, a data store <NUM>, and it can include other items <NUM> as well. Data store <NUM> is configured to store data for use by machine <NUM>, such as field data. Examples include, but are not limited to, field location data that identifies a location of the field to be operated upon by a machine <NUM>, field shape and topography data that defines a shape and topography of the field, crop location data that is indicative of a location of crops in the field (e.g., the location of crop rows), or any other data.

One example of control system <NUM> is illustrated in <FIG>, which will be described in conjunction with <FIG>. Control system <NUM> includes a communication controller <NUM> configured to control a communication system <NUM> to communicate between components of machine <NUM> and/or with other machines or systems, such as remote computing system(s) <NUM> and/or machine(s) <NUM>, either directly or over a network <NUM>. Also, machine <NUM> can communicate with other agricultural machine(s) <NUM> as well. Agricultural machine(s) <NUM> can be a similar machine type as machine <NUM>, and they can be different types of machines as well. Network <NUM> can be any of a wide variety of different types of networks such as the Internet, a cellular network, a local area network, a near field communication network, or any of a wide variety of other networks or combinations of networks or communication systems.

A remote user <NUM> is illustrated interacting with remote computing system <NUM>. Remote computing system <NUM> can be a wide variety of different types of systems. For example, remote computing system <NUM> can be a remote server environment, remote computing system that is used by remote user <NUM>. Further, it can be a remote computing system, such as a mobile device, remote network, or a wide variety of other remote systems. Remote computing system <NUM> can include one or more processors or servers, a data store, and it can include other items as well.

Communication system <NUM> can include wired and/or wireless communication logic, which can be substantially any communication system that can be used by the systems and components of machine <NUM> to communicate information to other items, such as between control system <NUM>, sensors <NUM>, controllable subsystems <NUM>, image capture system <NUM>, and image analysis system <NUM>. In one example, communication system <NUM> communicates over a controller area network (CAN) bus (or another network, such as an Ethernet network, etc.) to communicate information between those items. This information can include the various sensor signals and output signals generated by the sensor variables and/or sensed variables.

Control system <NUM> includes various control logic configured to control subsystem(s) <NUM> or other systems and components in architecture <NUM>. For example, control system <NUM> includes user interface component <NUM> configured to control interfaces, such as operator interface(s) <NUM> that include input mechanisms configured to receive input from an operator <NUM> and output mechanisms that render outputs to operator <NUM>. The user input mechanisms can include mechanisms such as a steering wheel, pedals, levers, joysticks, hardware buttons, dials, linkages, switches, keyboards, etc., as well as virtual mechanisms or actuators such as a virtual keyboard or actuators displayed on a touch sensitive screen. The output mechanisms can include speakers and/or display devices (e.g., screens) that display user actuatable elements, such as icons, links, buttons, etc. Where the display is a touch sensitive display, those user actuatable items can be actuated by touch gestures. Similarly, where mechanisms include speech processing mechanisms, then operator <NUM> can provide inputs and receive outputs through a microphone and speaker, respectively. Operator interface(s) <NUM> can include any of a wide variety of other audio, visual or haptic mechanisms.

Control system also includes feed rate control logic <NUM>, settings control logic <NUM>, route control logic <NUM>, ground speed control logic <NUM>, and it can include other items <NUM>. Some examples of controllable subsystems <NUM> are discussed above, and can include propulsion subsystem <NUM>, steering subsystem <NUM>, user interface mechanism(s) <NUM>, threshing subsystem <NUM>, separator subsystem <NUM>, cleaning subsystem <NUM>, residue subsystem <NUM>, material handling subsystem(s), header subsystem(s), and it can include a wide variety of other systems <NUM>, some of which were described above with respect to <FIG>.

Feed rate control logic <NUM> illustratively controls propulsion system <NUM> and/or any other controllable subsystems <NUM> to maintain a relatively constant feed rate, based upon the yield for the geographic location that machine <NUM> is about to encounter, or other characteristic(s) predicted by a predictive model. By way of example, if the predictive model indicates that the predicted yield in front of machine <NUM> (in the direction of travel) is going to be reduced, then feed rate control logic <NUM> can control propulsion system <NUM> to increase the forward speed of machine <NUM> in order to maintain the feed rate relatively constant. On the other hand, if the predictive model indicates that the yield ahead of machine <NUM> is going to be relatively high, then feed rate control logic <NUM> can control propulsion system <NUM> to slow down in order to, again, maintain the feed rate at a relatively constant level.

Settings control logic <NUM> can generate control signals to adjust machine settings or configuration. For instance, logic <NUM> can control actuators in order to change machine settings based upon the predicted characteristic of the field being harvested (e.g., based upon the predicted yield, or other predicted characteristic). By way of example, settings control logic <NUM> may actuate actuators that change, e.g., based upon the predicted yield or biomass to be encountered by machine <NUM>, the concave clearance machine <NUM>, thresher drum/rotor speed, conveyer speed, auger speed, concave clearance, sieve and chaffer settings, cleaning fan speed, etc..

Route control logic <NUM> can generate control signals and apply them to steering subsystem <NUM> to control steering of machine <NUM>.

Sensor(s) <NUM> can include any of a wide variety of different types of sensors. In the illustrated example, sensors <NUM> include forward-looking sensor <NUM>, speed sensor(s) <NUM>, position sensor(s) <NUM>, environmental sensor(s) <NUM>, yield sensor(s) <NUM>, and can include other types of sensors <NUM> as well.

Forward looking sensor <NUM> can be a variety of different sensors, including but not limited to, a camera, a stereo camera, a laser-based sensor, a lidar sensor, a radar sensor, a sonar sensor, an ultrasound-based sensor, a light emitting diode (LED) based lidar sensor, or any other sensor capable of measuring crop height, etc. In one example, forward looking sensor <NUM> is a laser system or stereo camera system and determines an average crop height across an area of interest. The area of interest is illustratively known and positioned a known distance in front of machine <NUM>. For instance, the area of interest can be centered a known distance in front of the machine <NUM>, as wide as the harvester head and one-quarter meter to one meter deep.

Position sensor(s) <NUM> are configured to determine a geographic position of machine <NUM> on the field, and can include, but are not limited to, a Global Navigation Satellite System (GNSS) receiver that receives signals from a GNSS satellite transmitter. It can also include a Real-Time Kinematic (RTK) component that is configured to enhance the precision of position data derived from the GNSS signal. Speed sensor(s) <NUM> are configured to determine a speed at which machine <NUM> is traveling the field during the spraying operation. This can include sensors that sense the movement of ground-engaging elements (e.g., wheels or tracks) and/or can utilize signals received from other sources, such as position sensor(s) <NUM>.

As shown in <FIG>, an image capture system <NUM> includes image capture components configured to capture one or more images of the area under consideration (i.e., the portions of the field to be operated upon by machine <NUM>) and image processing components configured to process those images. The captured images represent a spectral response captured by image capture system <NUM> that are provided to an image analysis system <NUM> and/or stored in data store <NUM>. A spectral imaging system illustratively includes a camera that takes spectral images of the field under analysis. For instance, the camera can be a multispectral camera or a hyperspectral camera, or a wide variety of other devices for capturing spectral images. The camera can detect visible light, infrared radiation, or otherwise.

In one example, the image capture components include a stereo camera configured to capture a still image, a time series of images, and/or a video of the field. An example stereo camera captures high definition video at thirty frames per second (FPS) with one-hundred-and-ten-degree wide-angle field of view. Of course, this is for sake of example only.

Illustratively, a stereo camera includes two or more lenses with a separate image sensor for each lens. Stereo images (e.g., stereoscopic photos) captured by a stereo camera allow for computer stereo vision that extracts three-dimensional information from the digital images. In another example, a single lens camera can be utilized to acquire images (referred to as a "mono" image).

Image capture system <NUM> can include one or more of an aerial image capture system <NUM>, an on-board image capture system <NUM>, and/or other image capture system <NUM>. An example of aerial image capture system <NUM> includes a camera or other imaging component carried on an unmanned aerial vehicle (UAV) or drone (e.g., block <NUM>). An example of on-board image capture system <NUM> includes a camera or other imaging component (e.g., sensor <NUM>) mounted on, or otherwise carried by, machine <NUM> (or <NUM>). An example of image capture system <NUM> includes a satellite imaging system. System <NUM> also includes a location system <NUM> and can include other items <NUM> as well. Location system <NUM> is configured to generate a signal indicative of geographic location associated with the captured image. For example, location system <NUM> can output GPS coordinates that are associated with the captured image to obtain geo-referenced images <NUM> that are provided to image analysis system <NUM>.

Image analysis system <NUM> illustratively includes one or more processors <NUM>, a communication system <NUM>, a data store <NUM>, target field data identification logic <NUM>, trigger detection logic <NUM>, crop state determination logic <NUM>, harvest data (e.g., map) processing logic <NUM> and can include other items <NUM> as well.

Communication system <NUM>, in one example, is substantially similar to communication system <NUM>, discussed above. Target field data identification logic <NUM> is configured to identify a target or subject field under analysis for which images <NUM> are being analyzed. Also, target field data identification logic <NUM> is configured to obtain or otherwise identify field data for the target field, such as, but not limited to, terrain data that identifies field topology, harvest data to indicates previous harvesting operations during the current growing season (i.e., what regions of the field have already been harvested), etc..

Trigger detection logic <NUM> is configured to detect a triggering criterion that triggers the image analysis. For example, in response to detection of a triggering criteria, logic <NUM> can communication instructions to image capture system <NUM> to capture images of the target field. These images are then processed by image analysis system <NUM>, and the results of the image analysis are utilized by crop state determination logic <NUM> to determine a state (e.g., standing crop, downed crop, crop stubble) of the crop represented in the capture images. For instance, as discussed below, logic <NUM> can perform geometric classification (e.g., height-based classification) of crop plants in the image. As discussed in further detail below, logic <NUM> is configured to determine that a height of crop plants represented in the captured images is below a threshold, and are thus classified as non-standing crop (i.e., downed or stubble). As used herein, the term "crop plant(s)" refers to crop in both a growing and non-growing state, such as after harvest (referred to as stubble).

Harvest data processing logic <NUM> is configured to obtain harvest data indicative of area(s) of the field that have already been harvested. In one example, this harvest data includes harvest yield data from particular areas of the field based on prior harvesting passes of machine <NUM> and/or other harvesting machines. Using this data, image/crop analysis results <NUM> are generated and output by system <NUM>.

Results <NUM> illustratively identify areas of the field that include standing crop, downed crop, and crop stubble (i.e., already harvested crop). In one example, results <NUM> are utilized to generate representation of an amount of plant material in the path of machine <NUM>. This can include any desired characteristic such as, but not limited to, mass, volume, weight, etc. of the crops to be harvested by machine <NUM>. In one example, the representation identifies one or more of a predicted biomass of the crop to be engaged by the header and threshed in machine <NUM>, a predicted crop grain yield, material other than grain (MOG), etc..

In one example, results <NUM> are provided to biomass system <NUM> that is configured to estimate a value of the crop that is about to be processed and to convert that volume into a biomass metric indicative of a biomass of the crop that is about to be engaged.

Crop biomass can be indicative of the predicted amount of biomass in the given section of the field. For example, biomass may be indicated as a mass unit (kg) over an area unit(m<NUM>). Crop grain yield can be indicative of the predicted amount of yield in the given section of the field. For example, crop grain yield may be indicated as a yield unit (bushel) over an area unit (acre).

In any case, this information can be utilized to control machine <NUM>. Examples of this are discussed in further detail below. Briefly, in one example, the machine speed can be controlled based on the estimated biomass and/or crop yield to maintain a desired throughput.

It is noted that, as illustrated by the dashed boxes in <FIG>, control system <NUM> can include some or all of image capture system <NUM>, image analysis system <NUM>, and/or biomass system <NUM>.

Biomass system <NUM> includes sensed area generator logic <NUM> that determines the area sensed by forward-looking sensor <NUM>. Volume generator logic <NUM> uses the area sensed and determines a crop volume or characteristic forward of machine <NUM>. Volume to biomass conversion logic <NUM> receives a crop volume or characteristic forward of machine <NUM> and estimates a biomass of the crop in the sensed volume. In one example, this is based on a conversion factor that is based on a sensed variable indicative of actual biomass (e.g., from sensors in machine <NUM>, such as rotor pressure sensor(s)). Datastore interaction logic <NUM> stores and retrieves information from data store <NUM>. Biomass system <NUM> also includes recommendation logic <NUM>, one or more processor(s) <NUM>, and can include other items <NUM>. Recommendation logic <NUM> is configured to generate a recommendation to maintain a desired throughput based on the estimated biomass. The recommendation can be performed either automatically by control system <NUM> and/or manually by operator <NUM>. Some recommendations that can be generated include changing the ground speed of machine <NUM>, and/or changing machine settings, such as concave settings, sieve and chaffer settings, cleaning fan speed, threshing rotor speed, conveyer/feed speed or cutter speed. Of course, other settings may be changed as well.

<FIG> is a block diagram illustrating one example of crop state determination logic <NUM>. As illustrated, logic <NUM> includes a geometric classifier <NUM> and is configured to receive images <NUM> and field data <NUM>. As discussed above, images <NUM> can be obtained by image capture system <NUM>, and can include aerial image capture system <NUM>, on-board image capture system <NUM>, etc. For example, the images can be obtained by sensor <NUM> of an area of the field in front of machine <NUM> in a direction of forward travel as machine <NUM> is harvesting the field.

Field data <NUM> represents characteristics or conditions of the field such as, but not limited to, harvest data, terrain data, etc. This can include a priori and/or in situ data, and can be obtained from a wide variety of different sources. For instance, it can be obtained from UAVs or drones, satellite systems, sensors <NUM> on machine <NUM>, from other agricultural machines <NUM>, or otherwise. The harvest data, in one example, includes a harvest map that identifies regions of the field that have already been harvested (e.g., by machine <NUM> and/or other machines <NUM>). The terrain data can comprise a terrain map that identifies a topology of the field (e.g., surface height, slope, etc.).

Using images <NUM> and field data <NUM>, crop state determination logic <NUM> outputs image/crop analysis results (e.g., results <NUM>) that represents the state of the crop in the path of machine <NUM>. For instance, areas of the field can be identified as having standing crop. In this case, logic <NUM> outputs a standing crop classification (block <NUM>), along with a height metric that represents a height of the standing crop plants above the field surface. Logic <NUM> can also output a downed crop classification (block <NUM>) indicating that the crop plants in a particular area are downed, that is they are unharvested, but not standing (i.e., have a have a height below a threshold). Also, logic <NUM> can output a crop stubble classification (block <NUM>) indicating that a particular area has already been harvested. An example operation of logic <NUM> is discussed below.

<FIG> is a block diagram <NUM> for determining crop state in a field and generating corresponding control signals to control the agricultural machine. For sake of illustration, but not by limitation, <FIG> will be described in the context of architecture <NUM> shown in <FIG>.

At block <NUM>, logic <NUM> detects a trigger to activate crop state determination. In one example, the trigger includes detecting that a harvesting operation is to be performed by machine <NUM>. This is represented by block <NUM>. Detection of the trigger can also include detecting manual inputs (block <NUM>), or can occur automatically (block <NUM>). Of course, the trigger can be detected in other ways as well. This is represented by block <NUM>.

At block <NUM>, a target field to be harvested is identified. In one example, this can include an input from operator <NUM> that identifies the target field to control system <NUM>. In another example, the target field can be identified automatically based on geographical position information that identifies a location of machine <NUM> (e.g., using signals from sensor(s) <NUM>).

At block <NUM>, field data for the target field is identified. As noted above, in one example field data <NUM> is obtained by crop state determination logic <NUM>, and can include a priori data (block <NUM>), in situ data (block <NUM>) or other data (block <NUM>). In any case, this data can be obtained by machine <NUM> (block <NUM>), or other machines or systems such as remote computing system <NUM>, machine(s) <NUM>, machine(s) <NUM>, etc. This is represented by block <NUM>.

The field data can include terrain data <NUM>, such as a topology map of the target field. The field data can also obtain harvest data <NUM> indicating a harvesting operation from the current growing season. For instance, a harvest map indicating yield data from the target field can be obtained.

Based on the harvest data, regions of the field having already harvested crop is identified. This is represented by block <NUM>. For example, a stubble signal at block <NUM> is generated that identifies regions of the field that are already harvested and are to be classified as crop stubble areas.

At block <NUM>, characteristic(s) of crop plants in a particular area of the field (in a path of machine <NUM>) are detected. Illustratively, a characteristic represents a height of the crop plants above the field surface. The characteristic can be detected based on data obtained by harvesting machine <NUM> (block <NUM>) and/or data obtained by other machines (such as those discussed above with respect to block <NUM>). This is represented by block <NUM>.

In one example of block <NUM>, images of the field are obtained at block <NUM> and geometric classification is performed at block <NUM> to obtain crop height data. In one example, a 3D point cloud is generated from stereo image data at block <NUM>.

At block <NUM>, a height metric representing height of the crop plants in the particular area is generated. In one example, the height metric generated at block <NUM> represents an average height of the crop plants above the field surface. This is indicated by block <NUM>.

It is noted that the particular area for which the crop plant characteristic is detected at block <NUM>, and the height metric generated at block <NUM>, can be identified in any of a number of ways. For instance, the area can be a pre-defined area in front of machine <NUM>. To illustrate, the predefined area can be defined as the width of the header, between zero to twenty feet in front of the header.

Also, the area can be selected arbitrarily and/or based on identified boundaries, identified from the image data. For instance, based on the image data, portions of the field having differing crop heights (beyond a height difference threshold) can be identified. In this way, the selected areas roughly follow the boundaries between standing crop, downed crop, and/or crop stubble.

At block <NUM>, a crop state in the particular area is determined based on the height metric generated at block <NUM> and the identified regions of already harvested crop, identified at block <NUM>. One example of block <NUM> is illustrated in <FIG>.

In the example of <FIG>, at block <NUM>, the height metric for the particular area is generated. The height metric represents the height of the crop plants in the particular area. For instance, the height metric can represent the average height of the crop plants in the particular area. This can be generated in any of a number of ways. In one example, the ground plane (represented the surface of the field) is estimated based on sensor data from sensor(s) <NUM> and/or based on remotely received data, such as a terrain map from system <NUM>.

At block <NUM>, a height threshold is selected or otherwise obtained. The height threshold can be user-selected (represented at block <NUM>). In another example, the height threshold is selected automatically. This is represented by block <NUM>. The height threshold can be based on any of a number of factors. For instance, the height threshold can be based on the crop type and/or conditions of the crop being harvested in the field. For instance, conditions such as maturity, moisture content, etc. can be utilized to determine an expected height of standing crop plants in the field. This is represented by block <NUM>. Also, environmental conditions can be used to select the height threshold. This is represented by block <NUM>. For instance, the environmental conditions can indicate current or prior wind speeds, precipitation, to name a few. Of course, the height threshold can be selected in any of a number of other ways and based on any of a number of other considerations. This is represented by block <NUM>.

At block <NUM>, logic <NUM> determines whether the height metric is above the threshold obtained at block <NUM>. If so, the particular area of the field is classified as standing crop. This is represented by block <NUM>. In this case, a standing crop classification result is output (e.g., block <NUM>) along with the height metric that represents a height of the standing crop in the particular area. This is represented by block <NUM>.

If the height metric is not above the threshold, then block <NUM> determines whether the area is within an already harvested region of the field. If so, the area is classified as a crop stubble area and crop stubble classification is output at block <NUM>.

If the area is not in an already harvested region, block <NUM> classifies the area as having downed crop and a downed crop classification is output. At this point, it is worth noting that, in the present example, differentiation between the crop stubble and downed crop at blocks <NUM> and <NUM>, respectively, can be achieved without requiring a trained image classifier to classify images of non-standing crop. As noted above, utilizing such image classifiers to distinguish between crop stubble and downed crop require extensive training with large amounts of training data, which in turn requires large amounts of processing bandwidth and time. Often, such classifiers can be inaccurate with results in incorrect biomass or yield predictions.

Returning again to <FIG>, at block <NUM>, a representation of a predicted amount of crop plant material in the path of the machine is generated. This representation can take any of a number of forms. For instance, block <NUM> can predict the biomass to be encountered by machine <NUM> as it moves across the field. This is represented by block <NUM>. Alternatively, or in addition, block <NUM> can predict the yield from the field. This is represented by block <NUM>. Of course, other measures can be utilized as well. This is represented by block <NUM>.

In one example, the representation generated at block <NUM> is based on the crop classification, that is, the determined crop state at block <NUM>. This is represented by block <NUM>. For instance, the predicted amount of crop plant material is based on whether the area is standing crop, downed crop, and/or crop stubble. Also, the predicted amount of crop plant material (mass flow) can be based on the crop height. This is represented by block <NUM>. As such, mass flow through the machine <NUM> can be determined based on the classified areas in front of machine <NUM>, as well as the sensed crop height within those areas. Settings (e.g., speed, etc.) of the machine can be controlled based on the determined mass flow.

Alternatively, or in addition, the predicted amount of crop plant material can be based on other crop conditions. This is represented at block <NUM>. For example, but not by limitation, predictions can be made based normalized difference vegetation index (NDVI) images that indicate growth stages, moisture stress (drought), plant stock diameter, weed presence, etc. Of course, other considerations can be taken into account in generating the representation of a predicted amount of crop plant material. This is represented by block <NUM>.

At block <NUM>, a recommendation to change operation of machine <NUM> is generated based on the predicted amount of crop plant material, generated at block <NUM>. For example, the recommendation can be generated based on a desired throughput of machine <NUM>. This is represented by block <NUM>. In other examples, the recommendation can be based on target performance of machine <NUM>, such as grain quality, productivity, fuel consumption, or any other performance metric.

The recommended change can include a change to the ground speed of machine <NUM>. This is represented by block <NUM>. Alternatively, or in addition, the recommended change can comprise a change to harvesting functionality of machine <NUM>. This is represented by block <NUM>. For example, changes to the harvesting functionality can include changes to the header, threshing subsystem <NUM>, separator subsystem <NUM>, cleaning subsystem <NUM>, residue subsystem <NUM>, or any other controllable subsystems of machine <NUM>. Of course, other recommendations can be generated as well. This is represented by block <NUM>.

At block <NUM>, a control signal is generated by control system <NUM> (e.g., using user interface component <NUM>, logic <NUM>, logic <NUM>, logic <NUM>, logic <NUM>, etc.) to control one or more of controllable subsystems <NUM>. For example, user interface mechanisms <NUM> can be controlled by user interface component <NUM> to output an indication of the predicted amount of crop plant material in the plant path of the machine (represented by block <NUM>) and/or to output the recommendation generated at block <NUM>. This is represented by block <NUM>. For example, the outputs at blocks <NUM> and/or <NUM> can be provided through a display device, speakers, etc..

Alternatively, or in addition, the control signals generated at block <NUM> can control actuators to adjust settings of controllable subsystem(s) <NUM>. This is represented by block <NUM>. For example, with respect to machine <NUM> illustrated in <FIG>, the position of header <NUM> can be adjusted, the rotor pressure of separator <NUM> can be adjusted, the operation of cleaning subsystem <NUM> can be adjusted. These, of course, are for sake of example only.

Also, a control signal can be generated to control propulsion system <NUM> to control the ground speed of machine <NUM>. This is represented by block <NUM>. Alternatively, or in addition, control systems can be generated to control steering system <NUM>, to control steering of machine <NUM>. This is represented by block <NUM>. Of course, machine <NUM> (or other machines and systems in architecture <NUM>) can be controlled in other ways as well. This is represented by block <NUM>. At block <NUM>, operation determines whether there are other areas to be harvested on the field. If so, operation can return to block <NUM> to harvest subsequent areas of the field in the path of machine <NUM>.

For sake of illustration, but not by limitation, the operations illustrated in <FIG> will be described in the context of <FIG>, which shows an example field <NUM> on which machine <NUM> is operating to harvest crop plants <NUM>. As shown, machine <NUM> is traveling along a path <NUM> in a forward direction represented by arrow <NUM>. Header <NUM> operates to harvest the crop plants <NUM> as machine <NUM> travels along path <NUM>. The field data obtained at block <NUM> includes harvest data that identifies a region <NUM> of the field that has already been harvested (e.g., by machine <NUM>, or another harvester, operating in field <NUM>).

Forward-looking sensor <NUM> (illustratively a stereo camera) captures images of crop plants <NUM> in path <NUM> ahead of machine <NUM>. Block <NUM> identifies characteristics of the crop plants in path <NUM> by performing geometric classification, block <NUM>, based on the images obtained by sensor <NUM>. Block <NUM> generates a first height metric representing a height of the crop plant materials in a first area (represented by dashed block <NUM>) and a second height metric representing a height of crop plants in a second area (represented by dashed block <NUM>). The first height metric indicates that the crop plants <NUM> in area <NUM> have an average height that is above a selected threshold. Thus, area <NUM> is classified as standing crop. The second height metric indicates that the crop plants <NUM> in area <NUM> have an average height that is below the selected threshold. Thus, area <NUM> is classified as non-standing crop.

Additionally, based on the identification of region <NUM> as being already harvested, area <NUM> is removed from the non-standing crop area <NUM>, and is classified as crop stubble. The remaining portion of area <NUM> (that is, area <NUM>) is identified as downed crop. In other words, area <NUM> is identified as an unharvested area having a height metric below the threshold. Using the information from classified areas <NUM>, <NUM>, and <NUM>, biomass system <NUM> can generate a representation of a predicted amount of crop material (e.g., a biomass) in path <NUM>.

<FIG> illustrates one example of calculating mass flow for machine <NUM> harvesting field <NUM>. Area <NUM> is identified as crop stubble based on prior harvesting data, and area <NUM> is identified as standing crop based on image processing determining that the height of crop within area <NUM> is above a standing threshold. In this example, area <NUM> is treated as a binary signal (i.e., it is either crop stubble or not crop stubble), and is thus ignored for purposes of mass flow determination. Conversely, area <NUM> is analyzed to determine contribution to mass flow. In one example, system <NUM> determines the amount of crop material based on the width <NUM> of area <NUM> that resides within the path <NUM> of header <NUM> and crop height (e.g., a height metric indicating average crop height in area <NUM>). Based on this determination, the speed (or other settings) of machine <NUM> is controlled.

It can thus be seen that the present system provides a number of advantages. For example, but not by limitation, the present system generates predictions of crop plant material for control of an agricultural harvesting machine, which improves operation of the machine such as by maintaining a desired throughput. Further, the present system uses geometric classification to determine whether areas of the field are standing crop, downed crop, or crop stubble by leveraging harvest data. This classification between downed crop and crop stubble is made by leveraging harvest data, without requiring a specific image classifier performing a classification task.

The present discussion has mentioned processors, processing systems, controllers and/or servers. In one example, these can include computer processors with associated memory and timing circuitry, not separately shown. They are functional parts of the systems or devices to which they belong and are activated by, and facilitate the functionality of the other components or items in those systems.

<FIG> is a block diagram of one example of the architecture shown in <FIG>, where machine <NUM> communicates with elements in a remote server architecture <NUM>. In an example, remote server architecture <NUM> can provide computation, software, data access, and storage services that do not require end-user knowledge of the physical location or configuration of the system that delivers the services. In various examples, remote servers can deliver the services over a wide area network, such as the internet, using appropriate protocols. For instance, remote servers can deliver applications over a wide area network and they can be accessed through a web browser or any other computing component. Software or components shown in <FIG> as well as the corresponding data, can be stored on servers at a remote location. The computing resources in a remote server environment can be consolidated at a remote data center location or they can be dispersed. Remote server infrastructures can deliver services through shared data centers, even though they appear as a single point of access for the user. Thus, the components and functions described herein can be provided from a remote server at a remote location using a remote server architecture. Alternatively, they can be provided from a conventional server, or they can be installed on client devices directly, or in other ways.

In the example shown in <FIG>, some items are similar to those shown in <FIG> and they are similarly numbered. <FIG> specifically shows that system <NUM>, system <NUM>, system <NUM>, and/or data store <NUM> can be located at a remote server location <NUM>. Therefore, agricultural machine <NUM>, machine(s) <NUM>, machine(s) <NUM>, and/or system(s) <NUM> access those systems through remote server location <NUM>.

<FIG> also depicts another example of a remote server architecture. <FIG> shows that it is also contemplated that some elements of <FIG> are disposed at remote server location <NUM> while others are not. By way of example, data store <NUM> can be disposed at a location separate from location <NUM>, and accessed through the remote server at location <NUM>. Alternatively, or in addition, one or more of systems <NUM>, <NUM>, and <NUM> can be disposed at location(s) separate from location <NUM>, and accessed through the remote server at location <NUM>.

Regardless of where they are located, they can be accessed directly by agricultural machine <NUM>, through a network (either a wide area network or a local area network), they can be hosted at a remote site by a service, or they can be provided as a service, or accessed by a connection service that resides in a remote location. Also, the data can be stored in substantially any location and intermittently accessed by, or forwarded to, interested parties. For instance, physical carriers can be used instead of, or in addition to, electromagnetic wave carriers. In such an example, where cell coverage is poor or nonexistent, another mobile machine (such as a fuel truck) can have an automated information collection system. As the agricultural machine comes close to the fuel truck for fueling, the system automatically collects the information from the machine or transfers information to the machine using any type of ad-hoc wireless connection. The collected information can then be forwarded to the main network as the fuel truck reaches a location where there is cellular coverage (or other wireless coverage). For instance, the fuel truck may enter a covered location when traveling to fuel other machines or when at a main fuel storage location. All of these architectures are contemplated herein. Further, the information can be stored on the agricultural machine until the agricultural machine enters a covered location. The agricultural machine, itself, can then send and receive the information to/from the main network.

Claim 1:
A computer-implemented method (<NUM>) comprising:
obtaining (<NUM>) harvest data indicative of a prior harvesting operation on a field;
identifying (<NUM>) a region of the field having already-harvested crop plants based on the harvest data;
detecting (<NUM>) a characteristic of crop plants in a field in a path of an agricultural harvesting machine (<NUM>) in a direction of travel;
generating (<NUM>), based on the detected characteristic, a height metric representing a height of the crop plants on a particular area of the field;
the method (<NUM>) being characterised in that it further comprises the steps of identifying the crop plants on the particular area of the field as downed crop based on a determination that the height metric indicates a crop height below a threshold and that the harvest data indicate that the particular area is not in an already harvested region; and
generating a control signal (<NUM>) to control the agricultural harvesting machine based on the identification of downed crop on the particular area of the field.