Patent Description:
There are a wide variety of different types of mobile work machine such as agricultural vehicles and construction vehicles. Some vehicles include harvesters, such as forage harvesters, sugar cane harvesters, combine harvesters, and other harvesters, that harvest grain or other crop. Such harvesters often unload into carts which may be pulled by tractors or semi-trailers as the harvesters are moving.

As one example, while harvesting in a field using a forage harvester, an operator attempts to control the forage harvester to maintain harvesting efficiency, during many different types of conditions. The soil conditions, crop conditions, and other things can all change. This may result in the operator changing control settings. This means that the operator needs to devote a relatively large amount of attention to controlling the forage harvester.

At the same time, a semi-truck or tractor-pulled cart is often in position relative to the forage harvester (e.g., behind the forage harvester or alongside the forage harvester) so that the forage harvester can fill the truck or cart while moving through the field. In some current systems, this requires the operator of the forage harvester to control the position of the unloading spout and flap so that the truck or cart is filled evenly, but not overfilled. Even a momentary misalignment between the spout and the truck or cart may result in hundreds of pounds of harvested material being dumped on the ground, or elsewhere, rather than in the truck or cart.

Other harvesters such as combine harvesters and sugar cane harvesters, can have similar difficulties.

As discussed above, it can be very difficult for an operator to maintain high efficiency in controlling a harvester, and also to optimally monitor the position of the receiving vehicle. This difficulty can even be exacerbated when the receiving vehicle is located behind the forage harvester, so that the forage harvester is executing a rear unloading operation, but the difficulty also exists in side-by-side unloading scenarios.

In order to address these issues, some automatic cart filling control systems have been developed to automate portions of the filling process. One such automatic fill control system uses a stereo camera on the spout of the harvester to capture an image of the receiving vehicle. An image processing system determines dimensions of the receiving vehicle and the distribution of the crop deposited inside the receiving vehicle. The system also detects crop height within the receiving vehicle, in order to automatically aim the spout toward empty spots and control the flap position to achieve a more even fill, while reducing spillage. Such systems can fill the receiving vehicle according to a fill strategy (such as front-to-back, back-to-front, etc.) that is set by the operator or that is set in other ways.

However, there may be reasons that an operator wishes to at least temporarily deviate from the fill strategy. For instance, if the operator notices a void in the receiving vehicle that is not as full as the operator wishes, the operator may wish to temporarily switch filling to the location of the void to fill it to a higher fill level. Similarly, if the operator sees a certain type of terrain ahead, the operator may wish to increase the load at a particular location in the receiving vehicle to increase traction or stability over the terrain. To do this, the operator currently needs to disable the automatic fill control system, manually reset spout and flap angles, wait until the spout and flap are repositioned, adjust the position if necessary, wait until the desired fill operation is complete (e.g., the void is filled) and then re-engage the automatic fill control system. This can be very cumbersome, especially while trying to simultaneously control the harvester.

Reference is made to the prior art of <CIT>, considered as generic and describing a forage harvester with a camera looking providing an image of the receiving container and image processing. An image of the container and the crop flow into the container is displayed to the operator and the operator can move or drag the displayed crop flow to another position if desired. The spout of the forage harvester is then controlled accordingly. This input can also be used to select an order in which several trailers are filled automatically or for readjustment of an automated transfer process.

<CIT> shows a harvesting machine with a crop unloader and a camera looking onto a container receiving the unloaded crop. The image is displayed to an operator to enable him to control the position of the receiving vehicle or used for automatic control thereof.

<CIT> shows a harvesting machine with a crop unloader and a camera looking onto a container receiving the unloaded crop. The image is displayed to an operator to enable him to control the position of the unloader.

<CIT> shows a harvesting machine with a crop unloader and a camera looking onto a container receiving the unloaded crop. The image is displayed to an operator together with a representation of the crop unloader to enable him to control the position of the unloader.

<FIG> is a pictorial illustration showing one example of a self-propelled forage harvester <NUM> filling a tractor-pulled grain cart (or receiving vehicle) <NUM>. Cart <NUM> thus defines an interior that forms a receiving vessel <NUM> for receiving harvested material through a receiving area <NUM>. In the example shown in <FIG>, a tractor <NUM>, that is pulling grain cart <NUM>, is positioned directly behind forage harvester <NUM>. Also, in the example illustrated in <FIG>, forage harvester <NUM> has a camera <NUM> mounted on the spout <NUM> through which the harvested material <NUM> is traveling. The spout <NUM> can be pivotally or rotationally mounted to a frame <NUM> of harvester <NUM>. Camera <NUM> can be a stereo-camera or a mono-camera that captures an image (e.g., a still image or video) of the receiving area <NUM> of cart <NUM>. In the example shown in <FIG>, the receiving area <NUM> is defined by an upper edge of the walls of cart <NUM>.

Harvester <NUM> in accordance with the invention has an automatic fill control system that includes image processing, as discussed above, the automatic fill control system can gauge the height of harvested material in cart <NUM>, and the location of that material. The system thus automatically controls the position of spout <NUM> and flap <NUM> to direct the trajectory of material <NUM> into the receiving area <NUM> of cart <NUM> to obtain an even fill throughout the entire length and width of cart <NUM>, while not overfilling cart <NUM>. By automatically, it is meant, for example, that the operation is performed without further human involvement except, perhaps, to initiate or authorize the operation.

<FIG> is a pictorial illustration showing another example of a self-propelled forage harvester <NUM>, this time loading a semi-trailer (or receiving vessel on a receiving vehicle) <NUM> in a configuration in which a semi-tractor is pulling semi-trailer <NUM> alongside forage harvester <NUM>. Therefore, the spout <NUM> and flap <NUM> are positioned to unload the harvested material <NUM> to fill trailer <NUM> according to a pre-defined side-by-side fill strategy. Again, <FIG> shows that camera <NUM> can capture an image (which can include a still image or video) of semi-trailer <NUM>. In the example illustrated in <FIG>, the field of view of camera <NUM> is directed toward the receiving area <NUM> of trailer <NUM> so that image processing can be performed to identify a landing point for the harvested material in trailer <NUM>.

In other examples, where machine <NUM> is a combine harvester, it may be that the spout <NUM> is not moved relative to the frame during normal unloading operations. Instead, the relative position of the receiving vehicle <NUM>, <NUM> and the combine harvester is changed in order to fill the receiving vessel <NUM> as desired. Thus, if a front-to-back fill strategy is to be employed, then the relative position of the receiving vessel, relative to the combine harvester, is changed so that the spout is first filling the receiving vessel at the front end, and then gradually fills the receiving vessel moving rearward. In such an example, the combine harvester and towing vehicle may have machine synchronization systems which communicate with one another. When the relative position of the two vehicles is to change, then the machine synchronization system on the combine harvester can send a message to the machine synchronization system on the towing vehicle to nudge the towing vehicle slightly forward or rearward relative to the combine harvester, as desired. By way of example, the machine synchronization system on the combine harvester may receive a signal from the fill control system on the combine harvester indicating that the position in the receiving vessel that is currently being filled is approaching its desired fill level. In that case, the machine synchronization system on the combine harvester can send a "nudge" signal to the machine synchronization system on the towing vehicle. The nudge, once received by the machine synchronization system on the towing vehicle, causes the towing vehicle to momentarily speed up or slow down, thus nudging the position of the receiving vessel forward or rearward, respectively, relative to the combine harvester.

<FIG> is a pictorial illustrating showing one example of an operator interface display <NUM> that can be displayed on a display mechanism <NUM>, for the operator in an operator compartment of forage harvester <NUM>. The operator interface display <NUM> in <FIG> shows a view of images (or video) captured by camera <NUM> of material <NUM> entering trailer <NUM>. An image processing system on harvester <NUM> illustratively identifies the perimeter of the opening <NUM> in trailer <NUM> and also processes the image of the material <NUM> in trailer <NUM> to determine the fill height relative to opening <NUM>. The perimeter defining opening <NUM> can be visually enhanced by overlaying a visual overlay over the opening <NUM> so that the operator can easily identify the opening <NUM>, as it is being recognized by the image processing system.

In some cases the operator sees that, while the material <NUM> is generally filling trailer <NUM> evenly, there may be voids in the trailer <NUM>, such as a void <NUM> at the forward end of trailer <NUM>. In that case, it may be that the operator wishes to fill void <NUM> with more material before continuing to fill the remainder of trailer <NUM>. Thus, as is discussed in greater detail below, a fill control system allows the operator to use a touch gesture (or other command input, such as a point and click input) selecting the area of trailer <NUM> on display <NUM> that corresponds to the void <NUM>. The fill control system thus allows the operator to provide an input through interface <NUM>, marking a location (such as the location of void <NUM>) where material <NUM> is to be directed.

For example, where the display screen on mechanism <NUM> is a touch sensitive display screen, then the operator may simply touch the screen in the area of void <NUM>. The touch gesture will be detected by the fill control system and the fill control system will automatically generate control signals to move spout <NUM> so that it is depositing material <NUM> in the area of void <NUM>.

Generating the control signals to reposition spout <NUM> can be done in different ways. For instance, once the operator touches or otherwise selects (such as with a point and click device) an area of display <NUM>, the control system identifies the pixel or pixel sets that were selected (e.g., touched or otherwise selected) and, from those pixels, identifies a corresponding physical area or landing point within trailer <NUM>. The control system can then calculate the position that spout <NUM> needs to be in in order to fill material <NUM> in that particular landing point in trailer <NUM>.

In accordance with the invention, the spout <NUM> keeps depositing material in the area of void <NUM> (or another location commanded by the operator input) until some criteria are met, at which point the fill control system moves spout <NUM> back to resume filling trailer <NUM> at the location it was at prior to moving to the area of void <NUM>.

The criteria for resuming the prior fill operation can be any of a variety of different criteria. In one example, the criteria may be timing criteria. For instance, when the user provides a reposition command to move spout <NUM> to begin filling in the area of void <NUM>, the control system may move spout <NUM> to fill in that area for a predetermined amount of time (such as <NUM> seconds, etc.). After that time has elapsed, then the control system may move spout <NUM> to resume filling trailer <NUM> at the previous location. In another example, the criteria may be fill level or threshold criteria. For instance, where the image processing system detects how full trailer <NUM> is at different locations, then spout <NUM> may be repositioned to fill the area of void <NUM> until the fill level in the area of void <NUM> reaches a threshold fill level. In another example, where the fill level is already at the threshold level prior to repositioning spout <NUM>, the threshold levels may be increased so that the trailer <NUM> in the area of void <NUM> can be filled with more material <NUM>.

It should also be noted that, in one example, forage harvester <NUM> may have an automatic fill control system (or active fill control system) which fills trailer <NUM> according to a fill strategy (such as a back-to-front fill strategy, front-to-back fill strategy, etc.). In that case, a current location indicator (such as indicator <NUM>) may be displayed to show the current location where material <NUM> is being loaded into trailer <NUM> through spout <NUM> and the direction that spout <NUM> will be moving relative to trailer <NUM> as the filling operation continues. It can be seen in <FIG>, for instance, that indicator <NUM> is an arrow pointing in the front-to-back direction. The location of arrow <NUM> on the representation of trailer <NUM> indicates the current fill position, while the direction of the arrow indicates the direction that spout <NUM> will be moved relative to trailer <NUM> in executing the selected front-to-back fill strategy. Therefore, in one example, when the operator taps the display screen in, or otherwise selects, the area of trailer <NUM> corresponding to void <NUM>, the indicator <NUM> remains in place to indicate the location to which spout <NUM> will return after it temporarily fills the area of trailer <NUM> corresponding to void <NUM>. These are just some examples of how the operator interface display <NUM> can be generated.

In another example, as is discussed in greater detail elsewhere, the operator may also select an active fill strategy using a touch gesture. For instance, at the beginning of a fill operation, or at any time during the fill operation, the operator may touch the representation of trailer <NUM> on display <NUM> and swipe in one direction or the other or provide another touch gesture. By way of example, assume that the operator touches the rear portion of trailer <NUM> and swipes forward toward the front portion of trailer <NUM>. In that case, the system can detect that touch gesture and interpret it as selecting a back-to-front fill strategy. The control system then begins filling the trailer toward the rear of the trailer and continues to fill the trailer moving toward the front of the trailer.

<FIG> is another example of an operator interface display <NUM> which can be generated for the operator of harvester <NUM>. Some items are similar to those shown in <FIG> and they are similarly numbered. Display <NUM> is a representation of a top-down view of trailer <NUM>. The top-down view is accompanied by a graph <NUM> that illustrates the fill level of the different portions of trailer <NUM>. By way of example, image processing can divide the area of the trailer <NUM> into bins or discrete volumes (some of which are illustrated by the dashed lines in <FIG> shows that the bins <NUM>, <NUM>, <NUM>, and <NUM> correspond to volumes defined by a cross section of the trailer <NUM>. Each of the bins has a corresponding bar graph in graph <NUM>, indicating the fill level of that bin in trailer <NUM>. The bar graphs corresponding to bins <NUM> and <NUM> are at zero showing that there is no material <NUM> at those locations in trailer <NUM>. The bar graphs corresponding to bins <NUM> and <NUM> show increasing levels of material <NUM> in those bins. The location of the bins may be displayed on display <NUM>, or they may be hidden.

In the example shown in <FIG>, the operator may see that the location of trailer <NUM> corresponding to bins <NUM> and <NUM> is empty. In that case, the operator may provide a reposition command input (such as tapping or touching the display <NUM> in the area of bin <NUM> or <NUM>) to indicate that the operator desires to have the spout <NUM> repositioned to a location which fills the volume corresponding to the selected bin in trailer <NUM>. For instance, assume that the operator taps the display <NUM> in the area of bin <NUM>. In that case, the control system automatically repositions the spout <NUM> from the position illustrated by indicator <NUM> to the position of bin <NUM>, and begins filling at that position. In accordance with the invention, an automatic fill control system is executing a fill strategy, and the automatic fill strategy is temporarily suspended, and trailer <NUM> is filled in the location of bin <NUM> until some resume criteria are met, at which point the spout <NUM> is controlled to resume filling at the location of indicator <NUM>, and the selected fill strategy is re-activated. Once the operator enters a command (such as by tapping the location of a bin on the display <NUM> or by tapping indicator <NUM>) the control system translates that pixel location into a physical bin location on trailer <NUM> and moves the spout <NUM> so that it begins filling the trailer at that bin location.

Again, as with the display shown in <FIG>, the operator may use a touch gesture (such as a touch and swipe touch gesture) to indicate a desired active fill strategy. The control system can interpret the touch gesture to identify the fill strategy and begin implementing that fill strategy.

<FIG> is a block diagram showing one example of a mobile work machine which comprises agricultural harvester <NUM>, in more detail. Agricultural harvester <NUM>, in the example shown in <FIG>, includes one or more processors or servers <NUM>, communication system <NUM>, data store <NUM>, sensors <NUM>, fill control system <NUM>, void fill control system <NUM>, operator interface mechanisms <NUM>, controllable subsystems <NUM>, and other harvester functionality <NUM>. Sensors <NUM> can include automatic fill control sensors <NUM> that are used by fill control system <NUM>. Sensors <NUM> can include camera <NUM> (which may be a mono-camera, stereo-camera or another type of camera) and other sensors <NUM>. The other sensors can include such things as Doppler sensors, RADAR sensors, other image sensors or any of a wide variety of other types of sensors. Sensors <NUM> can also include spout position sensor <NUM> and flap position sensor <NUM>. Spout position sensor <NUM> illustratively senses the position of spout <NUM> relative to the frame of harvester <NUM>. Sensor <NUM> can do this by sensing the position of an actuator that drives movement of spout <NUM> relative to the frame of harvester <NUM>, or sensor <NUM> can be a rotary position sensor, a linear sensor, a potentiometer, a Hall Effect sensor, or any other of a wide variety of sensors that can sense the position of spout <NUM> relative to the frame of harvester <NUM>. Similarly, flap position sensor <NUM> can be a sensor that senses the position of the flap <NUM>. Thus, sensor <NUM> can be a rotary position sensor, a linear sensor, a potentiometer, a Hall Effect sensor, a sensor that senses a position of an actuator that drives movement of flap <NUM>, or any of a wide variety of other sensors.

Sensors <NUM> can also include machine synchronization sensors <NUM>. Sensors <NUM> can include relative position sensors <NUM> that sense the relative position of the harvester, relative to the receiving vehicle. Such sensors can include RADAR sensors, Doppler sensors, image or other optical sensors, or a wide variety of other relative position sensors. The relative position sensors <NUM> can also include position sensors (such as a GPS receiver, or another GNSS sensor) that senses the position of harvester <NUM>. This can be used, in conjunction with another position sensor on the receiving vehicle, to determine the position of the two vehicles relative to one another. The machine synchronization sensors <NUM> can include other sensors <NUM>, and sensors <NUM> can include a wide variety of other sensors <NUM> as well.

Fill control system <NUM> illustratively controls operations of various parts of harvester <NUM> (and possibly the towing vehicle <NUM>) to fill the receiving vehicle <NUM>, <NUM>, as desired. Fill control system <NUM> includes automatic fill control system <NUM> (which, itself, can include fill strategy selector <NUM>, fill strategy implementation processor <NUM> and other items <NUM>), manual fill control system <NUM> (which, itself can include manual set point detector <NUM> and other items <NUM>), and/or machine synchronization fill control system <NUM>. Fill control system <NUM> can also include fill control signal generator <NUM> and other items <NUM>. Void fill control system <NUM> can include operator interface display generator <NUM>, operator interaction detection and processing system <NUM>, position control signal generator <NUM> and other items <NUM>. Operator interface mechanisms <NUM> can include interactive display mechanism <NUM> and a variety of other operator interface mechanisms <NUM>. Controllable subsystems <NUM> can include propulsion subsystem <NUM>, steering subsystem <NUM>, one or more spout actuators <NUM>, one or more flap actuators <NUM> and other items <NUM>. <FIG> also shows that operator <NUM> can interact through operator interface mechanism <NUM> to control and manipulate agricultural harvester <NUM>. Further, <FIG> shows that harvester <NUM> is connected over network <NUM> to receiving vehicle <NUM>, <NUM>, towing vehicle <NUM> and/or it can be connected to other systems <NUM>. Before describing the overall operation of agricultural harvester <NUM> in more detail, a brief description of some of the items in agricultural harvester <NUM>, and their operation, will first be provided.

Communication system <NUM> can facilitate communication among the items of harvester <NUM> and with other items over network <NUM>. Network <NUM> can be a wide area network, a local area network, a near field communication network, a cellular communication network, or any of a variety of other networks or combinations of networks. Therefore, communication system <NUM> can use a controller area network (CAN) bus or other controllers to facilitate communication of the items on harvester <NUM> with other items. Communication on system <NUM> can also be different kinds of communication systems, depending on the particular network or networks <NUM> over which communication is to be made.

Operator interface mechanisms <NUM> can be a wide variety of different types of mechanisms. Interactive display mechanism <NUM> can be a display mechanism, such as that shown in <FIG> and <FIG>, or mechanism <NUM> can be a display mechanism on a mobile device, such as a tablet computer, a smartphone, etc., that is carried by the operator <NUM> and/or mounted in the operator compartment of harvester <NUM>. Thus, interactive display mechanism <NUM> can be a touch sensitive display mechanism, a display mechanism that receives inputs through a point and click device, or other kinds of display mechanisms.

Other operator interface mechanisms <NUM> can include a steering wheel, levers, buttons, pedals, a microphone and speaker (where speech recognition and speech synthesis are provided), joysticks, or other mechanical, audio, visual or haptic mechanisms that can be used to provide outputs to operator <NUM> or to receive inputs from operator <NUM>.

Controllable subsystems <NUM> can be controlled by various different items on harvester <NUM>. Propulsion subsystem <NUM> can be an engine that drives ground-engaging elements (such as wheels or tracks) through a transmission, hydraulic motors that are used to drive ground-engaging elements, electric motors, direct drive motors, or other propulsion systems that are used to drive ground-engaging elements to propel harvester <NUM> in the forward and rearward directions. Propulsion subsystem <NUM> can illustratively be controlled with a throttle to increase or decrease the speed of travel of harvester <NUM>.

Steering subsystem <NUM> can be used to control the heading of harvester <NUM>. One or more spout actuators <NUM> are illustratively configured to drive rotation or movement of spout <NUM> relative to the frame of harvester <NUM>. Actuators <NUM> can be hydraulic actuators, electric actuators, pneumatic actuators, or any of a wide variety of other actuators. Similarly, one or more flap actuators <NUM> are used to drive the position of flap <NUM> relative to spout <NUM>. The flap actuators <NUM> can also be hydraulic actuators, electric actuators, pneumatic actuators, or any of a wide variety of other actuators.

Fill control system <NUM> in accordance with the invention uses an automatic fill control system <NUM> to perform automated fill control to automatically execute a fill strategy in filling one of the receiving vehicles <NUM>, <NUM>. Therefore, fill strategy selector <NUM> can detect a user input selecting a fill strategy, or another input selecting a fill strategy and access data store <NUM> for a stored fill algorithm that can be executed to perform the selected fill strategy. For instance, where the selected fill strategy is a back-to-front strategy, the algorithm will direct filling of the receiving vehicle beginning at the back of the receiving vehicle and moving to the front of the receiving vehicle. Other fill strategies can be selected as well. Fill strategy implementation processor <NUM> receives inputs from the automatic fill control sensors <NUM>, spout position sensor <NUM> and flap position sensor <NUM> and generates an output to fill control signal generator <NUM> based upon the inputs from the sensors, to execute the desired automatic fill control strategy. Fill control signal generator <NUM> can generate control signals to control any of the controllable subsystems <NUM> (or other items) to execute the fill strategy being implemented by fill strategy implementation processor <NUM>.

Manual fill control system <NUM> can use manual set point detector <NUM> to detect a manual input from operator <NUM> (e.g., through interactive display mechanism <NUM>) to identify a landing point in the receiving vehicle <NUM>, <NUM> where the operator <NUM> desires the filling operation to be performed. Manual fill control system <NUM> can then generate outputs to fill control signal generator <NUM> which generates control signals to control the controllable subsystems <NUM> so that filling commences at the manually identified landing point in the receiving vehicle <NUM>, <NUM>.

Machine synchronization fill control system <NUM> can receive operator inputs or other inputs, as well as sensor inputs from sensors <NUM> to generate outputs to fill control signal generator <NUM> in order to synchronize the positions of agricultural harvester <NUM> and receiving vehicle <NUM>, <NUM> so that a desired filling operation is performed. For instance, machine synchronization control system <NUM> can receive sensor inputs identifying that the current position that is being filled in receiving vehicle <NUM>, <NUM>, is at a desired fill level so that the receiving vehicle should move forward or rearward relative to agricultural harvester <NUM>. Machine synchronization fill control system <NUM> then generates an output to fill control signal generator <NUM> indicating this. Fill control signal generator <NUM> can generate an output either to controllable subsystems <NUM>, or communication system <NUM>, or both, based on the inputs from machine synchronization fill control system <NUM>. For instance, where the output from system <NUM> indicates that the receiving vehicle <NUM>, <NUM> should move forward relative to agricultural harvester <NUM>, then fill control signal generator <NUM> can control communication system <NUM> to communicate with a corresponding machine synchronization fill control system <NUM> on towing vehicle <NUM> indicating that towing vehicle <NUM> should "nudge" forward relative to the harvester <NUM> by momentarily increasing its ground speed and then returning to its current ground speed. Alternatively, or in addition, fill control signal generator <NUM> can generate control signals to control the propulsion subsystem <NUM> on agricultural harvester <NUM> to momentarily change the speed of agricultural harvester <NUM> so that the position of the receiving vehicle <NUM>, <NUM> relative to agricultural harvester <NUM> changes as desired.

Void fill control system <NUM> can use operator interface display generator <NUM> to generate an operator interface display indicative of the receiving vessel <NUM>, <NUM> and the amount of material in the receiving vessel <NUM>, <NUM>. The operator interface display may be similar to one of the displays shown in <FIG> and <FIG> or different. Operator interaction detection and processing system <NUM> can detect operator inputs through the interface display that was generated by generator <NUM> and displayed on interactive display mechanism <NUM>. The operator input may be a reposition command input that illustratively identifies a selected landing point in receiving vehicle <NUM>, <NUM> to which the operator wishes to reposition the filling operation, at least temporarily. Operator interaction detection and processing system <NUM> then translates the landing point selected by operator <NUM> on interactive display mechanism <NUM> to a landing point in receiving vehicle <NUM>, <NUM> that is to be filled. Processing system <NUM> identifies the actions needed to be taken in order to begin filling at that landing point.

For instance, system <NUM> can identify the spout and flap angles that are needed in order to commence filling at the selected landing point in receiving vehicle <NUM>, <NUM>. System <NUM> can output those angles to position control signal generator <NUM> which can then generate control signals to control spout actuators <NUM> and flap actuators <NUM> to commence the fill operation at the selected location. At the same time, system <NUM> can provide an output to fill control system <NUM> temporarily suspending fill control system <NUM> from controlling the fill operation. For instance, if the automatic fill control system <NUM> is executing an automatic fill strategy, then the signal from system <NUM> can temporarily suspend system <NUM> from executing that fill strategy until criteria are met at which point system <NUM> can resume that fill strategy. Similarly, if manual fill control system <NUM> is controlling the fill operation to fill at a previously selected manual landing point, then the signal from system <NUM> can temporarily suspend system <NUM> from maintaining the fill operation at that previously selected landing point, and allow position control signal generator <NUM> to reposition the spout and flap to commence filling at the newly selected landing point.

Also, where machine synchronization fill control system <NUM> is being used, the signal from system <NUM> can cause system <NUM> to generate output signals to move the filling operation to the operator-selected landing point in receiving vehicle <NUM>, <NUM>, until criteria are met. Alternatively, the signal from system <NUM> can cause machine synchronization fill control system <NUM> to suspend its control operation and allow position control signal generator <NUM> to control the propulsion subsystem <NUM> and/or steering subsystem <NUM> to change the relative position of harvester <NUM> and receiving vehicle <NUM>, <NUM> (to reposition the spout relative to the receiving vehicle based on the reposition command input) so that the filling operation commences at the operator-selected landing point.

<FIG> is a flow diagram illustrating one example of void fill control system <NUM>, in more detail. Some of the items in <FIG> are similar to those shown in <FIG>, and they are similarly numbered. However, <FIG> shows one example of operator interface display generator <NUM>, operator interaction detection and processing system <NUM>, and reposition control signal generator <NUM>, in more detail. Before describing the operation of void fill control system <NUM> in more detail, some of the items in system <NUM>, and their operation, will first be provided. Operator interface display generator <NUM> illustratively includes receiving vehicle display system <NUM> (which, itself, can include streaming video display generator <NUM>, top-down display generator <NUM>, and other items <NUM>), overlay generator <NUM>, current fill location marker system <NUM>, and other items <NUM>. Operator interaction detection and processing system <NUM> can include vessel bin/location assignment system <NUM>, operator input detector <NUM> (which, itself, can include point and click detector <NUM>, touch gesture detector <NUM>, and other items <NUM>), input processing system <NUM> (which, itself, can include set point location identifier <NUM>, resume detector <NUM>, fill strategy identifier <NUM>, and other items <NUM>), and other items <NUM>. Reposition control signal generator <NUM> can include automatic fill control suspension signal controller <NUM>, spout/flap control signal generator <NUM>, machine synchronization control signal generator <NUM>, resume criteria evaluation system <NUM>, and other items <NUM>.

Receiving vessel display system <NUM> generates an operator display that represents the receiving vessel, and a fill level of material that the harvester is transferring to the receiving vessel. Streaming video display generator <NUM> generates a streaming video display based on video information captured by camera <NUM>. Top-down display generator <NUM> can generate a top-down representation of the receiving vessel <NUM>, such as that shown in <FIG>. The top-down display can be a pictorial illustration or it can be generated in other ways. In addition, top-down display generator <NUM> can generate the fill level display portion <NUM>, or another display portion that shows the fill level of material <NUM> in the receiving vessel.

Overlay generator <NUM> can generate one or more different types of overlays. For instance, overlay generator <NUM> can generate an overlay that visually distinguishes the perimeter of the receiving vessel103, or the opening <NUM> of the receiving vessel <NUM> so that the opening can be quickly identified, visually, by the operator. Overlay generator <NUM> can also generate other overlay display elements, such as the bin display elements shown as dashed lines in <FIG> (which can also be depicted in <FIG>) and other overlays.

Current fill location marker system <NUM> generates the current fill location (such as marker <NUM> shown in <FIG> and <FIG>) that indicates the current fill position, or the landing point where spout <NUM> is currently depositing material into the receiving vessel <NUM>. In one example, the current fill location marker can be an arrow or other directional marker that identifies the direction which spout <NUM> is moving relative to the receiving vessel <NUM>, during the filling operation.

Operator interaction detection and processing system <NUM> detects information indicative of operator interactions with the operator interface display, and processes those interactions. Vessel bin/location assignment system <NUM> first performs image processing on the captured images to identify the receiving vessel <NUM> in the captured images. It then divides the receiving vessel <NUM> into a number of discrete bins or locations which correspond to consecutive volumes in the receiving vessel <NUM>. For instance, system <NUM> can analyze the image or representation shown in <FIG> and divide that image or representation into different bins, some of which are shown as bins <NUM>-<NUM> in <FIG>. The same can be done with respect to the image illustrated in <FIG>.

Operator input detector <NUM> then detects any operator inputs or interactions with the operator interface display. Point and click detector <NUM> can detect a point and click input on the operator interface display. Touch gesture detector <NUM> detects touch gestures on the operator interface display.

Input processing system <NUM> processes the detected operator inputs to identify an operator intent or an operator command corresponding to the detected input. Set point location identifier <NUM> can receive an indication of the detected operator input from operator input detector <NUM> and determine that the operator input is identifying a reposition command. The reposition command identifies a set point position (or landing point) indicating that the operator wishes the spout to be repositioned to commence filling at the commanded landing point in the receiving vessel <NUM>. For instance, assume that the operator taps on a pixel within the displayed representation of bin <NUM> in <FIG>. In that case, set point location identifier <NUM> can access a correlation between the pixel where the tap was detected and the bin <NUM>. Set point location identifier <NUM> can then generate an output indicating that the user has commanded the filling operation to reposition the spout <NUM> so that it commences filling bin <NUM> in receiving vessel <NUM>.

Resume detector <NUM> can detect criteria indicating that the fill control system should resume controlling the fill operation as it was doing prior to the detected operator reposition command. For instance, continuing with the example discussed above, assume that the operator has tapped on a location corresponding to bin <NUM> in <FIG>. Again, assume that the void fill control system <NUM> or fill control system <NUM> has, in response, repositioned the spout <NUM> so it commences filling bin <NUM>. At some point, the fill control system <NUM> detects that the material <NUM> has reached a desired fill level in bin <NUM>. This can be done by analyzing the captured image to identify the fill level of material <NUM> in bin <NUM>. This information is provided to resume detector <NUM> which detects that the resume criteria have been met so that filling automatically resumes at its prior landing point (prior to receiving the user reposition command) and the control signals are generated to move the spout <NUM> so that filling resumes at that previous landing point. In another example, resume detector <NUM> may detect other resume criteria. For instance, it may be that the operator again taps on location indicator <NUM>. This will be detected by touch gesture detector <NUM> and an indication of this will be provided to resume detector <NUM>. Resume detector <NUM> may interpret this input command as a command by the operator to resume filling at the prior landing point (the landing point prior to receiving the reposition command). Other resume criteria may be detected by resume detector <NUM> as well.

In another example, operator <NUM> may select the fill strategy using a touch gesture on the operator interface display. For instance, when a filling operation is about to commence, the operator may touch and swipe on the depiction of the receiving vessel, depicted on the operator interface display. The location of the touch, and the direction of the swipe, may be used by fill strategy identifier <NUM> to identify a fill strategy that is to be employed during the filling operation. For instance, if the operator touches the back portion of the receiving vessel and swipes in a forward direction, then this may be interpreted by fill strategy identifier <NUM> as identifying a back-to-front fill strategy. Thus, fill strategy identifier <NUM> may output an indication of this to fill strategy selector <NUM> in fill control system <NUM> so that fill strategy selector <NUM> selects the back-to-front fill strategy.

The outputs from operation interaction and processing system <NUM> can be provided to reposition control signal generator <NUM> which provides output signals to either fill control signal generator <NUM> or directly to controllable subsystems <NUM>, or both, to implement control for repositioning spout <NUM> based upon the operator reposition command, and to again resume filling at the prior landing point once resume criteria are detected. As discussed above, it may be that automatic fill control system <NUM> is implementing an automatic fill strategy (or an active fill strategy) when the operator reposition command is detected. In that case, automatic fill control suspension controller <NUM> temporarily suspends the automatic fill control strategy so that the spout <NUM> can be repositioned based on the operator reposition command. Automatic fill control suspension controller <NUM> enables the automatic fill control strategy again, once the resume criteria are detected.

Spout/flap control signal generator <NUM> generates control signals to control the spout actuators <NUM> and flap actuators <NUM>. In one example, generator <NUM> can calculate the spout and flap angles needed to reposition spout <NUM> to the commanded landing point and to resume at its prior landing point, and provide those angles to fill control signal generator <NUM> which, in turn, controls the spout and flap actuators <NUM> and <NUM>, respectively, to move to the commanded angles. In another example, spout/flap control signal generator <NUM> can directly provide control signals to actuate spout actuators <NUM> and flap actuators <NUM>. These and other architectures are contemplated herein.

In an example in which agricultural harvester <NUM> has machine synchronization fill control system <NUM>, machine synchronization control signal generator <NUM> generates control signals and provides them to machine synchronization fill control system <NUM> to reposition the spout <NUM> relative to the receiving vehicle, based upon the operator reposition command, and to again position spout <NUM> relative to the receiving vehicle to resume filling at the prior landing point when the resume criteria are met. Thus, when a signal is received from operator interaction detection and processing system <NUM> indicating that the operator has provided a reposition command, then machine synchronization control signal generator <NUM> provides control signals to machine synchronization fill control system <NUM> to reposition spout <NUM>, relative to the receiving vehicle, to commence filling at the landing point indicated by the reposition command. When the fill operation is to resume at its prior landing point (e.g., when resume criteria have been detected), then machine synchronization control signal generator <NUM> provides control signals to machine synchronization fill control system <NUM> to move the position of spout <NUM>, relative to the receiving vehicle, back to the position it was in prior to receiving the reposition command.

Resume criteria evaluation system <NUM> can evaluate the resume criteria detected by resume detector <NUM> to determine if and when to resume filling at the prior landing point in the receiving vehicle. For instance, if resume detector <NUM> detects that the fill level is within <NUM>% of a threshold value, then resume criteria evaluation system <NUM> may determine when and how quickly to move spout <NUM> to the prior landing point and resume filling according to a prior active fill strategy. This is just one example of how resume criteria evaluation system <NUM> can evaluate the resume criteria to resume a prior filling operation.

<FIG> and <FIG> (collectively referred to herein as <FIG>) show a flow diagram illustrating one example of the operation of agricultural harvester <NUM> in beginning a fill operation, detecting an operator reposition input, repositioning the location of spout <NUM> relative to the receiving vehicle based upon the operator reposition input, detecting resume criteria, and resuming the prior fill operation. It is first assumed that harvester <NUM> is beginning a harvesting operation, or that the harvester is operating, as indicated by block <NUM> in the flow diagram of <FIG>. As discussed above, agricultural harvester <NUM> may be a self-propelled forage harvester <NUM>, a combine harvester <NUM>, a sugarcane harvester <NUM> or another harvester <NUM>.

Camera <NUM> can capture images (such as video or other images) of the receiving vessel and vessel bin/location assignment system <NUM> processes those images and identifies discrete locations or bins in the receiving vessel. Identifying bins or defining discrete locations within the receiving vessel is indicated by block <NUM>. As discussed above, system <NUM> can divide the receiving vessel into bins by processing images captured by camera <NUM>, as indicated by block <NUM>. In another example, system <NUM> can identify a type of receiving vessel (such as a type of wagon) and then access the dimensions of that receiving vessel from memory or from other predetermined data. System <NUM> can then divide the receiving vessel into a desired number of bins based on the dimensions. In yet another example, each vessel type is already assigned a number and configuration of different bins so that all system <NUM> needs to do is identify the type of receiving vessel or receiving vehicle and then access the known bin configuration for that receiving vessel. Recognizing the receiving vessel to identify and assign bins is indicated by block <NUM>. Dividing the receiving vessel into a number of discrete locations or bins can be done in other ways as well, as indicated by block <NUM>.

Operator interface display generator <NUM> displays a representation of the receiving vehicle on an interactive display device <NUM>. Displaying the representation of the receiving vehicle is indicated by block <NUM> in the flow diagram of <FIG>. In one example, the representation can be displayed on a touch sensitive display device as indicated by block <NUM>. The representation can be a streaming video with the vessel opening and the bins overlaid on top of the images, as indicated by block <NUM>. In another example, the representation is a top-down graphical or pictorial representation of the vessel and it may also have the bins displayed thereon, as indicated by block <NUM>. The representation of the receiving vessel can be displayed on an interactive display in other ways as well, as indicated by block <NUM>.

Operator interaction detection and processing system <NUM> then detects an operator reposition command input on the interactive display, as indication be block <NUM>. As discussed above, the operator reposition command can be a touch input <NUM>, a point and click input <NUM>, or another type of input <NUM> on the representation of the receiving vessel on the interactive display.

Input processing system <NUM> then identifies a location where the spout <NUM> is to be repositioned based upon the pixel location on the interactive display that received the reposition command from the operator. Identifying a location where the spout <NUM> is to be repositioned based upon the reposition command input is indicated by block <NUM> in the flow diagram of <FIG>. This can be done in a variety of different ways. For instance, set point location identifier <NUM> can access a pre-defined pixel-to-bin correlation which correlates the assigned bins in the receiving vessel to the different pixel locations on the image. Then, by identifying the particular pixel or set of pixels that receive the reposition command input from the operator, the correlation outputs the corresponding bin in the receiving vessel. Accessing a pre-defined pixel-to-bin correlation is indicated by block <NUM> in the flow diagram of <FIG>. In another example, set point location identifier <NUM> makes a runtime correlation. By way of example, the location of the receiving vessel within the image being processed may change over time. Therefore, the correlation between a particular pixel on the display device and a bin assigned to the receiving vessel may also change over time. Thus, in one example, a real-time (or near real time) correlation between the pixels on the displayed representation of the receiving vessel and the bins in the receiving vessel is generated and that real-time correlation is used to identify the location within the receiving vessel that the user has commanded with the reposition command. Using a runtime correlation to identify where to reposition the spout <NUM>, relative to the receiving vessel <NUM>, based upon the reposition command is indicated by block <NUM> in the flow diagram of <FIG>.

Set point location identifier <NUM> can identify a landing point where the spout <NUM> is to be repositioned based upon the pixel location that received the reposition command in other ways as well, as indicated by block <NUM>.

Set point location identifier <NUM> then outputs the identified set point location (or commanded landing point) to reposition control signal generator <NUM>, as indicated by block <NUM> in the flow diagram of <FIG>. Signal generator <NUM> generates output signals to control the position of the spout <NUM> relative to the receiving vessel <NUM> to begin filling at the identified landing point location, as indicated by block <NUM> in the flow diagram of <FIG>. If an automatic fill strategy is being used, it is temporarily suspended by automatic fill control suspension controller <NUM>. Temporarily suspending an automatic fill strategy is indicated by block <NUM> in the flow diagram of <FIG>.

Spout/flap control signal generator <NUM> can generate control signals to control the spout and flap actuators based upon the set point location (or landing point) in the reposition command, as indicated by block <NUM>. Where machine synchronization fill control system <NUM> is being used, then machine synchronization control signal generator <NUM> can generate outputs to system <NUM> to reposition the spout relative to the receiving vessel <NUM>, as indicated by block <NUM>. Signal generator <NUM> can generate control signals to control the position of the spout (and/or flap) relative to the receiving vessel <NUM> to begin filling at the identified landing point in other ways as well, as indicated by block <NUM>.

Void fill control system <NUM> and/or fill control system <NUM> then perform the fill operation at the identified landing point (e.g., in the identified bin) in the receiving vessel <NUM> until resume criteria are met, as indicated by block <NUM> in the flow diagram of <FIG>. The resume criteria may be the elapse of a predetermined time period (such as <NUM> seconds) as indicated by block <NUM>. The resume criteria may be fill level criteria which are used to identify when sufficient material <NUM> has been filled at the landing point, as indicated by block <NUM>. The fill level may be identified using threshold values, fill level ranges, or other criteria. The resume criteria may include receiving a "resume input" from the operator, through the interactive display, as indicated by block <NUM>. The resume criteria may be other criteria <NUM> as well.

At some point, resume detector <NUM> detects the resume criteria and provides them to resume criteria evaluation system <NUM>. Resume criteria evaluation system <NUM> determines that the resume criteria have been met. Determining whether the resume criteria have been met is indicated by block <NUM> in the flow diagram of <FIG>. The output from system <NUM> is then used by fill control system <NUM> to resume the prior filling operation, as indicated by block <NUM>. Fill control system <NUM> then takes over operation and changes the position of spout <NUM> relative to the receiving vessel <NUM> to resume filling at the prior landing point, prior to repositioning the spout <NUM>. Adjusting the position of spout <NUM> relative to receiving vessel <NUM> to perform the filling operation at the prior landing point is indicated by block <NUM>. If automatic fill control system <NUM> was performing an automatic fill operation, and if that operation was temporarily suspended, then automatic fill control suspension controller <NUM> re-engages that automatic fill control system <NUM> so that it can resume where it left off. Re-engaging the automatic fill control system <NUM> and continuing with the prior automatic fill strategy is indicated by block <NUM> in the flow diagram of <FIG>. Resuming the prior filling operation can be done in other ways as well, as indicated by block <NUM>. Processing then reverts to block <NUM> where the system continues to display a representation of the receiving vessel <NUM> on an interactive display mechanism, and waits for another operator reposition command. Continuing this operation until the harvesting operation is complete is indicated by block <NUM> in the flow diagram of <FIG>.

<FIG> is a flow diagram illustrating one example of the operation of agricultural harvester <NUM> in receiving a touch gesture and selecting an automatic fill strategy based upon that touch gesture. Operator interface display generator <NUM> displays a representation of the receiving vehicle or receiving vessel on an interactive display mechanism, as indicated by block <NUM> in the flow diagram of <FIG>. Touch gesture detector <NUM> detects a touch gesture on the representation of the receiving vessel, as indicated by block <NUM>. The touch gesture may be a swipe gesture <NUM>, a tap or touch and swipe gesture <NUM>, a sequence of taps <NUM>, or another touch gesture <NUM>.

Fill strategy identifier <NUM> then detects the touch gesture as an automatic fill strategy selection input and provides an indication of that input to fill strategy selector <NUM>. Identifying an automatic fill strategy based upon the detected touch gesture is indicated by block <NUM> in the flow diagram of <FIG>. The fill strategy can be any of a wide variety of different fill strategies such as a front-to-back strategy <NUM>, a back-to-front strategy <NUM>, or another strategy <NUM>. Providing the identified automatic fill strategy to the fill strategy selector in automatic fill control system <NUM> is indicated by block <NUM> in the flow diagram of <FIG>. Fill strategy selector <NUM> then selects the fill strategy (e.g., loads the fill strategy algorithm for implementing that strategy), from data store <NUM>, or in another way. Fill strategy implementation processor <NUM> then generates outputs to control signal generator <NUM> to control the filling operation by implementing the selected fill strategy, as indicated by block <NUM>.

It can thus be seen that the present discussion has proceeded with respect to a system that can quickly reposition the filling operation to fill a different landing point in a filling vessel based upon an operator reposition input on an interactive display mechanism. The filling operation that takes place at the repositioned landing point can be performed by suspending an automatic fill strategy that is currently being implemented and repositioning the filling operation, or by changing a manually set landing point based upon the reposition command. Similarly, the filling operation can take place at the repositioned location temporarily, until resume criteria are met, at which point control can revert to performing the previous filling operation, at the previous landing point in the receiving vessel.

The present discussion has mentioned processors and servers. In one example, the processors and servers include computer processors with associated memory and timing circuitry, not separately shown. They are functional parts of the systems or devices to which they belong and are activated by, and facilitate the functionality of the other components or items in those systems.

The interface displays can take a wide variety of different forms and can have a wide variety of different user actuatable input mechanisms disposed thereon. For instance, the user actuatable input mechanisms can be text boxes, check boxes, icons, links, drop-down menus, search boxes, etc. The mechanisms can also be actuated in a wide variety of different ways. For instance, the mechanisms can be actuated using a point and click device (such as a track ball or mouse). The mechanisms can be actuated using hardware buttons, switches, a joystick or keyboard, thumb switches or thumb pads, etc. The mechanisms can also be actuated using a virtual keyboard or other virtual actuators. In addition, where the screen on which they are displayed is a touch sensitive screen, the mechanisms can be actuated using touch gestures. Also, where the device that displays them has speech recognition components, the mechanisms can be actuated using speech commands.

It will be noted the data stores can each be broken into multiple data stores.

Claim 1:
An agricultural harvester (<NUM>), comprising:
a spout (<NUM>) configured to be positioned at a first spout position relative to a receiving vehicle (<NUM>), the receiving vehicle defining a receiving vessel (<NUM>) and the spout (<NUM>), when positioned in the first spout position, directing harvested material (<NUM>) to fill the receiving vessel (<NUM>) with the harvested material;
a display device (<NUM>);
an operator interface display generator (<NUM>) configured to generate a display on the display device (<NUM>) showing a view of images captured by a camera (<NUM>), the display being representative of the receiving vehicle (<NUM>) and of the harvesting material (<NUM>);
an operator interaction detection system (<NUM>) configured to detect an operator reposition command on the displayed receiving vehicle (<NUM>) on the display device (<NUM>) and identify a second spout position relative to the receiving vehicle (<NUM>) based on the operator reposition command;
a fill control system (<NUM>) comprising automatic fill control sensors (<NUM>) and an automatic fill control system (<NUM>) configured to execute an automatic fill operation according to an automatic fill strategy;
a reposition control signal generator (<NUM>) comprising a spout control signal generator configured to generate a spout actuator control signal to control a spout actuator (<NUM>) to move the spout (<NUM>) relative to a frame of the agricultural harvester (<NUM>) and/or a machine synchronization control signal generator (<NUM>) configured to automatically change a position of the agricultural harvester (<NUM>) relative to the receiving vehicle (<NUM>), the reposition control signal generator (<NUM>) configured to temporarily suspend the automatic fill operation automatically before generating a control signal to automatically move the spout (<NUM>) to the second spout position relative to the receiving vehicle (<NUM>);
characterized in that a resume detector (<NUM>) is configured to detect resume criteria and that the reposition control signal generator (<NUM>) is configured to automatically move the spout (<NUM>) back to the first spout position and to automatically resume the automatic fill operation based on detection of the resume criteria.