Patent Description:
There are a wide variety of different types of mobile work machines such as agricultural vehicles and construction vehicles. Some vehicles are material loading vehicles that include harvesters, such as forage harvesters, sugar cane harvesters, combine harvesters, and other harvesters, that harvest grain or other crop. Such harvesters often unload material into receiving vehicles that may include carts which may be pulled by tractors or semi-trailers as the harvesters are moving. Some construction vehicles include vehicles that remove asphalt or other similar materials. Such machines can include cold planers, asphalt mills, asphalt grinders, etc. Such construction vehicles often unload material into a receiving vehicle, such as a dump truck or other vehicle with a receiving vessel.

It is not uncommon for there to be multiple receiving vehicles for every loading vehicle. For instance, in an example in which the material loading vehicle is a self propelled forage harvester, there may be a single harvester harvesting a field, but multiple receiving vehicles that are operating with that harvester. As one receiving vehicle becomes full, it drives away from the harvester to an unloading location, while another receiving vehicle takes its place adjacent the harvester so the harvester can continue unloading to that second receiving vehicle. Still other operations may have multiple harvesters in a single field, multiple harvesters in multiple different fields, multiple receiving vehicles per harvester, and multiple unloading locations where the receiving vehicles unload the material they are carrying.

In order to assist the operator of the harvester, the overall fill level of material in the receiving vehicle can be detected in a variety of different ways. For example, some automatic fill control systems have been developed to automate portions of the filling process. One such automatic fill control system uses an image capture device, such as a stereo camera, on the spout of the harvester to capture an image (a static image or video image) of the receiving vehicle. An image processing system determines the fill level in order to automatically aim the spout toward empty spots and control the flap position to achieve a more even fill, while reducing spillage. Such systems can fill the receiving vehicle according to a fill strategy (such as front-to-back, back-to-front, etc.) that is set by the operator or that is set in other ways. Thus, the overall fill level of the receiving vehicle can be detected using the automatic fill control system.

Also, in some current systems, the image captured by the automatic fill control system is displayed to the operator of the harvester in an attempt to show the operator the fill level of the receiving vehicle. In some such current systems, the image is a live video image showing a portion of the receiving vehicle that is being filled with harvested material from the harvester.

Reference is made to the prior art of <CIT>, proposing to display an image of a crop receiving vehicle with overlaid fill state on a graphical user interface display and <CIT> according to which a crop receiving vehicle can be shown as a synthetic image on a user interface display. <CIT> describes a harvester with a camera looking on to transport vehicle receiving crop from the harvester and using a model for determining the crop distribution and fill level on the vehicle, while <CIT>, <CIT>, <CIT> and <CIT> use this model for displaying the fill state to the operator of the harvesting machine. In the prior art, the image from the camera is used by the model to derive the crop distribution and the fill state, such that complicated computing operations are needed. <CIT> uses pre-stored patterns of containers to identify the particular trailer type and corresponding contour pattern for spout control purposes, not for controlling the display.

The present discussion proceeds with respect to a material loading vehicle being an agricultural harvester, but it will be appreciated that the present discussion is also applicable to material loading systems in which the material loading vehicle is a construction machine or other material loading vehicle as well, such as those discussed elsewhere herein.

An automatic fill control system or another system on the harvester detects the fill level of the receiving vehicle that is currently being filled, as well as a distribution of the material within the receiving vehicle. A rendering is then generated. The rendering shows a representation of the receiving vehicle and a fill level display showing the fill level of material in the receiving vehicle and the distribution of material in the receiving vehicle. The rendering generation system comprises a pre-loaded rendering retrieval system configured to retrieve a pre-loaded rendering corresponding to the fill level and material distribution. The rendering can be displayed on the harvester. The rendering and/or data representing the rendering can be transmitted to the receiving vehicle. The rendering can then be displayed on a mobile device or other display device for the operator of the receiving vehicle. In one example, the fill level is indicated by a two-dimensional rendering or a three-dimensional rendering. Because the rendering is a computer-generated rendering instead of an image captured by a camera, the rendering is not dependent on the field of view of the camera and accurately shows the fill level and material distribution for the entire receiving vehicle even when there are obscurants in the air.

<FIG> is a pictorial illustration showing one example of a material loading vehicle, which is a self-propelled forage harvester <NUM>, followed by a receiving vehicle <NUM>. Receiving vehicle <NUM> includes tractor <NUM> pulling grain cart <NUM>. Cart <NUM> thus defines an interior that forms a receiving vessel <NUM> for receiving harvested material through a receiving area <NUM>. In the example shown in <FIG>, towing vehicle (e.g., a tractor) <NUM>, that is pulling grain cart <NUM>, is positioned directly behind forage harvester <NUM> and has a mobile device <NUM> which may be a smart phone, tablet computer, etc. either mounted in the operator compartment of tractor <NUM>, or carried by the operator of tractor <NUM>. Also, in the example illustrated in <FIG>, forage harvester <NUM> has a camera <NUM> mounted on the spout <NUM> through which the harvested material <NUM> is traveling. The spout <NUM> can be pivotally or rotationally mounted to a frame <NUM> of harvester <NUM>. Camera <NUM> can be a stereo-camera or a mono-camera that captures an image (e.g., a still image or video) of the receiving area <NUM> of cart <NUM>. In the example shown in <FIG>, the receiving area <NUM> is defined by an upper edge of the walls of cart <NUM>.

When harvester <NUM> has an automatic fill control system that includes image processing, as discussed above, the automatic fill control system can gauge the height of harvested material in cart <NUM>, and the location of that material. The automatic fill control system can also generate a metric indicative of a fill level of cart <NUM> and the distribution of material in cart <NUM> based on the dimensions of cart <NUM> and the sensed level of material in cart <NUM>. The automatic fill control system also identifies a fill level at the location (material landing point) in cart <NUM> where the material is currently loading. The automatic fill control system thus automatically controls the position of spout <NUM> and flap <NUM> to direct the trajectory of material <NUM> into the receiving area <NUM> of cart <NUM> to obtain an even fill throughout the entire length and width of cart <NUM>, while not overfilling cart <NUM>. By automatically, it is meant, for example, that the operation is performed without further human involvement except, perhaps, to initiate or authorize the operation.

For example, when executing a back-to-front automatic fill strategy the automatic fill control system may attempt to move the spout and flap so the material begins landing at a first landing point in the back of vessel <NUM>. Then, once a desired fill level is reached in the back of vessel <NUM>, the automatic fill control system moves the spout and flap so the material begins landing just forward of the first landing point in vessel <NUM>. This continues until the vessel <NUM> reaches a desired fill level and distribution. The fill level of vessel <NUM> and distribution can be compared to a desired fill level and distribution (or a fill level and distribution threshold) which may be a default fill level and distribution, an operator-input fill level and distribution, or another fill level and distribution. The fill level and distribution in vessel <NUM> can then be used to generate a rendering that shows a representation of vessel <NUM>, the fill level of material in vessel <NUM>, and the distribution of material in vessel <NUM>. The rendering is, unlike prior systems, not an image of vessel <NUM> captured by a camera. Instead, the rendering is a graphical rendering generated by a computer. This avoids the difficulties encountered when using a camera with a field of view that is too small to capture the entire vessel <NUM> or when using a camera in environments that are dusty or otherwise contain visual obscurants. The rendering, or data representing the rendering, can be output to mobile device <NUM> for display to the operator of receiving vehicle <NUM>, such as on a mobile application running on mobile device <NUM>. The rendering or data representing the rendering can be sent to other mobile devices in other receiving vehicles as well so the operators of the other receiving vehicles, can better decide where and when to position the receiving vehicles.

It can be seen in the example of <FIG> that the camera <NUM> can capture an image of a portion of the cart <NUM>. For instance, it can capture an image of the forward portion <NUM> of cart <NUM>. Thus, in one example, optical or visual features of that forward portion <NUM> of cart <NUM> can be used by an image processor to uniquely identify cart <NUM>, or to identify the type of the cart <NUM>. A unique cart identifier, or type identifier, can be used to automatically identify a size of cart <NUM> or a model that identifies the dimensions and/or shape of the cart <NUM> which may be set manually or downloaded from a manufacturer database or obtained in other ways. The identity of cart <NUM> can also be used to access pre-loaded images as well. The fill level and distribution detected by harvester <NUM> can also be correlated to a specific receiving vehicle <NUM> using the identifier identifying the receiving vehicle.

<FIG> is a pictorial illustration showing another example of a self-propelled forage harvester <NUM>, this time loading a receiving vehicle <NUM> that includes a semi-tractor <NUM>, a semi-trailer (or receiving vessel) <NUM> in a configuration in which the semi-tractor <NUM> (that also has a mobile device <NUM>) is pulling semi-trailer <NUM> alongside forage harvester <NUM>. Therefore, the spout <NUM> and flap <NUM> are positioned to unload the harvested material <NUM> to fill trailer <NUM> according to a pre-defined side-by-side fill strategy. Again, <FIG> shows that camera <NUM> can capture an image (which can include a still image or video) of a portion of semi-trailer <NUM>. In some examples, the field of view of camera <NUM> cannot capture the entire semi-trailer <NUM>. In the example illustrated in <FIG>, the field of view of camera <NUM> is directed toward the front portion of the receiving area <NUM> of trailer <NUM> so that image processing can be performed to identify a landing point for the harvested material in trailer <NUM> and the height of material in trailer <NUM>. An automatic fill control system can thus determine the fill level of trailer <NUM> and the distribution of material in trailer <NUM>. The automatic fill control system on harvester <NUM> can also control spout <NUM> and flap <NUM> to fill trailer <NUM> as desired. Also, the fill level and distribution can be used to generate a graphic rendering of trailer <NUM> showing the fill level and material distribution. In one example, a graphic rendering is a rendering that shows a representation of the receiving vessel, the fill level, and the material distribution, other than a captured image that is captured during the harvesting operation. Thus, the rendering can clearly depict the fill level and material distribution in the receiving vehicle regardless of the field of view of camera or visual obscurants.

Also, in the example shown in <FIG>, it can be seen that camera <NUM> can be positioned to have a field of view that captures an image of a side portion <NUM> of trailer <NUM>. Thus, the visual or optical features of the side portion <NUM> of trailer <NUM> can be used to uniquely identify trailer <NUM>, or at least to identify the type of the trailer <NUM>. Based on the unique trailer identifier or the type identifier, the settings values for the automatic fill control system can be obtained (such as the dimensions of trailer <NUM>, the desired fill pattern, the desired fill level, the desired material distribution, etc.) so that the trailer <NUM> is filled in a trailer-specific way or in a trailer type-specific way, depending upon whether the trailer is uniquely identified or the trailer type is identified. For example, once the trailer or trailer type is identified, the desired fill level and material distribution for the trailer <NUM> can be retrieved and compared against the current fill level and material distribution.

In other examples, where machine <NUM> is a combine harvester, it may be that the spout <NUM> is not moved relative to the frame during normal unloading operations. Instead, the relative position of the receiving vehicle <NUM>, <NUM> and the combine harvester is changed in order to fill the receiving vehicle <NUM>, <NUM> as desired. Thus, if a front-to-back fill strategy is to be employed, then the relative position of the receiving vessel, relative to the combine harvester, is changed so that the spout is first filling the receiving vessel at the front end, and then gradually fills the receiving vessel moving rearward. This is just one example.

<FIG> is a pictorial illustration showing one example of an operator interface display <NUM> that can be displayed on a display mechanism <NUM>, for the operator in an operator compartment of forage harvester <NUM>. The display (or a portion of it), or the information used to generate the display, can also be sent to the mobile device <NUM> for use by the operator of the receiving vehicle <NUM>, <NUM>. The operator interface display <NUM> in <FIG> shows a view of images (static or video) captured by camera <NUM>. The image(s) show material <NUM> entering trailer <NUM>. The camera field view of camera <NUM> is not large enough to capture the entire trailer <NUM>. An image processing system on harvester <NUM> illustratively identifies the perimeter of a portion of the opening <NUM> in trailer <NUM> and also processes the image of the material <NUM> in trailer <NUM> to determine the fill height relative to opening <NUM> and the overall fill level and distribution of material in trailer <NUM>. The perimeter-defining opening <NUM> can be visually enhanced by overlaying a visual overlay over the opening <NUM> so that the operator can easily identify the opening <NUM>, as it is being recognized by the image processing system.

<FIG> also shows that, once the fill level and distribution of material in trailer <NUM> is detected and calculated, a fill level and distribution indicator <NUM> can be displayed and dynamically updated as trailer <NUM> is filled. In the example shown in <FIG>, the fill level and distribution indicator <NUM> is generated as a representation of a side view of the entire trailer <NUM> with contour indicator <NUM> that is visually updated as the trailer <NUM> is filled to indicate the fill level and distribution of material in trailer <NUM>. Also, the fill level and distribution trailer <NUM> can be compared to a threshold fill level and material distribution. Once the fill level and distribution in trailer <NUM> reaches the threshold fill level and material distribution in trailer <NUM>, then this can be indicated by the fill level and material distribution indicator <NUM> by changing the color of indicator <NUM>, blinking indicator <NUM>, or in another visual way.

It will be noted that fill level and material distribution indicator <NUM> is a display element generated by a computer system (discussed in greater detail below). Also, the fill level and material distribution can be detected using a LIDAR detection system or any other detector that detects the fill level and material distribution in the receiving vehicle. Therefore, even if the environment is dusty or otherwise contains obscurants, indicator <NUM> remains clearly visible. Similarly, even though the entire trailer <NUM> cannot be seen within the field of view of camera <NUM>, the indicator <NUM> shows the fill level and material distribution in the entire trailer <NUM>. This enhances the ability of the operator to make accurate decisions.

It should also be noted that, in an example in which forage harvester <NUM> has an automatic fill control system (or active fill control system) which fills trailer <NUM> according to a fill strategy (such as a back-to-front fill strategy, front-to-back fill strategy, etc.), or in other examples, a current landing position indicator (such as indicator <NUM>) may be displayed to show the current landing position where material <NUM> is being loaded into trailer <NUM> through spout <NUM> and the direction that spout <NUM> is, or should be, moving relative to trailer <NUM> as the filling operation continues. It can be seen in <FIG>, for instance, that indicator <NUM> is an arrow pointing in the front-to-back direction. The location of arrow <NUM> on the representation of trailer <NUM> indicates the current landing position, while the direction of the arrow <NUM> indicates the direction that spout <NUM> will be moved relative to trailer <NUM> in executing the selected front-to-back fill strategy.

<FIG> is a block diagram of one example of a material loading system which comprises agricultural system <NUM>. Agricultural system <NUM> may be on harvester <NUM> or on the receiving vehicle receiving material from harvester <NUM>, or in other locations, or dispersed among a variety of different locations. In the example described with respect to <FIG>, agricultural system <NUM> is deployed on harvester <NUM>. In the example shown in <FIG>, operator <NUM> can interact with agricultural system <NUM> in order to control and manipulate some of the items on agricultural system <NUM>. Agricultural system <NUM> can also communicate with other vehicles <NUM> and/or other systems <NUM> over network <NUM>. Therefore, network <NUM> may be a cellular network, a near field communication network, a wide area network, a local area network, or any of a wide variety of other networks or combinations of networks.

Other vehicles <NUM> can be other harvesters, other receiving vehicles, fuel trucks, or any of a wide variety of other vehicles. Other systems <NUM> may be farm manager systems, vendor systems, manufacturer systems, or other systems.

In the example shown in <FIG>, agricultural system <NUM> includes one or more processors or servers <NUM>, data store <NUM> (which can include pre-loaded renderings <NUM>, vehicle models <NUM>, and other items <NUM>), sensors <NUM> (which may include camera <NUM> and other sensors <NUM>), operator interface mechanisms <NUM>, fill level detection system <NUM>, rendering generation system <NUM>, communication system <NUM>, and other agricultural system functionality <NUM>. Fill level detection system <NUM> can include trigger detector <NUM>, receiving vehicle identifier <NUM>, image processing system <NUM>, fill level array generator <NUM>, other sensor processing systems <NUM>, and other items <NUM>. Rendering generation system <NUM> includes pre-loaded rendering retrieval system <NUM>, multiple view generator <NUM>, user interaction mechanism generator <NUM>, real-time rendering generator <NUM>, rendering output system <NUM>, and other rendering system functionality <NUM>. Real-time rendering generator <NUM> can include heat map generator <NUM>, photogrammetry system <NUM>, curve fitting system <NUM>, model population system <NUM>, smooth surface generator <NUM>, generic shape generator <NUM>, and other items <NUM>. Before describing the overall operation of agricultural system <NUM> in more detail, a brief description of some of the items in agricultural system <NUM>, and their operation, will first be provided.

Sensors <NUM> illustratively generate sensor signals indicative of the fill level of material <NUM> in the receiving vehicle and the distribution of the material <NUM> throughout the receiving vehicle. Therefore, in one example, sensors <NUM> can include camera <NUM> that captures an image (either a static image or a video) of a receiving vehicle. Sensors <NUM> can include other sensors <NUM> such as LIDAR-based sensors or other sensors that can sense the fill level of material <NUM> throughout the receiving vehicle (or at different points within the receiving vehicle) so that the material distribution within the receiving vehicle may be determined or estimated.

Fill level detection system <NUM> detects the fill level within the receiving vehicle and the distribution of material <NUM> within the receiving vehicle based upon the sensor signals from sensors <NUM>.

Trigger detector <NUM> detects a trigger indicating that system <NUM> is to detect the fill level and material distribution in the receiving vehicle. The trigger criteria detected by detector <NUM> may be time-based criteria. For instance, system <NUM> may detect the fill level and material distribution continuously or intermittently (such as periodically) The trigger criteria can be other criteria as well such as criteria based upon changes in the fill level or other criteria). Receiving vehicle identifier <NUM> can identify the particular receiving vehicle based upon inputs from sensors <NUM>. Image processing system <NUM> may be a computing system that processes the image captured by camera <NUM>. The image processing system <NUM> can process the image to identify the fill level of material <NUM> at different points within the receiving vehicle so that the material distribution can be determined or estimated based upon the fill levels at the different points within the receiving vehicle. Fill level array generator <NUM> may be used to generate one or more arrays of fill levels at different points within the receiving vehicle. Other sensor processing system <NUM> can process other sensor inputs, such as LIDAR sensor inputs, or other inputs. Fill level detection and material distribution detection system <NUM> then generates an output indicative of the fill level of material <NUM> within the receiving vehicle and indicative of the distribution of that material within the receiving vehicle. The output may be an array of fill level values that are correlated to different points within the receiving vehicle or the output from system <NUM> may take other forms as well.

Rendering generation system <NUM> receives the output from fill level and material distribution detection system <NUM> and generates a rendering that can be displayed to the operator of harvester <NUM>, to the operator of the receiving vehicle, or elsewhere. The rendering is illustratively generated by system <NUM> (which can be a computing system) instead of simply outputting the image captured by camera <NUM>. Pre-loaded rendering retrieval system <NUM> receives the fill level and material distribution output by system <NUM> and access data store <NUM> to obtain a pre-loaded rendering <NUM> corresponding to the detected fill level and material distribution. In one example, the pre-loaded renderings <NUM> are stored for different receiving vehicles and the identity of the receiving vehicle, output by receiving vehicle identifier <NUM>, can be used by pre-loaded rendering retrieval system <NUM> to retrieve the appropriate pre-loaded rendering <NUM>.

Multiple view generator <NUM> may generate multiple views showing the fill level and material distribution in the receiving vehicle. User interaction mechanism generator <NUM> can generate user interaction mechanisms (such as icons, buttons, links, menus, etc.) on the fill level and material distribution indicator so that the operator <NUM> can interact with the indicator. For instance, the user may be able to actuate a displayed actuator to magnify the fill level and material distribution indicator to see additional details about the fill level or material distribution.

Real-time rendering generator <NUM> may generate a real-time rendering, instead of accessing a pre-loaded rendering <NUM>. Real-time rendering generator <NUM> may receive the identity of the receiving vehicle from system <NUM> and access a vehicle model <NUM> which defines the dimensions of the particular receiving vehicle that is currently being processed. The real-time rendering generator <NUM> may generate any of a wide variety of different types of real-time renderings that vary dynamically, as the receiving vehicle is being filled. Heat map generator <NUM> can generate the rendering as a heat map showing a depiction of the receiving vehicle along with values, colors, or other visual indicia indicating the fill level at different places in the receiving vehicle. Photogrammetry system <NUM> can generate a three-dimensional (3D) representation of the receiving vehicle, showing a representation of material in the 3D representation of the receiving vehicle, and also showing a distribution of that material within the 3D representation of the receiving vehicle. Curve fitting system <NUM> may receive array values from fill level array generator <NUM> and fit lines and planes or curves to different array values to thereby generate a smooth contoured surface indicative of the surface of the material within the receiving vehicle. Model population system <NUM> can access the vehicle model <NUM> corresponding to the identified receiving vehicle and generate an image of the receiving vehicle and populate the image with a depiction of the material based upon the detected fill level and material distribution within the receiving vehicle. Smooth surface generator <NUM> can generate a smooth material surface showing how the material is distributed within the receiving vehicle, and generic shape generator <NUM> can generate a generic shape corresponding to the receiving vehicle and provide an indicator showing the fill level and distribution of material on the generic shape of the receiving vehicle being rendered.

Rendering output system <NUM> generates an output indicative of the rendering. The output can be data representing the rendering or data upon which the rendering was generated. The output can also be the rendering itself.

Operator interface mechanisms <NUM> can include any of a wide variety of operator interface mechanisms that operator <NUM> can use to interact with agricultural system <NUM>. Therefore, operator interface mechanisms <NUM> can include pedals, a steering wheel, joysticks, levers, buttons, knobs, keypads, keyboards, dials, a display screen, a touch sensitive display screen, lights, vibrating mechanisms, a speaker, a microphone where speech recognition and speech synthesis are provided, and any of a wide variety of other audio, visual, or haptic devices. Similarly, where a display screen is provided, user actuatable elements can be displayed on the display screen and actuated by operator <NUM>. Those user actuatable elements can be actuated using a touch gesture on a touch sensitive display or using a point and click device or other device. The user actuatable elements can include links, icons, buttons, meus, etc..

Communication system <NUM> illustratively facilitates communication among the various items on agricultural system <NUM> and communication with other vehicles <NUM> and other systems <NUM> over network <NUM>. Therefore, communication system <NUM> may include a controller area network - CAN - bus and bus controller, a cellular communication system, a near field communication system, a wide area network communication system, a local area network communication system, or any of a wide variety of other communication systems or combinations of communication systems.

<FIG> is a flow diagram illustrating one example of the operation of agricultural system <NUM> in detecting a fill level and material distribution in a receiving vehicle and generating a rendering of the receiving vehicle showing the fill level and material distribution in the receiving vehicle. It is first assumed that a machine filling operation is being performed with a machine loading vehicle (such as harvester <NUM>) and a receiving vehicle (such as receiving vehicle <NUM>), as indicated by block <NUM> in the flow diagram of <FIG>. Fill level and material distribution detection system <NUM> then detects a fill level and material distribution in the receiving vehicle, as indicated by block <NUM>. The fill level can be based on sensor inputs from sensors <NUM>, such as from a stereo camera <NUM>, a LIDAR sensor <NUM>, or a combination of different sensors <NUM>. The fill level detection can be triggered when trigger detector <NUM> detects a trigger. The trigger detector <NUM> can detect a trigger to detect the fill level and material distribution based on a wide variety of different trigger criteria. For instance, a trigger may be a detection frequency which is fixed or periodic. Every time the detection period lapses, then trigger detector <NUM> determines that fill level and material distribution detection system <NUM> is to perform a detection. Detection at a fixed frequency is indicated by block <NUM>. The detection frequency can also be variable based on other criteria, such as how closely the fill level is to a fill level threshold for the receiving vehicle. As the receiving vehicle gets closer to its threshold fill level, it may be that the fill level and material distribution detections performed by system <NUM> increase in frequency. Detecting based on a variable frequency is indicated by block <NUM>. The fill level and material distribution can be detected using image processing system <NUM>, using other sensor processing system(s)<NUM>, <NUM>, or in a wide variety of other ways, as indicated by block <NUM> in the flow diagram of <FIG>.

Rendering generation system <NUM> then generates a computer-generated rendering of the receiving vehicle showing the fill level and the material distribution in the receiving vehicle, as indicated by block <NUM> in the flow diagram of <FIG>. Pre-loaded rendering retrieval system <NUM> obtains the fill level and material distribution from system <NUM> as well as the identity of the receiving vehicle from receiving vehicle identifier <NUM>. Using the fill level, material distribution, and receiving vehicle identity, system <NUM> accesses pre-loaded renderings <NUM> to obtain a rendering that is indicative of the detected fill level and material distribution in the specific receiving vehicle that was identified. In another example, real-time rendering generator <NUM> can generate a real-time rendering based upon the detected fill level and material distribution and/or the detected vehicle identity. Generating the computer-generated rendering using pre-loaded or real-time generation is indicated by block <NUM> in the flow diagram of <FIG>. The rendering can be a two-dimensional rendering as indicated by block <NUM> or a three-dimensional rendering as indicated by block <NUM>. The rendering can be an orthogonal view <NUM> or a grid-based view <NUM>. The rendering may be a heat map <NUM> or the rendering can be one of a plurality of different user-selectable renderings as indicated by block <NUM>. Also, rendering generation system <NUM> can generate the rendering showing multiple views of the receiving vehicle and the fill level and material distribution, simultaneously, as indicated by block <NUM>. The computer-generated rendering can take a variety of other forms and can be rendered in other ways as well, as indicated by block <NUM>.

The rendering output system <NUM> then generates a control signal to output the rendering. For instance, the control signal can control a display device in operator interface mechanism <NUM> to display the computer-generated rendering, as indicated by block <NUM>. Rendering output system <NUM> can generate a control signal to control communication system <NUM> to send a representation of the rendering to the receiving vehicle or other systems or vehicles.

<FIG> show some examples of renderings that can be displayed on display device <NUM>. <FIG> shows a two-dimensional rendering <NUM> that has a representation <NUM> of the receiving vehicle along with a fill level and material distribution indicator <NUM>. Indicator <NUM> shows the fill level and how the material is distributed along the length of receiving vehicle <NUM>. In the example shown in <FIG>, the rendering <NUM> also includes the indicator <NUM> indicating a current position of receiving vehicle <NUM> that is being filled and the direction that the fill operation is proceeding relative to receiving vehicle <NUM>. The rendering <NUM> is a pre-loaded rendering <NUM> that is retrieved by pre-loaded rendering retrieval system <NUM>. In another not-claimed example, rendering <NUM> may be a real-time rendering in which fill level array generator <NUM> has generated an array of fill level values that are plotted on the representation <NUM> of the receiving vehicle and where curve fitting system <NUM> fits a curve corresponding to the level indicator <NUM> to the fill level values plotted on the representation of the receiving vehicle <NUM>. In another example, real-time rendering generator <NUM> can generate rendering <NUM> in other ways as well. User interaction mechanism generator <NUM> can also generate a user interaction input mechanism <NUM> which can be actuated by operator <NUM> to zoom in or zoom out of rendering <NUM>.

<FIG> shows another example of a computer-generated rendering <NUM>. Rendering <NUM> includes an outline <NUM> representing the perimeter of the receiving vehicle. The perimeter <NUM> is broken into cells, where each cell corresponds to a position in the receiving vehicle and has a fill value (in the form of a numeric value) indicating the fill level of material in the receiving vehicle in a location corresponding to that cell. For instance, cell <NUM> has a fill level indicator in the form of the number <NUM>. Cell <NUM> has a fill level indicator in the form of the number <NUM>. Similarly, cell <NUM> has a fill level indicator comprising a numeric value of <NUM>. In the example shown in <FIG>, the fill level detected in the receiving vehicle is similar to that displayed in <FIG>. The numeric values in each of the cells within the periphery <NUM> illustrate the fill level of material in the receiving vehicle and the location of those numbers in the particular cells of the grid structure shown in <FIG> indicate the material distribution within the receiving vehicle. For instance, those cells having a numeric value of <NUM> indicate a relatively low fill level that is low relative to a fill level threshold for the receiving vehicle. Those cells that have a numeric value of <NUM> indicate a relatively high fill level, one that meets or exceeds the fill level threshold corresponding to the receiving vehicle.

It will also be noted that in the example shown in <FIG>, instead of displaying numerical values in each of the cells, a color, shading, or other visual representation can be displayed in each cell and can correspond to the fill level of the material in that cell. Thus, the rendering <NUM> may be displayed as a heat map or other color-coded representation where the color of each cell corresponds to the material fill level in that cell.

<FIG> shows another example of a computer-generated rendering <NUM> that has an orthogonal or three-dimensional representation <NUM> of the receiving vehicle with a fill level and material distribution indicator <NUM> disposed on the representation <NUM> of the receiving vehicle. Rendering <NUM> represents an orthogonal or three-dimensional view of the receiving vehicle while indicator <NUM> illustrates a smooth surface corresponding to the fill level of the material in the receiving vehicle. The smooth surface can be generated in a wide variety of different ways. Smooth surface generator <NUM> can invoke a model that receives the array of data fill levels in the receiving vehicle and construct a smooth surface based on that data. In another example, photogrammetry system <NUM> can perform photogrammetry on the images captured by camera <NUM> or other sensors to generate the representation as a smooth surface.

<FIG> shows an example in which the computer-generated rendering <NUM> includes a plurality of separate renderings. The first rendering is rendering <NUM> shown and described above with respect to <FIG>. The second rendering is a bar graph rendering <NUM> which has a set of bar graphs that each correspond to a different portion of the receiving vehicle representation <NUM>. The height of the bar graph corresponds to the fill level of the material at a corresponding position in the receiving vehicle, and the location of the bars in the bar graph indicates the distribution of the material in the receiving vehicle. Multiple view generator <NUM> can generate multiple renderings as shown in <FIG> and juxtapose them relative to one another on the display device. The example of the multiple views or multiple renderings <NUM> and <NUM> is just one example and the multiple renderings could include three or more renderings, two-dimensional and three-dimensional renderings, grid-based renderings as well as other renderings.

Returning again to the flow diagram of <FIG>, after the computer-generated rendering is displayed, the display can be updated when fill level and material distribution detection system <NUM> detects a new fill level. Thus, if the filling operation for this receiving vehicle is not complete, as indicated by block <NUM> in the flow diagram of <FIG>, then processing reverts to block <NUM> where the fill level and distribution is again detected. If, at block <NUM>, the filling operation for this receiving vehicle is completed, the processing continues at block <NUM> where agricultural system <NUM> can perform any post fill operations, such as storing the fill level and material distribution either locally, such as in data store <NUM> or elsewhere. Storing the fill level and material distribution is indicated by block <NUM>.

Communication system <NUM> can also send the fill level and material distribution to other vehicle(s) <NUM> or other system(s) <NUM>, as indicated by block <NUM> in the flow diagram of <FIG>. System <NUM> can perform any of a wide variety of other post-fill operations as well, as indicated by block <NUM>.

<FIG> is a flow diagram illustrating one example of detecting a fill level and material distribution in a receiving vehicle by detecting the fill level at a plurality of different points in the receiving vehicle and generating a data array indicative of fill level values at the different points. In one example, fill level and material distribution detection system <NUM> first detects the fill level values at different points in a grid that corresponds to the receiving vehicle. For instance, receiving vehicle identifier <NUM> can identify the particular receiving vehicle and fill level array generator <NUM> can divide the area of the receiving vehicle into a grid of cells. The image processing system <NUM> or other sensor processing system <NUM> then detects the fill level in each grid of the array based on the sensor signal and outputs a fill level value indicative of the fill level in each grid of the array. Fill level array generator <NUM> then generates an array of those fill level values and provides the array to rendering generation system <NUM>. Detecting the fill level values at different points in an array of grid of cells is indicated by block <NUM> in the flow diagram of <FIG>. Rendering generation system <NUM> then generates the rendering based upon the fill level values, as indicated by block <NUM>.

In one example, heat map generator <NUM> generates a heat map using the values corresponding to each of the grid cell, as indicated by block <NUM>. In another example, photogrammetry system <NUM> uses photogrammetry to generate a three-dimensional reconstruction of the receiving vehicle showing the fill level and distribution of material, such as that shown in <FIG>, as indicated by block <NUM> in the flow diagram of <FIG>. Curve fitting system <NUM> can fit lines or curves to the values in the grid sections, as indicated by block <NUM>. Model population system <NUM> can obtain a model of the receiving vehicle, generate a representation of the receiving vehicle based on the model, and populate the representation of the receiving vehicles with a fill level indicator based upon the array of fill level values for the grid sections, as indicated by block <NUM>.

Smooth surface generator <NUM> can generate a smooth surface representing the surface of material distributed in the receiving vehicle, as indicated by block <NUM>. Generic shape generator <NUM> can generate a generic shape rendering corresponding to the receiving vehicle and populate that rendering based upon the fill level values, as indicated by block <NUM>. The rendering can be generated based upon the fill level values in other ways as well, as indicated by block <NUM>.

It can thus be seen that a computer-generated rendering of the receiving vehicle and the fill level and material distribution within the receiving vehicle can be generated based upon the detection of the fill level and distribution of the material in the receiving vehicle. The computer-generated rendering can be generated to clearly display the fill level and material distribution regardless of the environment of the receiving vehicle, such as whether it is dusty or has other visual obscurants around it. Similarly, the rendering of the entire receiving vehicle can be generated even where a field of view of a camera does not capture the entire receiving vehicle.

<FIG> is a block diagram of harvesters and receiving vehicles and other vehicles shown in <FIG>, except that they communicate with elements in a remote server architecture <NUM>. In one example, remote server architecture <NUM> can provide computation, software, data access, and storage services that do not require end-user knowledge of the physical location or configuration of the system that delivers the services. In various examples, remote servers can deliver the services over a wide area network, such as the internet, using appropriate protocols. For instance, remote servers can deliver applications over a wide area network and they can be accessed through a web browser or any other computing component. Software or components shown in <FIG> as well as the corresponding data, can be stored on servers at a remote location. The computing resources in a remote server environment can be consolidated at a remote data center location or they can be dispersed. Remote server infrastructures can deliver services through shared data centers, even though they appear as a single point of access for the user. Thus, the components and functionality described herein can be provided from a remote server at a remote location using a remote server architecture. Alternatively, the components and functionality can be provided from a conventional server, or they can be installed on client devices directly, or provided in other ways.

In the example shown in <FIG>, some items are similar to those shown in <FIG> and they are similarly numbered. <FIG> specifically shows that other systems <NUM> and rendering generation system <NUM> can be located at a remote server location <NUM>. Therefore, the harvester accesses those systems through remote server location <NUM>. Other portions of agricultural systems <NUM> can be located in remote server location <NUM> or elsewhere and the block diagram of <FIG> is just one example.

<FIG> also depicts another example of a remote server architecture. <FIG> shows that it is also contemplated that some elements of <FIG> are disposed at remote server location <NUM> while others are not. By way of example, data store <NUM> and/or other systems <NUM> can be disposed at a location separate from location <NUM>, and accessed through the remote server at location <NUM>. Regardless of where they are located, the items can be accessed directly by through a network (either a wide area network or a local area network), the items can be hosted at a remote site by a service, or they can be provided as a service, or accessed by a connection service that resides in a remote location. Also, the data can be stored in substantially any location and intermittently accessed by, or forwarded to, interested parties. For instance, physical carriers can be used instead of, or in addition to, electromagnetic wave carriers. In such an example, where cell coverage is poor or nonexistent, another mobile machine (such as a fuel truck) can have an automated information collection system. As a harvester or receiving vehicle comes close to the fuel truck for fueling, the system automatically collects the information from the harvester or other vehicle and transfers information to the harvester or receiving vehicle using any type of ad-hoc wireless connection. The collected information can then be forwarded to the main network as the fuel truck reaches a location where there is cellular coverage (or other wireless coverage). For instance, the fuel truck may enter a covered location when traveling to fuel other machines or when at a main fuel storage location. All of these architectures are contemplated herein. Further, the information can be stored on the harvester or receiving vehicle until the harvester or receiving vehicle enters a covered location. The harvester or receiving vehicle, itself, can then send the information to the main network.

It will also be noted that the elements of <FIG>, or portions of them, can be disposed on a wide variety of different devices. Some of those devices include servers, desktop computers, laptop computers, tablet computers, or other mobile devices, such as palm top computers, cell phones, smart phones, multimedia players, personal digital assistants, etc..

<FIG> is a simplified block diagram of one illustrative example of a handheld or mobile computing device that can be used as a user's or client's hand held device <NUM>, in which the present system (or parts of it) can be deployed. For instance, a mobile device can be deployed in the operator compartment of a harvester and/or as mobile device <NUM> in a receiving vehicle for use in generating, processing, or displaying the fill levels and material distributions. <FIG> are examples of handheld or mobile devices.

<FIG> provides a general block diagram of the components of a client device <NUM> that can run some components shown in previous FIGS. , that interacts with them, or both. In the device <NUM>, a communications link <NUM> is provided that allows the handheld device to communicate with other computing devices and in some examples provides a channel for receiving information automatically, such as by scanning. Examples of communications link <NUM> include allowing communication though one or more communication protocols, such as wireless services used to provide cellular access to a network, as well as protocols that provide local wireless connections to networks.

In other examples, applications can be received on a removable Secure Digital (SD) card that is connected to an interface <NUM>. Interface <NUM> and communication links <NUM> communicate with a processor <NUM> (which can also embody processors or servers from previous FIGS. ) along a bus <NUM> that is also connected to memory <NUM> and input/output (I/O) components <NUM>, as well as clock <NUM> and location system <NUM>.

I/O components <NUM>, in one example, are provided to facilitate input and output operations. I/O components <NUM> for various examples of the device <NUM> can include input components such as buttons, touch sensors, optical sensors, microphones, touch screens, proximity sensors, accelerometers, orientation sensors and output components such as a display device, a speaker, and or a printer port. Other I/O components <NUM> can be used as well.

System <NUM> can include, for instance, a global positioning system (GPS) receiver, a LORAN system, a dead reckoning system, a cellular triangulation system, or other positioning system. System <NUM> can also include, for example, mapping software or navigation software that generates desired maps, navigation routes and other geographic functions.

Memory <NUM> stores operating system <NUM>, network settings <NUM>, applications <NUM>, application configuration settings <NUM>, data store <NUM>, communication drivers <NUM>, and communication configuration settings <NUM>. Memory <NUM> can include all types of tangible volatile and non-volatile computer-readable memory devices. It can also include computer storage media (described below). Memory <NUM> stores computer readable instructions that, when executed by processor <NUM>, cause the processor to perform computer-implemented steps or functions according to the instructions. Processor <NUM> can be activated by other components to facilitate their functionality as well.

<FIG> shows one example in which device <NUM> is a tablet computer <NUM>. In <FIG>, computer <NUM> is shown with user interface display screen <NUM>. Screen <NUM> can be a touch screen or a pen-enabled interface that receives inputs from a pen or stylus. It can also use an on-screen virtual keyboard. Of course, computer <NUM> might also be attached to a keyboard or other user input device through a suitable attachment mechanism, such as a wireless link or USB port, for instance. Computer <NUM> can also illustratively receive voice inputs as well.

<FIG> is one example of a computing environment in which elements of previous FIGS. , or parts of them, (for example) can be deployed. With reference to <FIG>, an example system for implementing some embodiments includes a computing device in the form of a computer <NUM> programmed to operate as discussed above. Components of computer <NUM> may include, but are not limited to, a processing unit <NUM> (which can comprise a processor or server from previous FIGS. ), a system memory <NUM>, and a system bus <NUM> that couples various system components including the system memory to the processing unit <NUM>. The system bus <NUM> may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. Memory and programs described with respect to previous FIGS. can be deployed in corresponding portions of <FIG>.

Computer storage media includes hardware storage media including both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The computer <NUM> is operated in a networked environment using logical connections (such as a controller area network - CAN, local area network - LAN, or wide area network WAN) to one or more remote computers, such as a remote computer <NUM>.

Claim 1:
A material filling system (<NUM>), comprising:
a sensor (<NUM>) configured to detect material being loaded into a receiving vehicle (<NUM>) and to generate a sensor signal indicative of the detected material;
a fill level and material distribution detection system (<NUM>) configured to identify, based on the sensor signal, a fill level and material distribution of the detected material on the receiving vehicle (<NUM>) and to generate a fill level and material distribution signal;
a rendering generation system (<NUM>) configured to generate a computer-generated rendering of the receiving vehicle (<NUM>) showing the fill level and material distribution based on the fill level and material distribution signal; and
a rendering output system (<NUM>) configured to generate a display control signal to control a display mechanism (<NUM>) to display the computer-generated rendering;
wherein the rendering generation system comprises a pre-loaded rendering retrieval system;
characterized in that the pre-loaded rendering retrieval system (<NUM>) is configured to retrieve a pre-loaded rendering corresponding to the fill level and material distribution.