Patent Description:
Agricultural balers are used to consolidate and package crop material so as to facilitate the storage and handling of the crop material for later use. For example, when the crop is hay a mower-conditioner is typically used to cut and condition the crop material for windrow drying in the sun. As another example, when the crop is straw a combine harvester discharges non-grain crop material from the rear of the harvester defining the straw which is to be picked up by the baler. The cut crop material is usually dried, and a baler, such as a large square baler or round baler, straddles the windrows and travels along the windrows to pick up the crop material and form it into bales.

A large square baler typically comprise two main parts used in the formation of the bales, being a pre-compression chamber and a bale chamber. Crop material is gathered and pushed into the pre-compression chamber, where a slice of crop material is formed. The pre-compression chamber is linked to the bale chamber in such a manner that the slice of crop material is periodically transferred into the bale chamber. In the bale chamber, a plunger reciprocally moves, thereby pressing a square bale from subsequently fed slices.

When the bale is completed, a piece of twine is wrapped around the bale and tied to keep the bale together. While a new bale is formed in the bale chamber, the completed bale is pushed to the rear of the bale chamber by this new bale. Eventually, the completed bale is pushed through an outlet at the rear of the bale chamber and dropped on the field behind the agricultural baler.

In practice, not all bales come out the same. A length of the bale is determined by the number of slices that is used to form a bale, but also influenced by, e.g., crop type, crop humidity, and the filling level of the pre-compression chamber when the crop slices are transferred into the bale chamber. If the filling of the pre-compression chamber varies per slice and/or if the gathered crop is not evenly distributed over the width of the pre-compression chamber, the shape of the resulting bale may deviate from a perfect rectangular box.

Different systems for measuring the length of a square bale have been described and used in the past. Commonly, a measuring wheel is configured to rotate when the bale is pushed through the bale chamber and a sensor measures the rotation of the measuring wheel. Alternatively, a sensor may measure the amount of twine that is needed to wrap the completed bale and a bale length is calculated based on the required amount of twine. In, e.g., <CIT>, an optical sensor is mounted to the bale chamber wall for measuring the movement of the bale through the bale chamber and for deriving the total bale length therefrom. All these known methods for determining the bale length have their own inaccuracies and share the disadvantage that the bale length measurement is based on measurements made while the bale is still being formed and its size and shape are at least partly confined by the external pressure of the bale chamber walls.

<CIT> describes a tractor and baler combination. The round baler is foreseen with a bale size sensor associated with the baling chamber and an electronic baler controller. The baler controller is able top submit a halt signal to the tractor controller when the bale is of an appropriate size which is detected by the bale size sensor. The tractor controller then halts the tractor and the bale will be wrapped. Once wrapped, the rear door is opened automatically, but only if an obstacle sensor detects that no obstacle is present in the detection zone behind the baler.

<CIT> also discloses a tractor with a round baler and is directed to a system and method for controlling bale forming operations. The method includes a monitoring system for visually imaging a portion of a bale and providing a warning to an operator if the bale is wrapped or shaped incorrectly.

It is an aim of the present invention to address one or more disadvantages associated with the prior art.

According to an aspect of the invention there is provided a new method for monitoring bale shape. This new method comprises the steps of receiving a series of bale images from a camera, identifying the bale in the bale images, determining at least one bale shape parameter of the identified bale, and providing an electronic signal representative of the bale shape parameter. The bale images comprise a view of at least an outlet of a bale chamber of an agricultural baler, of a bale being ejected from the outlet, and of a field travelled by the agricultural baler during the ejection of the bale. The at least one bale shape parameter of the identified bale is then determined based on at least one of the bale images.

The method according to the invention provides for an easy way to determine the bale length when the bale has already been ejected from the bale chamber, just before it is dropped on the field. By including the view of at least the outlet of the bale chamber in the bale images, it is made possible to use the known dimensions and orientations of parts of the agricultural baler as a reference for determining the bale shape parameters that are derived from the captured images. When capturing the bale images during and just after the ejection of the bale from the outlet, the bale images show images of the bale when free of external pressure from the bale chamber walls. The bale can thus obtain its eventual shape before the at least one bale shape parameter is determined. This allows for a more accurate measurement of relevant bale shape parameters that better reflects the shape and dimensions of the bale as it is eventually left behind on the field. In addition to more accurate information about the produced bales, this allows for improved control of the operation of the agricultural baler in dependence of the observed shape of the bales that are produced.

Standard image processing techniques for identifying objects in images use edge detection and pattern recognition algorithms to separate the object from the background and recognise the object to be identified. In an embodiment of the invention, the identifying of the bale in the bale images comprises distinguishing the bale from the field based on an observed difference in displacement relative to the agricultural baler and between different bale images of the series. During use, the agricultural baler drives over the field, causing a displacement of the field relative to the agricultural baler between subsequent images of the series. The bale is carried by and generally travels at the same speed as the agricultural baler. Only when the plunger pushes against the bale (about once per second), the bale is displaced relative to the agricultural baler. According to this embodiment of the invention, the continuous and larger displacement of the field relative to the agricultural baler is distinguished from the periodic and smaller displacement of the bale, thereby improving the detection algorithm for identifying the bale.

The at least one bale shape parameter that is determined may comprise a bale length, a bale volume and/or a bale rectangularity of the identified bale. The latter may, e.g., be determined by fitting the identified bale to a quadrilateral bounding box. The rectangularity of the identified bale can then, e.g., be determined by counting pixels that are part of the bale but not of the bounding box, or vice versa. Optionally, the bale images further comprise a view of a reference part of the agricultural baler, and the at least one lateral edge of the quadrilateral bounding box is aligned with the reference part. When the exact orientation of the reference part is known, this knowledge can be used to ensure that the quadrilateral bounding box in the image represents an actual rectangle in 3D. Alternatively, stereo cameras may be used for determining a rectangularity of the bale.

In a preferred embodiment, the method further includes identifying a movable top door of the bale chamber and, based on at least one of the bale images, determining a position of the top door relative to a reference point on the agricultural baler. The movable top door of the bale chamber applies pressure to the top of the bale. If the pre-compression chamber is not filled to full capacity before new slices are fed into the bale chamber, the top door can push down further than when it is completely filled. When the filling level of the pre-compression chamber varies for subsequent slices of a single bale, an irregularly shaped bale may be the result. When capturing images of the bale from a top perspective view, such irregularity may not be easy to identify. By monitoring the position of the top door relative to the reference point while the bale is being formed, the average and varying height of the bale can be determined, and additional bale shape information is obtained.

In a preferred embodiment, the method for monitoring bale shape further comprises determining, based on at least two of the bale images, an extent of movement of the bale relative to the agricultural baler. As explained above, the bale normally only moves relative to the agricultural baler when it is pushed by the plunger. If, e.g., <NUM> bale images are captured per second and plunger strokes come at a rate of <NUM> per second, most images will show the bale in the same position as the immediately preceding images of the series. By monitoring the extent of movement of the bale relative to the agricultural baler between different images of the series, it can be determined if (and how much) the bale is moved in between plunger strokes. When the bale is found to continue moving when not in contact with the plunger, this is a clear sign that the bale is not properly clamped in the bale chamber. This problem can possibly be solved by adjusting one or more operational parameters of the agricultural baler.

In an advanced embodiment of the method for monitoring bale shape, trained neural networks and/or other artificial intelligence (AI) algorithms are used to identify the bale in the bale images. For example, training data sets may be provided by labelling bales in a plurality of bale images comprising a view of a bale against the background of a field. The labelling may, e.g., be performed by hand or using the bale identifying algorithm described above.

According to a further aspect of the invention, a computer program is provided comprising instructions which, when executed by a computer, cause the computer to carry out a method as described above.

According to a further aspect of the invention, a system is provided for monitoring bale shape in an agricultural baler. The system comprises a camera for capturing a series of bale images, and a controller, operatively coupled to the camera and configured to perform a method as described above.

According to yet a further aspect of the invention, an agricultural baler is provided comprising a bale chamber for forming a bale therein, the bale chamber comprising an outlet for ejection of the bale from the bale chamber. The agricultural baler further comprises a camera positioned to capture a series of bale images, the bale images comprising a view of the outlet,
a view of the bale while being ejected from the outlet, and a view of a field travelled by the agricultural baler during the ejection of the bale. A controller is operatively coupled to the camera for receiving the series of bale images therefrom and suitable for performing a method as described above. The agricultural baler may be a large square baler.

Embodiments of the invention will now be described by way of example with reference to the accompanying drawings.

<FIG> shows an agricultural baler <NUM> in the form of a large square baler. The baler <NUM> has a pickup unit or apparatus <NUM> for lifting crop material from windrows. The pickup apparatus <NUM> has a rotatable pickup roll (or rotor or cylinder) <NUM> with a plurality of pickup tines <NUM> to move the collected crop rearward towards a rotor cutter apparatus <NUM>. Optionally, a pair of stub augers (one of which is shown, but not numbered) is positioned above the pickup roll <NUM> to move the crop material laterally inward.

The rotor cutter apparatus <NUM> has a rotor assembly with rotor tines <NUM> that push the crop towards a knife rack with knives for cutting the crop and into a pre-compression chamber <NUM> to form a slice of crop material. The tines <NUM> intertwine the crop together and pack the crop within the pre-compression chamber <NUM>. The pre-compression chamber <NUM> and the rotor assembly with the tines <NUM> function as a first stage for crop compression.

Once the pressure in the pre-compression chamber <NUM> reaches a predetermined sensed value, a stuffer unit or apparatus <NUM> moves the slice of crop from the pre-compression chamber <NUM> to a bale chamber <NUM>. The stuffer apparatus <NUM> includes stuffer forks <NUM> which push the slice of crop directly in front of a plunger <NUM>, which reciprocates within the bale chamber <NUM> and compresses the slice of crop into a flake. The stuffer forks <NUM> return to their original state after the slice of material has been moved into the bale chamber <NUM>. The plunger <NUM> compresses the slices of crop into flakes to form a bale and, at the same time, gradually advances the bale toward an outlet <NUM> of the bale chamber <NUM>. The bale chamber <NUM> and plunger <NUM> function as a second stage for crop compression.

When enough flakes have been added and the bale reaches a full (or other predetermined) size, the knotters <NUM> are actuated which wrap and tie twine around the bale while it is still in the bale chamber. Needles <NUM> bring the lower twine up to the knotters <NUM> and the tying process then takes place. The twine is cut, and the formed bale is ejected from a discharge chute <NUM> as a new bale is formed.

A camera <NUM> is installed on the agricultural baler <NUM> at a position and orientation that allows it to capture a series of bale images comprising a view of the outlet <NUM> of the bale chamber <NUM> and the bale that is ejected therefrom. In the background of the captured images, a portion of the agricultural field that is being traversed will be visible too. An example of one of the images <NUM> of such a series is shown in <FIG>. The camera may be a standard 2D monochrome or multi-colour digital camera for capturing photos and/or video. More than one camera may be used for capturing images from different viewpoints, thereby increasing the accuracy with which bales can be identified and their exact 3D shape determined. Alternatively, the camera may be a stereo camera, a laser scanner, radar, or other type of 3D camera. In the following, an embodiment of the invention will be described, based on the use of a single monochrome or multi-colour 2D digital camera. It should, however, be clear that the invention is not limited to this embodiment.

<FIG> shows an exemplary bale image <NUM> captured by a camera <NUM> of an embodiment of a system according to the invention. <FIG> shows a schematic representation <NUM> of a bale image <NUM> as shown in <FIG>. The bale image <NUM> shown in <FIG> is just one of a series of images captured and processed in accordance with a method according to the invention. Typically, the camera <NUM> captures about <NUM> bale images per second. Higher capture rates and higher image resolutions can help to improve the accuracy with which the bales are identified and their exact dimensions in 3D are determined. However, higher capture rates and image resolutions also require more processing power and processing time. In practice, a balance needs to be found between cost and detection accuracy, while allowing the system to analyse the images in real time. Efficient and effective image processing algorithms can thereby assist to increase detection accuracy without increasing the required amount of processing power and time.

As can be seen in <FIG>, the agricultural baler <NUM> itself is easily discernible and clearly stands out from the other parts of the image <NUM>. The bale on the discharge chute <NUM> and the agricultural field in the background, however, have very similar colour and texture and are difficult to distinguish. This lack of contrast between the bale and the background may not just be a problem for a human observer looking at the image, but also makes it difficult for standard image recognition algorithms to identify the bale and accurately determine its exact shape.

According to an embodiment of the invention, the bale <NUM> may be identified in the bale images <NUM>, <NUM> by determining and comparing a displacement of specific pixels or groups of pixels in the image <NUM>, <NUM> between subsequent images of the series. During use, the agricultural baler <NUM> drives over the field <NUM>, causing a displacement of the field <NUM> relative to the agricultural baler <NUM> between subsequent images of the series. The bale <NUM> is carried by and generally travels at the same speed as the agricultural baler <NUM>. Only when the plunger <NUM> pushes against the bale <NUM> (about once per second), the bale <NUM> is displaced relative to the agricultural baler <NUM>. In the bale images <NUM>, <NUM>, the bale chamber outlet <NUM> and some other parts of the agricultural baler <NUM>, such as a bale chamber frame <NUM> and a bale chamber top door <NUM> may be visible. The bale chamber outlet <NUM> and bale chamber frame <NUM> keep in a constant position relative to the camera <NUM> and do not show any displacement between different images <NUM>, <NUM> in the series of bale images.

The observed difference in displacement between the bale <NUM> and the field <NUM> relative to the agricultural baler <NUM> may be used to distinguish the pixels and groups of pixels in the bale images <NUM>, <NUM> that represent the bale <NUM> and the field <NUM>.

Alternatively, trained neural networks and/or other artificial intelligence (AI) algorithms are used to identify the bale <NUM> in the bale images <NUM>, <NUM>. For example, training data sets may be provided by labelling bales <NUM> in a plurality of bale images <NUM>, <NUM> comprising a view of a bale <NUM> against the background of a field <NUM>. The labelling may, e.g., be performed by hand or using the above described bale identifying algorithm based on the difference in displacement relative to the agricultural baler <NUM>.

When the bale <NUM> is identified in the bale images <NUM>, <NUM>, the shape and dimensions of the bale <NUM> can be derived from those bale images too. Exemplary bale shape parameters that may be derivable from these bale images <NUM>, <NUM> are a bale length, a bale volume and/or a bale rectangularity of the identified bale.

Preferably, such bale shape parameters are derived from bale images <NUM>, <NUM> that show a complete top surface of the bale <NUM>. The complete top surface of the bale <NUM> is best visible in the period after the complete bale <NUM> has been ejected from the bale chamber outlet <NUM> and before it tips over the rear end of the bale chute <NUM> and is dropped onto the field <NUM>. Alternatively, for example if the rear end of the bale <NUM> is already visible before the bale has been fully ejected, the bale shape parameters may be derived from an image wherein a small portion of the top surface is hidden from view by the outlet <NUM>.

The viewing angle of the camera <NUM> on the top surface of the bale and general perspective distortion resulting from the choice of imaging equipment make it difficult to directly derive the relevant bale shape parameters from the bale images. Reference parts of the agricultural baler <NUM> may be identified in the same bale images <NUM> and used as a reference to allow a more accurate measurement of the bale shape parameters. For example, the bale chamber outlet <NUM> and the bale chamber frame <NUM> have known dimensions and a fixed orientation and are very suitable to function as a reference part. Additional markers <NUM> may be applied to such reference parts to further facilitate the bale shape parameter measurements.

Optionally, the images further show at least a portion of the movable top door <NUM> of the bale chamber and the position of the top door <NUM> is monitored while the bale <NUM> is being formed. During the formation of the bale, the movable top door <NUM> of the bale chamber applies pressure to the top of the bale <NUM>. If the pre-compression chamber is not filled to full capacity before new slices are fed into the bale chamber, the top door <NUM> can push down further than when it is completely filled. When the filling level of the pre-compression chamber varies for subsequent slices of a single bale <NUM>, an irregularly shaped bale <NUM> may be the result. When capturing images <NUM>, <NUM> of the bale <NUM> from a top perspective view, such irregularity may not be easy to identify. By monitoring the position of the top door <NUM> relative to a reference point <NUM> while the bale <NUM> is being formed, the average and varying height of the bale <NUM> can be determined, and additional bale shape information is obtained. As shown in <FIG>, the reference point may be embodied as reference part <NUM> with vertically arranged markers.

In order to accurately determine one or more bale shape parameters, the processing of the bale images may comprise fitting the identified bale <NUM> to a quadrilateral bounding box <NUM>. As can be seen in <FIG>, the quadrilateral bounding box <NUM> will generally not be rectangular in the bale image <NUM>, <NUM>. However, knowing that the bale <NUM> is supposed to be rectangular, and possibly making use of the reference parts <NUM>, <NUM> visible in the bale image <NUM>, a geometric transformation may be used to transform the quadrilateral bounding box <NUM> into a rectangular shape. The bale <NUM> identified in the image is then transformed accordingly.

A possible result of such transformations can be seen in <FIG> which, in fact, shows a reconstructed top view of a bale identified in one of the bale images <NUM>, <NUM>. In this top view, some of the available bale shape parameters are identified. A bale length <NUM> is defined by a distance between the leading and the trailing edge of the bale <NUM>. The rectangularity of the identified bale <NUM> may, e.g., be determined by counting pixels that are inside the bounding box <NUM> but do not form part of the bale <NUM>. A different measure of (non-)rectangularity may be a skew parameter <NUM>, defining how far one of the corners of the top surface is displaced from the corresponding corner of the bounding box <NUM>. An indent parameter <NUM> may indicate the largest distance of any edge of the top surface to the corresponding edge of the bounding box <NUM>. Other useful parameters for classifying the shape or rectangularity of the bale <NUM> may be used.

In addition to measuring bale shape parameters, the bale shape monitoring system may be configured to check if the bale <NUM> in the bale chamber <NUM> is properly clamped between the bale chamber walls. This may be done by determining an extent of movement of the bale <NUM> relative to the agricultural baler <NUM> between two consecutive images of the series of bale images <NUM>, <NUM>. As explained above, the bale <NUM> normally only moves relative to the agricultural baler <NUM> when it is pushed by the plunger <NUM>. If, e.g., <NUM> bale images are captured per second and plunger strokes come at a rate of <NUM> per second, most bale images <NUM>, <NUM> will show the bale <NUM> in the same position as the immediately preceding images <NUM>, <NUM> in the series. By monitoring the extent of movement of the bale <NUM> relative to the agricultural baler <NUM> between different images <NUM>, <NUM> in the series, it can be determined if (and how much) the bale <NUM> is moved in between two plunger strokes. When the bale <NUM> is found to continue moving when not in contact with the plunger <NUM>, this is a clear sign that the bale <NUM> is not properly clamped in the bale chamber <NUM>. This problem can possibly be solved by adjusting one or more operational parameters of the agricultural baler.

Claim 1:
A method for monitoring bale shape, the method comprising the steps of:
receiving a series of bale images (<NUM>, <NUM>) from a camera (<NUM>), the bale images (<NUM>, <NUM>) comprising
- a view of at least an outlet (<NUM>) of a bale chamber (<NUM>) of an agricultural baler (<NUM>),
- a view of a bale being ejected from the outlet (<NUM>),
characterised in that the bale images (<NUM>, <NUM>) further comprising a view of a field (<NUM>) travelled by the agricultural baler (<NUM>) during the ejection of the bale (<NUM>), and wherein the method further comprising the steps of:
identifying the bale (<NUM>) in the bale images (<NUM>, <NUM>),
based on at least one of the bale images (<NUM>, <NUM>), determining at least one bale shape parameter (<NUM>, <NUM>, <NUM>) of the identified bale (<NUM>),
providing an electronic signal representative of the at least one bale shape parameter (<NUM>, <NUM>, <NUM>).