Vision-assisted system and method for picking of rubber bales in a bin

The problem of picking tightly-pack generally uniformed products such as rubber bales in a bin is solved by sequentially selecting each one of the products based on the products depths in the bin, using a robot with a tool to grip each selected product, and moving on an output station each gripped product considering its position relative to the gripping tool. A first sensor system is used to determine the product depths in the bin. The sensor system can be mounted on the robot tool or be positioned above the bin. The position of each gripped product in the gripping tool is achieved by analyzing an image of the gripped product in the gripping tool.

FIELD

The present disclosure relates to bin picking, and more specifically to vision-assisted system and method for picking of rubber bales in a bin.

BACKGROUND

Rubber product manufacturers typically receive their raw material in the form of rubber bales that are usually received in large bins. Of course, at one point the bales have to be taken from the bins to feed the fabrication process. Traditionally, those bales are manually removed from the bin, which is physically difficult and time consuming. Some manufacturers use vacuum tools coupled with a gantry type hoist to pick the rubber bales. While this approach is not as physically demanding as the fully manual operation, it still requires manual labor to guide the hoist. Also, the bales located at the bottom of the bin are typically difficult to reach.

Various bin picking technologies are available, but they are designed for small identical parts that are randomly distributed in the bin. Those technologies are not suited to pick rubber bales because, while the bales are orderly placed in the bin, a challenge is to determine their position as the contour of each bale is not obvious to determine, even for the human eye. A sensor system and method typically used for picking of randomly placed objects in a bin would therefore be inappropriate in a rubber bale picking application.

The problems inherent in both identifying and gripping a rubber bale in a bin yield uncertainties in the rubber bale holding condition. This is a drawback considering that a picked bale is to be correctly positioned in an output station.

United States Patent publication no. 2011/0222995 A1, published on Sep. 15, 2011, naming Irie et al. as the inventors and being titled “Robot System and Transfer Method” describes a robot system and method where a shape sensor detects shape information of products randomly placed in a container and an inspection device determines the holding condition of the workpiece by the robot. In the case where the holding condition of the workpiece is found unacceptable, the workpiece is placed on a temporary placement table. The shape sensor is used a second time to detect the workpiece before instructing the robot to retake it.

While the approach taken by Irie et al may be appropriate to pick products that are piled in a bin in an unarranged manner, the proposed shape sensor would be inappropriate to detect edges of the bales considering that there is typically no gap between the rubber bales. This is in addition to Irie's gripping tool that would also be inappropriate for rubber bales. Furthermore, the use of a temporary placement table is detrimental for cycle time reduction purposes.

United States Patent publication no. 2012/0165986 A1, published on Jun. 28, 2012 to Fuhlbrigge et al. and being titles “Robotic picking of parts from a parts holding bin” describes a robot system and method where an image from one or more cameras are used to determine if a robot gripper has picked one or more parts and the position and orientation of the picked part.

The approach taken by Fuhlbrigge et al. is based on the possibility that the gripper can hold more than one part or to return the picked part in the bin if its position and orientation does not meet a predetermined criteria. It is designed for parts that are randomly distributed in a bin and that can therefore be interlocked with each other, which cannot happen for rubber bales. For cycle time purposes, it is preferable to have a system and a method that only picks one part and does not have to return it to the bin regardless of its position and orientation in the gripper.

A system and method that allows bin picking of orderly positioned rubber bales is thus desirable.

SUMMARY

The problem of picking tightly-pack generally uniformed products in a bin is solved by sequentially selecting each one of the products based on the products depths in the bin, using a tool to grip each selected product, and moving on an output station each gripped product considering its position relative to the gripping tool.

According to an illustrative embodiment, there is provided a system for picking products in a container, the system comprising:

a first sensor system to obtain an image of upper products in the container and to use the image in determining position-related information on the upper products; the upper products being the products in the container that are visible from above the container;

a robot coupled to the first sensor system for receiving therefrom the position-related information on the upper products and for using the position-related information on the upper products for sequentially gripping the upper products; and

a second sensor system for determining a position of each of the upper products relative to the robot when said each of the upper products is gripped by the robot; and

the robot being further configured to use information indicative of the position determined by the second sensor to position at a selected location the upper product that is gripped by the robot.

According to another illustrative embodiment, there is provided a robot, equipped with an end of arm tool, that uses position-related information on products in a bin that are visible from above the bin to sequentially pick the visible products, and, while each visible product is being picked, that moves said each visible product to a selected location while taking into consideration a position and an orientation of said each visible product relative to the end of arm tool.

According to still another illustrative embodiment, there is provided a method for picking tightly-pack generally uniformed products in an opened container comprising sequentially gripping each one of the products based on depths of the products in the container, determining a position of each gripped product relative to the tool and moving each gripped product on an output station in accordance with a position of the gripped product relative to the tool.

Other objects, advantages and features of the vision-assisted system and method for picking of rubber bales in a bin will become more apparent upon reading the following non restrictive description of preferred embodiments thereof, given by way of example only with reference to the accompanying drawings.

DETAILED DESCRIPTION

In the following description, similar features in the drawings have been given similar reference numerals, and in order not to weigh down the figures, some elements are not referred to in some figures if they were already identified in a precedent figure.

A system10for picking products12in a container14will now be described with reference toFIGS. 1 and 2.

The system10according to the first illustrated embodiment is adapted to picking, as an input, rubber bales12in a bin14.

The system10comprises a robot16equipped with a gripper18and guided by a first sensor system20to remove the bales12one by one from the bin14, and a second sensor system22to determine the final orientation and position of each bale in the gripper18and to instruct the robot16to properly transfer the bale12on an output conveyor (not shown) in a predetermined manner, yielding a flow of unitized bales aligned in a desired orientation on the output conveyor.

As will become more apparent upon reading the following description, the system10is not limited to picking rubber bales, and the expression “product” should be construed herein as including any type of bale, block, brick, pad, etc. The product12can be wrapped in a thin plastic film, but can be without it. Also, the plastic film can be in various colors.

The product dimensions may vary greatly between each different type of product. Typical dimensions (width×length×height) are between 6″×6″×5″ (15.25 cm×15.25 cm×12.7 cm) and 24″×36″×9″ (61 cm×91.5 cm×22.9 cm).

Typically, there are no substantial gaps between the products12in the incoming bin14. Even though the system10allows picking in the bin14products12that are tightly pack, it can be used to pick products that are spaced in a bin.

These dimensions are given for illustration purposes only. It is believed to be within the reach of a person skilled in the art to use the present teaching to adapt and more specifically to dimension the system10for products and/or bins having other dimensions.

The robot16is in the form of a standard 4 or 6 axis industrial articulated arm. A conventional robot arm can be used, such as ABB's or IRB 7600, FANUC's M900, or any similar robot arm.

The robot arm16is adapted for the application and, according to the first illustrative embodiment, is equipped with an end of arm tool in the form of the gripper18, which is adapted to securely pick and transfer the rubber bales12, wrapped or not in a thin plastic film, from the filled bin14to the output conveyor98.

According to the first illustrative embodiment, the end of arm tool18includes a series of hooks that can firmly grip the rubber bale12. According to another embodiment (not shown), the end of arm tool18includes needles, vacuum, screws or other means to pick the rubber bales12. According to still another embodiment, the end of arm tool18is configured to grip products having a different configuration than rubber bales12, such as, without limitations, boxes (not shown).

In the description and in the claim, the expressions ‘robot’ and ‘robot arm’ will be used interchangeably to means a programmable system including articulated and/or movable members that can receive, control and move a tool.

According to another embodiment, the articulated arm and tool forms are integrated in a single device, providing the combined features of the robot arm16and tool18.

The robot arm16includes other well-known systems and components that allow its operation, including a robot controller. Since these systems and components are believed to be well-known in the art, they will not be described herein in more detail for concision purposes.

The robot arm16is positioned adjacent a bin unloading station where the bin14that includes the rubber bales12to be picked by the robot arm16is positioned. The bin14can be placed in the bin unloading station by a lift truck, an input bin conveyor, or any other means to transport a filled bin14(all not shown).

According to the first illustrated embodiment, the first sensor system20includes one or more image acquisition sensors24that are positioned over the bin unloading station so as to include the bin14in its field of view.

The first sensor system20allows acquiring sufficient data to reconstruct the image of the upper products in the bin14, the upper products being the rubber bales12in the bin that are visible from above. According to the first illustrative embodiment, the image acquisition sensors24are time-of-flight type sensors, such as the Microsoft Kinect® camera. According to another embodiment, the sensors24are conventional industrial cameras generating 3D images (scanners, stereocameras, etc.).

The image acquisition sensors24are wired to a controller26or wirelessly coupled thereto (see lines28) and both are configured for the transfer of acquired image data between the sensors24and the controller26.

The controller26is configured or programmed for analyzing the image data acquired by the first sensor system20, determines products' characteristics and uses such characteristics to determine robot-readable coordinates of the individual upper products in the bin14. Examples of product characteristics determined by the controller26includes without limitations depth of the products within the bin, their positions and orientations within the bin, and the products horizontality. The controller26is wired or wirelessly coupled to the robot16(see lines30) and is configured to send the upper products coordinates thereto or robot-readable information indicative thereof.

The expression “image” should be construed in the description and in the claims as including any type of data that forms a more or less precise two-dimensional (2D) or three-dimensional (3D) representation of one or more subjects including 2D or 3D conventional grey-tone or color images, depth maps, topography images, height data, or any other partial or complete reproduction of the subjects.

The expression “controller” should be construed broadly as including one or more electronic devices, including for example one or more computers that are configured with components and/or programmed with instructions that produce one or more functionalities.

The first sensor system20is not limited to being fixed to a structure above the bin unloading station. According to a further embodiment, the first sensor system20is part of the tool18. According to such an embodiment, the first sensor system20includes for example four cameras or other sensors99that are mounted to the tool18so as that their positions define a rectangle or another shape. The images or depth values obtained from such a sensor system can be used in same side pairs to better assess, validate and/or correct gripping point on a selected bale12for the tool18.

According to still another embodiment, the image acquisition sensors24are in the form of a series of laser sensors mounted to the tool18and that measures the depths of the upper products12in the bin14for a plurality of points thereon. In this later case, the robot16moves the sensors24over the bin14via the tool18to grab the depths of the products. This is obviously achieved before a first product12is gripped. This yields a depth map of the upper surface of the products12in the bin which is used by the controller26to determine the coordinates of the upper products12.

The controller26may be programmed to request a new image acquisition by the first sensor system20according to predetermined criteria, such as the arrival of a new bin in the bin unloading station, a problematic scan, etc.

According to another embodiment, the controller26commands the robot16to move during the image acquisition so as to improve or supplement an initial field of view.

In some cases, the image acquisition sensors24cannot adequately identify the location and orientation of the individual products12on the top region of the filled bin14when those products12are so close to each other that there is no visible edge that separates them.

As an example which is illustrated inFIGS. 3aand 4a, two side-by-side rubber bales could be placed horizontally or vertically. The image acquisition sensors' controller26is configured to evaluate, in such cases, where a “half bale” is most probably located (see dashed lines32inFIGS. 3band 4b) and sends the corresponding coordinates to the robot16, which then picks the product12based on the position of the “half bale”. The evaluation of the most probable location of the half bale is determined using for example the theoretical pattern of the products in the bin. This can be provided by the product manufacturer.

It results from this last bale picking strategy that the exact orientation and position of the product12in the end of arm tool18are not known.

The situation shown inFIGS. 3a-3band 4a-4bis shown for illustration purposes only, and various other product positions could lead to the same difficulty of identifying the next product12to be picked.FIGS. 5a-5band 6a-6billustrate other examples of such a situation.

To cope with such situations, a second sensor system22, including an image acquisition sensor, is provided (see inFIG. 1) to determine precisely the product's position and orientation before it is placed on the output conveyor (not shown).

According to the first illustrative embodiment, such a bale position sensor system22includes an image acquisition sensor in the form of a 2D camera with front lighting. According to another embodiment, the sensor system22includes infrared light sources and infrared filter on the camera. According to still another embodiment, the sensor system22includes a conventional industrial 2D camera. Other type of sensor can also be used, such as time of flight, scanner, laser sheet, etc.

The second sensor system22is secured adjacent the robot16and bin unloading station and is oriented so as to take an image of the tool18and a picked bale12from below. A lighting system (not shown) is positioned in such a way to provide front lighting of the product12for the bale position vision system22so that it can see the product12and determine its position relative to the end of arm tool18. According to the first illustrated embodiment, this is achieved using edge detection and by measuring the differences in the distances Δx and Δy between the centers of the tool18and product12and of the angle Δθ therebetween as illustrated inFIG. 8. According to some embodiments, the sensor system22also confirms that a bale is actually picked by the end of arm tool. The relative position of the bale in the end of arm tool18is also considered when the robot arm16moves the image acquisition sensors24when it is mounted on the end of arm tool18to determine where the next bale to be picked should most probably be located.

The second sensor system22is wired to the controller26or wirelessly coupled thereto and is configured for the transfer of acquired image data to the controller26.

The controller26processes the image from the system22, so that the rubber bale12appears as a black rectangle in the image as shown inFIG. 8. By measuring the product's length and width, the controller26determines the product's position relative to the end of arm tool18by measuring Δx, Δy and Δθ.

Once the product's position relative to the tool18is determined by the controller26, the controller26calculates and sends to the robot arm16movement displacement instructions that will precisely place the rubber bale12at predetermined location and orientation on the output conveyor (not shown). These instructions take into consideration the position of the product12in the tool18.

According to another embodiment, the second sensor system22includes or is coupled to a dedicated controller that processes the image acquired thereby.

The output conveyor is in the form of a linear conveyor that is adapted to receive the products placed one by one thereon by the system10. According to another embodiment (not shown), the output conveyor is replaced by an output table or any other means adapted to receive the products, such as without limitations an automated guided vehicle (AGV). According to still another embodiment (not shown), two output conveyors (or more) or other output means are used.

The flow chart ofFIG. 9describes a vision-assisted bin picking method100according to a first illustrated embodiment which will now be described in more detail with reference also to the operation of the system10.

In step102, a bin14loaded with rubber bales12is forwarded next to the robot16at the bin unloading station. The first image acquisition sensors24take pictures of the upper products12in the bin14and more specifically obtains a depth map of the upper products12(step104). In step106, the controller26uses the depth map to determine the next product12to be removed among the upper products12.

According to some embodiment, the controller26chooses the highest product or, if all upper products12are at the same depth, than the controller chooses the product12whose depth has been determined with the highest certainty. This can be determined using a conventional edge detection method. Alternatively or additionally to such method, for example when it is inconclusive, a product12on the edge of the bin is selected.

Instructions are then sent to the robot16(step108) by the controller26which then grips the corresponding product12(seeFIG. 1) using the tool18, and moves it above the second sensor system22(step110) (seeFIG. 2).

When a product cannot be identified in step106, the bin14is considered empty (step118) and the system10waits for the arrival of a new bin14with products12(step120).

In step112, an image is taken by the second sensor system22and sent to the controller26, which then determines in step114the position and orientation of the product12relative to the tool18. The controller26uses this information in determining the displacement requires by the robot16to position the product12at a predetermined location and with a predetermined orientation on the output conveyor. In step116, the robot16moves the product12onto the output conveyor at the determined location.

Each time a product12is moved on the output conveyor, the method proceed with step118, wherein the controller26determines whether the bin is empty, and if so, wait for a new loaded bin (step120). If not then the method returns to step104. The bin14is determined to be empty when the highest depth equals the known distance of the bottom of the bin14for example.

A vision-assisted bin picking method according to a second illustrative embodiment will now be described. This method is executed using the previously mentioned embodiment of the system where the first sensor system20is located on the gripper18. Since the first and second illustrative embodiments of the method are similar, only the differences therebetween will now be described for concision purposes.

In step104, the depth measures of the upper products12in the bin14are taken as the robot16moves the gripper18to a series of positions so as to vary the point and field of view of the first sensor system20.

According to the position of the previously picked product12in the bin14and the product orientation determined by the second sensor system22, the controller26evaluates in step106the most probable position of the next product12to be picked as described hereinabove.

Once a product12is dropped on the output conveyor, the robot16moves the gripper18over the most probable position of the next product to be picked.

The first sensor system20is used to take depth measures at this most probable position in order to confirm the position of the next product to be picked.

If this confirmation is found to be true, the controller26commands the robot16to pick this next product12and the following steps described in the first embodiment are followed.

If this confirmation is found to be false, new depth measures are taken by the first sensor system20, but slightly offset from the previous ones.

Those last three steps are repeated until the sensor system22confirms the position of the next product to be picked or until the search process iterations exceeds a predetermined value, as inputted for example by the operator. In this later case, an alarm is sent to the operator for a manual intervention.

It is to be noted that many other modifications could be made to the vision-assisted bin picking systems and methods described hereinabove and illustrated in the appended drawings. For example:to increase the production rate, two robot arms can be used for example along with two or more output conveyors;for gripping the products, the end of arm tool can include needles, screw, cork-screws, vacuum pad, suction cups or a combination thereof;the articulated robot arm can be replaced by a gantry type system; andthe products that are brought at the picking station can be in a bin, a bin pallet, a box pallet, a pallet or any other means to transport such products.

It is to be understood that embodiments of the vision-assisted bin picking system and method are not limited in their application to the details of construction and parts illustrated in the accompanying drawings and described hereinabove. Other embodiments can be foreseen and practiced in various ways. It is also to be understood that the phraseology or terminology used herein is for the purpose of description and not limitation.