Abstract:
Disclosed are a bin-picking system and a bin-picking method. A bin-picking system includes a transformable bin-picking box; a supporting unit configured to support the bottom part of the bin-picking box and be movable upward and downward; and a control unit configured to change an alignment of at least one bin-picking candidate object by transforming the bin-picking box by controlling a movement of the supporting unit if no bin-picking target object is detected from the at least one bin-picking candidate object existing inside bin-picking box and placed on the supporting unit to facilitate detection of a bin-picking target object and bin-picking.

Description:
CROSS REFERENCE TO RELATED APPLICATION 
       [0001]    This application claims the benefit under 35 USC §119(a) of Korean Patent Application No. 10-2015-0041588 filed on Mar. 25, 2015 in the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes. 
       BACKGROUND 
       [0002]    1. Field 
         [0003]    The following description relates to a bin-picking system and a bin-picking method. 
         [0004]    2. Description of Related Art 
         [0005]    Industrial robots have been used for the automation of manufacturing processes. For example, industrial robots have been applied for assembly and pick and place, for example, such as packaging of objects (for example, components) that are placed in a certain location to improve productivity. A cell manufacturing system has been introduced for needs of flexible production lines. Thus, a great deal of development research is currently under way on bin-picking systems to assembly and pick and place. Bin-picking allows detecting a location of a target object to be assembled or packaged from randomly stacked objects and completing assembly or packaging by repeating pick and place processes. 
         [0006]    Typical bin-picking method estimates location and pose of objects using a 3-dimensional scanner and then picking an object which is on the top therefrom in a bin-picking box. A structured light and a stereo method using a 2D camera have being used to estimate location and pose of objects. 
         [0007]    Since objects placed in a bin-picking box are randomly oriented and overlapped with each other, errors are commonly caused in estimating location and pose of a bin-picking target object. Specially, if the surface material of objects such as plastics has a characteristic of reflecting light, it is very difficult to identify and detect location and pose of a target object from the objects overlapped with each other. 
       SUMMARY 
       [0008]    This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. 
         [0009]    A bin-picking system according to an example may facilitate detection and identification of a bin-picking target object. 
         [0010]    A bin-picking system according to an example may improve accuracy for estimating location and pose of a bin-picking target object. 
         [0011]    A bin-picking system according to an example may provide smooth bin-picking of a bin-picking target object. 
         [0012]    According to one general aspect, a bin-picking system includes a transformable bin-picking box; a supporting unit configured to support the bottom part of the bin-picking box and be movable upward and downward; and a control unit configured to change an alignment of at least one bin-picking candidate object by transforming the bin-picking box by controlling a movement of the supporting unit if no bin-picking target object is detected from the at least one bin-picking candidate object existing inside bin-picking box and placed on the supporting unit. 
         [0013]    The bin-picking box may be formed of a flexible material. 
         [0014]    The bin-picking box may include a back light configured to irradiate light to the bin-picking candidate objects. 
         [0015]    A cross-sectional object region of the bin-picking box may become narrower toward the bottom end. 
         [0016]    A cross-sectional object region of the supporting unit may be narrower than that of the top end of the bin-picking box. 
         [0017]    The bin-picking system may further include an image acquisition unit configured to take an image of the bin-picking candidate objects, wherein the control unit may detect the bin-picking target object from the image received from the image acquisition unit. 
         [0018]    The control unit may extract silhouette and edge information of the bin-picking candidate objects from the image received from the image acquisition unit and detect a bin-picking target object from the bin-picking candidate objects based on the extracted silhouette and edge information. 
         [0019]    The control unit may estimate location and pose of the bin-picking target object based on the extracted silhouette and edge information. 
         [0020]    The bin-picking system may further include a bin-picking unit configured to perform bin-picking for the bin-picking target object based on the estimated location and pose. 
         [0021]    The control unit may move the supporting unit down from a first location to a second location and then move the supporting unit up from the second location to the first location when the bin-picking target object is not detected from the bin-picking candidate objects. 
         [0022]    The control unit may control the image acquisition unit to take an image when the supporting unit is positioned at the first location. 
         [0023]    The image acquisition unit may include a visual sensor configured to take photos of the bin-picking candidate objects and a front light configured to irradiate light to the bin-picking candidate objects. 
         [0024]    The image acquisition unit may further include at least one of a polarizing filter configured to filter the light incident to the visual sensor and a diffusing panel configured to diffuse the light irradiated from the front light. 
         [0025]    According to another general aspect, a bin-picking method includes moving a supporting unit up to a first location to place at least one bin-picking candidate object on the supporting unit, while objects are placed in a bin-picking box, the bottom of the bin-picking box being supported by the supporting unit and the bin-picking box being transformed in accordance with a vertical movement of the supporting unit; taking an image of the at least one bin-picking candidate object and detecting a bin-picking target object from the at least one bin-picking candidate object based on the image; and rearranging the bin-picking candidate objects by moving the supporting unit down to a second location and then moving the supporting unit back to the first location when the bin-picking target object is not detected. 
         [0026]    The method may further include turning a back light, located at the bottom in the bin-picking box, on when the bin-picking target object is not detected. 
         [0027]    The detecting a bin-picking target object may include extracting silhouette and edge information of the bin-picking candidate objects from the image and detecting a bin-picking target object based on the extracted silhouette and edge information. 
         [0028]    The method may further include estimating location and pose of the bin-picking target object based on the extracted silhouette and edge information. 
         [0029]    The method may further include performing bin-picking for the bin-picking target object based on the estimated location and pose. 
         [0030]    A bin-picking system and a method according to an example may improve accuracy for estimating location and pose of a bin-picking target object. 
         [0031]    A bin-picking system and a method according to an example may facilitate detection and identification of a bin-picking target object. 
         [0032]    A bin-picking system and a method according to an example may be applied to the cell manufacturing process to improve productivity. 
         [0033]    A bin-picking system and a method according to an example may be applied to the assembly or packaging process to improve productivity. 
         [0034]    Other features and aspects will be apparent from the following detailed description, the drawings, and the claims. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0035]      FIG. 1  is a diagram illustrating an example of a bin-picking system. 
           [0036]      FIG. 2A ,  FIG. 2B  and  FIG. 2C  are diagrams illustrating examples of rearrangement of objects. 
           [0037]      FIG. 3  is a diagram illustrating an example of an image acquisition unit. 
           [0038]      FIG. 4A  and  FIG. 4B  are diagrams illustrating examples of a bin-picking box. 
           [0039]      FIG. 5  is a flowchart illustrating an example of a bin-picking method. 
           [0040]      FIG. 6  is a flowchart illustrating an example of a method for detecting a bin-picking target object. 
       
    
    
       [0041]    Throughout the drawings and the detailed description, unless otherwise described or provided, the same drawing reference numerals refer to the same elements, features, and structures. The drawings may not be to scale, and the relative size, proportions, and depiction of elements in the drawings may be exaggerated for clarity, illustration, and convenience. 
       DETAILED DESCRIPTION 
       [0042]    The following detailed description is provided to assist the reader in gaining a comprehensive understanding of the methods, apparatuses, and/or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatuses, and/or systems described herein will be apparent to one of ordinary skill in the art. The sequences of operations described herein are merely examples, and are not limited to those set forth herein, but may be changed as will be apparent to one of ordinary skill in the art, with the exception of operations necessarily occurring in a certain order. Also, descriptions of functions and constructions that are well known to one of ordinary skill in the art may be omitted for increased clarity and conciseness. 
         [0043]    The features described herein may be embodied in different forms, and are not to be construed as being limited to the examples described herein. Rather, the examples described herein have been provided so that this disclosure is thorough, complete, and conveys the full scope of the disclosure to one of ordinary skill in the art. 
         [0044]    It will be understood that, although the terms “first,” “second,” etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of the present disclosure. Similarly, when it is described that a method includes series of steps, a sequence of the steps is not a sequence in which the steps should be performed in the sequence, an arbitrary technical step may be omitted and/or another arbitrary step, which is not disclosed herein, may be added to the method. 
         [0045]      FIG. 1  is a diagram illustrating an example of a bin-picking system. 
         [0046]    Referring to  FIG. 1 , a bin-picking system may include a bin-picking box  100 , a supporting unit  210 , an image acquisition unit  300  and a bin-picking unit  500 . At least one of the elements can be omitted. 
         [0047]    The bin-picking box  100  may hold at least one object  10 . The bin-picking box  100  may be formed of an atypical flexible material, for example, such as fabric, plastic or silicone. 
         [0048]    The bottom of the bin-picking box  100  may be supported by the supporting unit  210  and thus the bin-picking box  100  may be transformed. The meaning of that the bottom of the bin-picking box  100  is supported by the supporting unit  210  is that the bottom of the bin-picking box  100  may be pressed by the supporting unit  210 . When the bin-picking box  100  is pressed by the supporting unit  210 , some of the objects  10  may be located on the supporting unit  210 . The meaning of that the objects  10  may be located on the supporting unit  210  is that the objects  10  are supported by the supporting unit  210  while they are still in the bin-picking box  100 . The object being in the bin-picking box  100  and supported by the supporting unit  210  are called as “a bin-picking candidate object” for convenience of description. 
         [0049]    The bin-picking candidate objects may be targets of which an image is to be taken in order to detect a bin-picking target object. The bin-picking target object may be an object to be bin-picked by the bin-picking unit  500  among the bin-picking candidate objects. 
         [0050]    The supporting unit  210  may function as a bench for a bin-picking process and be movable upward and downward by the driving unit  200 . The driving unit  200  may move the supporting unit  210  up and down under control of the control unit  400 . 
         [0051]    The supporting unit  210  may be moved up and down by the control unit  400  to transform shape of the bin-picking box  100 . 
         [0052]    The image acquisition unit  300  may take an image of bin-picking candidate objects. The image acquisition unit  300  may take an image of bin-picking candidate objects when the supporting unit  210  is located at a predetermined first position. Thus, image of the bin-picking candidate objects may be always taken at a constant distance. This may improve accuracy for estimating location and pose of the bin-picking target object. 
         [0053]    The control unit  400  may pre-process the image received from the image acquisition unit  300  and extract features of the bin-picking candidate objects from the pre-processed image. Image pre-processing may mean generation of at least one of a binary image and an edge image from the image taken by the image acquisition unit  300 . Features of the bin-picking candidate objects may include at least one of edge information and silhouette information of the bin-picking candidate objects. Edge information and silhouette information of the objects may be extracted from at least one of the binary image and the edge image. Edge information of the object may include, for example, an image or a text printed on the surface of the object. Silhouette of the object may be, for example, entire outline of the object. 
         [0054]    The control unit  400  may detect a bin-picking target object from the bin-picking candidate objects based on the extracted features. Detecting a bin-picking target object means detecting an object region where the bin-picking target object is. 
         [0055]    The control unit  400  may estimate location and pose of the bin-picking target object included in the detected object region. The control unit  400  may control the bin-picking unit  500  to perform bin-picking for the bin-picking target object based on the estimated location and pose. 
         [0056]    When the bin-picking target object is not detected because several bin-picking candidate objects are overlapped with each other, the control unit  400  may control the movement of the supporting unit  210  to change alignment of the objects  10  in the bin-picking box  100 . 
         [0057]    For example, the control unit  400  may move the supporting unit  210  down and then up to transform shape of the bin-picking box  100 . When the supporting unit  210  moves up and down, the objects  10  in the bin-picking box  100  may be rearranged. Thus, locations and poses of the bin-picking candidate objects may be changed to facilitate the detection of the bin-picking target object. 
         [0058]      FIG. 2A ,  FIG. 2B  and  FIG. 2C  are diagrams illustrating examples of rearrangement of objects. 
         [0059]    As shown in  FIG. 2A , it is assumed that four objects  10  are in the bin-picking box  100  and the bottom of the bin-picking box  100  is supported by the supporting unit  210 . When the supporting unit  210  is spaced-apart from the image acquisition unit  300  in a distance of d 1 , it is also assumed that the supporting unit  210  is at a first location. It is also assumed that bin-picking candidate objects are overlapped with each other so that feature extracting is quite difficult. 
         [0060]    The control unit may move the supporting unit  210  down to be spaced-apart from the image acquisition unit  300  in a distance of d 2  to transform shape of the bin-picking box  100  as shown in  FIG. 2B . Here, when the supporting unit  210  is spaced-apart from the image acquisition unit  300  in a distance of d 2 , it is assumed that the supporting unit  210  is at a second location. When the bin-picking box is formed of a flexible fabrics material and the supporting unit  210  moves down, the bin-picking box  100  may be drooped down toward the gravity direction to shuffle the objects in the bin-picking box  100 . 
         [0061]    The control unit may move the supporting unit  210  up to the first location to transform shape of the bin-picking box  100  to rearrange the objects  10  in the bin-picking box  100  as shown in  FIG. 2C . Accordingly, locations and poses of the bin-picking candidate objects may be changed. Referring to  FIG. 2C , it is noted that locations and poses of the bin-picking candidate objects are changed from the locations and the poses of the bin-picking candidate objects shown in  FIG. 2A . 
         [0062]    As described with reference to  FIG. 2A ,  FIG. 2B  and  FIG. 2C , when the supporting unit  210  is spaced-apart from the image acquisition unit  300  in a distance of d 1 , an image may be taken. 
         [0063]    A conventional bin-picking system detects and picks objects in a fixed bin-picking box. The number of objects placed in the bin-picking box becomes less and thus the distance between the object at the top position and a camera to take an image becomes farther which may increase an error when the location of the bin-picking target object is estimated. 
         [0064]    Since the image acquisition unit  300  takes an image when the supporting unit  210  is spaced-apart in the distance of d 1  therefrom, it may reduce an error when the location of the bin-picking target object is estimated. This may improve accuracy for estimating the distance for the bin-picking target object. 
         [0065]    A cross-sectional object region of the bin-picking box  100  may become narrower toward to the bottom. For example, the bin-picking box  100  may be formed in a cone shape having a narrower cross-sectional object region toward the bottom. 
         [0066]    Since a conventional bin-picking box is formed in a rectangular shape, objects at the edge cannot be easily detected and picked. 
         [0067]    When the cross-sectional object region of the bin-picking box  100  becomes narrower toward to the bottom, detection and picking of the object may be easier. For example, when the supporting unit  210  moved downward, the bin-picking box  100  may be drooped to the gravity direction to place the objects  10  at one place. When the supporting unit  210  moves up again, the bin-picking candidate objects may be located on the supporting unit  210  to facilitate detection and picking of the bin-picking target object. 
         [0068]    The supporting unit  210  may be formed to have a narrower cross-sectional object region compared to cross-sectional object region of a top end of the bin-picking box  100  to place some of the objects  10  in the bin-picking box  100  on the supporting unit  210  to increase the accuracy of estimating pose. 
         [0069]      FIG. 3  is a diagram illustrating an example of an image acquisition unit. 
         [0070]    Referring to  FIG. 3 , the image acquisition unit may include a visual sensor  310  and a front light  320 . The front light  320  can be omitted. 
         [0071]    The visual sensor  310  may take an image of the bin-picking candidate objects. The visual sensor  310  may be a 2-dimensional (2D) or 3-dimensional (3D) sensor. The visual sensor  310  may be a high resolution industrial camera to acquire accurate edge and silhouette information of the objects. 
         [0072]    The front light  320  may be turned on or off by the control unit or an operator. When the front light  320  irradiates light to the bin-picking candidate objects, edge information of the bin-picking candidate objects may be easily extracted. 
         [0073]    The image acquisition unit may be formed to minimize impact from light reflection by surface characteristics of the objects, and lighting irradiated from the front light  320 . For example, a polarizing filter may be formed around lens of the visual sensor  310  which is the part where light is incident to the visual sensor  310 . Highlights caused by light reflection may be eliminated since the light incident to the visual sensor  310  is filtered. Thus, impact caused from light reflection by surface characteristics of the object may be minimized. 
         [0074]    For example, a diffusing panel may be also formed to diffuse light around the front light  320 . The diffusing panel may minimize impact to the visual sensor  310  from the light from the front light  320  by diffusing the light from the front light  320 . 
         [0075]      FIG. 4A  and  FIG. 4B  are diagrams illustrating examples of a bin-picking box. 
         [0076]    Referring to  FIG. 4A , a back light  110  may be formed at the bottom of the bin-picking box  100 . 
         [0077]    When an object is detected by using a visual sensor, specific surface features of the objects may be detected to estimate location and pose of the object. When the object is covered by a package having characteristics of light reflection or refraction, for example, such as plastic, it may be difficult to detect silhouette and edge information of the object properly. 
         [0078]    The back light  110  may be formed to facilitate estimation of location and pose of the object. The back light  110  may be turned on or off by the control unit or an operator. 
         [0079]    As shown in  FIG. 4B , contour of the bin-picking candidate objects is appeared considerably by light irradiated form the back light  110 . Thus, silhouette of the bin-picking candidate objects is extracted more accurately. 
         [0080]    The back light  110  may be formed to have a cross-sectional object region similar to that of the supporting unit  210 . 
         [0081]      FIG. 5  is a flowchart illustrating an example of a bin-picking method. At least one step illustrated in  FIG. 5  can be omitted. 
         [0082]    In step  501 , image of a bin-picking candidate object may be taken when a supporting unit is positioned at a first location. The bin-picking candidate object may be an object in the bin-picking box and placed on the supporting unit. The bin-picking candidate object may be one or more. Photographing may be performed when the front light is turned on. 
         [0083]    In step  503 , detecting a bin-picking target object from the image may be performed. When the bin-picking target object is detected, it may proceed to a step  505 , while it is not detected, it may proceed to a step  511 . The bin-picking target object may be an object to be bin-picked from the bin-picking candidate objects. The detecting a bin-picking target object may be explained with reference to  FIG. 6  below. 
         [0084]    In step  505  when the bin-picking target object is detected, the bin-picking target object may be bin-picked. 
         [0085]    On the other hands, in step  511  when the bin-picking target object is not detected, the supporting unit may move down to the second location to transform shape of the bin-picking box to shuffle the objects in the bin-picking box. 
         [0086]    In step  513 , the supporting unit may move back to the first location to rearrange the objects placed in the bin-picking box. Accordingly, the bin-picking candidate objects on the supporting unit may be also rearranged to change locations and poses of the bin-picking candidate objects. 
         [0087]    In step  515 , the back light at the bottom in the bin-picking box may be turned on. If the back light is already turned on, the back light may be kept in an on state. 
         [0088]    Then, it may proceed to step  501  to repeat the bin-picking process. 
         [0089]      FIG. 6  is a flowchart illustrating an example of a method for detecting a bin-picking target object. A sequence of the steps is not a sequence in which the steps should be performed in the sequence and/or an arbitrary technical step can be omitted. 
         [0090]    In step  601 , image pre-processing may be performed. The image pre-processing may include, for example, distortion correction and image filtering processes. Binary image may be generated using an adaptive binarization algorithm during the image pre-processing. 
         [0091]    In step  603  and step  605 , silhouette and edge information of bin-picking candidate objects may be obtained from the image (the pre-processed image). General methods may be used to obtain the silhouette and edge information. 
         [0092]    In step  607 , a bin-picking target object may be detected. Detecting a bin-picking target object may mean detecting an object region where the bin-picking target object exists. When the bin-picking candidate object is more than two, a bin-picking candidate object with the best silhouette and edge information may be detected as the bin-picking target object. 
         [0093]    As described above, the image may be taken when at least one of the front light and the back light is turned on. The image, which is taken when the front light is turned on, may provide better edge information than that taken when the front light is turned off. The image, which is taken when the back light is turned on, may provide better silhouette information than that taken when the back light is turned off. Therefore, the image, which is taken when both the front light and the back light are turned on, may provide better detection for the bin-picking target object. 
         [0094]    In step  609 , location and pose of the bin-picking target object may be estimated. Location and pose of the bin-picking target object may be estimated using the silhouette and edge information obtained in the previous processes. 
         [0095]    According to an example, transformation parameters may be extracted for the bin-picking target object. Location and pose of the bin-picking target object may be estimated based on the extracted transformation parameters. Location of the bin-picking target object may be a 3-dimensional coordinate based on the image acquisition unit or the supporting unit. Pose of the bin-picking target object may be a 3-dimensional rotation angle of the object. Pose of the bin-picking target object may include a rotation angle based on any side of the bin-picking target object. 
         [0096]    According to an example, pose of the bin-picking target object may be estimated through machine learning. For example, when images of many bin-picking target objects are continuously taken, feature types (including at least one of silhouette information and edge information) may be classified and stored based on the classification. Thus, when a feature type of the bin-picking target object is recognized, the location and pose stored according to the recognized feature type may be estimated as location and pose of the bin-picking target object which is currently taken. 
         [0097]    In step  611 , the bin-picking target object may be bin-picked. 
         [0098]    The exemplary embodiment of the present disclosure can be implemented by various method. For example, the exemplary embodiment of the present disclosure can be implemented by using hardware, software or its combination. When they are implemented by software, they may be implemented as software executing in more than one processors using various operating systems or platforms. In addition, the software may be created by using any language among various appropriate programming languages or be compiled in machine language codes or intermediate codes executable in a framework or virtual machine. 
         [0099]    In addition, when the exemplary embodiment of the present disclosure is executed in more than one processors, the exemplary embodiment of the present disclosure may be implemented by processor readable media such as a memory, a floppy disk, a hard disk, a compact disk (CD), an optical disk or a magnetic tape, or the like in which more than one programs are recorded to conduct the implementation of various exemplary embodiments of the present disclosure.