Information processing apparatus, control method thereof and storage medium

An information processing apparatus for performing recognition processing by a recognizer for a position and orientation of a work subject to undergo work by a working unit of a robot arm. The information processing apparatus including an obtaining unit adapted to obtain, for each of a plurality of positions and orientations of the work subject, a position and an orientation of the working unit in which the working unit can perform the work, and a restriction unit adapted to restrict a position and an orientation of the work subject used in the recognition processing by the recognizer to a position and an orientation of the work subject corresponding to the position and the orientation of the working unit that have been obtained by the obtaining unit.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an information processing apparatus, control method thereof, and storage medium and, more particularly, to an information processing apparatus which generates a recognizer for recognizing a work subject and estimates the three dimensional positions and orientations of piled work subjects using the generated recognizer in order to perform predetermined work with a robot

2. Description of the Related Art

In the field of recognition using visual information, various researches and developments have been made in regard to a method of estimating the three dimensional position and orientation of a subject. In the field of industrial robots or experimental humanoid robots, three dimensional information is often used for the purpose of random picking and the like, and its necessity is growing. When the orientation of a target subject to be handled has a high degree of freedom, various orientations of the target subject need to be estimated three-dimensionally. As for a target subject with a known shape, its position and orientation are estimated using a three dimensional sensor such as a stereo camera or laser range finder. The correspondence between a three dimensional feature amount obtained from the three dimensional sensor and a three dimensional feature amount regarding a plurality of feature points on a model is obtained. Then, the position and orientation of the subject are calculated using rigid transformation. The position and orientation of a target subject are also estimated using a monocular camera. There is a method of recognizing various orientations as a multi-class classification problem.

Even if a target subject has a three dimensional degree of freedom, it may suffice to recognize only restricted orientations for practical use. In gripping work for a target subject with a robot hand, a detected target subject in an estimated orientation may not be able to be gripped owing to the relative positional relationship with the robot. A task to detect such a target subject is wasteful and can be ignored from the beginning without any problem. Especially in the field of industrial robots, this restriction is often essential. Taking the trouble to detect a target subject in an orientation incapable of gripping increases the memory capacity and prolongs the detection processing time in a recognizer used for detection.

In Japanese Patent No. 2555823, when collating parts based on the contours of images of piled parts, a collation limit value indicating a mismatch range permitted for a collation model in a reference orientation is set based on a tolerance limit angle in a grippable range. This method does not set a high degree of freedom of the orientation, and a target subject is detected by relaxing the collation limit value from one reference orientation to permit variations of the orientation from the reference orientation.

The method disclosed in Japanese Patent No. 2555823 takes account of an orientation range considering grippability, but does not examine a case in which the degree of freedom of the orientation is high. Further, it is difficult to apply this method when the appearance of a target subject greatly changes depending on the orientation.

In consideration of the aforementioned problems, the present invention provides a technique of reducing the memory capacity of a recognizer used in actual work for a target subject with a high degree of freedom of the orientation, and shortening the recognition processing time when detecting a target subject.

SUMMARY OF THE INVENTION

According to one aspect of the present invention, there is provided an information processing apparatus for performing recognition processing by a recognizer for a position and orientation of a work subject to undergo work by a working unit of a robot arm, comprising: an obtaining unit adapted to obtain, for each of a plurality of positions and orientations of the work subject, a position and orientation of the working unit in which the working unit can perform the work; and a restriction unit adapted to restrict a position and orientation of the work subject used in the recognition processing by the recognizer to a position and orientation of the work subject corresponding to the position and orientation of the working unit that have been obtained by the obtaining unit.

According to one aspect of the present invention, there is provided a method of controlling an information processing apparatus which includes an obtaining unit and a restriction unit, and performs recognition processing by a recognizer for a position and orientation of a work subject to undergo work by a working unit of a robot arm, comprising: causing the obtaining unit to obtain, for each of a plurality of positions and orientations of the work subject, a position and orientation of the working unit in which the working unit can perform the work; and causing the restriction unit to restrict a position and orientation of the work subject used in the recognition processing by the recognizer to a position and orientation of the work subject corresponding to the position and orientation of the working unit that have been obtained by the obtaining unit.

DESCRIPTION OF THE EMBODIMENTS

First Embodiment

An outline of an overall system using an information processing apparatus according to the present invention will be described with reference toFIG. 1. First, an outline of the system will be explained, and then details will be described. As an operation in online actual work, an image capturing unit R300captures an image of a work subject serving as the work target of a robot system R200. The image capturing information is sent to an information processing apparatus R100. The information processing apparatus R100estimates the position and orientation of the work subject. The information processing apparatus R100sends a work instruction generated based on the estimation result to a robot control unit R210of the robot system R200. The robot control unit R210operates a robot arm R220in accordance with the received work instruction, and performs predetermined work on the work subject. The information processing apparatus R100generates a recognizer offline in advance to estimate the position and orientation of a work subject in actual work. At this time, a position/orientation range to be obtained is restricted and set based on the orientation of the work subject based on the workability of the robot arm R220, and then the recognizer is generated.

A detailed hardware arrangement of the system will be exemplified with reference toFIG. 2. A robot controller A210and robot arm A220are building components of the robot system R200inFIG. 1. The robot controller A210corresponds to the robot control unit R210, and the robot arm A220corresponds to the robot arm R220. Work subjects A400are subjects to undergo work by the robot arm A220. A plurality of work subjects A400are placed on a tray A500. A camera A300corresponds to the image capturing unit R300inFIG. 1. The camera A300may be a monocular camera for obtaining image information, a stereo camera or TOF sensor for obtaining distance information, or an apparatus formed from a combination of a camera and projector using a light-section method, spatial coding method, or the like. The robot arm A220is equipped with an end effector A230for performing predetermined work (for example, gripping work) on a target subject. A computer A100includes the information processing apparatus R100inFIG. 1.

In online actual work, information about work subjects piled on the tray A500that has been obtained by image capturing by the camera A300is sent to the computer A100. The computer A100executes calculation using the recognizer, and estimates the positions and orientations of the work subjects A400on the tray A500. An instruction to perform predetermined work is encoded based on the position and orientation of a designated work subject A400, and is sent to the robot controller A210. The robot controller A210decodes the received instruction, and operates the robot arm A220and end effector A230to perform predetermined work on the recognized work subject A400. The recognizer in the information processing apparatus R100is a class classifier for classifying three dimensional positions and orientations of the work subject A400. The recognizer recognizes the position and orientation of a work subject by determining a class to which information obtained from the image capturing unit R300belongs. The embodiment explains one type of work subject, but work subjects are not always limited to one type. When recognizing a plurality of types of work subjects, recognizers can also be generated for the respective types of work subjects by increasing the number of classes. The information processing apparatus R100generates this recognizer offline in advance before actual work. At this time, to reduce the memory capacity of the recognizer and shorten the recognition processing time when detecting a work subject, the following processing is performed to restrict the position/orientation range of a work subject to be detected.

The functional arrangement of the information processing apparatus R100for restricting the position/orientation range of a work subject to be detected as described above will be explained with reference toFIG. 3. The information processing apparatus R100includes an orientation setting unit S1010, work state setting unit S1020, data storage unit D1030, virtual position setting unit S1040, robot parameter storage unit D1050, obtaining unit S1060, setting unit S1070, learning data generation unit S1080, recognizer generation unit S1090, recognizer storage unit D1100, recognition processing unit S1110, work instruction generation unit S1120, and calibration result storage unit D1130.

The orientation setting unit S1010generates an orientation set Θ={θj} (j=1, . . . , N) which may be handled by the recognizer. N is the total number of orientations which express all classes. The orientation setting unit S1010sends the generated orientation set Θ to the obtaining unit S1060.

The work state setting unit S1020sets the state of work on the work subject A400by the end effector A230. The work state is determined by predetermined work contents. For example, when the work contents indicate gripping work with fingers, the work state is expressed by the relative position and orientation of the work subject A400and end effector A230in a state in which the end effector A230grips the work subject A400with fingers at a gripping position.

The data storage unit D1030stores in advance model data of the work subject A400and data of the end effector A230as three dimensional model data. The physical coordinate system of the work subject A400and the end effector coordinate system of the end effector A230are set. The virtual position setting unit S1040sets a virtual position within a work area to be described later.

The robot parameter storage unit D1050stores known values determined by design values as characteristic parameters of the robot arm R220, such as the limit values of the link length and joint rotation angle. The obtaining unit S1060functions as an obtaining means for obtaining the orientation set Θ calculated by the orientation setting unit S1010, the relative position p and relative orientation EHpset by the work state setting unit S1020, and the virtual position Xkset by the virtual position setting unit S1040, and calculating and obtaining, based on them, the orientation of the work subject A400to be detected.

The setting unit S1070functions as a restriction means for setting restricted orientations necessary to generate a recognizer, based on workability in each orientation calculated by the obtaining unit S1060, so as to restrict the orientation range of the work subject A400. The learning data generation unit S1080generates learning data of the work subject A400in a restricted orientation. The recognizer generation unit S1090generates a recognizer using the learning data generated by the learning data generation unit S1080.

The recognizer storage unit D1100stores the recognizer generated by the recognizer generation unit S1090. The recognition processing unit S1110recognizes the position and orientation of the work subject A400using image data obtained by image capturing by the image capturing unit R300and the recognizer stored in the recognizer storage unit D1100. The recognition processing unit S1110sends the position and orientation of the work subject A400recognized by the recognition processing unit S1110to the work instruction generation unit S1120.

Based on the estimated position and orientation of the work subject A400recognized by the recognition processing unit S1110, the work instruction generation unit S1120generates an instruction to perform work on the work subject A400. The calibration result storage unit D1130stores information about the relative positional relationship between the camera and the robot. The work instruction generation unit S1120sets the target position of the robot from the relative positional relationship information, and encodes the target position as a robot instruction. The work instruction generation unit S1120transmits the encoded robot instruction to the robot control unit R210.

Range setting processing for the position and orientation of a work subject in the information processing apparatus R100will be described in detail. Processing of setting the position/orientation range of a work subject to be recognized in the embodiment is executed offline before actual work. This processing can be implemented by calculation inside the computer A100without connecting the image capturing unit R300and robot system R200to the information processing apparatus R100.

First, the orientation setting unit S1010generates the orientation set Θ={θj} (j=1, . . . , N) which may be handled by the recognizer. N is the total number of orientations which express all classes. Generation of the orientation set Θ will be explained with reference toFIG. 4. The three dimensional orientation set of the work subject A400is generated by combining a geodesic dome401and in-plane rotation402. The geodesic dome401is a well-known method of uniformly discretizing and expressing a spherical surface by recursively dividing triangular surface elements of a regular polyhedron into triangles of the same area. When the center of the geodesic dome401is regarded as a center404of the work subject, vertices of the regular polyhedron obtained from the geodesic dome401can be regarded as viewpoints403when looking down the work subject A400from various positions. Variations of the appearance of the work subject A400obtained from the respective viewpoints403in the geodesic dome401are accompanied by patterns of the in-plane rotation402. For example, when rotation patterns are given to the geodesic dome401having 162 viewpoints in every Π/4 within the plane, the orientation set contains N=162×8=1296. In general, the position and orientation of a subject are expressed by a transformation for moving a model511to an observation value512, as shown inFIG. 5. More specifically, the position can be expressed by a translation vector PWto a subject coordinate system C102in a camera coordinate system C101. The orientation can be expressed by an orientation matrix EW=[eWX, eWY, eWZ] based on a set of direction vectors along the respective axes of the subject coordinate system C102in the camera coordinate system C101. The elements eWX, eWY, and eWZof the orientation matrix EWare unit string vectors with a length of 1. Expression of the orientation of the recognizer need not consider the position, and the orientation matrix in the orientation θjof a work subject is given by EWj=[eWjX, eWjY, eWjZ]. The axes of the subject coordinate system are rotated within the plane using each viewpoint direction of the geodesic dome401as an axis, and the inclination of the viewpoint direction of the geodesic dome is added, obtaining the orientation matrix of each orientation. The generated orientation set Θ is sent to the obtaining unit S1060.

The work state setting unit S1020sets the state of work on the work subject A400by the end effector A230. The work state setting unit S1020reads model data of the work subject A400and data of the end effector A230from the data storage unit D1030. The data storage unit D1030stores in advance, as three dimensional model data, model data of the work subject A400and data of the end effector A230. The physical coordinate system of the work subject A400and the end effector coordinate system of the end effector A230are set.FIG. 6shows the relationship between an end effector coordinate system C103and a robot arm distal end coordinate system C104. To simplify calculation of the position and orientation of the distal end of the robot arm (to be described later), the end effector coordinate system C103may be set to coincide with the robot arm distal end coordinate system C104in an end effector connection state601in which the end effector is connected to the robot arm. In the following description, the end effector coordinate system C103and robot arm distal end coordinate system C104coincide with each other upon connection. The user designates a state in which the end effector A230performs predetermined work when the subject coordinate system C102of the work subject A400is set as a reference. The designation method does not depend on the interface format. For example, the user may adjust the state using a three dimensional display GUI or set it by inputting a numerical value.

Setting of the work state will be explained with reference toFIG. 7. As a relative position701of the work subject A400and end effector A230in a set work state, p=[Xp, Yp, Zp]Tis calculated as the position of the origin of the end effector coordinate system using the subject coordinate system C102as a reference. The superscript “T” means the transpose of a matrix.

Also, the orientation matrix EHp=[eHpX, eHpY, eHpZ] of the end effector A230using the subject coordinate system C102as a reference is calculated as a relative orientation702of the work subject A400and end effector A230. The calculated p and EHpare sent to the obtaining unit S1060. The work state is determined by predetermined work contents. For example, when the work contents indicate gripping work with fingers, as shown inFIG. 2or7, the work state is expressed by the relative position and orientation of the work subject A400and end effector A230in a state in which the end effector A230grips the work subject A400with fingers at a gripping position. When the work contents indicate chucking work with a nozzle, which will be described later with reference toFIG. 9B, the work state is expressed by the relative position and orientation of the work subject A400and end effector A230in a state in which the chucking surface of the work subject A400is chucked by the nozzle. The work contents are not limited to only a picking operation for the purpose of gripping or chucking the work subject A400. For example, the work contents may indicate fitting work to fit a fitting part A402gripped by the end effector A230as shown inFIG. 18Ainto a fitted part A401as shown inFIG. 18B. In this case, the fitted part A401can be regarded as the work subject A400, and the relative position and orientation of the end effector A230and fitted part A401in the fitting state as shown inFIG. 18Bcan be regarded as the work state. Work states to be set are not limited to one type. When a plurality of work states are possible, for example, when a work subject can be gripped at a plurality of angles, a plurality of work states can be designated. In this case, a plurality of relative positions p of the work subject A400and end effector A230, and a plurality of end effector orientation matrices EHpusing the subject coordinate system C102as a reference are set. If a relative approach path to the work subject A400is determined in picking work, fitting work, or the like, its intermediate path may be additionally set as a work state. For example, in fitting work as inFIGS. 18A and 18B, when the fitted part A401and fitting part A402need to have a positional relationship as shown inFIG. 18Aas a previous step to the fitting state shown inFIG. 18B, the state inFIG. 18Ais set as an additional work state.

After that, the virtual position setting unit S1040sets a virtual position in the work area.FIG. 8explains the virtual position. The virtual position is an arbitrary position within the work area and is a position where the center of a part is arranged. The work area is a range where the work subject A400is arranged before work, and is defined by, for example, the internal space of the tray A500. In some cases, the virtual position setting unit S1040selects a plurality of virtual positions801by a plurality of times of loops, which will be described later. Xkrepresents a virtual position801set by the kth setting, and is expressed using a robot coordinate system C107as a reference. As an initial setting (k=1), virtual position X1is set at one point at an end of the work area, the center of gravity of the work area, or the like. The set Xkis sent to the obtaining unit S1060.

The obtaining unit S1060obtains the orientation set Θ calculated by the orientation setting unit S1010, the relative position p and relative orientation EHpset by the work state setting unit S1020, and the virtual position Xkset by the virtual position setting unit S1040. Based on them, the obtaining unit S1060calculates the orientation of the work subject A400to be detected. The orientation of the work subject A400to be detected is calculated considering a position and orientation in which the distal end (working unit) of the robot arm can work. Assume that the origin of the physical coordinate system of the work subject A400is arranged at the virtual position Xk. A case is examined in which the orientation of the work subject A400is set to θjwhile fixing an origin901of the physical coordinate system, as shown inFIG. 9A. The orientation θjto be determined includes all orientations defined by Θ in general. However, the orientation θjmay be restricted in advance using known environmental information such as restrictions on the arrangement of the work subject A400, the characteristics of the end effector A230, and the tray arrangement. For example, when the arrangement of the work subject A400in the work area is determined to a certain degree by a part feeder or the like, the orientation θjto be determined may be restricted within the range. Alternatively, when the end effector is a chucking one using a nozzle911, as shown inFIG. 9B, and the orientation range where the end effector can chuck the work subject A400is experimentally known, the orientation θjto be determined may be restricted within the range. When the position of the tray A500in the work area is known, an orientation in which a tray wall surface921and the end effector A230interfere with each other with respect to the virtual position Xk, as shown inFIG. 9C, may be excluded in advance from orientations to be determined.

Whether the robot arm R220can work in the orientation θjto be determined is determined by solving inverse kinematics of the robot arm to determine whether the position and orientation of the distal end (working unit) of the robot arm allow work in terms of the robot structure. Although the analytic solution of inverse kinematics depends on the robot structure, it will be explained on the premise of an RPP-RPR six-axis multi-joint robot. Note that R is a rotational joint and P is a prismatic joint.

FIG. 10is a view for explaining the structure of the RPP-RPR robot. Assume that the robot coordinate system C107coincides with the coordinate system of a joint J1. For each joint Ji, the link length between Jiand Ji+1, is defined as li. Note that the length from the joint J6to the distal end of the robot arm is l6. Assume that J1and J2are at the same position and l1=0. φiis the right-handed rotation angle of the joint Ji, and a state in which all φiare 0 rad is defined as the initial state of the robot arm R220. The limit values of the link length and joint rotation angle are known values determined by design values as characteristic parameters of the robot arm R220, and are read from the robot parameter storage unit D1050.

In general, the rotation angles of rotational joints (φ1, φ4, and φ6in this example) do not have a limit value (or even if they have limit values, their ranges are wide). However, prismatic joints (φ2, φ3, and φ5in this example) often have narrow angle ranges owing to physical limitations posed by interference with adjacent links. The orientation matrix of each joint Jiis given by Ei=[eiX, eiY, eiZ] and is defined such that the orientation matrix Eiin the initial state of the robot arm becomes a unit matrix. The position of the joint Jiin the robot coordinate system is expressed by Qi=[Xi, Yi, Zi]T. Solving inverse kinematics equals calculating each joint angle φiwhen the position and orientation of the distal end (working unit) of the robot arm are determined. Letting QT=[XT, YT, ZT]Tbe the position of the distal end (working unit) of the robot arm and ET=[eTX, eTY, eTZ] be the orientation, two values are obtained as solutions of φ1in accordance with equations (1):

ϕ1={ATAN⁢⁢2⁢(Y5,X5)ATAN⁢⁢2⁢(Y5,X5)+π(1)
where ATAN 2 (a, b) is an arc tangent function which gives θ satisfying equations (2):

Note that equations (3) and (4) take the double sign in the same order. α, β, and γ are given by equations (5), (6), and (7):

The position Q5of the joint J5can be calculated from equation (8):
Q5=QT−(l5+l6)eTZ(8)

The joint angle φ4is an angle defined by the J3axis and J5axis, and is obtained by equation (9):
φ4=ATAN 2(e3Y·e5Y,(e3Y×e5Y)·e3Z)  (9)
where • is the inner product of vectors and × is the outer product of vectors. The vectors e3Y, e3Z, and e5Ycan be obtained from equations (10), (11), and (12):

Since no prismatic joint exists between the joint J5and the distal end of the robot arm and the orientation matrices E5and ETare equal, equation (16) holds:
e5Z=eTZ(16)

Similarly, since no prismatic joint exists between the joint J6and the distal end of the robot arm and the orientation matrices E6and ETare equal, equation (17) holds:
e6Y=eTY(17)

From sign inversion of equation (12), each of φ4, φ5, and φ6has two solutions because of the double sign in the same order. Therefore, a combination of φ1to φ6has eight solutions for one robot arm distal end position QTand one orientation ET.

A case in which the orientation of the work subject A400is θjwhen the center of the work subject A400in the subject coordinate system C102is set at Xkin the robot coordinate system C107will be examined. The position QHof the end effector A230in the robot coordinate system C107is obtained from the orientation matrix Ejand the relative position vector p of the work subject A400and end effector A230in accordance with equation (18):
QH=Xk+Ejp(18)

Further, an end effector orientation matrix EHusing the robot coordinate system C107as a reference can be obtained from equation (19):
EH=EHpEj(19)

As defined above, when the robot arm distal end coordinate system C104coincides with the end effector coordinate system C103, QT=QHand ET=EH. From this, a joint angle for the position Xkand orientation θjof the work subject A400can be analytically obtained. From equation (6), whether the solution of φ2can be obtained can be determined based on whether the value of cos β falls within [−1, 1]. Also from equations (6) and (7), whether the solution of φ3can be obtained can be determined. If the obtained values of φ1to φ6do not fall within the design movable range of the robot arm A220, it is determined that they fall outside the movable range. If φ1to φ6within the design movable range of the robot arm are obtained as a result of the determination, they are considered to satisfy the joint conditions. It can therefore be determined that when the work subject A400exists at the position Xkand takes the orientation θj, the distal end (working unit) of the robot arm can work. If the work state setting unit S1020sets a plurality of work states, inverse kinematics for p and ETpin each work state are calculated for the virtual position Xkand orientation θj. If there is even one orientation in which the distal end (working unit) of the robot arm can work, it is determined that they can work in the orientation θjat the virtual position Xk. The above-described solution of inverse kinematics changes depending on a combination of joints of the robot arm. A description of a solution to a robot arm having another arrangement will be omitted, and the solution is not limited to one for a robot arm having the above-mentioned arrangement.

Based on workability in each orientation calculated by the obtaining unit S1060, the setting unit S1070sets restricted orientations necessary to generate a recognizer, and sets the position/orientation range of the work subject A400. A processing sequence in the setting unit S1070will be explained with reference toFIG. 21.

First, a work possible/impossible vector calculation unit S1071functioning as a determination means and work information calculation means sets a restricted orientation and calculates a work possible/impossible vector based on the restricted orientation (work information calculation processing). An orientation θjfor which the obtaining unit S1060has determined that the distal end (working unit) of the robot arm can work at the virtual position Xkis set as a restricted orientation at the virtual position Xk. Based on this, a work possible/impossible vector Fk(Nth-order vector) to the virtual position Xkis defined. The jth element Fkjof the work possible/impossible vector Fkis defined such that Fkj=1 when the orientation θjis a restricted orientation; otherwise, Fkj=0. That is, the work possible/impossible vector Fkexpresses the presence/absence of a restricted orientation at the virtual position Xkby a binary vector. When the angle of view of the camera A300in the work area on an image is too narrow to ignore the perspective, the virtual position setting unit S1040may set only one virtual position Xk(k=1), and the obtained work possible/impossible vector F1may be set as a position/orientation range to be obtained. When the work area is captured at a wide angle of view, the position/orientation range to be obtained may change depending on the position within the frame. In this case, first, the virtual position setting unit S1040sets a plurality of types of virtual positions Xkof the work subject A400within the work area. Then, the virtual position setting unit S1040calculates the work possible/impossible vector Fkat each virtual position Xkin accordance with the determination result of each orientation by the obtaining unit S1060.

Initial virtual positions Xkare set roughly. For example, X1to X4may be set at four corners of a work area1101, as shown inFIG. 11A. When the height of a pile of the work subjects A400varies greatly so that the perspective by height cannot be ignored, a work area1102may be defined as a cubic area, and X1to X8may be defined at eight vertices, as shown inFIG. 11B.

As shown inFIG. 12A, a Voronoi boundary1211is set using, as centers, a plurality of virtual positions801Xkprojected within a work area set in a space in the image coordinate system, and divides the work area into a plurality of areas (area division processing). The Voronoi boundary is an area boundary obtained when the area is divided on the assumption that an arbitrary point X belongs to Xknearest to the point X.

A work possible/impossible state distance calculation unit S1072functioning as a determination means, work information calculation means, and area division means calculates a work possible/impossible state distance as the difference work possible/impossible vectors between respective positions. When two virtual positions Xkand Xlare adjacent to each other via the above-described Voronoi boundary, the Hamming distance between the work possible/impossible vectors Fkand Flat the virtual positions Xkand Xlis calculated and defined as a work possible/impossible state distance. When the work possible/impossible state distance between Fkand Flis nonzero, the distance between the virtual positions Xkand Xlin the image coordinate system is measured. The virtual position setting unit S1040newly adds and sets a virtual position if the distance is larger than an image search width (one pixel generally). It suffices to set a new virtual position1221at a middle point between Fkand Fl, as shown inFIG. 12B. The obtaining unit S1060determines workability, and a work possible/impossible vector is newly generated. After repeating this processing, if the distance between two points having a nonzero work possible/impossible state distance in the image coordinate system is equal to or smaller than the image search width for all existing virtual positions, virtual position addition processing ends.

A restricted orientation map generation unit S1073functioning as a generation means assigns the same index to areas having the same work possible/impossible vector based on the results obtained by the work possible/impossible state distance calculation unit S1072, and integrates them as an identical area. Accordingly, a restricted orientation map1231for respective positions at image coordinates is generated, as shown inFIG. 12C. The restricted orientation map indicates position/orientation ranges in the work area, and represents the work possible/impossible information distribution.FIG. 12Cexemplifies a restricted orientation map obtained by dividing the work area into a plurality of areas1234. Although a geodesic dome1232schematically shows a restricted orientation range1233, the range is determined including even in-plane rotation in practice. When there is only one virtual position, the restricted orientation map1231having the same restricted orientation in the entire work area is generated. Learning data is generated from the thus-obtained restricted orientation map, generating a recognizer used in actual work.

The learning data generation unit S1080generates learning data of the work subject A400in a restricted orientation. The learning data is created based on a three dimensional model.FIG. 13shows generation of a learning image. A projection matrix to the image plane when the work subject A400takes the restricted orientation θjis obtained from the relative positional relationship between the robot coordinate system C107and the camera coordinate system C101. For this reason, learning data1301can be generated by an existing computer graphics technique using three dimensional model data stored in the data storage unit D1030, and the calibration result between the robot and the camera that is stored in the calibration result storage unit D1130. Although learning data is explained as an image inFIG. 13, it essentially depends on an input data format used in the recognizer and is not limited to an image. For example, in recognition based on a depth map using a distance sensor, the depth is calculated from three dimensional model data to generate a depth map as learning data. The calibration result stored in the calibration result storage unit D1130includes a transformation matrix representing the relative positional relationship between the robot coordinates and the camera coordinates, a camera lens distortion parameter, and a projection matrix for representing the correspondence between the camera coordinates and the image coordinates. These calibration methods suffice to be known methods, and a detailed description thereof will be omitted.

The recognizer generation unit S1090generates a recognizer using learning data generated by the learning data generation unit S1080. Recognizers having different restricted orientations for respective areas of the restricted orientation map may be generated. Alternatively, restricted orientations in which the distal end (working unit) of the robot arm can work in all work areas may be selected to generate recognizers. At this time, the number of classes serving as estimated orientation variations in generated recognizers equals the number of orientations obtained as restricted orientations.

The recognizer can adopt any existing method, and the present invention is not limited by any method. For example, the recognizer may be an identifier using a well-known technique such as SVM (Support Vector Machine) or Randomized Tree. For example, when the recognizer employs SVM, positions and orientations to be obtained are restricted, decreasing the number of classes to be learned. In this case, the number of learning data used for learning decreases, and a higher learning speed can be expected. Since the class identification boundary decreases, a smaller number of support vectors can be expected and a sparser expression becomes possible. Hopes are high for a smaller-size recognizer, and a higher detection speed and higher detection accuracy in actual work.

When generating recognizers separately for the respective areas1234on the restricted orientation map1231, learning data of restricted orientations in which the distal end (working unit) of the robot arm can work in the respective areas are selected from learning data generated by the learning data generation unit S1080, and recognizers are generated for the respective areas. For example, when the restricted orientation combination pattern is divided into five areas, as shown inFIG. 12C, five recognizers are generated using learning data of restricted orientations corresponding to the respective five areas.

When selecting a restricted orientation in which the distal end (working unit) of the robot arm can work in all work areas, for example, in the case ofFIG. 12C, restricted orientations in all areas are ANDed, obtaining an orientation in which they can work in all work areas. Then, one recognizer is generated using learning data about a restricted area obtained by ANDing of all areas. A recognizer generated by the recognizer generation unit S1090is stored in the recognizer storage unit D1100. After that, offline processing ends.

Next, online processing will be explained. In online actual work, the recognition processing unit S1110first sends an image capturing signal to the image capturing unit R300to capture an image of the work area. The captured image data is then transmitted to the recognition processing unit S1110. The recognition processing unit S1110recognizes the position and orientation of the work subject A400using a recognizer stored in the recognizer storage unit D1100. When recognizers are prepared for respective areas, a recognizer is selected based on an image coordinate position in search. The position and orientation of the work subject A400that are recognized by the recognition processing unit S1110are sent to the work instruction generation unit S1120.

Based on the estimated position and orientation of the work subject A400that have been obtained from the recognition processing unit S1110, the work instruction generation unit S1120generates an instruction to perform work on the work subject A400. The target position of the robot is set in accordance with the relative positional relationship between the camera and the robot that has been obtained from the calibration result storage unit D1130. The target position is encoded as a robot instruction. The encoded robot instruction is transmitted to the robot control unit R210.

The robot control unit R210decodes the instruction received from the work instruction generation unit S1120to operate the robot arm R220and perform work on the work subject A400by the robot system.

According to the first embodiment, orientations of a work subject to be recognized can be restricted based on the workability of the robot arm for a work subject having a high degree of freedom of the orientation. The embodiment can therefore reduce the memory capacity of a recognizer used in actual work, shorten the recognition processing time when detecting a target subject, and expect higher recognition accuracy.

Second Embodiment

Learning data generated by the learning data generation unit S1080is data generated from three dimensional model data in the first embodiment, but learning data in the present invention is not limited to this. Learning data generated by the learning data generation unit S1080may be an image actually captured using the image capturing unit R300. An apparatus arrangement when generating learning data by actual image capturing will be described with reference toFIG. 14.

The arrangement and processing contents other than a learning data generation unit S1080and learning image storage unit D1140are the same as those in the first embodiment, and a description thereof except for these processing units will not be repeated.

The learning data generation unit S1080obtains actually captured images lvof a work subject that are obtained in advance at a plurality of viewpoints v (v=1, . . . , V) toward the work subject by using an image capturing unit R300. The learning data generation unit S1080stores the images lvin the learning image storage unit D1140. The image capturing interval between a plurality of viewpoints v is set smaller than the granularity of an orientation pattern generated by an orientation setting unit S1010. When obtaining these work subject images, the image capturing unit R300preferably has the same settings as those in online actual work, but need not always have them.

After obtaining images, the learning data generation unit S1080first obtains the three dimensional model of a work subject A400from a data storage unit D1030which stores CAD data. Based on the three dimensional model, the learning data generation unit S1080associates image coordinates on a learning image obtained from each viewpoint with camera coordinates. By matching processing manually or using a tracking tool based on a well-known technique, the learning data generation unit S1080calculates the position and orientation, in the camera coordinate space, of the work subject on the learning image lvread out from the learning image storage unit D1140. Accordingly, the position XVand orientation θvof the work subject A400on the learning image in the camera coordinate space are obtained. By perspectively projecting CAD data, a work subject area on the image is obtained. The position of the work subject A400on the image with respect to the center of the subject coordinate system C102is normalized. The area of the work subject A400is extracted and used as a learning image.

For the obtained learning image data, an image lvin an orientation closest to a restricted orientation θjcalculated by a setting unit S1070is handled as a learning image in the orientation θj. At this time, θjis updated as θv. A recognizer generation unit S1090performs learning for a recognizer using the assigned learning image as learning data.

According to the second embodiment, orientations of a work subject to be recognized can be restricted based on the workability of the robot arm for a work subject having a high degree of freedom of the orientation. The embodiment can reduce the memory capacity of a recognizer used in actual work, shorten the recognition processing time when detecting a target subject, and expect higher recognition accuracy.

Third Embodiment

Unlike the first and second embodiments, the third embodiment is not limited to an arrangement in which all processes by an obtaining unit S1060are executed by calculation inside a computer A100. Processes by the obtaining unit S1060may be implemented by actually operating a robot arm R220.

FIG. 19shows the functional arrangement of an information processing apparatus R100when obtaining a position/orientation range by actually operating the robot arm R220. The functions of processing units except for the obtaining unit S1060are the same as those in the first or second embodiment, and a description thereof except for the obtaining unit S1060will not be repeated.

The obtaining unit S1060calculates the orientation of a work subject A400to be detected, based on the orientation set Θ set by an orientation setting unit S1010, the relative position p and relative orientation EHpset by a work state setting unit S1020, and the virtual position Xkset by a virtual position setting unit S1040. The orientation of the work subject A400to be detected is calculated considering a position and orientation in which the distal end (working unit) of the robot arm can work.

Assume that the origin of the physical coordinate system of the work subject A400is arranged at the virtual position Xk. Letting Ejbe the orientation matrix of the orientation θjfor which it is determined whether the distal end (working unit) of the robot arm can work, the target position QTand target orientation ETof the distal end (working unit) of the robot arm in the robot coordinate system are given by equations (20) and (21), respectively:
QT=Xk+Ejp(20)
ET=EHpEj(21)

The obtaining unit S1060sends an instruction to a robot control unit R210to operate the robot arm R220to the target position QTand target orientation ET. When the robot control unit R210has successfully moved the robot arm R220to the target position QTand target orientation ET, the obtaining unit S1060determines that the orientation θjis a workable orientation. To the contrary, when the robot control unit R210has failed in moving the robot arm R220to the target position QTand target orientation ET, that is, when an error occurs during movement of the robot arm, the obtaining unit S1060determines that θjis an unworkable orientation.

According to the third embodiment, orientations of a work subject to be recognized can be restricted based on the workability of the robot arm for a work subject having a high degree of freedom of the orientation. The embodiment can reduce the memory capacity of a recognizer used in actual work, shorten the recognition processing time when detecting a target subject, and expect higher recognition accuracy.

Fourth Embodiment

The present invention is not limited to the arrangement described in the first embodiment, and can take various arrangements. An orientation need not always be calculated in processing by the orientation setting unit S1010in the first embodiment. For example, an orientation storage unit D1010may replace the orientation setting unit S1010as in an arrangement shown inFIG. 15A. In this case, the orientation storage unit D1010stores a predetermined orientation set, and an obtaining unit S1060reads out the orientation set from the orientation storage unit D1010.

Similarly, a virtual position need not always be calculated in processing by the virtual position setting unit S1040in the first embodiment. For example, a virtual position storage unit D1040may replace the virtual position setting unit S1040as in an arrangement shown inFIG. 15B. In this case, the virtual position storage unit D1040stores a plurality of virtual positions set in advance, and the obtaining unit S1060reads out a virtual position from the virtual position storage unit D1040.

Also, processing by the work state setting unit S1020in the first embodiment need not always be set by the user via a user interface. For example, a work state storage unit D1020may replace the work state setting unit S1020as in an arrangement shown inFIG. 15C. In this case, the work state storage unit D1020stores the relative position/orientation relationship between a work subject and an end effector in work that is set by CAD or the like, and the obtaining unit S1060reads out the relative position/orientation relationship from the work state storage unit D1020.

Note that combinations of the orientation setting unit S1010or orientation storage unit D1010, the virtual position setting unit S1040or virtual position storage unit D1040, and the work state setting unit S1020or work state storage unit D1020are arbitrary. Hence, various arrangements (not shown) are conceivable.

To visualize and confirm a restricted orientation map obtained by a setting unit S1070, a display unit S1150may be added to the apparatus arrangement to display the restricted orientation map, as shown inFIG. 17.

FIG. 16shows a display example of the restricted orientation map on the display unit S1150. A window1601displayed by the display unit S1150represents a restricted orientation. By using a geodesic dome401and the computer graphics-based appearance of a work subject A400, the display unit S1150displays the contents in the range of geometrical information selected by the setting unit S1070in an area corresponding to a work area selected by the user on the screen. The user can visually confirm a set restricted orientation, and check a setting error, data registration error, and the like before actual work.

Offline processing by the information processing apparatus R100described in the first, second, and third embodiments can also be implemented as a series of information processes. The processing sequence will be explained with reference to the flowchart ofFIG. 20.

In orientation setting step P1010, the orientation setting unit S1010generates an orientation set which may be handled by the recognizer.

In work state setting step P1020, the work state setting unit S1020sets the state of work on the work subject A400by an end effector A230.

In virtual position setting step P1040, the virtual position setting unit S1040sets a virtual position within the work area.

In obtaining step P1060, the obtaining unit S1060calculates the orientation of the work subject A400to be detected in consideration of possible positions and orientations of the distal end (working unit) of the robot arm.

In setting step P1070, the setting unit S1070sets restricted orientations necessary to generate a recognizer, and sets the position/orientation range of the work subject A400.

In learning data generation step P1080, the learning data generation unit S1080generates learning data of the work subject A400in a restricted orientation.

In recognizer generation step P1090, the recognizer generation unit S1090generates a recognizer using the learning data generated by the learning data generation unit S1080. Then, the process end.

According to the fourth embodiment, orientations of a work subject to be recognized can be restricted based on the workability of the robot arm for a work subject having a high degree of freedom of the orientation. The embodiment can reduce the memory capacity of a recognizer used in actual work, shorten the recognition processing time when detecting a target subject, and expect higher recognition accuracy.

Other Embodiments

This application claims the benefit of Japanese Patent Application No. 2011-087698 filed on Apr. 11, 2011, which is hereby incorporated by reference herein in its entirety.