Patent ID: 12186920

PREFERRED MODE FOR CARRYING OUT THE INVENTION

An embodiment of the present invention will be described below.FIG.1is a diagram illustrating a configuration of a robot system1. As illustrated inFIG.1, the robot system1includes a robot2, an arm3, a visual sensor4, a visual sensor controller5, and a robot controller6. For example, the robot system1recognizes the position of a target W based on an image of the target W captured by the visual sensor4, and performs operations such as handling the target W and processing the target W.

The arm3of the robot2has a hand or a tool attached to the distal end of the arm3. The robot2performs operations such as handling or processing the target W under control of the robot controller6. The visual sensor4is attached to the distal end of the arm3of the robot2.

The visual sensor4captures images of the target W under control of the visual sensor controller5. The visual sensor4may be configured as a two-dimensional camera that has an imaging element constituted by a charge coupled device (CCD) image sensor and an optical system including a lens. Alternatively, the visual sensor4may be configured as a stereo camera or the like that is capable of conducting three-dimensional measurement.

The robot controller6executes a motion program of the robot2to control motion of the robot2. When controlling the motion of the robot2, the robot controller6compensates the motion of the robot2to cause the robot2to perform a predetermined operation with respect to the position of the target W detected by the visual sensor controller5.

FIG.2is a diagram illustrating the configuration of the visual sensor controller5and the configuration of the robot controller6. The visual sensor controller5includes a storage unit51and a control unit52. The storage unit51is a storage device including, for example, read only memory (ROM) that stores an operating system (OS), application programs, etc., random access memory (RAM), a hard disk drive or a solid state drive (SSD) that stores various other pieces of information, etc.

The storage unit51includes a model pattern storage unit511and a calibration data storage unit512. The model pattern storage unit511stores a model pattern produced by modeling an image of the target W. For example, the model pattern represents the characteristics of the image of the target W.

The calibration data storage unit512stores calibration data for associating a robot coordinate system with an image coordinate system. The robot coordinate system serves as a reference based on which the motion of the robot2is controlled. The image coordinate system serves as a reference based on which the visual sensor4performs measurement processing. For the calibration data, various formats and various methods for determining the formats have been proposed. Any of the proposed format and methods may be employed.

The control unit52is a processor such as a central processing unit (CPU), and executes programs stored in the storage unit51to function as a first acquisition unit521, a second acquisition unit522, a determination unit523, an estimation unit524, a calibration unit525, and an annunciation unit526.

The first acquisition unit521acquires a first position Pw of the target W in the robot coordinate system, based on a first image of the target W captured by the visual sensor4and the calibration data.

The second acquisition unit522acquires a second position Ps of the target W in the robot coordinate system, based on the first image and a second image of the target W.

The determination unit523determines whether a difference between the first position Pw and the second position Ps is within a predetermined range.

In a case where the determination unit523determines that the difference is outside the predetermined range, the estimation unit524estimates a cause of an abnormality in the motion compensation of the robot2, based on the first position Pw, the second position Ps, and positions of the robot2corresponding to the first position Pw and the second position Ps.

Alternatively, in the case where the determination unit523determines that the difference is outside the predetermined range, the visual sensor4captures, at a plurality of locations, a plurality of images of the target W. The estimation unit524then estimates a cause of the abnormality in the motion compensation of the robot2, based on the plurality of images and the plurality of locations.

The calibration unit525performs calibration of the visual sensor4. Details of the calibration of the visual sensor4will be described later. In the case where the determination unit523determines that the difference is outside the predetermined range, the annunciation unit526announces the presence of an abnormality in the motion compensation of the robot2. Further, the annunciation unit526announces the cause of the abnormality estimated by the estimation unit524. For example, the annunciation unit526provides the annunciation in such a manner that an abnormality message is displayed on a display (not shown) or the like.

The robot controller6includes a motion control unit61. The motion control unit61executes the motion program of the robot2to control the motion of the robot2.

Next, the details of the operation performed by the visual sensor controller5will be described with reference toFIGS.3to8. First, the calibration unit525calibrates the visual sensor4in the following manner, for example.FIG.7is a diagram illustrating a calibration jig7for calibration.FIG.8is a diagram illustrating the calibration that is performed using the calibration jig7. The calibration jig7has a dot pattern illustrated inFIG.7, which is recognizable to the visual sensor4. The dot pattern is composed of large and small dots arranged in a grid pattern. The large dots are arranged in an L-shape and represent a coordinate system of the calibration jig7.

The calibration unit525presets a position of the calibration jig7viewed from the robot coordinate system. The intervals between the dots of the calibration jig7are known in advance from, for example, the design drawing of the calibration jig7. Accordingly, the calibration unit525specifies the known values as the intervals between the dots and stores data regarding the intervals between the dots as a part of the calibration data in the calibration data storage unit512.

The visual sensor4captures an image of the calibration jig7. The calibration unit525acquires a position where the robot was when the image of the calibration jig7was captured. The motion control unit61causes the robot2to perform vertical motion with respect to the calibration jig7, and the visual sensor4then captures an image of the calibration jig7. The calibration unit acquires a position where the robot was when the image of the calibration jig7was captured.

The calibration unit525calibrates the visual sensor4by using information regarding the positions of the plurality of dots of the calibration jig7in the image coordinate system and information regarding the positions of the plurality of dots of the calibration jig7in the robot coordinate system, the information both being based on the captured images of the calibration jig7.

The calibration of the visual sensor4allows external parameters and internal parameters of the visual sensor4to be determined. Here, the external parameters are information regarding the position and posture of the visual sensor, whereas the internal parameters are information regarding conditions of the optical system of the visual sensor, such as the focal length of a lens, distortion of the lens, and the dimension of a light-receiving element.FIG.3is a diagram illustrating movement of the visual sensor4. First, the motion control unit61of the robot controller6moves the robot2such that the target W is within an imaging range. The visual sensor4then captures a first image of the target W. At this time, the first acquisition unit521stores the position where the robot2was at the time of capturing the image.

The first acquisition unit521detects the target W from a predetermined range of the captured first image, by referring to the model pattern stored in the model pattern storage unit511. In this way, the first acquisition unit521acquires the position and posture of the target W in the image coordinate system.

The first acquisition unit521transforms the position of the target W in the image coordinate system to a three-dimensional position Pw in the robot coordinate system (world coordinate system), based on the calibration data stored in the calibration data storage unit512and the position of the robot2at the time of capturing the image. In this way, the first acquisition unit521acquires the first position Pw of the target W in the robot coordinate system.

Here, when a two-dimensional camera is used to detect the target W, the first acquisition unit521acquires the first position Pw as a three-dimensional position, while assuming that the location at which the target W is detected is on a certain plane (compensation plane). This compensation plane is typically set by a user using an operation panel (not shown) or the like connected to the visual sensor controller5.

Usually, the motion compensation of the robot2is implemented using the first position Pw of the target W. At this time, absolute position compensation and relative position compensation are employed as the motion compensation methods. According to the absolute position compensation, the motion control unit61moves a tool center point (TCP)8(seeFIG.8) of the robot2to the first position Pw. According to the relative position compensation, the control unit52determines in advance a position as a reference (reference position) of the target and calculates, as a compensation value, a difference between the reference position and the first position Pw. The robot controller6then implements motion compensation of the robot2by adding the calculated compensation value to a preset motion of the robot2.

In some cases, the above-described settings of the motion compensation are determined incorrectly due to various setting errors. To identify an error in the motion compensation of the robot2, the robot system1according to the present embodiment performs the following processing.

As illustrated inFIG.3, the motion control unit61of the robot controller6moves the robot2from the reference position by a predetermined distance D in a direction orthogonal to an optical axis of the visual sensor4. The visual sensor4captures a second image of the target W, at a location as a result of movement of the robot2from the reference position by the predetermined distance D. Here, the predetermined distance D refers to a distance from the reference position to a position where the target W is located after being sufficiently moved and where the target W can be detected by the visual sensor4.

FIG.4is a diagram illustrating the second position Ps.FIG.5is a diagram illustrating the relationship between the first position Pw and the second position Ps. The second acquisition unit522detects the target W from the first image of the target W captured at the location corresponding to the reference position and from the second image of the target W captured at the location as a result of movement of the robot2by the predetermined distance D. As a result, the position of the target in the image coordinate system and viewed from two viewpoints A and B is determined.

The second acquisition unit522then calculates two lines of sight from the position of the target W in the image coordinate system. The second acquisition unit522determines an intersection of the two lines of sight or a point where the distance between the two lines of sight is the shortest, as the second position Ps of the target W.

In the example illustrated inFIG.4, the two lines of sight do not intersect. Accordingly, the second acquisition unit522acquires, as the second position Ps of the target W, the point where the distance between the two lines of sight is the shortest. An image capturing unit may capture a plurality of images as the second images. In this case, the image capturing unit captures an image of the target W at a location as a result of the movement of the robot2by the predetermined distance D, and then captures another image of the target W at another location as a result of further movement of the robot2by the predetermined distance D from the prior image capturing position. This process is repeated.

In the case where the image capturing unit captures a plurality of images (e.g., three or more images) as the second images, the second acquisition unit522calculates three or more viewpoints from the position of the target W in the image coordinate system. In this case, the second acquisition unit522calculates, as the second position Ps of the target W, the intersection of the three or more lines of sight or the position where the distance between the three or more line of sight is the shortest by using the least-squares method.

Here, since the target W remains unmoved, while the position of the target W varies in the images, a positional discrepancy (difference) between the first position Pw and the second position Ps should be within a tolerance. Therefore, the determination unit523determines whether the difference between the first position Pw and the second position Ps is within a predetermined range.

If the determination unit523determines that the difference is within the predetermined range, the estimation unit524determines that the motion compensation of the robot2is normally implemented. If the determination unit523determines that the difference is outside the predetermined range, the estimation unit524determines that there is abnormality in the motion compensation of the robot2, and estimates a cause of the abnormality in the motion compensation of the robot2, based on the first position Pw, the second position Ps, the positions of the robot2corresponding to the first position Pw and the second position Ps.

Specifically, in the case where the determination unit523determines that the difference is outside the predetermined range and the two lines of sight do not intersect with each other (e.g., as inFIG.4), the estimation unit524estimates that the calibration data stored in the calibration data storage unit512is incorrect.

In the case where the determination unit523determines that the difference is outside the predetermined range and the two lines of sight intersects with each other, the estimation unit524estimates that a compensation plane is incorrect as illustrated inFIG.5.

When the calibration data is estimated to be incorrect, calibration is performed according to a different calibration method, whereby the incorrect item of the calibration data can be estimated.FIG.6illustrates movement of the visual sensor4. As illustrated inFIG.6, the motion control unit61moves the robot2from the reference position along a grid pattern in directions orthogonal to the optical axis of the visual sensor4. The visual sensor4captures images of the target W, at a plurality of locations as a result of movement of the robot2. In this way, the plurality of images of the target W are captured by the visual sensor4.

The calibration unit525detects the target W from each of the plurality of captured images, and acquires a position Pci of the target W in the image coordinate system. Further, the calibration unit525determines a position Psi of the target W as viewed from the position of a flange of the robot2at the time of capturing each of the images. As can be seen, the motion control unit61moves the robot2ntimes (i=1, 2, 3, . . . n), and the visual sensor4captures images of the target W n times. In this way, the calibration unit525acquires n pairs of (Pci, Psi), and calibrates the visual sensor4using the n pairs of (Pci, Psi).

The estimation unit524compares external parameters and internal parameters of the visual sensor4determined in this manner, with the aforementioned external parameters and internal parameters determined in advance by the calibration unit525(seeFIGS.7and8), and thereby can determine an abnormality in the external parameters and internal parameters.

For example, when the external parameters are incorrect, the estimation unit524estimates that the internal parameters determined by way of the calibration are incorrect or that the position of the calibration jig7is incorrect. When the internal parameters are incorrect, the estimation unit524estimates that the specified intervals between the dots are incorrect or that inappropriate dots have been used in the calibration.

FIG.9is a flowchart illustrating processing performed by the visual sensor controller5. In Step S1, the motion control unit61moves the robot2such that the target W is within the imaging range, and the visual sensor4captures a first image of the target W. At this time, the first acquisition unit521stores the position where the robot2was at the time of capturing the image.

In Step S2, the first acquisition unit521detects the target W from a predetermined range of the captured first image, by using the model pattern stored in the model pattern storage unit511. The first acquisition unit521transforms the position of the target W in the image coordinate system to a first position Pw in the robot coordinate system, based on the calibration data stored in the calibration data storage unit512and the position where the robot2was at the time of capturing the image.

In Step S3, the motion control unit61of the robot controller6moves the robot2from the reference position by a predetermined distance D in a direction orthogonal to the optical axis of the visual sensor4. The visual sensor4captures a second image of the target W, at a location as a result of movement of the robot2by the predetermined distance D.

In Step S4, the second acquisition unit522detects the target W from the first image of the target W captured at the location corresponding to the reference position and from the second image of the target W captured at the location as a result of the movement of the robot2by the predetermined distance D.

The second acquisition unit522calculates two lines of sight from the position of the target W in the image coordinate system. The second acquisition unit522acquires an intersection of the two lines of sight or a point where the distance between the two lines of sight is the shortest, as the second position Ps of the target W.

In Step S5, it is determined whether the difference between the first position Pw and the second position Ps is within a predetermined range. If the difference is within the predetermined range (YES), the process proceeds to Step S6. If the difference is outside the predetermined range (NO), the process proceeds to Step S7.

In Step S6, the estimation unit524determines that the motion compensation of the robot2is normally implemented, and then, the processing ends.

In Step S7, the estimation unit determines that there is an abnormality in the motion compensation of robot2, and estimates a cause of the abnormality in the motion compensation of the robot2, based on the first position Pw, the second position Ps, and the positions of the robot2corresponding to the first position Pw and the second position Ps. In Step S8, the annunciation unit526announces the presence of the abnormality in the motion compensation of the robot. Further, the annunciation unit526announces the cause of the abnormality estimated by the estimation unit524.

In Step S9, the motion control unit61moves the robot2from the reference position along a grid pattern in directions orthogonal to the optical axis of the visual sensor4. The visual sensor4captures an image of the target W at each of a plurality of locations as a result of movement of the robot2. In this way, the visual sensor4captures a plurality of images of the target W.

In Step S10, the calibration unit525detects the target W from each of the plurality of captured images and acquires a position Pci of the target W in the image coordinate system. Further, the calibration unit525determines a position Psi of the target W as viewed from the position of the flange of the robot2at the time of capturing each image. The calibration unit525calibrates the visual sensor4by using n pairs (Pci, Psi) acquired in the above-described manner.

According to the present embodiment, the robot system1includes: the visual sensor4that captures the first image of the target W, at a location corresponding to the reference position of the robot2, and captures the second image of the target W, at a location as a result of movement of the robot2from the reference position by the predetermined distance D; the calibration data storage unit512that stores the calibration data for associating the robot coordinate system with the image coordinate system, the robot coordinate system serving as a reference based on which motion of the robot2is controlled, the image coordinate system serving as a reference based on which the visual sensor4performs measurement processing; the first acquisition unit521that acquires the first position Pw of the target W in the robot coordinate system, based on the first image and the calibration data; the second acquisition unit522that acquires the second position Ps of the target W in the robot coordinate system, based on the first image and the second image; and the determination unit523that determines whether the difference between the first position Pw and the second position Ps is within the predetermined range. Due to this feature, the robot system1can determine an abnormality in the motion compensation of the robot2, based on the difference between the first position Pw and the second position Ps.

The robot system1further includes the estimation unit524that estimates, in the case where the determination unit523determines that the difference is outside the predetermined range, a cause of the abnormality in the motion compensation of the robot2, based on the first position, the second position, and the positions of the robot2corresponding to the first position and the second position. When there is an abnormality in the motion compensation of the robot2, this feature allows the robot system1to estimate a cause of the abnormality. Thus, the robot system1can properly implement the motion compensation of the robot2, based on the estimated cause of the abnormality in the motion compensation of the robot2.

In the robot system1, in the case where the determination unit523determines that the difference is outside the predetermined range, the visual sensor4captures, at a plurality of locations, a plurality of images of the target W, and the estimation unit524estimates a cause of the abnormality in the motion compensation of the robot2, based on the plurality of images and the plurality of positions. Due to this feature, the robot system1can properly implement the motion compensation of the robot2.

The visual sensor4may capture a plurality of second images of the target W, and the second acquisition unit522may acquire a plurality of second positions of the target W in the robot coordinate system, based on the first image and the plurality of second images.

Due to this feature, the robot system1can determine the cause of the abnormality in the position compensation of the robot2, using the plurality of second images and the plurality of second positions. The robot system1may further include a setting change unit that changes the setting of the motion compensation of the robot, based on the cause of the abnormality in the motion compensation of the robot, the cause having been estimated by the estimation unit524. This feature allows the robot system1to appropriately change the setting of the motion compensation of the robot.

In the embodiment described above, the robot system1uses the first image and the second image. However, the robot system may use only an image or images corresponding to the second image, for example. For example, the robot system1may include: a first acquisition unit521that acquires a first position of the target W in the robot coordinate system, based on a plurality of second images and calibration data; and a second acquisition unit522that acquires a second position of the target W in the robot coordinate system based on the plurality of second images.

Instead of the configuration of the embodiment described above, the determination unit523of the robot system1can use, for example, a method for checking respective elements of a difference in three-dimensional position between the first position Pw and the second position, or a method for checking a distance. In other words, the determination unit523may check whether a relationship between the first position Pw and the second position Ps is within a predetermined range, based on a difference between the first position Pw and the second position Ps. Further, instead of the configuration of the embodiment described above, the determination unit523of the robot system1may use a method for measuring a distribution or standard deviation of three or more positions. In other words, the determination unit523may determine whether a relationship between the first position and a plurality of positions is within a predetermined range, based the first position and the plurality of positions.

It should be noted that the embodiments described above are not intended to limit the present invention. The effects described in the embodiments are merely most favorable effects exerted by the present invention. The effects of the present invention are not limited to those described above.

EXPLANATION OF REFERENCE NUMERALS

1: Robot system2: Robot3: Arm4: Visual sensor5: Visual sensor controller6: Robot controller61: Motion control unit511: Model pattern storage unit512: Calibration data storage unit521: First acquisition unit522: Second acquisition unit523: Determination unit524: Estimation unit525: Calibration unit526: Annunciation unit