Patent ID: 12236663

PREFERRED MODE FOR CARRYING OUT THE INVENTION

An embodiment of the present invention will be described below.FIG.1is a diagram illustrating a configuration of an image processing system1according to the embodiment of the present invention. The image processing system1shown inFIG.1detects an image of an object W from an image taken by a visual sensor4serving as an imaging device whose position relative to the object W is changed by a robot2. In the following description, an “image coordinate system” refers to a coordinate system (in two dimensions) defined in an image, and a “sensor coordinate system” refers to a coordinate system (in three dimensions) as viewed from the visual sensor4. A “robot coordinate system” (mechanical coordinate system) is a coordinate system (in three dimensions) as viewed from the robot2.

The image processing system1of the present embodiment includes the robot2, an arm3, the visual sensor4, a visual sensor control device5, a robot control device6, and a console7. The image processing system1recognizes the position of the object W based on, for example, an image of the object W taken by the visual sensor4, and handles or processes the object W.

A hand or a tool is attached to a tip of the arm3of the robot2. The robot2performs work, such as handling and processing of the object W, under control of the robot control device6. The visual sensor4is also attached to the tip of the arm3of the robot2.

The visual sensor4is an imaging device that takes an image of the object W under control of the visual sensor control device5. The visual sensor4may be a two-dimensional camera having an image pickup device including a charge coupled device (CCD) image sensor and a lens, or a stereoscopic camera that can perform three-dimensional measurement. In the present embodiment, the visual sensor4takes an image of the object W fixed on a worktable8.

The robot control device6executes an operation program of the robot2to control the operation of the robot2. The robot control device6operates the robot2, and the position of the visual sensor4relative to the object W changes.

The console7is a reception unit via which a user handles the image processing system1. The user enters various types of commands to the visual sensor control device5via the console7.

FIG.2is a view illustrating the configurations of the visual sensor control device5and the robot control device6.FIG.3is a functional block diagram schematically illustrating the functions related to the image processing executed by a controller. The visual sensor control device5of the present embodiment includes a storage51and a controller52.

The storage51is a storage device, such as a read only memory (ROM) that stores an operating system (OS) and application programs, a random-access memory (RAM), and a hard disk drive and a solid-state drive (SSD) that store various types of information.

The storage51includes a model pattern storage511and a calibration data storage512.

The model pattern storage511will be described below. The model pattern storage511stores a model pattern which is a modeled image of the object W. Examples of the model pattern will be described later.

The calibration data storage512stores calibration data that associates a robot coordinate system which is a standard for control of the operation of the robot2with an image coordinate system which is a standard for measurement by the visual sensor4. The calibration data may have any format, and may be calculated by any proposed method.

The controller52is a processor such as a central processing unit (CPU), and functions as an image processing unit that executes various types of control of the image processing system1.

FIG.3is a functional block diagram schematically illustrating the functions related to the image processing executed by the controller52. As shown inFIG.3, the controller52includes, as functional units, a feature point extraction unit521, a calibration unit522, a determination unit523, and a display processing unit524. The functional units of the controller52work when the program stored in the storage51runs.

The feature point extraction unit521extracts feature points from an input image taken by the visual sensor4. Various types of methods can be used to extract the feature points. In the present embodiment, edge points that have a great luminance gradient in the image and can be used to acquire a contour shape of the object are extracted as the feature points. An image of a contour line of the object W generally has a great luminance gradient. Thus, the contour shape of the object W can be acquired by using the edge points as the feature points. A Sobel filter or a Canny edge detector may be used for the extraction of the edge points.

The calibration unit522executes transformation between the positions of two-dimensional points in the image coordinate system and the positions of three-dimensional points in the robot coordinate system based on the positional relationship between the visual sensor4and the object W and the calibration data stored in the calibration data storage512. For example, when data of the three-dimensional points in the robot coordinate system is given, the calibration unit522calculates the position of an image of the three-dimensional points in the image taken by the visual sensor4, i.e., the two-dimensional points in the image coordinate system. When data of the two-dimensional points in the image coordinate system is given, the calibration unit522calculates a line of sight in the robot coordinate system (world coordinate system). The line of sight is a three-dimensional straight line passing a point of regard and a focus of the visual sensor4. The point of regard is a three-dimensional point of the object W in the robot coordinate system (three-dimensional positional information of the feature point). The calibration unit522executes transformation of the data of the two-dimensional points into the three-dimensional points, i.e., data indicating the three-dimensional position, based on the calculated line of sight.

The determination unit523compares a feature point cloud (an edge point cloud) extracted from the input image acquired by the imaging device with the model pattern stored in the model pattern storage511, and detects the object based on the degree of matching between them.

The display processing unit524executes a process of displaying, on the console7, the result of determination by the determination unit523and an operation screen for setting a compensation plane (imaginary plane) which will be described later.

The robot control device6includes an operation controller61. The operation controller61runs an operation program of the robot2in accordance with a command from the visual sensor control device5to control the operation of the robot2.

How the feature point extraction unit521creates the model pattern in the image coordinate system will be described below.FIG.4is a view illustrating the model pattern formed of a plurality of feature points. As shown inFIG.4, the model pattern used in the present embodiment is a model pattern formed of a plurality of feature points P_i. As shown inFIG.4, the model pattern is formed of the plurality of feature points P_i (i=1 to NP). In this example, the feature points P_i forming the model pattern are stored in the model pattern storage511.

The position and posture of the feature points P_i forming the model pattern may be represented in any form. For example, a coordinate system of the model pattern is defined (will be hereinafter referred to as a “model pattern coordinate system”), and the position and posture of each of the feature points P_i forming the model pattern are represented by a position vector or a direction vector viewed from the model pattern coordinate system.

The origin O of the model pattern coordinate system may be defined as appropriate. For example, any one point selected from the feature points P_i forming the model pattern may be defined as the origin, or the center of gravity of all the feature points P_i forming the model pattern may be defined as the origin.

The posture (axial direction) of the model pattern coordinate system may be defined as appropriate. For example, the posture may be defined so that the image coordinate system and the model pattern coordinate system are parallel to each other in an image in which the model pattern is created. Alternatively, any two points may be selected from the feature points forming the model pattern so that a direction from one of the two points to the other is defined as an X-axis direction and a direction orthogonal to the X-axis direction is defined as a Y-axis direction. The posture may be defined so that the image coordinate system and the model pattern coordinate system are parallel to each other in an image in which a model pattern50is created. Thus, the model pattern coordinate system and the origin O can be suitably changed depending on the circumstances.

An example of how the model pattern is created will be described below.FIG.5is a flowchart of a procedure for creating the model pattern.FIG.6is a view illustrating a model pattern specification region specified in an image.

First, the object W, which the user wishes to teach as a model pattern, is arranged in a field of view of the visual sensor4so that an image of the object W can be taken. Thus, an input image (an image for teaching) including the object W is acquired (Step S101). In this step, the position of the visual sensor4relative to the object W is preferably the same as the position of the visual sensor4when detecting the object W in actual use.

Then, a region where the object W is found in the taken image is specified as a model pattern region (Step S102). The region specified in Step S102will be hereinafter referred to as a model pattern specification region60. The model pattern specification region60of the present embodiment is specified by a rectangle or a circle surrounding the object W. The model pattern specification region60may be stored as operator-created information in the model pattern storage511.

Next, the feature points are extracted (Step S103). As described above, the feature points form the model pattern. A plurality of feature points P_i (i=1 to NP) is extracted from the model pattern specification region60.

In Step S103, physical quantities of the edge points are calculated. Examples of the physical quantities of each edge point include the position of the edge point, and the direction and magnitude of the luminance gradient of the edge point. When the direction of the luminance gradient of the edge point is defined as the posture of the feature point, the posture of the feature point can be defined together with its position. The physical quantities of the edge point, i.e., the position of the edge point, the posture (direction of the luminance gradient) of the edge point, and the magnitude of the luminance gradient of the edge point, are stored as the physical quantities of the feature point.

Then, a model pattern coordinate system is defined in the model pattern specification region60and represented by posture vectors v_Pi and position vectors t_Pi of the feature points P_i based on the model pattern coordinate system and the origin O.

Then, the model pattern50is created based on the extracted physical quantities of the feature points P_i (Step S104). In Step S104, the extracted physical quantities of the feature points P_i are stored as the feature points P_i forming the model pattern. The feature points P_i constitute the model pattern. In the present embodiment, the model pattern coordinate system is defined in the model pattern specification region60, and the position and posture of the feature points P_i are stored as values transformed from the values represented in the image coordinate system (seeFIG.6) into the values represented in the model coordinate system (seeFIG.4).

If necessary, the model pattern50is corrected (Step S105). The correction of the model pattern in Step S105is carried out by an operator or the image processing unit32. Alternatively, the correction may be carried out automatically by machine learning. If the correction of the model pattern is unnecessary, Step S105may be skipped. The model pattern in the image coordinate system is created by the above-described series of processes.

Transformation of the model pattern into three-dimensional points will be described below. The three-dimensional points constitute three-dimensional positional information that identifies the three-dimensional positions of the feature points forming the model pattern.FIG.7is a flowchart of the transformation of the model pattern into the three-dimensional points.FIG.8is a diagram illustrating a relationship between the line of sight of the visual sensor4as the imaging device and a compensation plane.

A plane where the model pattern exists is specified as a compensation plane (Step S201). The compensation plane is an imaginary plane. The compensation plane can be specified by various types of methods. For example, a user sets the compensation plane via the console7by adding touch-ups to the compensation plane viewed from the robot coordinate system or the sensor coordinate system with a robot. The compensation plane is not necessarily a single plane, and may include two or more planes or curved surfaces.

The line of sight toward each feature point of the model pattern is acquired based on the calibration data of the visual sensor4and the positional information of the robot2(Step S202).

As shown inFIG.8, a point of intersection Pw of the compensation plane acquired in Step S201with the line of sight acquired in Step S202is acquired, and the three-dimensional points of the feature points are acquired based on the acquired point of intersection Pw (Step S203). The three-dimensional points constitute the three-dimensional positional information of the feature points. The three-dimensional positional information of the feature points forming the model pattern is stored as information used for the matching in the model pattern storage511.

As can be seen in the foregoing description, the image processing system1of the present embodiment includes: the controller52that acquires the positional relationship between the visual sensor4and the object W based on the positional information of the robot2used to identify the position of the visual sensor4in the robot coordinate system and the positional information indicating the position of the object W in the image coordinate system; and the storage51that stores the model pattern, which is formed of the feature points extracted from an image for teaching, in the form of three-dimensional positional information based on the model pattern and the positional relationship between the visual sensor4and the object W when the image for teaching was taken. The controller52executes a detection process of detecting the object W from a detected image including the object W based on the result of matching between the feature points extracted from the detected image and the model pattern. The matching process is performed based on the model pattern stored in the form of the three-dimensional positional information. This can avoid a situation where the taught relative positions of the visual sensor4and the object W differ from the detected relative positions of them. This can keep the detection of the object W from failing or taking time, and can detect the object W more correctly and efficiently than known techniques.

The image processing system of the present embodiment acquires the three-dimensional positional information of the model pattern stored in the storage51based on the point of intersection of the compensation plane, which is the imaginary plane assumed to include the feature points extracted from the image for teaching, with the line of sight of the visual sensor4toward the feature points of the object W. This allows the acquisition of the three-dimensional positional information of the model pattern using the compensation plane which is the imaginary plane, and thus, allows more correct detection using the compensation plane. In the present embodiment, the object W is assumed to be detected on a certain plane (compensation plane). This assumption allows the acquisition of the three-dimensional positional information of each of the feature points (edge points) forming the model pattern.

The image processing system1of the present embodiment detects the image of the object W from the input image including the object to be detected based on the three-dimensional positional information of each of the feature points forming the model pattern. The detection of the object W using the three-dimensional positional information of each feature point forming the model pattern can be carried out by a number of methods.

First, a matching process of a first example will be described with reference toFIG.9.FIG.9is a flowchart of the matching process of the first example.

In Step S301, the controller52acquires an input image taken by the visual sensor4. The input image is a detected image of a matching target including the object W to be detected. The image of the matching target is different from the image for teaching used to create the model pattern, and is newly acquired by the visual sensor4.

In Step S302, the controller52extracts the edge points as the feature points from the input image. The edge points can be extracted by the method described above.

In Step S303, the controller52acquires the compensation plane when the image was taken and the position of the visual sensor4. The compensation plane is an imaginary plane similar to the imaginary plane set in the creation of the model pattern. The position of the visual sensor4mentioned herein is the position of the visual sensor4relative to the object W. The position of the visual sensor4is acquired based on the calibration data, the positional information of the robot2, the compensation plane, and the positional information of the object W in the image coordinate system. For example, when the visual sensor4is a two-dimensional camera, the point of intersection of the line of sight toward the edge points with the compensation plane is acquired on the assumption that the feature points (edge points) are present on the compensation plane. When the visual sensor4is a three-dimensional sensor, information of a distance to the position of the edge points is acquired to acquire the three-dimensional positional information.

In Step S304, the edge points, which are the feature points extracted from the input image of the matching target (detected image), are projected on the compensation plane to acquire three-dimensional points as the three-dimensional positional information of the edge points. Thus, data of a three-dimensional point cloud extracted from the input image of the matching target is acquired.

In Step S305, a matching process is performed, i.e., the three-dimensional points extracted from the input image of the matching target (detected image) are compared with the three-dimensional points forming the model pattern. In this process, an image of the object is detected from the input image of the matching target.

In the detection process of the first example, the feature points in the detected image are acquired in the form of the three-dimensional positional information based on the point of intersection of the compensation plane, which is the imaginary plane assumed to include the feature points extracted from the detected image, with the line of sight of the visual sensor4toward the feature points of the object W. Then, the object W is detected from the detected image based on the result of the matching process of comparing the feature points in the detected image acquired in the form of the three-dimensional positional information with the feature points based on the model pattern stored in the form of the three-dimensional positional information. In the first example, the detection process can correctly reflect the three-dimensional positional information of the model pattern, allowing the detection of the object W from the detected image based on the suitable positional relationship. When three-dimensional rotation is added to the model pattern in the matching process, the detection can be achieved irrespective of a three-dimensional change in posture. Thus, the first example can address the three-dimensional change in posture.

Next, a matching process of a second example will be described with reference toFIG.10.FIG.10is a flowchart of the matching process of the second example.

In Step S401, the controller52acquires an input image taken by the visual sensor4. The input image is an image of a matching target including the object W to be detected. The image of the matching target is different from the input image used to create the model pattern, and is newly acquired by the visual sensor4.

In Step S402, the feature points are extracted from the input image. The controller52extracts the edge points as the feature points from the input image. The edge points can be extracted by the method described above.

In Step S403, the controller52acquires the compensation plane when the image was taken and the position of the visual sensor4.

In Step S404, the controller52executes a process of projecting the three-dimensional points of the model pattern on the compensation plane on the assumption that the object W is detected at any part of the compensation plane.

In Step S405, the controller52performs matching between the feature points in the input image in the image coordinate system and the feature points of the projected model pattern. Specifically, the two-dimensional points of the feature points in the input image are compared with the feature points obtained by two-dimensionally transforming the three-dimensional points of the model pattern to acquire the degree of matching. The degree of matching can be calculated by any known method, such as the above-described Hankel Fourier transformation.

In Step S406, whether a termination condition is met is determined, and the processes of Steps S404and405are repeated until the termination condition is met. When the process returns from Step S406to Step S404, the three-dimensional points of the model pattern are projected on the compensation plane on the assumption that the object W is detected at any part different from the part assumed last time, and then the process of Step S405is performed. Thus, the image of the object W is detected based on the matching result having the high degree of matching among the matching results. Various types of conditions can be set as the termination condition, e.g., the detection result showed the high degree of matching, or a predetermined time has passed.

Thus, the detection process of the second example includes repeating the projection process of projecting the model pattern on the imaginary plane to acquire the feature points in the image coordinate system on the assumption that the feature points extracted from the detected image are detected at a certain part of the imaginary plane and the matching process of comparing the feature points in the image coordinate system extracted from the detected image with the feature points in the image coordinate system based on the model pattern acquired in the projection process, thereby detecting the object from the detected image based on the result of the matching process having a high degree of matching. In the second example, the detection process can correctly reflect the three-dimensional positional information of the model pattern, allowing the detection of the object W from the detected image based on the suitable positional relationship. In this example, the difference in how the object looks depending on the viewing position can be addressed by repeating the projection process and the matching process. Thus, the matching process can be carried out in the two-dimensional coordinate system.

Embodiments of the present invention have just been described above, but the present invention is not limited to those exemplary embodiments. The advantages described in the embodiments are merely listed as the most suitable advantages derived from the present invention, and do not limit the advantages of the present invention.

It has been described in the embodiment that the position of the visual sensor4as the imaging device relative to the object W is changed by the robot2, but the present invention is not limited to this configuration. For example, the present invention is applicable to a configuration in which the imaging device is fixed and the object is moved by the operation of the robot. Specifically, a fixed camera may take an image of the object held by the robot. In this case, the object moves when the robot is operated, changing the positional relationship between the imaging device and the object.

In the embodiment, the model pattern formed of a plurality of edge points has been described as an example. However, the model pattern is not limited to this example. For example, pixels may be regarded as the feature points, and the model pattern may be formed in an image format.

EXPLANATION OF REFERENCE NUMERALS

1Image processing system2Robot4Visual sensor (imaging device)51Storage52Controller