Patent Description:
<CIT> discloses a gripping method for gripping a plurality of bulk-loaded workpieces with a hand of a robot. In this method, a distance of the workpiece is measured by a distance sensor fixedly disposed on the workpiece and a measurement result and a 3D CAD model of the workpiece are collated, so that a three-dimensional position and a posture of the individual workpiece are recognized. After the three-dimensional position and the posture of the individual workpiece are recognized, a gripping operation of the workpiece starts. Patent Application <CIT> relates to robot control employing deep learning via e.g. a (convolutional) neural network ((C)NN), and in particular, to control an end effector (e.g. gripping end) of an arm of a robot to grip objects (workpieces). The NN is used to predict the probability that candidate motion data for the end effector results in a successful grasp of the object. Vision sensors are mounted on the robot, taking images related to shape, color, and depth of the object. Images of the grasp attempt by an end effector and the pose of the end effector are stored at time instances. The end effector's pose is determined from data of joint position sensors of joints. The end effector is positioned by generating control commands by providing a current image, an additional image, and a candidate motion vector (CMV) as input to the trained NN and applying a measure indicating the probability that the CMV results in a successful grasp. The NN provides, as result, an end effector motion vector based on the end effector's pose at the instance and the end of the grasp attempt. Based on the measure, a grasp command for the end effector is generated. The conference contribution by <NPL> relates to robot control of grasping via machine learning and, in particular, to data set enhancement used for the training of a convolutional neural network (CNN). The robot has gripping arms, and grasping data sets are generated by inputting an image of the gripper to the CNN to predict a grasp likelihood for different grasp directions with reference to a robot's grasp point and randomly chosen angles. Images from all the cameras, robot arm trajectories, and gripping history are stored during the training. The randomly trained CNN represents a model for the robot grasping and is used as a prior. The CNN is subsequently trained, using data collected based on previously seen objects and novel objects, allowing for correcting incorrect grasp modalities. The article by <NPL>, relates to learning-based hand-eye coordination for robot grasping employing big data collection. For that purpose, a neural network is trained so as to servo a robotic gripper to poses that are likely to produce successful grasps and generates corresponding motor commands. The environment and possible perturbations are included in the training through sensors so that the grasp may be adjusted to maximize the probability for the grasp being successful. Further, a spatial relationship between the gripper and the graspable object is determined from visual cues. Patent Application <CIT> relates to a deep-learning method for grasping an object by an end effector of a robot, in particular, to training of a grasping model. The grasping model is used to predict a measure indicating whether motion data for the end effector results in a successful grasp of the object. A vision sensor is mounted to the robot, and has a field of view of the workspace including the graspable object. The grasping model may include multiple networks, each of which is trained. The grasp NN is trained using stored training data of Images taken at instances, an end effector motion vector from the pose at that instance (current pose), the final pose, and a grasp success label. The current pose may also be determined by joint position sensors of the robots joints. A vision sensor takes a current and an additional image, generates a candidate motion vector, and subsequently inputs the trained grasp model along with a measure for a successful grasp. A corresponding end effector grasp command is then generated for controlling the end effector grasp.

An object of the present disclosure is to provide a gripping system, a gripping robot, a learning device, a gripping method, and a model manufacturing method that are effective for improving efficiency of a gripping operation of a robot.

According to the present disclosure, it is possible to provide a gripping system, a learning device, a gripping method, and a model manufacturing method that are effective for improving efficiency of a gripping operation of a robot.

Hereinafter, embodiments will be described in detail with reference to the drawings. In the description of the drawings, the same elements or elements having the same function are denoted by the same reference numerals and redundant explanation may be omitted.

<FIG> is a schematic diagram illustrating an example of an entire configuration of a gripping system <NUM>. The gripping system <NUM> shown in <FIG> is a system for causing a robot <NUM> to execute an operation of gripping workpieces W to be gripping objects and automating various operations such as processing and assembling. The gripping system <NUM> performs learning regarding the gripping operation of the robot <NUM> and commands the robot <NUM> to execute the gripping operation based on a learning result.

The gripping system <NUM> includes a robot <NUM>, a hand <NUM>, an image sensor <NUM>, a robot controller <NUM> (an example of a robot control module), an operation command device <NUM>, and a learning device <NUM>.

The robot <NUM> supports the hand <NUM> gripping the workpiece W and changes at least one of a position and a posture of the hand <NUM>. The robot <NUM> is, for example, a multi-axis (for example, six-axis or seven-axis) serial link type vertical articulated robot and is configured to execute various operations in a state where the hand <NUM> is supported at a tip portion 2a. The robot <NUM> may be a robot capable of freely changing the position and the posture of the hand <NUM> in a predetermined range and is not necessarily limited to a <NUM>-axis vertical articulated robot. For example, the robot <NUM> may be a seven-axis vertical articulated robot in which one redundant axis is added to six axes.

The hand <NUM> is an end effector that grips the workpiece W. An example of the hand <NUM> is a gripper that grips the gripping object by opening and closing operations of a pair of claw members 3a. The hand <NUM> may have a gripping function and is not limited to the gripper having the pair of claw members. For example, the hand <NUM> may be a gripper having three or more claw members or may be an adsorption type end effector.

As one example, the robot <NUM> grips one workpiece W among a plurality of workpieces W arranged in a bucket <NUM>. The workpieces W have various shapes and sizes. Examples of the workpieces W include a bolt, a nut, an electronic component, and the like. The workpieces W in the bucket <NUM> are in an unaligned state (a so-called bulk-loaded state). The workpieces W may be workpieces that can be gripped and a quantity, a shape, a size, and an arrangement are not limited. For example, only one workpiece W may be arranged in the bucket <NUM>. The plurality of workpieces W arranged in the bucket <NUM> may have the same shape or may be a combination of a plurality of types of workpieces having different shapes. The workpieces W in the bucket <NUM> may be aligned. The workpieces W are not limited to rigid bodies and may be elastic bodies. The workpieces W are not limited to the workpieces arranged in the bucket <NUM> and may be workpieces arranged on a workbench or the like.

The image sensor <NUM> is a detector that acquires image information. Examples of the image sensor <NUM> include a camera, a charge-coupled device (CCD) image sensor, a complementary MOS (CMOS) image sensor, and the like. The image sensor <NUM> acquires a pixel value as an example of image information. The pixel value is color information such as a color tone and a gradation and is, for example, a brightness value.

The image sensor <NUM> acquires image information from a viewpoint interlocked with at least one of the position and the posture of the hand <NUM>. The viewpoint is a position that becomes a starting point of an imaging direction of the image sensor <NUM>. The viewpoint interlocked with at least one of the position and the posture of the hand <NUM> means that the viewpoint is changed in accordance with at least one of the position and the posture of the hand <NUM>. As an example, the image sensor <NUM> is fixed to the hand <NUM>. In this case, the image sensor <NUM> acquires image information from a viewpoint interlocked with both the position and the posture of the hand <NUM>. The image sensor <NUM> may be fixed to the tip portion 2a of the arm portion <NUM> of the robot <NUM>. As an example, the image sensor <NUM> may be arranged such that the tip of the hand <NUM> is included in the image information.

The robot controller <NUM> operates the robot <NUM> based on an operation command of the robot <NUM>. The operation command is information for operating the robot <NUM>. The operation command includes information regarding the gripping operation. The operation command is, for example, a gripping target position and a target posture angle. The gripping target position is a final position of the hand <NUM> at timing when the workpiece W is gripped. The gripping target position is defined in a robot coordinate system set to the robot <NUM>, for example. As an example of the robot coordinate system, a direction vertical to an arrangement surface on which the robot <NUM> is arranged may be set to a Z direction, a direction parallel to the arrangement surface may be set to an X direction, and a direction orthogonal to the X direction and the Z direction may be set to a Y direction. In addition, for example, a point where the robot <NUM> is fixed to the arrangement surface may be a fixed point P and the fixed point P may be set to an original point of the robot coordinate system. The gripping target position may be represented by a relative position. The robot controller <NUM> calculates a joint angle target value (angle target value of each joint of the robot <NUM>) for matching the position and the posture of the hand <NUM> with the gripping target position and the target posture angle and operates the robot <NUM> in accordance with the joint angle target value. The operation command may not be the gripping target position.

The operation command device <NUM> is a device that generates the operation command. The operation command device <NUM> is configured to operate the operation command of the robot <NUM>, based on the image information acquired by the image sensor <NUM>, hand position information representing at least one of the position and the posture of the hand <NUM>, and a model.

The model is, for example, a program module that generates an output for an input. As an example, the model is a neural network that is specified by a node and a parameter. The model is an object constructed by machine learning based on collection data and corresponds to at least a part of a process of specifying the operation command of the robot <NUM>. For example, the process of specifying the operation command includes at least one of a recognition process of recognizing the workpiece W from the image information, an extraction process of extracting a feature amount corresponding to the workpiece W from the image information, a command output process of outputting an operation command based on the feature amount corresponding to the workpiece W and the position of the hand <NUM>, and a gripping probability calculation process of calculating a gripping probability based on the feature amount corresponding to the workpiece W and the operation command. The gripping probability is a probability that the workpiece W can be gripped. Although the neural network is exemplified as the model, a Bayesian network (intelligent system that supports decision making of a user under a reliable information environment) may be used or an input/output table may be used.

At least a part of the process of specifying the operation command of the robot <NUM> may be the extraction process. At least a part of the process of specifying the operation command of the robot <NUM> may be the command output process. At least a part of the process of specifying the operation command of the robot <NUM> may be the extraction process and the command output process. At least a part of the process of specifying the operation command of the robot <NUM> may be the extraction process, the command output process, and the gripping probability calculation process.

The collection data is prior data collected before learning and is data regarding the gripping operation. The collection data is, for example, history data of the robot <NUM> or learning data for recognizing the workpiece W. The history data of the robot <NUM> includes the image information acquired by the image sensor <NUM>, the feature amount acquired from the image information by the operation command device <NUM>, the operation command and the gripping probability output by the operation command device <NUM>, the hand position information output by the robot controller <NUM>, and the like. The hand position information is information representing at least one of the position and the posture of the hand <NUM>. The hand position information is not limited to the position coordinates as long as it is information capable of guiding the position and the posture of the hand <NUM> and the hand position information may be a relative vector or a motor torque.

The learning device <NUM> is a device for constructing the model. The learning device <NUM> is configured to acquire image information imaged from a viewpoint interlocked with at least one of the position and the posture of the hand <NUM> gripping the workpiece W and hand position information representing at least one of the position and the posture of the hand <NUM> and to construct a model, by machine learning based on collection data including the image information and the hand position information, corresponding to at least a part of the process of specifying the operation command of the robot <NUM> based on the image information and the hand position information, by machine learning based on collection data including the image information and the hand position information.

<FIG> is a block diagram illustrating an example hardware configuration of the operation command device <NUM>. As shown in <FIG>, the operation command device <NUM> includes a circuit <NUM> that has one or more processors <NUM>, a storage module <NUM>, a communication port <NUM>, and an input/output port <NUM>. The storage module <NUM> includes a memory <NUM> and a storage <NUM>. The storage <NUM> records a program for configuring a functional module of the operation command device <NUM>. The storage <NUM> may be any storage as long as it can be read by a computer. Specific examples include a hard disk, a non-volatile semiconductor memory, a magnetic disk, an optical disk, and the like. The memory <NUM> temporarily stores a program loaded from the storage <NUM>, an operation result of the processor <NUM>, and the like. The processor <NUM> executes the program in cooperation with the memory <NUM> to configure each functional module.

The communication port <NUM> inputs and outputs an electric signal into/from the robot controller <NUM>, the image sensor <NUM>, and the learning device <NUM>, in accordance with a command from the processor <NUM>. The input/output port <NUM> inputs and outputs an electric signal into/from a user interface <NUM>, in accordance with a command from the processor <NUM>. The user interface <NUM> includes a monitor <NUM> and an input device <NUM>.

<FIG> is a block diagram illustrating an example hardware configuration of the learning device <NUM>. As shown in <FIG>, the learning device <NUM> includes a circuit <NUM> that has one or more processors <NUM>, a storage module <NUM>, a communication port <NUM>, and an input/output port <NUM>. The storage module <NUM> includes a memory <NUM> and a storage <NUM>. The storage <NUM> records a program for configuring the functional module of the learning device <NUM>. The storage <NUM> may be any storage as long as it can be read by a computer. Specific examples include a hard disk, a non-volatile semiconductor memory, a magnetic disk, an optical disk, and the like. The memory <NUM> temporarily stores a program loaded from the storage <NUM>, an operation result of the processor <NUM>, and the like. The processor <NUM> executes the program in cooperation with the memory <NUM> to configure each functional module.

The communication port <NUM> inputs and outputs an electric signal between the operation command device <NUM> and the communication port <NUM>, according to a command from the processor <NUM>. The input/output port <NUM> inputs and outputs an electric signal between the user interface <NUM> and the input/output port <NUM>, according to a command from the processor <NUM>. The user interface <NUM> includes a monitor <NUM> and an input device <NUM>.

<FIG> is a block diagram illustrating an example functional configuration of the operation command device <NUM>. As shown in <FIG>, the operation command device <NUM> has an operation module <NUM> and a model acquisition module <NUM>.

The operation module <NUM> executes an operation command of the robot <NUM> based on the image information, the hand position information, and the model corresponding at least a part of the process of specifying the operation command of the robot <NUM>, constructed by machine learning based on the collection data. The operation module <NUM> acquires the image information from the image sensor <NUM> and acquires the hand position information of the hand <NUM> from the robot controller <NUM>. The operation module <NUM> sequentially executes the operation command during the gripping operation of the robot <NUM>.

The sequential operation of the operation command by the operation module <NUM> will be described using <FIG> are schematic diagrams illustrating an example of the gripping operation of the robot <NUM>. <FIG> is a diagram illustrating a position and a posture of the robot <NUM> at the time of starting the gripping operation. In <FIG>, a gripping operation start time is set to <NUM>. <FIG> is a diagram illustrating a position and a posture of the robot <NUM> at time t (<NUM> < t < T). <FIG> is a diagram illustrating a position and a posture of the robot <NUM> at time T. At the time T, the robot <NUM> is in a gripping state in which the robot <NUM> grips the workpiece W. At the time t, the operation module <NUM> generates a gripping target position to be a final position of the hand <NUM> at timing (time T) at which the workpiece W is gripped, based on the image information at the time t. The operation module <NUM> acquires the image information at each time t and generates the gripping target position. As described above, the operation module <NUM> sequentially derives the operation command at each time t and outputs the operation command to the robot controller <NUM>.

For example, the operation module <NUM> includes a position generation module <NUM>, a determination module <NUM>, an extraction module <NUM>, an output module <NUM>, and a calculation module <NUM>. The operation module <NUM> does not need to include all of the position generation module <NUM>, the determination module <NUM>, the extraction module <NUM>, the output module <NUM>, and the calculation module <NUM> and include at least one of the position generation module <NUM>, the determination module <NUM>, the extraction module <NUM>, the output module <NUM>, and the calculation module <NUM>.

The position generation module <NUM> generates a recognition result of the workpiece W, based on the image information acquired by the image sensor <NUM>. The recognition result of the workpiece W includes position information of the workpiece W. The position information of the workpiece W shows a position of the workpiece W in an image generated based on the image information. The recognition result of the workpiece W includes the position and the size of the workpiece W, as an example. The position and the size of the workpiece W may be represented using bounding boxes.

<FIG> illustrates an example recognition result of the workpiece W. An image G1 shown in <FIG> is an image generated based on the image information acquired by the image sensor <NUM>. In the image G1, a plurality of workpieces (for example, a first workpiece W1, a second workpiece W2, and a third workpiece W3) are drawn. In the image G1, a bounding box <NUM> which is a recognition result of the first workpiece W1 is displayed. A position and a size of the first workpiece W1 are represented by the coordinates of a left corner of the bounding box <NUM> on a coordinate axis of the image and an aspect ratio thereof. In <FIG>, the claw members 3a which are the tips of the hand <NUM> are included in the image information.

The recognition result of the workpiece W may include a type of the workpiece W. As the recognition result of the workpiece W, one type is selected from a plurality of types set in advance. The recognition result of the workpiece W may include a gripping expectation value. The gripping expectation value is an index showing easiness of gripping. The position generation module <NUM> receives the image information at the time t and outputs recognition results of all workpieces W drawn in the image at the time t. That is, when the plurality of workpieces W are drawn in the image, the position generation module <NUM> generates a recognition result of each of the plurality of workpieces W based on the image information.

The position generation module <NUM> may generate the recognition result of the workpiece W using a position model. The position model is at least a part of the process of specifying the operation command of the robot <NUM>. The position model is a model corresponding to the recognition process of recognizing the workpiece W from the image information. The position model receives the image information and outputs the recognition result of the workpiece W. As an example, the position model receives the image information and outputs position information of the plurality of workpieces W and respective gripping expectation values of the plurality of workpieces W. The position model is stored in a position model storage module <NUM>.

For example, the position model is composed of a neural network. <FIG> illustrates an example neural network configuring a model. As illustrated in <FIG>, a position model MD1 is composed of an aggregation of nodes <NUM>. Each of the nodes <NUM> is connected to at least one node <NUM>. Weights are set between the connected nodes. The aggregation of the nodes includes an aggregation that functions as an input layer <NUM> receiving data, an aggregation that functions as an intermediate layer <NUM> performing an operation using a weight, and an aggregation that functions as an output layer <NUM> outputting a result. The input layer <NUM> has the number of nodes corresponding to the number of input data. The output layer <NUM> has the number of nodes corresponding to the number of contents of an output result. The intermediate layer <NUM> is appropriately set corresponding to the number of the input layer <NUM> and the output layer <NUM>. The neural network may include a plurality of intermediate layers <NUM>. In the position model MD1, the input layer <NUM> receives the image information and the output layer <NUM> outputs the recognition result of the workpiece W.

The determination module <NUM> determines one workpiece W to be a gripping object. As an example, the determination module <NUM> determines a gripping object based on the recognition result of the workpiece W recognized by the position generation module <NUM>. As a more specific example, the determination module <NUM> determines a gripping object based on a gripping expectation value of the workpiece W recognized by the position generation module <NUM>. For example, the determination module <NUM> determines a workpiece having a highest gripping expectation value among the plurality of workpieces as the gripping object.

The extraction module <NUM> extracts a feature amount corresponding to the workpiece W, based on the image information acquired by the image sensor <NUM> and the position information of the workpiece W included in the image information. The workpiece W from which a feature amount is to be extracted is, for example, one workpiece W determined by the determination module <NUM>. For example, the extraction module <NUM> extracts a feature amount corresponding to the workpiece W, based on the image information acquired by the image sensor <NUM> and the position information of the workpiece W recognized by the position generation module <NUM>. The feature amount corresponding to the workpiece W is a value derived from the image information of the workpiece W. The feature amount is not limited to information which is recognizable by people, such as an outline, a shape, a size, and a color. That is, the feature amount does not need to be associated with information visually recognized by people. The feature amount is, for example, a feature amount of the workpiece W obtained from the image information at the time t, that is, a viewpoint at the time t.

The extraction module <NUM> may extract a feature amount corresponding to the workpiece W, based on a plurality of image information having different viewpoints and position information of the same workpiece W included in each of the plurality of image information. The plurality of image information having the different viewpoints are a plurality of image information including at least image information imaged from a first viewpoint and image information imaged from a second viewpoint different from the first viewpoint. The image information imaged from the first viewpoint is, for example, the image information acquired by the image sensor <NUM> at the time of starting the gripping operation (<FIG>; time <NUM>). The image information imaged from the second viewpoint is, for example, the image information acquired by the image sensor <NUM> during the gripping operation (<FIG>; time t). The same workpiece W means that a workpiece recognized based on image information captured from a workpiece recognized based on the image information imaged from the first viewpoint is the same as a workpiece recognized based on the image information imaged from the second viewpoint different from the first viewpoint.

The extraction module <NUM> may extract a feature amount using an extraction model. The extraction model is at least a part of the process of specifying the operation command of the robot <NUM>. The extraction model is a model corresponding to the extraction process of extracting the feature amount corresponding to the workpiece W from the image information. The extraction model receives the image information and the position information of the workpiece W and outputs the feature amount corresponding to the workpiece W. The extraction model is stored in an extraction model storage module <NUM>.

For example, the extraction model is composed of a neural network. A configuration of the neural network has an input layer, an intermediate layer, and an output layer, similar to the position model MD1 of <FIG>. In the extraction model, the input layer receives the image information and the position information of the workpiece W and the output layer outputs the feature amount corresponding to the workpiece W.

The output module <NUM> outputs an operation command of the robot <NUM> based on the feature amount of the workpiece W extracted from the image information acquired by the image sensor <NUM> and the hand position information. As an example, the output module <NUM> outputs an operation command of the robot <NUM> based on the feature amount of the workpiece W extracted by the extraction module <NUM> and the hand position information. The hand position information is information of at least one of the position and the posture of the hand <NUM> at the time when the image information has been acquired. For example, the output module <NUM> receives the image information at the time t and the hand position information at the time t as an input. The output module <NUM> outputs the gripping target position as an example of the operation command.

The output module <NUM> may output an operation command using a command model. The command model is at least a part of the process of specifying the operation command of the robot <NUM>. The command model is a model corresponding to the command output process of outputting the operation command of the robot <NUM> based on the feature amount of the gripping object and the position of the hand <NUM>. The command model receives the feature amount of the workpiece W extracted from the image information acquired by the image sensor <NUM> and the hand position information and outputs the operation command of the robot <NUM>. The command model is stored in a command model storage module <NUM>.

For example, the command model is composed of a neural network. A configuration of the neural network has an input layer, an intermediate layer, and an output layer, similar to the position model MD1 of <FIG>. In the command model, the input layer receives the feature amount of the workpiece W extracted from the image information acquired by the image sensor <NUM> and the hand position information and the output layer outputs the operation command of the robot <NUM>.

In the case where the hand <NUM> is a gripper, the command model may receive an opening/closing degree of the gripper in addition to the feature amount of the workpiece W and the hand position information and may output a target opening/closing degree of the gripper in addition to the operation command. The target opening/closing degree is a target value of an interval of the claw members 3a of the gripper. In this case, the output module <NUM> outputs the operation command and the target opening/closing degree based on the feature amount of the workpiece W, the hand position information, and the opening/closing degree, using the command model. The robot controller <NUM> operates the gripper in accordance with the target opening/closing degree.

The calculation module <NUM> calculates a gripping probability representing a probability that the workpiece can be gripped by the hand <NUM>, based on the operation command of the robot <NUM> and the feature amount corresponding to the workpiece W. As an example, the calculation module <NUM> calculates a gripping probability representing a probability that the workpiece can be gripped by the hand <NUM>, based on the operation command of the robot <NUM> output by the output module <NUM> and the feature amount corresponding to the workpiece W, extracted by the extraction module <NUM>.

The calculation module <NUM> may output the gripping probability using a gripping probability model. The gripping probability model is at least a part of the process of specifying the operation command of the robot <NUM>. The gripping probability model is a model corresponding to the gripping probability calculation process of calculating the gripping probability based on the feature amount corresponding to the workpiece W and the operation command of the robot <NUM>. The gripping probability model receives the feature amount corresponding to the workpiece W and the operation command of the robot <NUM> and outputs the gripping probability. The gripping probability model is stored in a gripping probability model storage module <NUM>.

For example, the gripping probability model is composed of a neural network. A configuration of the neural network has an input layer, an intermediate layer, and an output layer, similar to the position model MD1 of <FIG>. In the gripping probability model, the input layer receives the feature amount corresponding to the workpiece W and the operation command of the robot <NUM> and the output layer outputs the gripping probability.

The operation module <NUM> outputs data calculated in the process of specifying the operation command of the robot <NUM> as history data to a history storage module <NUM>. The history data includes the time at which the hand position information has been acquired by the output module <NUM>, image information acquired from the image sensor <NUM> by the position generation module <NUM>, a position of the workpiece W generated by the position generation module <NUM>, a feature amount of the workpiece W extracted by the extraction module <NUM>, an operation command generated by the output module <NUM>, a gripping probability calculated by the calculation module <NUM>, a gripping success/failure, and the like. The history data is configured such that the history data can be referred to by the learning device <NUM>.

The gripping probability stored in the history storage module <NUM> is used to determine whether or not the gripping operation of the robot <NUM> is re-executed, for example. For example, the output module <NUM> determines whether the gripping probability tends to increase or tends to decrease. When the gripping probability tends to decrease, the output module <NUM> outputs an operation command to re-execute the gripping operation of the robot <NUM>. As an example, the output module <NUM> outputs an operation command to operate the robot <NUM>, such that the robot <NUM> returns to the position of the hand <NUM> a predetermined time before. Thereby, the robot controller <NUM> may operate the robot <NUM> based on the gripping probability calculated by the calculation module <NUM>.

The gripping probability may be used as a parameter for determining whether or not the robot controller <NUM> commands the hand <NUM> to approach the workpiece W. For example, when the gripping probability is equal to or more than a predetermined threshold, the robot controller <NUM> commands the robot <NUM> to perform the gripping operation and when the gripping probability is less than the threshold, the robot controller <NUM> may re-execute commanding the extraction module <NUM> to extract the feature amount corresponding to the workpiece W and commanding the output module <NUM> to output the operation command of the robot <NUM> by the output module <NUM>. As an example, the robot controller <NUM> may move the hand <NUM> by a predetermined distance to separate the position of the hand <NUM> from the workpiece W and may re-execute the operation of commanding the extraction module <NUM> to extract the feature amount corresponding to the workpiece W, and re-execute the operation of commanding the output module <NUM> to output the operation command of the robot <NUM>. Alternatively, after the passage of update timing of the weight of the neural network learned by the learning device <NUM>, the robot controller <NUM> may re-execute the operation of commanding the extraction module <NUM> to extract the feature amount corresponding to the workpiece W, and re-execute the operation of commanding the output module <NUM> to output the operation command of the robot <NUM>.

The model acquisition module <NUM> acquires a learning result from the learning device <NUM>. The model acquisition module <NUM> acquires network configurations and weight data of the position model, the extraction model, the command model, and the gripping probability model as the learning result. The model acquisition module <NUM> stores the acquired position model in the position model storage module <NUM>. The model acquisition module <NUM> stores the acquired extraction model in the extraction model storage module <NUM>. The model acquisition module <NUM> stores the acquired command model in the command model storage module <NUM>. The model acquisition module <NUM> stores the acquired gripping probability calculation model in the gripping probability model storage module <NUM>.

<FIG> is a block diagram illustrating an example functional configuration of the learning device <NUM>. As shown in <FIG>, the learning device <NUM> includes an acquisition module <NUM> and a construction module <NUM>.

The acquisition module <NUM> acquires history data from the history storage module <NUM> of the operation command device <NUM>. As an example, the acquisition module <NUM> acquires collection data including image information imaged from a viewpoint interlocked with at least one of the position and the posture of the hand <NUM> gripping the workpiece W and hand position information representing at least one of the position and the posture of the hand <NUM>. The acquisition module <NUM> stores the history data as the collection data in the history data storage module <NUM>.

The collection data is not limited to the data acquired from the operation command device <NUM> and may be acquired from other devices. For example, when the gripping system <NUM> includes at least the robot <NUM>, the hand <NUM>, the operation module <NUM>, and the robot controller <NUM> as one set of devices, the acquisition module <NUM> may acquire an operation history of another set of devices, different from the one set, as the collection data. That is, the acquisition module <NUM> may acquire an operation history of the other set included in the gripping system <NUM> as the collection data or may acquire an operation history of a set of devices included in another gripping system as the collection data.

The construction module <NUM> constructs a model corresponding to at least a part of the process of specifying the operation command of the robot <NUM> based on the image information acquired by the image sensor <NUM> and the hand position information representing at least one of the position and the posture of the hand, by machine learning based on the collection data.

The construction module <NUM> has a position model construction module <NUM>, an extraction model construction module <NUM>, a command model construction module <NUM>, and a gripping probability model construction module <NUM>. The construction module <NUM> does not need to include all of the position model construction module <NUM>, the extraction model construction module <NUM>, the command model construction module <NUM>, and the gripping probability model construction module <NUM> and include at least one of the position model construction module <NUM>, the extraction model construction module <NUM>, the command model construction module <NUM>, and the gripping probability model construction module <NUM>.

The position model construction module <NUM> constructs a position model by machine learning based on workpiece learning data. The workpiece learning data is training data for recognizing the workpiece W and is previously stored in a workpiece learning data storage module <NUM>. Examples of the workpiece learning data include data in which image information of an image of the workpiece W and a profile (for example, a position, a type, and easiness of gripping of the workpiece W in the image) are associated with each other. The position model construction module <NUM> adjusts the weight of the neural network such that the workpiece W can be recognized from the image information of the image of the workpiece W. The position model construction module <NUM> stores the constructed position model in a position model storage module <NUM>. The position model storage module <NUM> is configured such that the operation command device <NUM> can refer to the position model storage module <NUM>. The position model storage module <NUM> is used in order to update the position model storage module <NUM>.

The gripping probability model construction module <NUM> constructs a gripping probability model by machine learning based on the gripping probability learning data. The gripping probability learning data is training data for calculating the gripping probability and is stored in a gripping probability learning data storage module <NUM> in advance. Examples of the gripping probability learning data include data in which image information of an image of the workpiece W, an operation command, and a gripping success/failure are associated with each other. The gripping probability model construction module <NUM> adjusts the weight of the neural network such that a degree of the gripping probability will be more closely related to a gripping success/failure of a history. In the case where history data sufficient for learning is accumulated in the history data storage module <NUM>, the gripping probability model construction module <NUM> further adjusts the gripping probability model by machine learning based on the history data stored in the history data storage module <NUM>. Examples of the history data referred to by the gripping probability model construction module <NUM> include the feature amount of the workpiece W, the operation command, and the gripping success/failure. The gripping probability model construction module <NUM> further adjusts the weight of the neural network such that the degree of the gripping probability will be more closely related to the gripping success/failure of the history, based on the feature amount of the workpiece W and the operation command. The gripping probability model construction module <NUM> stores the constructed gripping probability model in the gripping probability model storage module <NUM>. The gripping probability model storage module <NUM> is configured such that the operation command device <NUM> can refer to the gripping probability model storage module <NUM>. The gripping probability model storage module <NUM> is used to update the gripping probability model storage module <NUM>.

The extraction model construction module <NUM> constructs an extraction model by machine learning based on the history data stored in the history data storage module <NUM>. The extraction model construction module <NUM> constructs the extraction model when history data sufficient for learning is accumulated in the history data storage module <NUM>. Examples of the history data referred to by the extraction model construction module <NUM> include the image information, the position information of the workpiece W, and the gripping probability. The history data to be referred to may include the gripping success/failure instead of the gripping probability. The extraction model construction module <NUM> adjusts the weight of the neural network such that the feature amount of the workpiece W which improves the gripping probability can be extracted from the image information, based on the position information of the workpiece W. The extraction model construction module <NUM> stores the constructed extraction model in the extraction model storage module <NUM>. The extraction model storage module <NUM> is configured such that the operation command device <NUM> can refer to the extraction model storage module <NUM>. The extraction model storage module <NUM> is used to update the extraction model storage module <NUM>.

The command model construction module <NUM> constructs a command model by machine learning based on the history data stored in the history data storage module <NUM>. The command model construction module <NUM> constructs the command model when history data sufficient for learning is accumulated in the history data storage module <NUM>. Examples of the history data referred to by the command model construction module <NUM> include the feature amount of the workpiece W, the hand position information, the operation command, and the gripping probability. The history data may include the gripping success/failure instead of the gripping probability. The command model construction module <NUM> adjusts the weight of the neural network such that the operation command which improves the gripping probability can be output, based on the feature amount of the workpiece W and the hand position information. The command model construction module <NUM> stores the constructed command model in the command model storage module <NUM>. The command model storage module <NUM> is configured such that the operation command device <NUM> can refer to the command model storage module <NUM>. The command model storage module <NUM> is used to update the command model storage module <NUM>.

The gripping system <NUM> is not limited to the hardware configuration described above and may have any hardware configuration compatible with the functions of the robot controller <NUM>, the operation command device <NUM>, and the learning device <NUM> described above. For example, the operation command device <NUM> and the learning device <NUM> may be one hardware resource, the robot controller <NUM> and the operation command device <NUM> may be one hardware resource, the robot controller <NUM> and the learning device <NUM> may be one hardware resource, or the robot controller <NUM>, the operation command device <NUM>, and the learning device <NUM> may be one hardware resource. One hardware resource is a resource that is integrated in appearance.

The internal hardware configurations of the operation command device <NUM> and the learning device <NUM> need not to be separated for eachof the functional configurations described above. The hardware configurations of the operation command device <NUM> and the learning device <NUM> are not limited to the configurations in which each function is exhibited by executing a program. For example, at least a part of each functional module may be configured by a logic circuit specialized for the function thereof or may be configured by an application specific integrated circuit (ASIC) in which the logic circuit is integrated.

The models of the operation command device <NUM> and the learning device <NUM> may be composed of one neural network. One neural network may be constructed by a series of machine learning.

As an example of a gripping method, a gripping procedure executed by the operation command device <NUM> will be described. <FIG> illustrates a flowchart of an example gripping procedure.

As shown in <FIG>, first, the operation command device <NUM> executes step S10. In step S10, the position generation module <NUM> acquires image information from the image sensor <NUM>.

Next, the operation command device <NUM> executes step S12. In step S12, the output module <NUM> acquires hand position information from the robot controller <NUM>.

Next, the operation command device <NUM> executes step S14. In step S14, the position generation module <NUM> generates position information of the workpiece W based on the image information acquired in step S10.

Next, the operation command device <NUM> executes step S16. In step S16, the determination module <NUM> determines one workpiece W based on the position information of the workpiece W generated in step S14.

Next, the operation command device <NUM> executes step S18. In step S18, the extraction module <NUM> extracts a feature amount of one workpiece W determined in step S16, based on the image information acquired in step S10.

Next, the operation command device <NUM> executes step S20. In step S20, the output module <NUM> calculates an operation command of the robot <NUM> based on the hand position information and the feature amount of the workpiece W extracted in step S18.

Next, the operation command device <NUM> executes step S22. In step S22, the output module <NUM> outputs the operation command calculated in step S20 to the robot controller <NUM>.

Next, the operation command device <NUM> executes step S24. In step S24, the calculation module <NUM> calculates a gripping probability based on the feature amount of the workpiece W extracted in step S18 and the operation command calculated in step S20.

Next, the operation command device <NUM> executes step S26. In step S26, the output module <NUM> determines whether or not the gripping probability calculated in step S24 tends to decrease.

In step S26, when it is determined that the gripping probability calculated in step S24 tends to decrease, the operation command device <NUM> executes step S28. In step S28, the output module <NUM> outputs an operation command such that the robot returns to the position of the hand <NUM> a predetermined time before.

In step S26, when it is determined that the gripping probability calculated in step S24 does not tend to decrease or when step S28 ends, the operation command device <NUM> ends the process.

In <FIG>, step S10 may be executed at a timing that is earlier than step S14 and may be executed between step S12 and step S14. Step S12 may be executed at any timing earlier than step S20. Step S22 may be executed after step S24. In this case, step S26 and step S28 are not executed and when the gripping probability calculated in step S24 is equal to or more than a predetermined threshold, step S22 may be executed and when the gripping probability is less than the threshold, the process may be re-executed from step S14, step S16, or step S18.

As an example of the learning method, a position model learning procedure (an example of a model manufacturing method) executed by the learning device <NUM> will be described. <FIG> illustrates a flowchart of an example position model learning procedure.

As shown in <FIG>, first, the learning device <NUM> executes step S30. In step S30, the position model construction module <NUM> reads workpiece learning data from the workpiece learning data storage module <NUM>.

Next, the learning device <NUM> executes step S32. In step S32, the position model construction module <NUM> constructs a position model. The position model construction module <NUM> adjusts the weight of the neural network such that the workpiece W can be recognized from the image information of the image of the workpiece W. The position model construction module <NUM> stores a learning result in the position model storage module <NUM>.

When step S32 ends, the learning device <NUM> ends the process.

As an example of the learning method, a gripping probability model learning procedure (an example of a model manufacturing method) executed by the learning device <NUM> will be described. <FIG> illustrates a flowchart of an example gripping probability model learning procedure.

As shown in <FIG>, first, the learning device <NUM> executes step S40. In step S40, the gripping probability model construction module <NUM> reads gripping probability learning data from the gripping probability learning data storage module <NUM>. Examples of the gripping probability learning data include the image information, the operation command, and the gripping success/failure.

Next, the learning device <NUM> executes step S42. In step S42, the gripping probability model construction module <NUM> constructs a gripping probability model. The gripping probability model construction module <NUM> adjusts the weight of the neural network such that the degree of the gripping probability will be more closely related to the gripping success/failure of the history, based on the image information, the operation command, and the gripping success/failure. The gripping probability model construction module <NUM> stores a learning result in the gripping probability model storage module <NUM>.

When step S42 ends, the learning device <NUM> ends the process.

As an example of the learning method, a learning procedure (an example of a model manufacturing method) of the extraction model, the command model, and the gripping probability model executed by the learning device <NUM> will be described. <FIG> illustrates a flowchart of a learning procedure of the extraction model, the command model, and the gripping probability model.

As shown in <FIG>, first, the learning device <NUM> executes step S50. In step S50, each of the extraction model construction module <NUM>, the command model construction module <NUM>, and the gripping probability model construction module <NUM> determines whether or not history data sufficient for learning (a predetermined amount of data) is accumulated in the history data storage module <NUM>. When the extraction model construction module <NUM>, the command model construction module <NUM>, and the gripping probability model construction module <NUM> perform learning as a unit, as an example, the extraction model construction module <NUM> determines whether or not the history data sufficient for learning (a predetermined amount of data) is accumulated in the history data storage module <NUM>.

In step S50, when it is determined that the history data sufficient for learning (a predetermined amount of data) is accumulated in the history data storage module <NUM>, the learning device <NUM> executes step S52. In step S52, each of the extraction model construction module <NUM>, the command model construction module <NUM>, and the gripping probability model construction module <NUM> reads the history data. For example, the extraction model construction module <NUM> reads the position information, the image information, and the gripping probability of the workpiece W to be the history data. For example, the command model construction module <NUM> reads the feature amount of the workpiece W, the hand position information, and the history data of the gripping probability. The gripping probability model construction module <NUM> reads the feature amount of the workpiece W, the operation command, and the gripping success/failure.

Next, the learning device <NUM> executes step S54. In step S54, the extraction model construction module <NUM> constructs an extraction model. The extraction model construction module <NUM> adjusts the weight of the neural network such that the feature amount of the workpiece W which improves the gripping probability can be extracted from the image information, based on the image information, the position information of the workpiece, and the gripping probability. The extraction model construction module <NUM> stores the learning result in the extraction model storage module <NUM>.

Next, the learning device <NUM> executes step S56. In step S56, the command model construction module <NUM> constructs a command model. The command model construction module <NUM> adjusts the weight of the neural network such that the operation command which improves the gripping probability can be output, based on the feature amount of the workpiece, the hand position information, and the gripping probability. The command model construction module <NUM> stores the learning result in the command model storage module <NUM>.

Next, the learning device <NUM> executes step S58. In step S58, the gripping probability model construction module <NUM> constructs a gripping probability model. The gripping probability model construction module <NUM> further adjusts the weight of the neural network such that the degree of the gripping probability will be more closely related to the gripping success/failure of the history, based on the feature amount of the workpiece W and the operation command. The gripping probability model construction module <NUM> stores the constructed gripping probability model in the gripping probability model storage module <NUM>.

When it is determined in step S50 that the history data sufficient for learning (a predetermined amount of data) is not accumulated in the history data storage module <NUM> or when step S58 ends, the learning device <NUM> ends the process.

In steps S52 to S58 described above, the extraction model construction module <NUM>, the command model construction module <NUM>, and the gripping probability model construction module <NUM> may perform learning as a unit. In this case, as an example, the extraction model construction module <NUM> reads the position information of the workpiece W, the image information, the hand position information, the gripping probability, and the gripping success/failure. In addition, the extraction model construction module <NUM>, the command model construction module <NUM>, and the gripping probability model construction module <NUM> are configured as one neural network and perform learning by a series of machine learning. As described above, the extraction model, the command model, and the gripping probability model may be combined and learning may be performed.

As described above, the gripping system <NUM> includes the hand <NUM> that grips the workpiece W, the robot <NUM> that supports the hand <NUM> and changes at least one of the position and the posture of the hand <NUM>, the image sensor <NUM> that acquires the image information from the viewpoint interlocked with at least one of the position and the posture of the hand <NUM>, the construction module <NUM> that constructs the model corresponding to at least a part of the process of specifying the operation command of the robot <NUM> based on the image information acquired by the image sensor <NUM> and the hand position information representing at least one of the position and the posture of the hand <NUM>, by machine learning based on the collection data, the operation module <NUM> that derives the operation command of the robot <NUM>, based on the image information, the hand position information, and the model, and the robot controller <NUM> that operates the robot <NUM> based on the operation command of the robot <NUM> operated by the operation module <NUM>.

The model is constructed based on the image information acquired from the viewpoint interlocked with the hand <NUM> and the position of the hand <NUM>, by machine learning by the construction module <NUM>, and the operation command is output from the model. Because the operation command is acquired by machine learning based on the image information acquired from the viewpoint interlocked with at least one of the position and the posture of the hand <NUM>, a distance sensor and a 3D CAD model do not need to be provided and a collation process between a detection result of the distance sensor and the 3D CAD model becomes unnecessary. In addition, machine learning is performed, so that teaching by operators becomes unnecessary. Therefore, the gripping system <NUM> is effective for improving efficiency of the gripping operation of the robot.

The operation module <NUM> may include the extraction module <NUM> that extracts the feature amount corresponding to the workpiece W, based on the image information and the position information of the workpiece W included in the image information, and the output module <NUM> that outputs the operation command of the robot <NUM> based on the feature amount extracted by the extraction module <NUM> and the hand position information. In this case, the feature amount can be extracted from the image information and the position information of the workpiece W by the extraction module <NUM> and the operation command of the robot <NUM> can be output from the extraction amount and the hand position information. Therefore, learning efficiency is improved and a gripping success probability is also improved.

The extraction module <NUM> may extract a feature amount corresponding to the workpiece W, based on a plurality of image information having different viewpoints and position information of the same workpiece W included in each of the plurality of image information. In this case, by using the positions of the workpieces W included in the images having the different viewpoints, the feature amount can be extracted more accurately and a more accurate operation command can be output.

The construction module <NUM> may include the extraction model construction module <NUM> that constructs the extraction model receiving the image information and the position information and outputting the feature amount by machine learning based on the collection data and the command model construction module <NUM> that constructs the command model receiving the feature amount and the hand position information and outputting the operation command by machine learning based on the collection data, the extraction module <NUM> may extract the feature amount using the extraction model, and the output module <NUM> may output the operation command using the command model. In this case, the extraction of the feature amount of the gripping object and the determination of the operation for gripping the gripping object can be automatically performed. In particular, according to the combination of the two models, accuracy of the operation command is improved.

The hand <NUM> may be a gripper, the command model construction module <NUM> may construct the command model receiving the opening/closing degree of the gripper in addition to the feature amount and the hand position information and outputting the target opening/closing degree of the gripper in addition to the operation command, the output module <NUM> may output the operation command and the target opening/closing degree based on the feature amount, the hand position information, and the opening/closing degree, using the command model, and the robot controller <NUM> may operate the gripper in accordance with the target opening/closing degree. In this case, it is possible to output an appropriate opening/closing degree of the gripper with respect to the gripping object.

The image sensor <NUM> may be arranged such that the tip of the hand <NUM> is included in the image information. In this case, the tip shape of the hand <NUM> can be recognized from the image information and machine learning can be performed. Therefore, gripping accuracy can be further improved. Particularly, when the hand <NUM> is the gripper, accuracy of opening/closing of the gripper is also improved.

The operation module <NUM> may further include the calculation module <NUM> that calculates the gripping probability representing the probability that the workpiece can be gripped by the hand, based on the operation command of the robot <NUM> and the feature amount corresponding to the workpiece W. The robot controller <NUM> may operate the robot <NUM> based on the gripping probability calculated by the calculation module <NUM>. In this case, the robot can be operated in accordance with the gripping probability. At this time, accuracy of the gripping probability can be improved by using the feature amount.

The construction module <NUM> may further have the gripping probability model construction module <NUM> that constructs the gripping probability model receiving the operation command of the robot <NUM> and the feature amount corresponding to the workpiece W and outputting the griping probability, by machine learning based on the collection data, and the calculation module <NUM> may calculate the gripping probability using the gripping probability model. In this case, the gripping probability by the determined operation can be automatically acquired by machine learning.

When the gripping probability is equal to or more than the predetermined threshold, the robot controller <NUM> may command the robot <NUM> to perform the gripping operation and when the gripping probability is less than the threshold, the robot controller <NUM> may re-execute commanding the extraction module <NUM> to extract the feature amount corresponding to the workpiece and commanding the output module <NUM> to output the operation command of the robot. In this case, the probability of the gripping failure can be reduced.

The operation module <NUM> may further include the position generation module <NUM> that generates the position information of the plurality of workpieces W and the respective gripping expectation values of the plurality of workpieces, based on the image information acquired by the image sensor <NUM>, and the determination module <NUM> that determines one workpiece of the gripping object, based on the gripping expectation values, and the extraction module <NUM> may extract the feature amount corresponding to the one workpiece, based on the image information and the position information of one workpiece. By using the gripping expectation values, the gripping object can be determined in consideration of easiness of gripping. As a result, the gripping success probability can be improved.

The construction module <NUM> may further include the position model construction module <NUM> that constructs the position model, by machine learning based on the collection data, that receives the image information, and that outputs the position information of the plurality of workpieces and the respective gripping expectation values of the plurality of workpieces, and the position generation module <NUM> may generate the position information and the gripping expectation values using the position model. In this case, the gripping expectation value of each workpiece can be automatically acquired by machine learning.

The gripping system <NUM> may include at least the robot <NUM>, the hand <NUM>, the operation module <NUM>, and the robot controller <NUM> as one set and the construction module may construct the model by machine learning, based on collection data of an operation history of another set different from the one set as the collection data, and the operation module <NUM> of the one set may operate the operation command of the robot <NUM> of the one set, based on the model. In this case, because a learning result obtained from other robot can be diverted, portability of the learning result can be enhanced.

Claim 1:
A gripping system (<NUM>) comprising:
a robotic hand (<NUM>) configured to grip a workpiece (W);
a robot (<NUM>) configured to support the robotic hand (<NUM>) and to change at least one hand configuration selected from a group comprising a position of the robotic hand (<NUM>) and a posture of the robotic hand (<NUM>);
an image sensor (<NUM>) configured to acquire image information from a viewpoint interlocked with the at least one hand configuration;
characterized by
an operation module (<NUM>) configured to derive an operation command of the robot (<NUM>) based on
- the image information,
- hand position information representing at least one of the position of the hand (<NUM>) and the posture of the hand (<NUM>), and
- a command model configured to output the operation command of the robot (<NUM>) in response to an input including a feature amount of the workpiece extracted from the image information and the hand position information,
wherein the command model is constructed by machine learning performed by a learning device (<NUM>) according to claim <NUM>; and
a robot control module (<NUM>) configured to operate the robot (<NUM>) based on the operation command of the robot (<NUM>) derived by the operation module (<NUM>).