Machine learning device and machine learning method for learning optimal object grasp route

A machine learning device according to the present invention learns an operation condition of a robot that stores a plurality of objects disposed on a carrier device in a container using a hand for grasping the objects. The machine learning device includes a state observation unit for observing the positions and postures of the objects and a state variable including at least one of cycle time to store the objects in the container and torque and vibration occurring when the robot grasps the objects during operation of the robot; a determination data obtaining unit for obtaining determination data for determining a margin of each of the cycle time, the torque, and the vibration against an allowance value; and a learning unit for learning the operation condition of the robot in accordance with a training data set constituted of a combination of the state variable and the determination data.

This application is a new U.S. patent application that claims benefit of JP 2016-188857 filed on Sep. 27, 2016, the content of 2016-188857 is incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a machine learning device and a machine learning method, and more specifically relates to a machine learning device and a machine learning method for learning an optimal object grasp route when a robot grasps objects disposed on a carrier device.

2. Description of Related Art

Object carrier systems that sequentially grasp a plurality of objects carried by a carrier device, i.e., a conveyor, one-by-one with a hand (multi-fingered hand) that can hold multiple objects, and which put the objects in containers similarly carried by another conveyor are known. Conventionally, when grasping and taking out the objects, basically, the objects are grasped in the order in which the objects are located downstream of the conveyor.

For example, there is proposed a method in which a conveyor is divided in two along a conveying direction of objects, and the objects are grasped and put in containers in the order in which the objects are located downstream thereof (for example, Japanese Unexamined Patent Publication (Kokai) No. 2014-104524, hereinafter referred to as “Patent Document 1”).

FIG. 1shows a conventional method for grasping a plurality of objects (a1to a3and b1to b3) disposed on a conveyor10with a hand (not shown) provided in a robot20, and putting the objects in containers (not shown). The objects are disposed on a plane of the conveyor10in random orientations (postures), as indicated by the arrows of the objects (a1to a3and b1to b3) themselves.

The hand has the functions of grasping three objects and putting the objects in a container. When the conveyor10moves from the left to the right ofFIG. 1, the objects are located downstream of the conveyor10in order of a1, a2, and a3. Thus, the hand grasps the objects a1, a2, and a3in this order and puts the objects in a container, and thereafter grasps the objects b1, b2, and b3in this order and puts the objects in another container.

Conventionally, since the objects are grasped in the order in which the objects are located downstream, the hand may move back and forth, as indicated by the arrows inFIG. 1.

Furthermore, since the postures of the objects are not considered, the hand of the robot20may rotate significantly, as shown inFIG. 1. For example, since the objects a1and a3have similar postures, the hand rotates minimally when grasping the objects in order of a1, a3, and a2. However, since the hand grasps the objects in the order in which the objects are located downstream, the hand grasps the objects in order of a1, a2, and a3, as shown inFIG. 1. As a result, the hand of the robot20rotates about 180 degrees to grasp the object a2after having grasped the object a1, and thereafter again rotates about 180 degrees to grasp the object a3.

The method of Patent Document 1, as described above, grasps objects in the order in which the objects are located downstream, without considering the positions of the objects in the width direction of the conveyor10and the orientations of the objects. Therefore, the movement time of the hand varies widely, and in some instances, the robot may fail to put the objects in a container while the container is passing in front of the robot.

It is conceivable to stop the conveyor for containers whenever the robot grasps objects. However, this is difficult to adopt in actual production sites, because there are cases where production volume is predetermined within a certain time period and conveyance of the containers cannot be stopped due to a relationship with subsequent steps.

Patent Document 1 describes a method in which an area is further divided into smaller regions, and grasping is performed in each region to reduce carry distance. However, when the conveyor is wide, dividing the area may have little effect. Furthermore, since the orientations (postures) of the objects are not considered, the robot may grasp an object that has a completely different posture from a posture which is favorable for grasping.

Furthermore, the method of Patent Document 1 does not consider the inherent ability of the robot (for example, the strength of a mechanism and the like) and the difference in carrying capacity owing to difference in the locations of the objects on the conveyor.

SUMMARY OF THE INVENTION

The present invention aims at providing a machine learning device and a machine learning method for reducing the burden on a robot having a hand that has the function of grasping a plurality of objects, as well as minimizing the cycle time that is the time required for the robot to store the objects in a container.

A machine learning device according to an embodiment of the present invention learns an operation condition of a robot that stores a plurality of objects disposed on a carrier device in a container using a hand for grasping the objects. The machine learning device includes a state observation unit for observing the positions and postures of the objects and a state variable including at least one of cycle time to store the objects in the container and torque and vibration occurring when the robot grasps the objects, during operation of the robot; a determination data obtaining unit for obtaining determination data for determining a margin of each of the cycle time, the torque, and the vibration against an allowance value; and a learning unit for learning the operation condition of the robot in accordance with a training data set constituted of a combination of the state variable and the determination data.

A machine learning method according to an embodiment of the present invention learns an operation condition of a robot that stores a plurality of objects disposed on a carrier device in a container using a hand for grasping the objects. The machine learning method includes the steps of observing the positions and postures of the objects and a state variable including at least one of cycle time to store the objects in the container and torque and vibration occurring when the robot grasps the objects, during operation of the robot; obtaining determination data for determining a margin of each of the cycle time, the torque, and the vibration against an allowance value; and learning the operation condition of the robot in accordance with a training data set constituted of a combination of the state variable and the determination data.

DETAILED DESCRIPTION OF THE INVENTION

A machine learning device and a machine learning method according to the present invention will be described below with reference to the drawings.

First Embodiment

A machine learning device according to a first embodiment will be described with reference to the drawings.FIG. 2is a block diagram of a machine learning device101according to the first embodiment of the present invention.FIG. 3is a drawing showing a method for grasping a plurality of objects disposed on a carrier device (conveyor) with a hand provided in a robot to put the objects in containers, by the machine learning device according to the first embodiment of the present invention. The machine learning device101according to the first embodiment of the present invention learns an operation condition of a robot20that stores a plurality of objects p1to p6disposed on a carrier device10in containers (not shown) using a hand (not shown) for grasping the objects.

The machine learning device101includes a state observation unit11, a determination data obtaining unit12, and a learning unit13.

The state observation unit11observes the positions and postures of the objects (p1to p6), and a state variable including at least one of cycle time to store the objects in a container and torque and vibration occurring when the robot20grasps the objects, during operation of the robot20. The positions and postures of the objects may be analyzed based on an image captured by a camera (not shown). In this case, the analysis of the positions and postures of the objects is preferably completed by the time the robot20begins to grasp the objects (p1to p6). The camera is thereby preferably located upstream of the conveyor10from the robot20. Note that, the conveyor10carries the objects from the left to the right ofFIGS. 3 and 5at a constant speed.

The cycle time refers to the time from when the robot begins storing a prescribed number of objects in a container until the robot completes storing the objects in the container. The prescribed number of objects refers to objects the grasp order (route) of which is to be determined, and objects contained in a certain area30ofFIG. 3. The robot20preferably has a timer for measuring the cycle time.

The torque occurs when the hand moves to the positions of the objects, and when the hand rotates depending on the postures of the objects. The torque is calculated based on currents flowing through motors for driving the hand and an arm (not shown) of the robot20. The robot20preferably has ammeters for measuring the currents flowing through the motors. Note that, after the hand has grasped one object, the hand rotates while moving to a different position to grasp another object. In other words, after the hand has grasped one object, the hand moves while rotating so as to have a suitable angle for grasping the next object.

Vibration occurs when the hand moves to and stops at the positions of the objects, and when the hand rotates and stops rotating depending on the postures of the objects. To measure the vibration, the hand preferably has an acceleration sensor. The vibration is calculated based on an acceleration detected by the acceleration sensor.

The determination data obtaining unit12obtains determination data for determining a margin of each of the cycle time, the torque, and the vibration against its allowance value. The allowance values of the cycle time, the torque, and the vibration may be stored in a memory (not shown). All of the cycle time, the torque, and the vibration are preferably equal to or less than the allowance values.

The learning unit13learns the operation condition of the robot in accordance with a training data set that is constituted of a combination of the state variable and the determination data. When all of the cycle time, the torque, and the vibration are equal to or less than the allowance values, the objects are preferably grasped in the order in which the cycle time is minimized.

Next, the operation of grasping objects with a robot by the machine learning device according to the first embodiment of the present invention will be described with reference to the flowchart ofFIG. 4. First, in step S101, the present positions and postures of a plurality of objects (p1to p6) are obtained. The objects the grasp order of which is to be determined are objects disposed within a certain area30of a conveyor10, as indicated by the dashed lines ofFIG. 3. In the example ofFIG. 3, the grasp order of the objects (p1to p6) is to be determined. The certain area30of the conveyor10is preferably an area that contains an optimal number of objects based on a balance between the time required for calculating the cycle time, the torque, and the vibration in every combination of grasp order of the objects and the speed of the conveyor to move the objects.

Next, in step S102, the grasp order of the objects is assigned based on learning results. Next, in step S103, in response to a request from the robot20, the grasp order of the objects is transmitted from the machine learning device101to the robot20.

FIG. 5is a drawing showing a method for grasping the objects disposed on the conveyor with the hand provided in the robot to put the objects in the containers, after the machine learning device according to the first embodiment of the present invention has performed learning. As the results of learning, as indicated by arrows, the object p1is grasped first, the object p3is grasped next, and the object p2is grasped last.

Grasping the objects in this order allows for a reduction in movement distance of the hand to grasp the objects, as compared with the case of the conventional art shown inFIG. 1as an example. Thus, the time required to grasp the three objects (p1to p3) is reduced. Therefore, it is possible to shorten the cycle time, i.e., the time required to store all the objects in the container.

The hand grasps three objects in this embodiment, but not limited thereto, the number of the objects grasped by the hand may be two or four or more. “Grasp” includes “suction” by the hand.

According to the present invention, the hand rotates by a smaller angle to grasp the object p3, after having grasped the object p1, than the angle by which the hand rotates to grasp the object a2after having grasped the object a1, as described in the conventional art (seeFIG. 1). Accordingly, it is possible to reduce the torque and the vibration that occur while the machine learning device is grasping the three objects (p1to p3). Therefore, it is possible to reduce the total values of the torque and the vibration that occur while the machine learning device is storing all the objects in the container.

Next, the configuration of the learning unit13will be described. As shown inFIG. 2, the learning unit13includes a reward calculation unit14for calculating a reward based on the determination data, and a value function update unit15that updates a value function used for estimating the grasp order of objects so as to reduce at least one of the cycle time, the torque, and the vibration, based on the reward.

The learning unit13updates an action-value table corresponding to the grasp order of objects, based on the state variable including at least one of the cycle time, the torque, the vibration and the reward.

The learning unit13may update an action-value table that corresponds to at least one of the cycle time, the torque, and the vibration when another robot having the same configuration as the robot20stores other objects in a container, based on the state variable and reward of the other robot.

The reward calculation unit14calculates a reward based on at least one of the cycle time, the torque, and the vibration. Furthermore, when failing to grasp an object, in other words, when a grasping error occurs, the reward calculation unit14may provide a negative reward.

The learning unit13preferably further includes a decision determination unit16that determines the grasp order of objects in accordance with results of learning in accordance with a training data set.

Next, a method for calculating a reward will be described.FIG. 6is a flowchart showing a method for calculating a reward in the machine learning device according to the first embodiment of the present invention. First, in step S201, the state observation unit11obtains data on each of the cycle time, the torque, and the vibration amount.

Next, in step S202, the reward calculation unit14determines whether or not the cycle time is shorter than a reference value. When the cycle time is shorter than the reference value, a positive reward is provided in step S203. On the other hand, when the cycle time is equal to or longer than the reference value, the operation proceeds to step S205and no reward is provided. The reference value of the cycle time is the average value of the cycle time when the robot was operated for a certain period of time in the past. Furthermore, the reference value may be adjusted in accordance with learning results, using the average value as an initial value.

Next, in step S204, the reward calculation unit14determines whether or not the torque has increased. When the torque is equal to or less than a reference value, the operation proceeds to step S205and no reward is provided. On the other hand, when the torque is more than the reference value, a negative reward is provided in step S207. The reference value of the torque is the average value of the torque when the robot was operated for a certain period of time in the past. Furthermore, the reference value may be adjusted in accordance with learning results, using the average value as an initial value.

Next, in step S206, the reward calculation unit14determines whether or not the vibration amount has increased. When the vibration amount is equal to or less than a reference value, the operation proceeds to step S205and no reward is provided. On the other hand, when the vibration amount is more than the reference value, a negative reward is provided in step S207. The reference value of the vibration amount is the average value of the vibration amounts when the robot was operated for a certain period of time in the past. Furthermore, the reference value may be adjusted in accordance with learning results, using the average value as an initial value.

Next, a reward is calculated in step S208. When Rcrepresents a reward based on the cycle time, Rrrepresents a reward based on the torque, and Rvrepresents a reward based on the vibration, the total value R of the rewards is calculated by R=α×Rc+β×Rr+γ×Rv, using prescribed coefficients, which define weights.

Next, in step S209, the learning unit13updates an action-value table that corresponds to the grasp order of objects, based on a state variable of at least one of the cycle time, the torque, the vibration and the reward.

The learning unit13preferably performs computation of a state variable observed by the state observation unit11in a multilayer structure, and updates an action-value table in real time. As a method for performing computation of a state variable in a multilayer structure, for example, a multilayer neural network, as shown inFIG. 8, is usable.

The machine learning device101shown inFIG. 2will be described in detail. The machine learning device101has the function of learning knowledge, as well as the functions of extracting useful rules, knowledge representations, criteria, and the like from a data group inputted to the device by analysis and outputting determination results. There are various methods therefor, which are broadly divided into “supervised learning”, “unsupervised learning”, and “reinforcement learning”. Moreover, when realizing these methods, a method called “deep learning” is used for learning the extraction of features themselves.

In “supervised learning”, a large amount of data pairs of an input and a result (label) are provided to the learning device. The machine learning device learns features from the data set, and heuristically obtains a model to predict a result from an input, i.e., the relationship therebetween. “Supervised learning” can be realized using an algorithm such as a neural network, described later.

In “Unsupervised learning”, only a large amount of input data is provided to the machine learning device. The machine learning device learns the distribution of the input data and applies compression, classification, alignment, or the like to the input data, without being supplied with corresponding output data as supervisors. The features of the data set can be subjected to clustering by analogy. With the use of this result, while providing a certain criterion, an output is assigned so as to optimize the criterion, and this allows for a prediction of the output. There is also a method called “semi-supervised learning” as an intermediate problem setting between “supervised learning” and “unsupervised learning”, in which part of data includes pairs of an input and an output while the other includes only inputs. In this embodiment, data that can be obtained without actually operating the robot is used in the unsupervised learning, in order to improve learning efficiency.

Problems in the reinforcement learning are determined as follows.The robot observes an environment state and determines an action.The environment varies in accordance with some rule, and the actions of the robot itself sometimes vary the environment.A reward signal returns whenever an action is performed.The aim is to maximize a total (discounted) reward in the future.The learning is started from a state in which a result brought about by an action is not or is insufficiently known. The robot obtains the result as data, only after the robot is actually operated. In other words, it is necessary to search for an optimal action through trial and error.The learning may be started from a good start point by performing conventional learning (using a method of the above-described supervised learning or inverse reinforcement learning) so as to mimic the action of a human in an initial state.

“Reinforcement learning” is a method for learning optimal actions based on the interactions between an action and an environment by learning actions, as well as determining and classifying, in other words, a learning method to maximize a total reward obtained in the future. In this embodiment, this indicates that actions having an effect on the future can be obtained. The following description takes Q learning as an example, but is not limited thereto.

Q learning is a method for learning a Q(s, a) value for selecting an action “a” in a certain environment state “s”. In other words, in a certain state “s”, an action “a” having the highest Q(s, a) value is selected as an optimal action. However, as to a combination of a state “s” and an action “a”, a correct Q(s, a) value is not known at all in the beginning. Thus, an agent chooses various actions “a” in a certain state “s”, and is provided with a reward for each action “a”. Therefore, the agent learns to select a better action, i.e., a correct Q(s, a) value.

The aim is to maximize a total reward to be obtained in the future, i.e., to obtain Q(s, a)=E [Σγtrt], as a result of actions (an expected value is taken when a state changes in accordance with optimal actions; the optimal actions have not been known, as a matter of course, and hence have to be found while learning.). For example, an update equation for a Q(s, a) value is represented as follows:

where strepresents an environment state at a time t, and at represents an action at the time t. By executing the action at, the state changes to st+1. “rt+1” represents a reward provided by the state change. A term with “max” represents the product of a Q value when an action “a” having the highest Q value that has been known at that time is selected in the state st+1, and γ. “γ” is a parameter of 0<γ≤1 called discount factor. “α” is a learning rate in the range of 0<α≤1.

This equation indicates a method for updating a Q (st, at) value of an action at in a state stbased on a reward rt+1that has returned as a result of the action at. This update equation indicates that, as compared with a Q(st, at) value of an action “a” in a state “s”, if a Q (st+1, max at+1) value of an optimal action “max a” in the next state derived from a reward rt+1plus the action “a” is higher, Q(st, at) is increased. If not, Q(st, at) is decreased. In other words, the value of an action in a certain state is approximated to an optimal action-value in the next state that is derived from a reward returning immediately as a result of the action and the action itself.

There are two methods for representing Q(s, a) values in a computer, that is, a method in which the Q values of all state-action pairs (s, a) are retained in a table (action-value table) and a method in which a function for approximating Q(s, a) values is prepared. In the latter method, the above-described update equation can be realized by adjusting a parameter for an approximation function using a stochastic gradient descent method or the like. As the approximation function, a neural network can be used as described later.

As an approximation algorithm for a value function in supervised learning, unsupervised learning, and reinforcement learning, a neural network can be used. The neural network is constituted of, for example, an arithmetic unit, memory, and the like that imitate a neuron model as shown inFIG. 7.FIG. 7is a schematic diagram of a neuron model.

As shown inFIG. 7, a neuron produces an output y in response to a plurality of inputs x (for example, inputs x1to x3). Each of the inputs x1to x3is applied with a weight w (w1to w3) corresponding to the input x. Thus, the neuron produces the output y represented by the following equation. Note that, all of the input x, the output y, and the weight w are vectors.
y=fk(Σi=1nxiwi−θ)

where θ is a bias, and fkis an activation function.

Next, a three-layer neural network having weights of three layers, which is constituted of a combination of the above-described neurons, will be described with reference toFIG. 8.FIG. 8is a schematic diagram of a neural network having weights of three layers D1to D3.

As shown inFIG. 8, a plurality of inputs x (for example, inputs x1to x3) are inputted from the left of the neural network, and a result y (for example, results y1to y3) is outputted from the right thereof.

To be more specific, the inputs x1to x3are inputted to each of the three neurons N11to N13while being weighted correspondingly. The weights applied to the inputs are collectively indicated by W1.

The neurons N11to N13output vectors Z11to Z13, respectively. The vectors Z11to Z13are collectively indicated by a feature vector Z1, which is regarded as a vector that extracts a feature amount from the input vector. The feature vector Z1is a feature vector between the weight W1and a weight W2.

The vectors Z11to Z13are inputted to each of two neurons N21and N22while being weighted correspondingly. The weights applied to the feature vectors are collectively indicated by W2.

The neurons N21and N22output vectors Z21and Z22, respectively. The vectors Z21and Z22are collectively indicated by a feature vector Z2. The feature vector Z2is a feature vector between the weight W2and a weight W3.

The feature vectors Z21and Z22are inputted to each of three neurons N31to N33while being weighted correspondingly. The weights applied to the feature vectors are collectively indicated by W3.

The neural network has a learning mode and a value prediction mode. In the learning mode, the weight W is learned using a learning data set. In the value prediction mode, the action of the robot is determined through the use of the parameter (the word of “prediction” is used for the sake of convenience, but various tasks including detection, classification, inference, and the like can be performed).

In the value prediction mode, data that has been obtained by actual operation of the robot may be immediately learned and reflected in the next action (on-line learning). Also, learning may be collectively performed using a data group collected in advance, and a detection mode may be performed thereafter using the parameter (batch learning). In an intermediate manner, the learning mode may be performed whenever a certain amount of data is accumulated.

The weights W1to W3can be learned using an error back propagation algorithm (backpropagation algorithm). Information about an error enters from the right and propagates to the left. The error back propagation algorithm is a method in which each weight is adjusted (learned) with respect to each neuron so as to minimize the difference between an output y and an actual output y (supervisor) in response to an input x.

Such a neural network may have layers of more than three (called deep learning). An arithmetic unit that performs feature extraction from inputs in stages and regression of results can be automatically acquired from only supervisor data.

Accordingly, in order to perform the Q learning described above, as shown inFIG. 2, the machine learning device101according to this embodiment includes the state observation unit11, the learning unit13, and the decision determination unit16. However, a machine learning method applied to the present invention is not limited to Q learning. For example, in the instance of applying supervised learning, a value function corresponds to a learning model, and a reward corresponds to an error.

As shown inFIG. 2, the state of the robot20includes a state indirectly changed by an action and a state directly changed by an action. The state indirectly changed by an action includes the cycle time, the torque, and the vibration. The state directly changed by an action includes a current value.

The learning unit13updates an action-value that corresponds to a present state variable and an action to be taken in the action-value table, based on the update equation and the reward.

In the example ofFIG. 2, the own action-value table is updated using the action-value table updated by the learning unit of the own machine learning device, but is not limited thereto. The own action-value table may be updated using an action-value table updated by a learning unit of another machine learning device different from the own machine learning device.

Second Embodiment

A machine learning device according to a second embodiment of the present invention will be described.FIG. 9is a block diagram of the machine learning device according to the second embodiment of the present invention. The difference between a machine learning device102according to the second embodiment of the present invention and the machine learning device101according to the first embodiment is that the machine learning device102is connected to a first robot21and a second robot22through a communication unit18, i.e., a network, and the state observation unit11obtains a present state variable through the network. The other configurations of the machine learning device102according to the second embodiment are the same as that of the machine learning device101according to the first embodiment, and thus a detailed description thereof is omitted.

The machine learning device102is preferably installed in a cloud server.

It may be difficult for a robot to perform learning during high speed operation, owing to a heavy processing load. According to the configuration of the machine learning device of the second embodiment of the present invention, learning is performed by a device other than a robot controller, thus reducing the burden on the robot.

Third Embodiment

Machine learning devices according to a third embodiment of the present invention will be described.FIG. 10is a block diagram of the machine learning devices according to the third embodiment of the present invention. The difference between machine learning devices103-1and103-2according to the third embodiment of the present invention and the machine learning device101according to the first embodiment is that a machine learning device is provided in each of a plurality of robot controllers connected through a hub19. The other configurations of the machine learning devices103-1and103-2according to the third embodiment are the same as that of the machine learning device101according to the first embodiment, and thus a detailed description thereof is omitted.

A first machine learning device103-1is provided in a first robot controller201for controlling a first robot21. A second machine learning device103-2is provided in a second robot controller202for controlling a second robot22.

This structure allows the shared use of an action-value table created by one of the machine learning devices between the machine learning devices, thus improving the efficiency of learning.

Fourth Embodiment

A machine learning device according to a fourth embodiment of the present invention will be described.FIG. 11is a block diagram of the machine learning device according to the fourth embodiment of the present invention. The difference between a machine learning device104according to the fourth embodiment of the present invention and the machine learning device101according to the first embodiment is that the machine learning device104records and collects the positions and postures of objects disposed on a conveyor, and reproduces offline the positions and postures of the objects (p1to p6) on the conveyor10in order to learn the optimal grasp order of the objects. The other configurations of the machine learning device104according to the fourth embodiment are the same as that of the machine learning device101according to the first embodiment, and thus a detailed description thereof is omitted.

A first robot controller201controls a first robot21, while a second robot controller202controls a second robot22. The machine learning device104is provided outside the first robot controller201and the second robot controller202. The machine learning device104receives data (“LOG”) about the positions and postures of the objects, which is obtained from images captured by cameras (not shown) each provided in the vicinity of each of the first robot21and the second robot22, and learns the optimal grasp order of the objects. The learning results are transmitted to the first robot controller201and the second robot controller202, so that the first robot21and the second robot22can grasp the objects in the optimal order.

According to the machine learning device and the machine learning method of the embodiments of the present invention, it is possible to reduce the burden on the robot, as well as minimize the cycle time that is the time required for the robot to store the plurality of objects in the container using the hand having the function of grasping the objects.