Robot system and computer program product

A robot system includes a robot including a plurality of fingers for holding a target object and a control device configured to control a motion of the robot. The control device includes one or more processors. The processors acquire an image of a first target object and a second target object taken by an imaging device. The processors control the motion of the robot based on the image such that the robot moves the first target object with at least one finger included in the fingers in a direction in which a gap is formed between the first target object and the second target object, inserts at least one finger included in the fingers into the gap, and holds the first target object.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority from Japanese Patent Application No. 2020-020644, filed on Feb. 10, 2020; the entire contents of which are incorporated herein by reference.

FIELD

Embodiments described herein relate generally to a robot system and a computer program product.

BACKGROUND

In factories and distribution centers, robots take out holding target objects, such as parts and products, from cardboard boxes and shelves. If a holding target object is relatively large and has a flat surface, a robot can take out the holding target object by causing it to adhere to a suction cup provided to a robot hand. If a holding target object is a cylindrical object, such as a bolt, and the objects are loaded in bulk, the robot can hold the holding target object by sandwiching it with two fingers of the robot hand.

The robots according to the conventional technologies, however, may possibly fail to appropriately hold the holding target object. If the flat surface of the holding target object is small, and a plurality of holding target objects are placed with no gap therebetween, or if a plurality of holding target objects fall down and overlap one another, for example, the robots fail to cause the holding target object to be held next alone to adhere to the suction cup. In this case, the robots also fail to hold the holding target object by inserting the two fingers of the robot hand between the holding target objects.

DETAILED DESCRIPTION

According to one embodiment, a robot system according to an embodiment includes a robot including a plurality of fingers for holding a target object and a control device configured to control a motion of the robot. The control device includes one or more processors. The processors acquire an image of a first target object and a second target object taken by an imaging device. The processors control the motion of the robot based on the image such that the robot moves the first target object with at least one finger included in the fingers in a direction in which a gap is formed between the first target object and the second target object, inserts at least one finger included in the fingers into the gap, and holds the first target object.

Exemplary embodiments of a robot system according to the present invention are described below in greater detail with reference to the accompanying drawings.

First Embodiment

A robot system according to a first embodiment uses a finger (third finger) provided besides fingers (first finger and second finger) for holding a holding target object and moves (e.g., tilts, translates (slides), and rotates) the holding target object to a position where it can be held with the first finger and the second finger.

If there is no gap between the holding target object and another object, for example, the robot system moves the holding target object with the third finger to create a gap between the holding target object and the other object. By inserting the first finger or the second finger into the gap, the robot system can hold the holding target object. If a plurality of holding target objects fall down and overlap one another (e.g., if the holding target objects fall over like dominos), the robot system moves and raises the holding target object with the third finger. The robot system according to the present embodiment has a function of learning a model (e.g., a neural network) for deriving a motion sequence of a robot to enable the motions described above.

With this configuration, the robot system can avoid a situation where manual holding needs to be performed because holding by the robot is difficult. As a result, the robot system can significantly progress automation in holding work (picking work) for the holding target object in factories and distribution centers, for example.

In the present embodiment, it is not necessary to include an actuator (e.g., a motor) that independently operates the third finger and can move the third finger by an actuator common to the first finger and the second finger, for example. Consequently, the motions described above can be achieved with a simpler configuration.

In the present embodiment, the holding target object is held by being sandwiched (clamped) between two fingers (the first finger and the second finger). Holding includes causing the holding target object to adhere to a suction cup, for example. Examples using the suction cup will be described in a second embodiment.

FIG.1is a block diagram of an example of the entire configuration of the robot system according to the first embodiment. As illustrated inFIG.1, the robot system includes a control device100, two robots200M and200S, and a camera300.

The devices (the control device100, the robots200M and200S, and the camera300) are connected via a network, such as a local area network (LAN) and the Internet. The network may be a wired or wireless network. Instead of or besides the network, the devices may be connected via dedicated lines.

The robot200M and the robot200S operate as a master and a slave, respectively, and have the same configuration. The robots200M and200S may be simply referred to as a robot200when they need not be distinguished from each other.

The camera300is an example of an imaging device that images a motion of the robot200(robot200S inFIG.1) holding a holding target object400. The camera300may be mounted on the robot200.

The control device100controls motions of the robot200. When a user operates the robot200M, for example, the control device100controls the robot200S such that the robot200S performs the same motion as the robot200M. The robots200M and200S can be used to learn robot motions by imitation learning, for example. In imitation learning, the motion of the robot200S is learned by imitating the motion of the robot200M corresponding to the operation performed by the user, for example. In imitation learning, both of the robots200M and200S are required. In operation using the learning result (after learning), for example, at least one robot200to be operated (robot200S inFIG.1) is required.

The control device100includes storage121, an image acquirer101, a motion information acquirer102, a controller103, and a learner104.

The storage121stores therein various kinds of information used for various kinds of processing in the control device100. The storage121, for example, stores therein images acquired by the image acquirer101, motion information acquired by the motion information acquirer102, and information indicating a neural network to be learned (e.g., parameters, such as weight, of the neural network). The storage121may be any generally used storage medium, such as a flash memory, a memory card, a random access memory (RAM), a hard disk drive (HDD), and an optical disc.

The image acquirer101acquires an image taken by the camera300. The image acquirer101, for example, acquires time-series images by imaging a motion of the robot200holding the holding target object400. The acquired images are sequentially stored in the storage121, for example.

The motion information acquirer102acquires motion information indicating a motion state of the robot200. The motion information includes the joint angle of an arm joint included in the robot200, for example. In learning, as illustrated inFIG.1, the motion information acquirer102acquires the motion information on the robot200M serving as a master. In operation using the learning result (after learning), the motion information acquirer102acquires the motion information on the robot200being operated.

The controller103controls motions of the robot200. In learning, for example, the controller103refers to the motion information on the robot200M acquired by the motion information acquirer102and operates the robot200S such that the robot200S gets into a motion state indicated by the motion information. In operation using the learning result (after learning), the controller103operates the robot200such that the robot200gets into a motion state indicated by the motion information derived using the learned neural network.

The learner104learns a neural network. The neural network is an example of a model that receives input information based on an image of the holding target object and outputs the motion information on the robot200. The input information based on an image may be an image taken by the camera300or image information obtained by compressing the taken image (e.g., information indicating characteristics of the image). The learner104, for example, inputs the image acquired by the image acquirer101to the neural network and learns the parameters (e.g., weight) of the neural network such that the error between the motion information output from the neural network and the motion information (corresponding to correct data) on the robot200M acquired by the motion information acquirer102is smaller. An exemplary configuration of the neural network and an example of the learning method will be described later.

The units described above (the image acquirer101, the motion information acquirer102, the controller103, and the learner104) are provided by one or more processors, for example. The units may be provided by causing a processor such as a central processing unit (CPU) to execute a computer program, that is, by software. The units may be provided by a processor such as a dedicated integrated circuit (IC), that is, by hardware. The units may be provided by a combination of software and hardware. If a plurality of processors are used, the processors may each provide one of the units or two or more of them.

The units may be provided to different devices. Functions necessary for learning (e.g., the learner104) and the other functions may be provided to two different devices in a distributed manner, for example.

The following describes an exemplary configuration of the robot200.FIG.2is a schematic diagram of an exemplary configuration of the robot200. The robot200illustrated inFIG.2is an example of an articulated robot including five joints. The number of joints is not limited to five.

The robot200includes a housing201, a plurality of arm joints211to214, a support member202, and a plurality of fingers221,222, and223. The arm joints211to214are coupled to each other with any one of the joints interposed therebetween. The finger221includes a base221a, a telescopic mechanism221b, and an end portion221c. The finger222includes a base222a, a telescopic mechanism222b, and an end portion222c. Arrows231to235indicate motion directions of the five joints. The joint angles of the five joints are an example of the motion information and are acquired by the motion information acquirer102and stored in the storage121, for example.

“Joint angles θ1 to θ5” illustrated inFIG.1indicate the joint angles of the five joints corresponding to the motion information stored as described above. As illustrated inFIG.1, the motion information may be stored in association with an image taken at corresponding time.

The arm joints211to214are coupled in series. The support member202and the fingers221,222, and223are provided at the end of the arm joint214. The support member202supports the fingers221,222, and223. The fingers221,222, and223correspond to the first finger, the second finger, and the third finger, respectively.

The telescopic mechanisms221band222bextend and contract the end portions221cand222cin a direction away from the support member202(extension) and a direction closer to the support member202(contraction). The telescopic mechanisms221band222bcorrespond to a moving member that moves the fingers221and222relatively to the support member202. The support member202and the fingers221,222, and223can be considered to constitute a hand provided at the end of the arm joint214.

The arm joints211to214and the hand (the support member202and the fingers221,222, and223) are an example of movable members. The positions and the postures of the respective movable members can be changed and maintained by operations of an actuator (driver) that drives the movable members. While the actuator is a motor, for example, it is not limited thereto. The actuator may be a pump with a motor, a solenoid, an electromagnetic valve with a solenoid, or a piston cylinder, for example. A drive controller (servo amplifier) that controls drive of the actuator is provided to the robot200, for example.

The position and the posture of the hand can be changed within a movable range of the arm joints211to214. If the user moves the arm joints while holding the end of the hand of the robot200M serving as a master in learning, for example, the motion is transmitted to the robot200S serving as a slave.

A motor (driver) that changes the space between the finger221and the finger222may be provided as one motor, for example.FIG.3is a schematic diagram of an exemplary configuration of the motor. As illustrated inFIG.3, the motor that changes the space between the finger221and the finger222rotates about a rotation axis301, thereby changing the space between the finger221and the finger222with gears.

The following describes the telescopic mechanisms221band222bthat extend and contract the fingers221and222corresponding to the first finger and the second finger.FIGS.4and5are schematic diagrams for explaining motions of the telescopic mechanisms221band222b.FIGS.4and5do not illustrate the finger223for convenience of explanation.

FIG.4illustrates an exemplary motion of the telescopic mechanisms221band222bperformed when the space between the fingers221and222is wide (when the fingers are open). The telescopic mechanisms221band222bextend the fingers221and222in a direction away from the support member202by gravity acting on themselves and the end portions221cand222c. When the end portions221cand222ccome into contact with the holding target object400, and the hand is moved in the direction toward the holding target object, the telescopic mechanisms221band222bcontract the fingers221and222in a direction closer to the support member202by force from the holding target object400.

FIG.5illustrates an exemplary motion of the telescopic mechanisms221band222bperformed when the space between the fingers221and222is narrow (when the fingers are closed). The motion process of the telescopic mechanisms221band222bis the same as that inFIG.4.

As described above, in the present embodiment, the telescopic mechanisms221band222bserving as passive mechanisms extended by gravity extend and contract the fingers221and222. Consequently, it is not necessary to include actuators that extend and contract the fingers221and222. The telescopic mechanisms221band222bmay extend the fingers221and222with an elastic member such as a spring.

The following describes a motion of the finger223serving as the third finger.FIGS.6and7are schematic diagrams for explaining a motion of the finger223and the telescopic mechanisms221band222b. The finger223is used mainly for tilting the holding target object400and moving (translating) it in the horizontal direction.

FIG.6illustrates an exemplary motion of the finger223and the telescopic mechanisms221band222bperformed when the space between the fingers221and222is wide. As illustrated inFIG.6, the ends of the end portions221cand222care positioned in an outward direction with respect to the end of the finger223when they extend. The ends of the end portions221cand222care positioned in an inward direction with respect to the end of the finger223when they contract. The outward (inward) direction indicates a direction away from the support member202(direction closer to the support member202) when viewed in the extension direction of the fingers221and222with respect to the support member202, for example.

When the end portions221cand222ccome into contact with the holding target object400, and the hand is moved in the direction toward the holding target object, the telescopic mechanisms221band222bcontract the fingers221and222in the direction closer to the support member202. When the finger223comes into contact with the holding target object400, contraction of the fingers221and222stops. The finger223is not extended or contracted because it is not connected to any telescopic mechanism or the like. By changing the position and the posture of the hand with the finger223in contact with the holding target object400, the robot200can move (e.g., tilt, translate, and rotate) the holding target object400.

FIG.7illustrates an exemplary motion of the finger223and the telescopic mechanisms221band222bperformed when the space between the fingers221and222is narrow. The motion process of the finger223and the telescopic mechanisms221band222bis the same as that inFIG.6.

Holding the holding target object400using the finger223is performed as follows, for example. First, the controller103operates the robot200such that the finger223comes into contact with the holding target object400with the fingers221and222closed (in a state where the space between the fingers221and222is the initial value). Subsequently, the controller103changes the position of the finger223and moves the holding target object400such that the holding target object400gets into the position and the posture where it can be held with the fingers221and222as needed. Subsequently, the controller103operates the robot200to gradually widen the space between the fingers221and222(such that the space becomes a value larger than the initial value). When the space between the fingers221and222is made wider than the width of the holding target object400(value at which the holding target object400can be held), the fingers221and222extend by gravity and move to the position where they can hold the holding target object400. Subsequently, the controller103operates the robot200to narrow the space between the fingers221and222, thereby holding the holding target object400with the fingers221and222. Specific examples of the motion of holding the holding target object400will be described later.

The following describes learning performed by the control device100according to the first embodiment having the configuration described above.FIG.8is a flowchart of an example of learning according to the first embodiment.

If learning is started, the image acquirer101acquires an image of the holding target object400and the hand of the robot200S, for example, taken by the camera300(Step S101). The motion information acquirer102acquires the motion information from the robot200M serving as a master that operates based on the operations performed by the user (Step S102). The learner104learns the neural network using the acquired image and the motion information (Step S103). The learner104, for example, inputs the acquired image to the neural network and learns the parameters (e.g., weight) of the neural network such that the error between the motion information output from the neural network and the acquired motion information on the robot200M is smaller.

Learning is repeatedly performed by changing conditions, such as the number of holding target objects400, the positions and the postures of a plurality of holding target objects400, and the relative positional relation of the holding target objects400. By learning with a number of conditions, the control device100can learn the neural network such that it can output the motion information that enables appropriately holding the holding target object400under a variety of conditions.

The learner104, for example, learns the neural network using the motion information obtained when the robot200M is operated as follows: when there is no gap between the holding target object400(first target object) and another object (second target object), the robot200M moves the holding target object400with the finger223in a direction in which a gap is formed between the holding target object400and the other object, inserts the finger221or222into the formed gap, and holds the holding target object400. By using the neural network learned as described above, the control device100can control the robot200such that the robot200forms a gap into which the finger221or222can be inserted and appropriately holds the holding target object400in the same situation described above. In other words, the robot200can more appropriately hold the holding target object.

The following describes control of the robot200performed by the control device100according to the first embodiment having the configuration described above. Control is processing for controlling the motion of the robot200using the learned neural network.FIG.9is a flowchart of an example of control according to the first embodiment.

The image acquirer101acquires an image of the holding target object400and the hand of the robot200S, for example, taken by the camera300(Step S201). The controller103inputs the acquired image to the neural network and acquires the motion information output from the neural network (Step S202). The controller103controls the motion of the robot200such that the robot200gets into a motion state indicated by the output motion information (Step S203).

As described above, by using the neural network learned with a variety conditions, the control device100can control the robot200such that the robot200more appropriately holds the holding target object under the variety of conditions (states). When there is no gap between the holding target object400(first target object) and another object (second target object), the controller103, for example, can operate the robot200as follows: the robot200moves the holding target object400with the finger223in a direction in which a gap is formed between the holding target object400and the other object, inserts the finger221or222into the formed gap, and holds the holding target object400.

The following describes specific examples of the motion of holding the holding target object400in greater detail.FIG.10is a schematic diagram of an example of the holding motion performed when a configuration not provided with the finger223(third finger) is used. In this configuration, if the holding target object400is in contact with an obstacle1001(e.g., another object), no gap is present between the holding target object400and the obstacle1001. As a result, the robot200fails to insert the end portions221cand222c. In other words, the robot200fails to hold the holding target object400with the fingers221and222. The obstacle1001is not limited to another object and may be a container accommodating the holding target object400, for example.

FIG.11is a schematic diagram of an example of the holding motion performed when a configuration provided with the finger223(third finger) is used that the present embodiment discloses. In this configuration, by tilting the holding target object400using the finger223, for example, the robot200can form a gap between the obstacle1001and the holding target object400. The robot200can insert the end portion221cor222cinto the gap to hold the holding target object400.

The following describes a specific example of the motion of holding the holding target object400and moving it into a second container with reference toFIGS.12to21. The following describes an example where the controller103operates the robot200using the learned neural network. This example can be considered to be an example indicating a process of operating the master robot200M by the user in learning.

As illustrated inFIG.12, the controller103operates the robot200such that the finger223comes closer to the upper surface of the holding target object400with the fingers221and222closed. At this time, the end portions221cand222care lowered by gravity, and the positions of the ends are closer to the holding target object400than the finger223is.

FIG.13illustrates a state where the end portions221cand222care in contact with the upper surface of the holding target object400.FIG.14illustrates a state where the finger223is also in contact with the upper surface of the holding target object400. Subsequently, as illustrated inFIG.15, the controller103causes the finger223to act on the holding target object400, thereby tilting the holding target object400. As illustrated inFIG.16, the controller103widens the space between the fingers221and222.

When the space between the fingers221and222is made larger than the width of the holding target object400, the fingers221and222extend by the effect of gravity as illustrated inFIG.17. Subsequently, as illustrated inFIG.18, the controller103closes the fingers221and222to hold the holding target object400.

As illustrated inFIG.19, the controller103lifts the holding target object400while holding it. As illustrated inFIG.20, the controller103moves the holding target object400to the outside of the container. As illustrated inFIG.21, the controller103places the holding target object400in a second container2101. As described above, the robot200can hold the holding target object400placed with no gap therearound.

The following describes another specific example of the motion of holding the holding target object400and moving it into a second container with reference toFIGS.22to25.FIGS.22to25are an example of a process of holding the holding target object400that has fallen down.

As illustrated inFIG.22, when the holding target object400falls down, it is difficult to hold the holding target object400if it remains in the state. To address this, the controller103closes the fingers221and222to bring them into contact with the finger223and changes the position of the finger223, thereby raising the holding target object400.FIGS.23to25illustrate a process in which the holding target object400is being raised to stand. The subsequent process of moving the holding target object400into the second container (e.g., the container2101illustrated inFIG.21) is the same as that described with reference toFIGS.12to21.

The following describes a model for deriving the motion information on the robot200.FIG.26is a diagram of an exemplary configuration of the model for deriving the motion information on the robot200.

As described above, the robot system according to the present embodiment has a master-slave configuration. If the user operates the master robot200M, the robot system can operate the slave robot200S in the same manner as the operated robot200M. The camera300images the motion of the robot200S (arm joint and hand) and the holding target object400and outputs an image. The taken image is stored in the storage121in association with the joint angles of the respective joints of the robot200obtained when the image is taken.

The image may be used as data input to the model without any change, or a compressed image may be used as data input to the model. An image of 128×128 pixels, for example, is compressed into 256-dimensional image information using a technology such as variational auto encoder (VAE). In this case, the image information is represented by one point in the 256-dimensional space. Movement of the robot200and the holding target object400means continuous movement of one point in the 256-dimensional space. Movement of the point in the 256-dimensional space can be considered to represent movement of the holding target object400and movement of the hand.

If the number of joints of the robot200is five, the posture (motion information) of the robot200is represented by one point in five-dimensional space. Movement of the robot200means continuous movement of one point in the five-dimensional space.

The model for deriving the motion information from an image corresponds to a mapping function in which one point in the 256-dimensional space corresponds to one point in the five-dimensional space.FIG.26illustrates an example of mapping f for converting image information x into a joint angle τ (an example of the motion information).

When the robot200is moving, movement of one point in the 256-dimensional space and movement of one point in the five-dimensional space are synchronized time-series data. The mapping function for associating these pieces of time-series data (time-series data on a plurality of images or time-series data on a plurality of pieces of motion information) can be provided by a neural network, for example. Examples of the neural network include, but are not limited to long short-term memory (LSTM), etc.

FIG.27is a diagram of an exemplary configuration of the VAE and the LSTM. The VAE receives an image2701(e.g., an image of 128×128 pixels) taken by the camera300. The VAE is learned such that the error between output from the VAE and the image2701is smaller. In the process performed by the VAE, compressed image information (latent variable z) is generated. The VAE is also a neural network and requires a number of images for learning. Consequently, learning is performed by causing the master-slave configuration to perform the holding motion a plurality of times, creating data on a number of images and a number of joint angles (motion information), and inputting these pieces of data into the neural network.

The VAE has a function of forming average (μ) and variance (σ) of data inside thereof and can form a mapping function using a number of pieces of data. By using this function, the VAE can smoothly connect movement of one point in the 256-dimensional space, thereby deriving the latent variable z.

The LSTM receives the latent variable z derived as described above as compressed image information. The method for deriving the compressed image information (latent variable z) is not limited to the VAE. Furthermore, a non-compressed image (e.g., the image2701) may be input to the LSTM.

If a detection result (tactile information) can be acquired by a sensor or the like provided separately from the camera300, the tactile information may also be input to the LSTM. Data obtained by combining the latent variable z and the tactile information, for example, may be input to the LSTM. If no tactile information is acquired, the compressed image information (latent variable z) alone may be input to the LSTM.

The LSTM outputs a value indicating the joint angles of the respective joints of the robot200, for example. Also for the LSTM, data obtained by performing the holding motion for a number of times is used. In the example of the model illustrated inFIG.27, a mixed density network (MDN) is connected to the LSTM. The MDN statistically processes distribution of data on a number of joint angles output from the LSTM and outputs it. As described above, the model according to the present embodiment has a function of outputting information indicating distribution of predicted values (motion information such as the joint angle) for the input information.

FIGS.28to30are diagrams for explaining the processing performed by the MDN.FIG.28is a diagram of an example of distribution of the joint angles. A circle corresponds to a value of the joint angle of a certain joint at a certain time. The curved line inFIG.28indicates a line approximating variations of a number of values. The MDN can more appropriately express the distribution of the joint angles.

In other words, the MDN represents the distribution of the joint angles by superposition of Gaussian distribution as indicated by Expressions (1) and (2). In the following expressions, c represents the number of joints of the robot200.

FIG.29is a diagram of an example of distribution represented by superposition of Gaussian distribution. The shape of the Gaussian distribution is determined by: μ1, μ2, μ3, . . . , μmcorresponding to the peak positions of the respective distributions, α1, α2, α3, . . . , αmcorresponding to the heights of the respective distributions, and σ1, o2, σ3, . . . , σmcorresponding to the spreads of the respective distributions.FIG.29illustrates an example of the distribution where m, which represents the number of distributions, is 3.

FIG.30is a diagram of an example of correspondence between input and output to and from the MDN. InFIG.30, x represents input to the MDN (output from the LSTM), and α, μ, and σ represent output from the MDN and correspond to α, μ, and σ described with reference toFIG.29.FIG.30illustrates an example of output corresponding to the joint angle of a certain joint, that is, one-dimensional joint angle information. If five joints are used, for example, five pieces of data illustrated inFIG.30(corresponding to five-dimensional joint angle information) are output.

By using the MDN, the control device100can smoothly connect and process pieces of data on a plurality of taught joint angles. As a result, the control device100can support joint angles not taught in learning, thereby improving the robustness in movement of the robot200.

As described above, the robot system according to the first embodiment can move the holding target object to a position where it can be held using the finger provided besides the fingers for holding the holding target object. Consequently, the robot system can prevent the robot from failing to appropriately hold the holding target object.

Second Embodiment

A second embodiment describes an example of the robot system including a suction cup that can cause the holding target object to adhere thereto.

FIG.31is a schematic diagram of an exemplary configuration of the hand of the robot included in the robot system according to the present embodiment. As illustrated inFIG.31, a finger223-2corresponding to the third finger according to the present embodiment includes a suction cup226-2at the end. Because the other components are the same as those according to the first embodiment, they are denoted by like reference numerals, and explanation thereof is omitted.

With the suction cup226-2, the robot can more reliably hold the upper surface of the holding target object400. Consequently, the finger223-2according to the present embodiment can translate the holding target object400in a case where the finger223with no suction cup according to the first embodiment fails to translate the holding target object400, for example.

The lengths of the fingers221,222, and223-2may be shorter than those illustrated inFIG.31, for example.FIG.32illustrates an example of a configuration including the fingers221,222, and223-2having shorter lengths than those illustrated inFIG.31.

One of the fingers221and222is not necessarily provided.FIG.33illustrates an example of a configuration not including the finger222. With this configuration, the robot also can stably hold the holding target object400with the suction cup226-2and the finger221, for example.

Third Embodiment

A third embodiment describes an example of the robot system including a sensor different from the camera300.

FIG.34is a schematic diagram of an exemplary configuration of the hand of the robot included in the robot system according to the present embodiment. As illustrated inFIG.34, in the present embodiment further includes displacement sensors241-3and242-3and touch sensors251-3,252-3, and253-3.

The displacement sensors241-3and242-3detect the amount of displacement (amount of movement) of end portions221c-3and222c-3, respectively. The touch sensors251-3,252-3, and253-3are provided at the ends of the end portions221c-3and222c-3and a finger223-3, respectively, and detect contact of an object or the like with themselves or the corresponding fingers.

Data resulting from measurement (detection) by these sensors can be used as part of the motion information. The data resulting from detection by the sensors can be used as the tactile information illustrated inFIG.27. Consequently, the robot system can introduce the contact state of the hand with the holding target object400into learning, thereby enabling a more robust motion.

All the sensors (the displacement sensors241-3and242-3and the touch sensors251-3,252-3, and253-3) are not necessarily provided, and only part thereof may be provided. One of the displacement sensors241-3and242-3and the touch sensors251-3,252-3, and253-3may be provided, for example. Only the detection information of part of the provided sensors may be used as the motion information.

As described above, the first to the third embodiments can reduce the possibility that the robot fails to appropriately hold the holding target object.

The following describes a hardware configuration of the control device according to the first to the third embodiments with reference toFIG.35.FIG.35is a diagram for explaining an exemplary hardware configuration of the control device according to the first to the third embodiments.

The control device according to the first to the third embodiments includes a control device such as a central processing unit51, storage devices such as a read only memory (ROM)52and a random access memory (RAM)53, a communication I/F54connected to a network to perform communications, and a bus61that connects these units.

The computer program executed in the control device according to the first to the third embodiments is embedded and provided in the ROM52, for example.

The computer program executed in the control device according to the first to the third embodiments may be recorded in a computer-readable recording medium, such as a compact disc read only memory (CD-ROM), a flexible disk (FD), a compact disc recordable (CD-R), and a digital versatile disc (DVD), as an installable or executable file and provided as a computer program product.

The computer program executed in the control device according to the first to the third embodiments may be stored in a computer connected to a network, such as the Internet, and provided by being downloaded via the network. Furthermore, the computer program executed in the control device according to the first to the third embodiments may be provided or distributed via a network such as the Internet.

The computer program executed in the control device according to the first to the third embodiments can cause a computer to function as the units of the control device described above. The CPU51of the computer can read and execute the computer program from a computer-readable storage medium on a main memory.