Patent Description:
Techniques for detecting a posture of a user and changing a posture of a robot to a posture corresponding to the detected posture have been researched and developed.

In this regard, a robot system, in which (i) the posture of the user is detected by a user apparatus worn by the user and the posture of the robot is changed to the posture corresponding to the detected posture, (ii) a process corresponding to detection information detected by various sensors provided in the robot is performed by the user apparatus, and (iii) information corresponding to the detection information is provided to the user, is known (see Patent Document <NUM> and Non-Patent Document <NUM>).

In Non-Patent Document <NUM>, haptic assistance functions for a telepresence system are presented and assessed. These assistance functions are based on the augmentation of exchanged position and force data, and they are intended to increase the transparency of the telepresence system while maintaining stability. The authers present the concept and implementation of different assistance functions. Furthermore, the authers show the setup and results of a psychophysical experiment, which was designed to evaluate the effects of the assistance functions on perceived realism. As a result, the position assistance can increase stability and safety without negatively affecting transparency, and the force assistance can even increase the feeling of presence under certain conditions.

Non-Patent Document <NUM> presents a new control strategy based on the Time Domain Passivity Control approach which copes with the active nature of delayed communication channels. Describing the system by means of network elements, the energy of the communication channel can be computed in real time and subsequently dissipated, thus providing stable operation. This is done bilaterally, since the system energy may flow from master to slave and from slave to master. The approach is accompanied with some experiments which validate the method.

<CIT> discloses a control method may be applied to a surgical robot system including a slave robot having a robot arm to which a main surgical tool and an auxiliary surgical tool are coupled, and a master robot having a master manipulator to manipulate the robot arm. The control method includes acquiring data regarding a motion of the master manipulator, predicting a basic motion to be performed by an operator based on the acquired motion data and results of learning a plurality of motions constituting a surgical task, and adjusting the auxiliary surgical tool so as to correspond to the operator basic motion based on the predicted basic motion. The control method allows an operator to perform surgery more comfortably and to move or fix all required surgical tools to or at an optimized surgical position.

In such a robot system, the user can operate the robot by changing their own posture (i.e., by moving) while wearing the user apparatus. However, when the robot is caused to execute a certain task, in the conventional robot system, it is sometimes difficult for a user who is not familiar with operating the robot to cause the robot to execute the task with a high accuracy that a user who is familiar with operating the robot can achieve.

Accordingly, an object of the present invention is to provide a control apparatus, a robot control method, and a robot control system that can assist a user in operating a robot.

A control apparatus according to the first aspect of the present invention comprises a first information acquisition part that acquires first user posture information indicating a posture of a first user operating a robot, a second information acquisition part that acquires pre-change posture information indicating a pre-change posture, which is a posture of the robot before changing the posture of the robot on the basis of the first user posture information, and a determination part that determines, as the posture of the robot, a target posture different from the posture of the first user on the basis of the pre-change posture information and the first user posture information acquired by the first information acquisition part at the time when the robot is in the pre-change posture indicated by the pre-change posture information.

The control apparatus comprises a specification part that specifies, among a plurality of pieces of reference posture information used to change the posture of the robot, subject reference posture information corresponding to the pre-change posture information and the first user posture information acquired by the first information acquisition part at the time when the robot is in the pre-change posture indicated by the pre-change posture information.

The specification part selects, as the subject reference posture information, the reference posture information corresponding to one piece of reference information selected from among the plurality of pieces of reference information in which the pre-change posture information is associated with the reference posture information indicating the posture of the second user differing from the first user acquired at a previous point in time when the robot is in the pre-change posture indicated by the pre-change posture information.

The specification part may select, from among the plurality of pieces of reference information, one piece of reference information having a relatively high probability of changing the posture of the robot to the target posture intended by the first user, on the basis of the first user posture information acquired by the first information acquisition part and the pre-change posture information acquired by the second information acquisition part.

The specification part may estimate, on the basis of the first user posture information and the pre-change posture information, the target posture intended by the first user, and select one piece of reference information having a relatively high probability of changing the posture of the robot to the estimated target posture.

The specification part may specify the subject reference posture information by inputting the pre-change posture information and the first user posture information into a machine-learning model created by using a plurality of pieces of pre-change posture information and the plurality of pieces of reference posture information as teacher data.

The specification part may generate temporary target posture information indicating a rotation angle, an angular velocity, an angular acceleration, and a torque of each of one or more motors of the robot on the basis of the first user posture information acquired by the first information acquisition part, and specify the subject reference posture information on the basis of the generated temporary target posture information and the pre-change posture information.

The control apparatus may further comprises a third information acquisition part that acquires second user posture information indicating the posture of the second user differing from the first user, and a generation part that generates the plurality of pieces of reference information by associating the pre-change posture information with the second user posture information serving as the reference posture information.

The generation part may generate a machine-learning model capable of outputting the reference posture information suitable for the inputted pre-change posture information and the inputted first user posture information, using the pre-change posture information and the second user posture information as teacher data.

The control apparatus may further comprises a fourth information acquisition part that acquires post-change posture information indicating a post-change posture after the robot changes posture on the basis of the second user posture information, and a generation part that generates the plurality of pieces of reference posture information by specifying the second user posture information on the basis of the pre-change posture information and the post-change posture information and associating the pre-change posture information with the specified second user posture information.

The first information acquisition part acquires the first user posture information in association with a time, the second information acquisition part acquires the pre-change posture information in association with a time, and the determination part determines the target posture as the posture of the robot on the basis of the pre-change posture information and the first user posture information associated with the same time.

Claim <NUM> defines a robot control method according to the second aspect of the present invention, the method comprises the computer-implemented steps of acquiring first user posture information indicating a posture of a first user operating a robot, acquiring pre-change posture information indicating a pre-change posture, which is a posture of the robot before changing the posture of the robot on the basis of the first user posture information, selecting, as a subject reference posture information corresponding to the pre-change posture information and the first user posture information, a reference posture information corresponding to one piece of reference information selected from among a plurality of pieces of the reference information in which the pre-change posture information is associated with the reference posture information indicating the posture of a second user differing from the first user acquired at a previous point in time when the robot is in the pre-change posture indicated by the pre-change posture information; and determining, as the posture of the robot, a posture corresponding to the selected subject reference posture information.

A robot control system according to the third aspect of the present invention, the system comprises a robot, a user apparatus that detects information about a first user operating the robot, a robot control apparatus that controls the robot, and a control apparatus as described above capable of communicating with the user apparatus and the robot control apparatus, wherein the user apparatus transmits first user posture information indicating the posture of the first user to the control apparatus, the robot control apparatus controls a posture of the robot so that the robot reaches a posture determined by the control apparatus.

According to the present invention, it is possible to provide a control apparatus capable of assisting a user in operating a robot.

First, an outline of a robot system <NUM> will be described. The robot system <NUM> detects a posture of a user by a user apparatus worn by the user, and changes a posture of a robot to a posture corresponding to the detected posture. Further, the robot system <NUM> causes the user apparatus to perform a process according to detection information detected by various sensors provided in the robots, and provides information according to the detection information to the user.

Specifically, the robot system <NUM> includes a user apparatus (a user apparatus <NUM> in an example described below) for detecting information about a user, a robot (a robot <NUM> in the example described below), and a robot control apparatus (a robot control apparatus <NUM> in the example described below) for controlling the robot. The robot system <NUM> acquires first information from the user apparatus <NUM>, the first information including user posture information indicating the user's posture, which is the posture of the user. The robot system <NUM> changes, on the basis of the acquired first information, a pre-change posture, which is the posture of the robot before changing the posture, to a target posture, which is the posture corresponding to the user's posture indicated by the user posture information included in the acquired first information, and causes the user apparatus <NUM> to perform a process corresponding to the detection information detected by the robot. The robot system <NUM> causes the user apparatus <NUM> to perform the process to provide information corresponding to the detection information to the user.

In the robot system <NUM>, the user can operate the robot by changing their own posture (that is, by moving the user) while wearing the user apparatus <NUM>. The robot system <NUM> is characterized in that even a user who is not familiar with operating the robot can perform an operation equivalent to a user who is familiar with operating the robot.

Hereinafter, for convenience of description, the user who is not familiar with operating the robot provided in the robot system <NUM> will be referred to as a first user, and the user who is familiar with the operation will be referred to as a second user. That is, the first user is, for example, a layman in the operation, and the second user is, for example, an expert in the operation. In the following description, for the sake of convenience, a robot operated by the first user is referred to as a first robot and a robot operated by the second user is referred to as a second robot. It should be noted that the first robot and the second robot may be the same robot or different robots.

In order to solve the problem of the conventional robot system, the robot system <NUM> includes a control apparatus <NUM>, which will be described later. The control apparatus <NUM> acquires the first information including first user posture information indicating the first user's posture, which is the posture of the first user. In addition, the control apparatus <NUM> acquires second information including at least pre-change posture information among (i) the pre-change posture information indicating a pre-change posture, which is the posture of the first robot before changing the posture on the basis of the acquired first information and (ii) the above-described detection information.

The control apparatus <NUM> determines a target posture corresponding to the first user's posture indicated by the first user posture information included in the first information, which is the posture of the first robot, on the basis of (i) the reference information including information in which, for each piece of the second user posture information indicating the second user's posture, which is the posture of the second user U2, the second user posture information is associated with the reference posture information indicating the posture serving as the reference, (ii) the acquired first information, and (iii) the acquired second information. The control apparatus <NUM> can thereby assist the user in operating the robot. The robot control apparatus changes the posture of the first robot to the target posture determined by the control apparatus <NUM>. Hereinafter, the configuration of the robot system <NUM> including the control apparatus <NUM> and the process performed by the control apparatus <NUM> will be described in detail.

Next, a configuration and the outline of the robot system <NUM> will be described.

<FIG> is a diagram illustrating an exemplary configuration of the robot system <NUM> according to an embodiment. The robot system <NUM> includes the user apparatus <NUM>, the robot <NUM>, and a server SB housing a database DB. Further, in the robot system <NUM>, the robot <NUM> includes the robot control apparatus <NUM>. It should be noted that the robot <NUM> may be configured to be connected to the robot control apparatus <NUM> installed outside the robot <NUM> so as to enable communication with each other by wire or radio, instead of the configuration in which the robot control apparatus <NUM> is incorporated therein.

Hereinafter, an example in which the robot control apparatus <NUM> includes the above-mentioned control apparatus <NUM> in the robot system <NUM> will be described. Further, in the robot system <NUM>, the control apparatus <NUM> may be provided with any of the user apparatus <NUM>, the robot <NUM>, and the server SB instead of the robot control apparatus <NUM>.

In the robot system <NUM>, the user apparatus <NUM> and the robot <NUM> are communicably connected to each other via a network N. It should be noted that the user apparatus <NUM> and the robot <NUM> may be communicably connected to each other by wire or radio without using the network N.

The user apparatus <NUM> and the server SB are communicably connected to each other via the network N. It should be noted that the user apparatus <NUM> and the server SB may be communicably connected to each other by wire or radio without using the network N.

The robot <NUM> and the server SB are communicably connected to each other via the network N. It should be noted that the robot <NUM> and the server SB may be communicably connected to each other by wire or radio without the network N.

The network N may be any communication network. The network N is, for example, the Internet, a mobile communication network, a private line communication network, or the like.

The user apparatus <NUM> detects information about the user. The information about the user includes the user posture information indicating the user's posture, which is the posture of the user. It should be noted that the information about the user may include other information in addition to the user posture information. The user apparatus <NUM> according to the embodiment is an apparatus worn by the user.

The user apparatus <NUM> may be an apparatus that is not worn by the user, as long as it is an apparatus capable of detecting the information about the user. In the example shown in <FIG>, the user apparatus <NUM> is worn by a first user U1. Accordingly, the user apparatus <NUM> detects the information about the first user U1 (that is, information including the first user posture information, which is the user posture information indicating the first user's posture, which is the posture of the first user U1). Hereinafter, a case where the user apparatus <NUM> is mounted on the first user U1, as shown in <FIG>, will be described as an example. That is, in the following description, the robot <NUM> is an example of the above-described first robot.

The user apparatus <NUM> includes various sensors for detecting the user's posture, and detects the user's posture by performing motion capturing based on values outputted from these sensors. The user apparatus <NUM> detects the user's posture at a predetermined time interval. The user apparatus <NUM> generates the user information including the user posture information indicating the detected user's posture, and outputs the generated user information to the robot <NUM> via the network N. As a result, the user apparatus <NUM> changes the posture of the robot <NUM> to the target posture, which is the posture corresponding to the user's posture indicated by the user posture information included in the user information, by the process of the robot control apparatus <NUM> that has acquired the user information.

Since the user apparatus <NUM> is worn by the first user U1, the first user's posture, which is the posture of the first user U1, is detected at the predetermined time interval. The user apparatus <NUM> generates the user information including the first user posture information indicating the detected first user's posture, and outputs the generated user information to the robot <NUM> via the network N. As a result, the user apparatus <NUM> changes the posture of the robot <NUM> to the target posture corresponding to the first user's posture indicated by the first user posture information included in the user information by the process of the robot control apparatus <NUM> that has acquired the user information.

The predetermined time interval is, for example, <NUM> milliseconds. The predetermined time interval may be shorter than <NUM> milliseconds or longer than <NUM> milliseconds.

More specifically, the user apparatus <NUM> includes a head-mounted display <NUM>, a data glove <NUM> which is a data glove for the right hand, a data glove <NUM> which is a data glove for the left hand, and a user apparatus control part, which is not shown in the figures.

The head-mounted display <NUM> is mounted on a user's head. The head-mounted display <NUM> includes a display part, which is not shown in the figures, and causes the display part to display an image. The display part is a display panel that covers a part or all of the field of view of the user when the head-mounted display <NUM> is mounted on the head of the user. As a result, the head-mounted display <NUM> allows the user to see the image displayed on the display part. The display panel is a liquid crystal display panel, an organic electroluminescence (EL) display panel, or the like, but may be another display panel. The image is, for example, a moving image, but may be a still image instead.

The head-mounted display <NUM> includes a speaker for outputting sound. Further, the head-mounted display <NUM> includes a sensor for detecting a user's head posture, which is the posture of the user's head. The sensor may be any sensor as long as it is capable of detecting the posture of the user's head. The head-mounted display <NUM> detects the posture of the user's head by means of the sensor in response to a request from the user apparatus control part. The head-mounted display <NUM> outputs information indicating the detected posture of the user's head to the user apparatus control part. The head-mounted display <NUM> may be configured to detect the direction of the user's line of sight instead of the posture of the user's head. In these cases, the head-mounted display <NUM> includes a sensor for detecting the direction of the user's line of sight.

Here, since the head-mounted display <NUM> is mounted on the head of the first user U1, the image displayed on the display part can be displayed to the first user U1. The head-mounted display <NUM> detects the first user's head posture, which is the posture of the head of the first user U1, in response to the request from the user apparatus control part. The head-mounted display <NUM> outputs information indicating the detected first user's head posture to the user apparatus control part.

The data glove <NUM> is mounted on the user's right hand. The data glove <NUM> includes various sensors for detecting a user's right hand finger posture, which is the posture of each finger of the right hand, and a user's right arm posture, which is the posture of the right arm of the user. The various sensors may be any sensors capable of detecting the user's right hand finger posture and the user's right arm posture. The data glove <NUM> detects the user's right hand finger posture and the user's right arm posture in response to the request from the user apparatus control part. The data glove <NUM> outputs information indicating the detected user's right hand finger posture and information indicating the detected user's right arm posture to the user apparatus control part.

It should be noted that the data glove <NUM> may be configured to detect the posture of another part of the user in place of either or both of the user's right hand finger posture and the user's right arm posture, or may be configured to detect the posture of the other part of the user in addition to either one or both of the user's right hand finger posture and the user's right arm posture. In these cases, the data glove <NUM> includes a sensor for detecting the posture of the other part.

The data glove <NUM> includes a pressure device, which is not shown in the figures, for applying a pressure to each of one or more predetermined parts among the user's parts. The one or more predetermined parts may be, for example, fingertips of the user's right hand. Alternatively, the one or more predetermined parts may be other predetermined parts among the user's parts. In response to the request from the user apparatus control part, the pressure device applies a pressure indicated by the request to a part indicated by the request among the above-mentioned predetermined parts.

Here, since the data glove <NUM> is worn on the right hand of the first user U1, the data glove <NUM> detects the first user's right hand finger posture, which is the posture of each finger of the right hand of the first user, and the first user's right arm posture, which is the posture of the right arm of the first user, in response to the request from the user apparatus control part. The data glove <NUM> outputs information indicating the detected first user's right hand finger posture and information indicating the detected first user's right arm posture to the user apparatus control part.

The data glove <NUM> is mounted on the user's left hand. The data glove <NUM> includes various sensors for detecting a user's left hand finger posture, which is the posture of each finger of the left hand, and a user's left arm posture, which is the posture of the left arm of the user. The various sensors may be any sensors capable of detecting the user's left hand finger posture and the user's left arm posture. The data glove <NUM> detects the user's left hand finger posture and the user's left arm posture in response to the request from the user apparatus control part. The data glove <NUM> outputs information indicating the detected user's left hand finger posture and information indicating the detected user's left arm posture to the user apparatus control part.

It should be noted that the data glove <NUM> may be configured to detect the posture of another part of the user in place of either one or both of the user's left hand finger posture and the user's left arm posture, or may be configured to detect the posture of the another part of the user in addition to either one or both of the user's left hand finger posture and the user's left arm posture. In these cases, the data glove <NUM> includes a sensor for detecting the posture of the other part.

The data glove <NUM> includes a pressure device, which is not shown in the figures, for applying a pressure to each of one or more predetermined parts among the user's parts. The one or more predetermined parts may be, for example, fingertips of the user's left hand. Alternatively, the one or more predetermined parts may be other predetermined parts among the user's parts. In response to the request from the user apparatus control part, the pressure device applies a pressure indicated by the request to a part indicated by the request among the above-mentioned predetermined parts.

Here, since the data glove <NUM> is worn on the left hand of the first user U1, the data glove <NUM> detects the first user's left hand finger posture, which is the posture of each finger of the left hand of the first user, and the first user's left arm posture, which is the posture of the left arm of the first user, in response to the request from the user apparatus control part. The data glove <NUM> outputs information indicating the detected the first user's left hand finger posture and information indicating the detected first user's left arm posture to the user apparatus control part.

The user apparatus control part controls the entire user apparatus <NUM>. The user apparatus control part may be built in any of the head-mounted display <NUM>, the data glove <NUM>, and the data glove <NUM>, or may be installed outside each of the head-mounted display <NUM>, the data glove <NUM>, and the data glove <NUM> and connected to each of the head-mounted display <NUM>, the data glove <NUM>, and the data glove <NUM> so as to enable communication with each other by wire or radio.

The user apparatus control part causes the head-mounted display <NUM> to detect the user's head posture, causes the data glove <NUM> to detect the user's right hand finger posture and the user's right arm posture, and causes the data glove <NUM> to detect the user's left hand finger posture and the user's left arm posture at the predetermined time interval. The user apparatus control part acquires information indicating the user's head posture from the head-mounted display <NUM>, acquires information indicating the user's right hand finger posture and information indicating the user's right arm posture from the data glove <NUM>, and acquires information indicating the user's left hand finger posture and information indicating the user's left arm posture from the data glove <NUM>.

Here, in the embodiment, the user's posture is represented by each piece of the information indicating the user's head posture, the information indicating the user's right hand finger posture, the information indicating the user's right arm posture, the information indicating the user's left hand finger posture, and the information indicating the user's left arm posture. The user posture information may include a part of the information indicating the user's head posture, the information indicating the user's right hand finger posture, the information indicating the user's right arm posture, the information indicating the user's left hand finger posture, and the information indicating the user's left arm posture, or may include other information.

The user apparatus control part generates the user information including the user posture information represented by each piece of the acquired information indicating the user's head posture, the acquired information indicating the user's right hand finger posture, the acquired information indicating the user's right arm posture, the acquired information indicating the user's left hand finger posture, and the acquired information indicating the user's left arm posture. The user information may include other information in addition to the user posture information. The user apparatus control part outputs the generated user information to the robot <NUM> via the network N.

Since the user apparatus <NUM> is worn by the first user U1, the first user's head posture is detected by the head-mounted display <NUM>, the first user's right hand finger posture and the first user's right arm posture are detected by the data glove <NUM>, and the first user's left hand finger posture and the first user's left arm posture are detected by the data glove <NUM> at the predetermined time interval. The user apparatus control part acquires the information indicating the first user's head posture from the head-mounted display <NUM>, acquires the information indicating the first user's right hand finger posture and the information indicating the first user's right arm posture from the data glove <NUM>, and acquires the information indicating the first user's left hand finger posture and the information indicating the first user's left arm posture from the data glove <NUM>.

The user apparatus control part generates the user information including the first user posture information represented by each piece of the acquired information indicating the first user's head posture, the acquired information indicating the first user's right hand finger posture, the acquired information indicating the first user's right arm posture, the acquired information indicating the first user's left hand finger posture, and the acquired information indicating the first user's left arm posture. The user apparatus control part outputs the generated user information to the robot <NUM> via the network N. The user information is an example of each piece of the first information, third information, and fifth information.

The user apparatus control part acquires the detection information detected by the robot <NUM> from the robot <NUM> via the network N. The user apparatus <NUM> performs a process according to the acquired detection information. In this way, the user apparatus <NUM> can provide various types of information to at least some of the five senses of the user. In this example, the user apparatus <NUM> provides the various types of information to each of touch, sight, and hearing among the five senses of the user.

The detection information includes visual information including an image captured by an image capturing part provided to the robot <NUM>, tactile information including an output value from a tactile sensor provided to the robot <NUM> (the tactile sensation may include a force sensation but does not need to include the force sensation), and auditory information including an output value from a sound detection part provided to the robot <NUM>. The image may be a moving image or a still image. Hereinafter, a case where the image is a moving image will be described as an example.

It is acceptable for the detection information to include only a part of the visual information, the tactile information, and the auditory information, and the detection information may include other information (such as olfactory information or taste information) in addition to some or all of the visual information, the tactile information, and the auditory information. It is acceptable for the detection information to include only the other information in place of some or all of the visual information, the tactile information, and the auditory information. In these cases, the user apparatus <NUM> provides the various types of information for a sense corresponding to the information included in the detection information among the five senses of the user.

The user apparatus control part causes the display part of the head-mounted display <NUM> to display an image included in the visual information on the basis of the visual information included in the acquired detection information. The user apparatus control part operates the pressure devices of the data glove <NUM> and the data glove <NUM> on the basis of the tactile information included in the detection information, and applies a pressure corresponding to the output value included in the tactile information to the user. The user apparatus control part causes the speaker of the head-mounted display <NUM> to output sound corresponding to the output value included in the auditory information on the basis of the auditory information included in the detection information.

With such a configuration, the user apparatus <NUM> can provide information indicating an object viewed by the robot <NUM> (captured by the image capturing part) to the user's sense of sight as an image included in the visual information, provide information indicating a touch detected by the robot <NUM> while touching an object (detected by the tactile sensor) to the user's sense of touch as a pressure corresponding to an output value included in the tactile information, and provide sound that the robot <NUM> heard (detected by the sound detection part) to the user's sense of hearing as sound corresponding to an output value included in the auditory information. As a result, the user can cause the robot <NUM> to perform the next action desired by the user on the basis of the various types of information provided to at least some of the five senses of the user.

Here, since the user apparatus <NUM> is worn by the first user U1, it is possible to provide the information indicating the object viewed by the robot <NUM> (captured by the image capturing part) to the sense of sight of the first user U1 as the image included in the visual information, provide the information indicating the touch detected by the robot <NUM> while touching the object (detected by the tactile sensor) to the sense of touch of the first user U1 as the pressure corresponding to the output value included in the tactile information, and provide the sound that the robot <NUM> heard (detected by the sound detection part) to the sense of hearing of the first user U1 as the sound corresponding to the output value included in the auditory information. As a result, the first user U1 can cause the robot <NUM> to perform the next action desired by the first user U1 on the basis of the various types of information provided to at least some of the five senses of the first user U1.

The robot <NUM> is, for example, a dual-arm robot. The robot <NUM> includes, as movable parts, a head part <NUM> corresponding to a head part of the user (in this example, the first user U1) on which the user apparatus <NUM> is mounted, a right arm part <NUM> which is a robot arm (manipulator) corresponding to the right arm of the user, and a left arm part <NUM> which is a robot arm (manipulator) corresponding to the left arm of the user. Further, the robot <NUM> also includes a support part <NUM> for supporting each of the head part <NUM>, the right arm part <NUM>, and the left arm part <NUM>. It should be noted that the robot <NUM> may have a configuration without the head part <NUM>, may be a single-arm robot having only one robot arm (manipulator), may be a multi-arm robot having three or more robot arms, or may have another configuration.

The head part <NUM> is supported by the support part <NUM> for rotation about each of one or more predetermined axes. In the following, as an example, a case will be described in which the head part <NUM> is supported by the support part <NUM> so as to be rotatable about each of two predetermined axes. The head part <NUM> includes an actuator for rotating the head part <NUM> about each of the two axes. Here, the posture of the head part <NUM> is determined by each of a rotation angle, an angular velocity, an angular acceleration, and a torque of one or more actuators included in the head part <NUM>. The head part <NUM> operates the actuator in response to a request from the robot control apparatus <NUM> to change the posture of the head part <NUM>. In response to the request, the head part <NUM> outputs, to the robot control apparatus <NUM>, information indicating each of the rotation angle, the angular velocity, the angular acceleration, and the torque of the one or more actuators as information indicating the posture of the head part <NUM>.

Further, the head part <NUM> includes the image capturing part described above. The image capturing part is, for example, a stereo camera equipped with a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS), or the like, which is an imaging element for converting condensed light into electric signals. Since the image capturing part is provided on the head part <NUM>, a range in which the image capturing part can capture images varies depending on the posture of the head part <NUM>. The image capturing part captures a moving image of the range, but may instead capture a still image of the range. The image capturing part outputs the image obtained by capturing the image of the range as the visual information to the robot control apparatus <NUM>. The image capturing part may be a monocular camera instead of the stereo camera, or may be a compound eye camera with three or more lenses.

The head part <NUM> includes the sound detection part of the robot <NUM>. The sound detection part is, for example, a sensor for detecting sounds around the head part <NUM>. The head part <NUM> outputs information including the output value indicating the sounds detected by the sound detection part to the robot control apparatus <NUM> as the above-mentioned auditory information.

The right arm part <NUM> is a robot arm having a plurality of joints, and is a robot arm having, as an end effector, a robot hand corresponding to the right hand of the user (in this case, the first user U1) wearing the user apparatus <NUM>. It should be noted that the number of joints of the right arm part <NUM> may be any number as long as the posture of the right arm part <NUM> can be changed to a posture corresponding to each of the user's right arm posture and the user's right hand finger posture. Each joint included in the right arm part <NUM> includes an actuator for rotating the joint.

Here, the posture of the right arm part <NUM> is determined by each of the rotation angle, the angular velocity, the angular acceleration, and the torque of each of the plurality of actuators included in the right arm part <NUM>. The right arm part <NUM> operates the actuator in response to the request from the robot control apparatus <NUM> to change the posture of the right arm part <NUM>. In response to the request, the right arm part <NUM> outputs, to the robot control apparatus <NUM>, the information indicating each of the rotation angle, the angular velocity, the angular acceleration, and the torque of the one or more actuators as the information indicating the posture of the right arm part <NUM>.

A tactile sensor, which is not shown in the figures, is provided at the tip of each finger of the robot hand of the right arm part <NUM>. The finger part is a part corresponding to each finger of the right hand of the user (in this example, the first user U1) wearing the user apparatus <NUM> among the parts of the robot hand. The tactile sensor outputs the information including the output value indicating the detected pressure to the robot control apparatus <NUM> as the tactile information.

The left arm part <NUM> is a robot arm having a plurality of joints, and is a robot arm having, as an end effector, a robot hand corresponding to the left hand of the user (in this case, the first user U1) wearing the user apparatus <NUM>. It should be noted that the number of joints of the left arm part <NUM> may be any number as long as the posture of the left arm part <NUM> can be changed to a posture corresponding to each of the user's left arm posture and the user's left hand finger posture. Each joint included in the left arm part <NUM> includes an actuator for rotating the joint.

Here, the posture of the left arm part <NUM> is determined by each of the rotation angle, the angular velocity, the angular acceleration, and the torque of each of the plurality of actuators included in the left arm part <NUM>. The left arm part <NUM> operates the actuator in response to the request from the robot control apparatus <NUM> to change the posture of the left arm part <NUM>. In response to the request, the left arm part <NUM> outputs, to the robot control apparatus <NUM>, the information indicating each of the rotation angle, the angular velocity, the angular acceleration, and the torque of the one or more actuators as information indicating the posture of the left arm part <NUM>.

A tactile sensor, which is not shown in the figures, is provided at the tip of each finger of the robot hand of the left arm part <NUM>. The finger part is a part corresponding to each finger of the left hand of the user (in this example, the first user U1) wearing the user apparatus <NUM> among the parts of the robot hand. The tactile sensor outputs the information including the output values indicating the detected pressures to the robot control apparatus <NUM> as the tactile information.

Here, the posture of the robot <NUM> is represented by the posture of the head part <NUM>, the posture of the right arm part <NUM>, and the posture of the left arm part <NUM>. That is, a change of the posture of the robot <NUM> means that at least a part of the posture of the head part <NUM>, the posture of the right arm part <NUM>, and the posture of the left arm part <NUM> has changed.

The robot control apparatus <NUM> acquires the user information from the user apparatus <NUM> via the network N. Further, the robot control apparatus <NUM> acquires pre-change robot information from the robot <NUM>. The pre-change robot information is information including pre-change posture information indicating the pre-change posture, which is the posture of the robot <NUM> before the robot control apparatus <NUM> changes the posture on the basis of the acquired user information. The pre-change robot information may include other information in addition to the pre-change posture information. Hereinafter, a case where the detection information detected by the robot <NUM> is included in the pre-change robot information together with the pre-change posture information will be described as an example. As described above, the detection information is information including each of the visual information, the tactile information, and the auditory information.

The robot control apparatus <NUM> acquires, from the robot <NUM>, (i) the pre-change posture information including each piece of information indicating the posture of the head part <NUM> before the robot control apparatus <NUM> changes the posture on the basis of the acquired user information, information indicating the posture of the right arm part <NUM> before the change, and information indicating the posture of the left arm part <NUM> before the change, and (ii) the detection information including each piece of the visual information before the change, the tactile information before the change, and the auditory information before the change. Here, the robot control apparatus <NUM> acquires three pieces of information from the head part <NUM>, namely the information indicating the posture of the head part <NUM> before the change, the visual information before the change, and the auditory information before the change. Further, the robot control apparatus <NUM> acquires two pieces of information from the right arm part <NUM>, namely the information indicating the posture of the right arm part <NUM> before the change and the tactile information detected by the right arm part <NUM> which is the tactile information before the change. Furthermore, the robot control apparatus <NUM> acquires two pieces of information from the left arm part <NUM>, namely the information indicating the posture of the left arm part <NUM> before the change and tactile information detected by the left arm part <NUM> which is the tactile information before the change. It should be noted that the pre-change robot information is an example of the second information. The detection information is an example of each of first detection information, third detection information, and fifth detection information.

The robot control apparatus <NUM> reads the reference information stored in advance in the database DB of the server SB from the server SB via the network N. The reference information includes information in which, for each piece of the second user posture information indicating the second user's posture, which is the posture of the second user U2, the second user posture information is associated with the reference posture information indicating a posture serving as the reference. The posture indicated by the reference posture information associated with a certain piece of second user posture information is a posture corresponding to the second user's posture indicated by the second user posture information among the postures of the second robot which is a robot operated by the second user. The second robot may be the robot <NUM> or a robot different from the robot <NUM>. Hereinafter, a case where the second robot is the robot <NUM> will be described as an example.

The reference information is history information in which the second user posture information obtained when the second user U2, who is a user more familiar with the operation of the robot <NUM> that is the operation of the robot system <NUM> than the first user U1, previously operated the robot <NUM> is associated with the reference posture information, which is information indicating the posture of the robot <NUM> operated by the second user U2, which indicates the posture serving as the reference. It should be noted that the reference information may include other information in addition to information in which, for each piece of the second user posture information, the second user posture information is associated with the reference posture information. Further, the second user may be the same person as the first user U1.

On the basis of the user information corresponding to the first user acquired from the user apparatus <NUM> mounted on the first user U1, the pre-change robot information acquired from the robot <NUM>, and the reference information read from the database DB of the server SB, the robot control apparatus <NUM> determines a target posture, which is the posture of the robot <NUM>, corresponding to the user's posture indicated by the user posture information included in the user information. Then, the robot control apparatus <NUM> operates the robot <NUM> and changes the posture of the robot <NUM> to the determined target posture. As a result, the robot control apparatus <NUM> can assist the first user U1 in operating the robot <NUM>.

In this example, since the user apparatus <NUM> is worn by the first user U1, the robot control apparatus <NUM> determines the target posture corresponding to the first user's posture, which is the posture of the robot <NUM>, indicated by the first user posture information included in the user information on the basis of (i) the reference information read from the database DB of the server SB, (ii) the user information acquired from the user apparatus <NUM>, and (iii) the pre-change robot information acquired from the robot <NUM>. Then, the robot control apparatus <NUM> operates the robot <NUM> and changes the posture of the robot <NUM> to the determined target posture. As a result, the robot control apparatus <NUM> can assist the first user U1 in operating the robot <NUM>.

The server SB is a storage device realized by an information processing device such as a desktop personal computer (PC), a workstation, or the like. The server SB is communicably connected to the user apparatus <NUM> and the robot <NUM> (that is, the robot control apparatus <NUM>) via the network N. As described above, the server SB houses the database DB in advance. In response to the request from the robot control apparatus <NUM>, the database DB of the server SB outputs the reference information to the robot control apparatus <NUM>.

Hereinafter, the functional configuration of the robot control apparatus <NUM> will be described while referencing <FIG> is a diagram illustrating an exemplary functional configuration of the robot control apparatus <NUM>.

The robot control apparatus <NUM> includes (i) a storage part <NUM> such as a hard disk drive (HDD), a solid state drive (SSD), or the like, (ii) a communication part <NUM> which is a communication port including a digital input/output port such as a universal serial bus (USB), an Ethernet (registered trademark) port, or the like, (iii) a robot control part <NUM>, and (iv) a control apparatus <NUM>. As described above, the robot control apparatus <NUM> may be separated from the control apparatus <NUM>.

The robot control part <NUM> controls the entire robot control apparatus <NUM>. Further, the robot control part <NUM> operates the robot <NUM> to change the posture of the robot <NUM> to the target posture determined by the control apparatus <NUM>. The robot control part <NUM> is realized by, for example, a central processing unit (CPU), which is not shown in the figures, executing various programs stored in the storage part <NUM>. The robot control part <NUM> may be a hardware-functional unit such as large scale integration (LSI) or an application specific integrated circuit (ASIC), or the like.

The control apparatus <NUM> includes a user information acquisition part <NUM>, a pre-change robot information acquisition part <NUM>, a post-change robot information acquisition part <NUM>, a specification part <NUM>, a determination part <NUM>, a generation part <NUM>, and a storage control part <NUM>. These functional parts included in the control apparatus <NUM> are realized by, for example, a CPU, which is not shown in the figures, executing various programs stored in the storage part <NUM>. Some or all of the functional parts may be hardware-functional units such as LSI and an ASIC.

The user information acquisition part <NUM> acquires the user information from the user apparatus <NUM> via the network N. The user information acquisition part <NUM> is an example of a first information acquisition part and a third information acquisition part.

The pre-change robot information acquisition part <NUM> acquires the pre-change robot information from the robot <NUM>. The pre-change robot information acquisition part <NUM> is an example of a second information acquisition part.

The post-change robot information acquisition part <NUM> acquires the post-change robot information from the robot <NUM>. The post-change robot information is information including post-change information which is the posture of the robot <NUM> after the posture of the robot <NUM> is changed to a posture corresponding to the user's posture indicated by the user posture information included in the user information acquired by the user information acquisition part <NUM>. The post-change robot information is information used by the generation part <NUM>, which will be described later, to generate the above-described reference information. The post-change robot information acquisition part <NUM> is an example of a fourth information acquisition part. The post-change robot information is an example of fourth information and sixth information.

The specification part <NUM> reads the reference information from the database DB of the server SB via the network N. The specification part <NUM> specifies, as the subject reference posture information, the reference posture information corresponding to the user posture information included in the user information acquired by the user information acquisition part <NUM> and the pre-change posture information included in the pre-change robot information acquired by the pre-change robot information acquisition part <NUM> from among the read reference information.

The determination part <NUM> determines a target posture corresponding to the user's posture indicated by the user posture information included in the user information acquired by the user information acquisition part <NUM> on the basis of the subject reference posture information specified by the specification part <NUM>.

The generation part <NUM> generates the reference information to be stored in the database DB on the basis of the user information acquired by the user information acquisition part <NUM> and the post-change robot information acquired by the post-change robot information acquisition part <NUM>.

The storage control part <NUM> stores the reference information generated by the generation part <NUM> in the database DB.

Hereinafter, the process of the robot control apparatus <NUM> operating the robot <NUM> will be described while referencing <FIG> is a diagram illustrating an example of a process of the robot control apparatus <NUM> operating the robot <NUM>. In the following, a case will be described in which the robot control apparatus <NUM> receives a start operation for starting the operation of the robot <NUM> by the user (in this case, the first user U1) wearing the user apparatus <NUM> at a timing before step S110 is executed.

The specification part <NUM> reads the reference information from the database DB of the server SB (step S110). Here, the database DB may be configured to store a plurality of pieces of the reference information. In this case, identification information required for the specification part <NUM> to read the reference information desired by the user is associated with each piece of the reference information. The identification information is, for example, information including at least one of user identification information indicating the second user, proficiency information indicating the second user's proficiency in operating the robot <NUM>, and task information indicating the type of task to be executed by the robot <NUM>. It should be noted that the identification information may include other information instead of some or all of the user identification information, the proficiency information, and the task information, and may include other information in addition to some or all of the user identification information, the proficiency information, and the task information.

The specification part <NUM> receives the identification information from the user in advance at a timing before step S110 is executed. In step S110, the specification part <NUM> reads the reference information associated with the received identification information from the database DB. The specification part <NUM> may be configured to read the reference information associated with the identification information most similar to the received identification information from the database DB on the basis of a machine-learning algorithm. In this case, the machine-learning algorithm needs to learn combinations of the identification information and the reference information corresponding to the identification information. The machine-learning algorithm may be a known algorithm such as deep learning, or may be an algorithm to be developed.

Subsequently, the robot control part <NUM> and the control apparatus <NUM> repeat the processes from step S130 to step S180 at a predetermined time interval (step S120).

The user information acquisition part <NUM> acquires the user information from the user apparatus <NUM> via the network N (step S130). Then, the pre-change robot information acquisition part <NUM> acquires the pre-change robot information from the robot <NUM> (step S140). It should be noted that, in the robot control apparatus <NUM>, the processes of step S130 and step S140 may be performed in reverse order or may be performed in parallel.

Next, the specification part <NUM> specifies, as the subject reference posture information, the reference posture information corresponding to (i) the first user posture information included in the user information acquired by the user information acquisition part <NUM> in step S130 and (ii) the pre-change posture information included in the pre-change robot information acquired by the pre-change robot information acquisition part <NUM> in step S140 from among the reference information read from the database DB in step S110 (step S150). Here, the process of step S150 will be described.

The specification part <NUM> converts the first user posture information included in the user information acquired from the user information acquisition part <NUM> in step S130 into temporary target posture information, which is information represented by the rotation angle, the angular velocity, the angular acceleration, and the torque of each of one or more motors included in the robot <NUM>, on the basis of inverse kinematics. The specification part <NUM> uses (as input parameters) the temporary target posture information obtained by converting the first user posture information and the pre-change posture information included in the pre-change robot information acquired from the pre-change robot information acquisition part <NUM> in step S140, and specifies, as the subject reference posture information, the reference posture information most likely to be the reference posture information corresponding to the temporary target posture information and the pre-change posture information from among the reference information read from the database DB in step S <NUM>, on the basis of the machine-learning algorithm.

In this case, the machine-learning algorithm needs to learn combinations of the temporary target posture information, the pre-change posture information, and the reference posture information corresponding to the temporary target posture information and the pre-change posture information. The machine-learning algorithm may be a known algorithm such as deep learning, or may be an algorithm to be developed.

It should be noted that the reference information is information including information in which the second user posture information is associated with the reference posture information for each piece of the second user posture information, but may instead be information including the reference posture information and not including the second user posture information. In this case, the specification part <NUM> uses the temporary target posture information (as an input parameter) to specify, as the subject reference posture information, the reference posture information most likely to be the reference posture information corresponding to the temporary target posture information from among the reference information read out from the database DB in step S <NUM>, on the basis of the machine-learning algorithm. In addition, the reference information may be information in which the second user posture information, the reference posture information, and other information are associated with each other for each piece of the second user posture information.

In this case, the specification part <NUM> uses the temporary target posture information, the pre-change posture information, and the other information (as the input parameters), and specifies, as the subject reference posture information, the reference posture information most likely to be the reference posture information corresponding to the temporary target posture information from among the reference information read out from the database DB in step S110, on the basis of the machine-learning algorithm. Such other information is, for example, second detection information that is the detection information detected by the second robot (in this example, the robot <NUM>).

After the process of step S150 is performed, the determination part <NUM> determines the target posture corresponding to the posture indicated by the first user posture information acquired by the user information acquisition part <NUM> in step S130, on the basis of the temporary target posture information converted by the specification part <NUM> in step S150 and the subject reference posture information specified by the specification part <NUM> in step S150 (step S160). The target posture is a desired posture with which the posture of the robot <NUM> is desired to coincide if the posture of the first user U1 is the user's posture indicated by the first user posture information. Here, the process of step S160 will be described.

As an example, the determination part <NUM> calculates a difference between the temporary target posture information converted by the specification part <NUM> in step S150 and the subject reference posture information specified by the specification part <NUM> in step S150. Here, the temporary target posture information is represented by a vector having, as components, a plurality of rotation angles, a plurality of angular velocities, a plurality of angular accelerations, and a plurality of torques, each of which represents the posture indicated by the temporary target posture information. Further, the subject reference posture information is represented by a vector having, as components, a plurality of rotation angles, a plurality of angular velocities, a plurality of angular accelerations, and a plurality of torques, each of which represents the posture indicated by the subject reference posture information. The determination part <NUM> calculates the difference vector of these vectors as a temporary correction amount used for calculating a correction amount for correcting the temporary target posture information.

The determination part <NUM> uses the calculated temporary correction amount and the temporary target posture information to calculate, on the basis of the machine-learning algorithm, the correction amount most likely to be the correction amount for correcting the temporary target posture information. In this case, the machine-learning algorithm needs to learn combinations of the temporary target posture information, the temporary correction amount, and the correction amount corresponding to the temporary target posture information and the temporary correction amount. The machine-learning algorithm may be a known algorithm such as deep learning, or may be an algorithm to be developed. The determination part <NUM> corrects the temporary target posture information by adding the calculated correction amount to the temporary target posture information, and determines the posture indicated by the corrected temporary target posture information as the target posture. Here, the correction amount is a correction amount for reducing the difference between the temporary target posture information and the subject reference posture information.

It should be noted that the determination part <NUM> may calculate the correction amount most likely to be the correction amount for correcting the temporary target posture information on the basis of the machine-learning algorithm using the temporary target posture information converted by the specification part <NUM> in step S150 and the subject reference posture information specified by the specification part <NUM> in step S150, without calculating the difference vector. The machine-learning algorithm needs to learn combinations of the temporary target posture information, the subject reference posture information, and the correction amount corresponding to the temporary target posture information and the subject reference posture information.

After the process of step S160 is performed, the robot control part <NUM> operates the robot <NUM>, and changes the posture of the robot <NUM> to the target posture on the basis of the information indicating the target posture determined by the determination part <NUM> in step S160 (step S170). Subsequently, the robot control part <NUM> determines whether or not a termination operation for terminating the operation of the robot <NUM> has been received from the user (step S180). If it is determined that the termination operation has been received from the user (step S180-YES), the robot control part <NUM> terminates the process. On the other hand, if the robot control part <NUM> determines that the termination operation has not been received from the user, the step is transferred to step S130, and the user information acquisition part <NUM> acquires the user information from the user apparatus <NUM> via the network N.

As described above, the robot control apparatus <NUM> acquires the first information (the user information in the embodiment), and acquires the second information (the pre-change robot information in the embodiment) including at least the pre-change posture information from among the pre-change posture information indicating the posture before change and the first detection information. Then, the robot control apparatus <NUM> determines a target posture corresponding to the first user's posture indicated by the first user posture information included in the first information, which is the posture of the first robot (the robot <NUM> in this example) on the basis of (i) the reference information including information in which, for each piece of the second user posture information indicating the second user's posture, which is the posture of the second user, the second user posture information is associated with the reference posture information indicating the posture serving as the reference, (ii) the acquired first information, and (iii) the acquired second information. Accordingly, the robot control apparatus <NUM> can assist the first user who is not familiar with the operation of the robot <NUM> so that the first user can perform an operation equivalent to the preferable operation corresponding to the reference information.

The robot system <NUM> also performs the process of generating the reference information described above. Therefore, the processing will be described below. In the following explanation, the user apparatus <NUM> is attached to a third user. The third user is, for example, a user who can perform a preferable operation as a reference. It should be noted that the third user may be the same person as the second user or the first user U1. Further, the robot system <NUM> may have a configuration in which this process is not performed. In this case, the reference information is stored in the database DB of the server SB by another information processing apparatus. Further, in this instance, the robot control apparatus <NUM> (or the control apparatus <NUM>) does not include a generation part <NUM>, which will be described in detail below.

The robot system <NUM> does not need to perform the process of the flowchart shown in <FIG>. In this case, the reference information stored in the database DB of the server SB is used by another information processing apparatus. In this instance, the robot control apparatus <NUM> (or the control apparatus <NUM>) does not include the specification part <NUM> and the determination part <NUM>.

Hereinafter, the process of generating the reference information by the robot control apparatus <NUM> will be described while referencing <FIG> is a diagram illustrating an example of a process of the robot control apparatus <NUM> generating the reference information. The flowchart shown in <FIG> shows the process performed by the robot control apparatus <NUM> between the time when the third user starts operating the third robot and the time when the third user terminates the operation of the third robot. However, in the flowchart, the process relating to the operation of the third robot by the third user is omitted. The third robot is a robot operated by the third user. Hereinafter, a case where the third robot is the robot <NUM> will be described as an example. It should be noted that the third robot may be another robot instead of the robot <NUM>.

The robot control part <NUM> and the control apparatus <NUM> repeat the processes from step S220 to step S260 at a predetermined time interval from the time when the third user starts operating the robot <NUM> (step S210).

The user information acquisition part <NUM> acquires the user information, which is information including the third user posture information indicating the third user's posture, which is the posture of the third user, from the user apparatus <NUM> via the network N (step S220). Subsequently, the post-change robot information acquisition part <NUM> acquires the post-change robot information from the robot <NUM> (step S230). In the robot control apparatus <NUM>, the processes of step S220 and step S230 may be performed in reverse order or may be performed in parallel. The robot control apparatus <NUM> may be configured to acquire the post-change robot information from the robot <NUM> and acquire the user information including the user posture information indicating the user's posture corresponding to the posture indicated by the post-change posture information included in the post-change robot information.

Next, on the basis of the third user information acquired by the user information acquisition part <NUM> in step S220 and the post-change robot information acquired by the post-change robot information acquisition part <NUM> in step S230, the generation part <NUM> generates the reference information including information in which the third user posture information is associated with the reference posture information indicating the posture serving as the reference (step S240). Here, the process of step S240 will be described.

The generation part <NUM> generates the reference information, on the basis of the machine-learning algorithm, using the user information acquired by the user information acquisition part <NUM> in step S220 and the post-change robot information acquired by the post-change robot information acquisition part <NUM> in step S230. Specifically, using the third user posture information included in the user information and the post-change posture information included in the post-change robot information, the generation part <NUM> calculates, as the reference posture information indicating the posture serving as the reference and corresponding to the third user's posture, the posture most likely to be the posture corresponding to the third user's posture indicated by the third user posture information, on the basis of the machine-learning algorithm. The generation part <NUM> generates the reference information including information in which the calculated reference posture information is associated with the third user posture information. Here, the machine-learning algorithm needs to learn combinations of the third user posture information, the post-change posture information, and the reference posture information. The machine-learning algorithm may be a known algorithm such as deep learning, or may be an algorithm to be developed.

It should be noted that the generation part <NUM> may specify, as the reference posture information, the post-change posture information included in the post-change robot information acquired by the post-change robot information acquisition part <NUM> in step S230, and generate the reference information including information in which the specified reference posture information is associated with the third user posture information included in the user information acquired by the user information acquisition part <NUM> in step S220.

Further, in a case where the previously generated reference information can be acquired from the database DB of the server SB, the generation part <NUM> may acquire the reference information, and calculate, on the basis of the machine-learning algorithms, the posture most likely to be the posture corresponding to the third user posture information included in the user information using (i) the acquired reference information, (ii) the user information acquired by the user information acquisition part, and (iii) the post-change robot information acquired by the post-change robot information acquisition part. In this instance, the generation part <NUM> specifies the information indicating the calculated posture as the reference posture information to be associated with the third user posture information, and generates the reference information including information in which the specified reference posture information is associated with the third user posture information.

After the process of step S240 is performed, the storage control part <NUM> stores the reference information generated by the generation part <NUM> in step S240 in the database DB of the server SB via the network N (step S250). Subsequently, the robot control part <NUM> determines whether or not a termination operation for terminating the operation of the robot <NUM> has been received from the user (the third user in this example) wearing the user apparatus <NUM> (step S260). If it is determined that the termination operation has been received (step S260-YES), the robot control part <NUM> terminates the process. On the other hand, if the robot control part <NUM> determines that the termination operation has not been received, the step is transferred to step S220, and the user information acquisition part <NUM> acquires the user information from the user apparatus <NUM> via the network N.

Here, the processes from step S210 to step S260 described above will be briefly summarized. The processes of step S220 and step S230 in the flowchart shown in <FIG> is, in other words, a process of acquiring teacher data for the control apparatus <NUM> to generate the reference information using the machine-learning algorithm. The processes from step S240 to step S250 are processes for generating the reference information as a model generated by the machine-learning algorithm using the acquired teacher data.

Conventionally, in order to acquire the teacher data, a pilot (preliminary) robot system <NUM> for acquiring the teacher data has to be operated. However, in the robot system <NUM>, since the control apparatus <NUM> performs the processes of step S220 to step S230, the control apparatus <NUM> can acquire the teacher data without requiring the operation of the pilot robot system <NUM>, and can acquire the teacher data while the robot <NUM> is performing a non-pilot task (for example, an actual task).

The control apparatus <NUM> is not configured to store the user information and the post-change robot information acquired as the teacher data in the flowchart shown in <FIG> in the server SB, but may instead be configured to store the user information and the post-change robot information in the server SB. As a result, the control apparatus <NUM> can construct big data about the teacher data relating to the operation of the robot <NUM>. The reference information generated on the basis of the big data constructed in this way can be used to assist the operation of the robot by the user, and also can be used as information for the robot control apparatus <NUM> to automatically operate the robot <NUM> without requiring the operation by the user.

When the robot control apparatus <NUM> is enabled to automatically operate the robot <NUM> for executing a certain task, in the robot system <NUM>, the user operates the robot <NUM> to cause the robot <NUM> to execute a task different from this certain task and acquires the teacher data required for the control apparatus <NUM> to automatically cause the robot <NUM> to execute this certain task. By repeating such a process, the robot system <NUM> eventually can automatically cause the robot <NUM> to execute almost all of the tasks performed by the user.

It should be noted that the robot system <NUM> may include a plurality of user apparatuses <NUM> and robots <NUM> corresponding to the respective user apparatuses <NUM>. That is, in the robot system <NUM>, a portion of the robots <NUM> corresponding to each of the plurality of user apparatuses <NUM> can be made to automatically execute a predetermined task (which may be different for each robot <NUM>), a robot <NUM> other than this portion of the robots <NUM> corresponding to each of the plurality of user apparatuses <NUM> can be operated by the user, and a robot <NUM> that does not perform a desired action among this portion of the robots <NUM> can be assisted by the robot <NUM> operated by the user.

The reference information generated on the basis of the constructed big data may be used to improve the action of the robot <NUM> operated by the user to an action suitable for the task being executed by the user. In this case, in the robot system <NUM>, the robot control apparatus <NUM> (that is, the control apparatus <NUM>) improves (changes), on the basis of the reference information, the action of the robot <NUM> in accordance with the operation of the user to an action suitable for the task. This enables the robot control apparatus <NUM> to make the robot <NUM> operate appropriately, even if an unskilled user operates the robot.

The reference information generated on the basis of the constructed big data may be used to complement at least a portion of the body functions of the robot <NUM> operated by the user. In this instance, the robot control apparatus <NUM> (that is, the control apparatus <NUM>) complements at least the portion of the body functions of the robots <NUM> operated by the user on the basis of the reference information. In this case, for example, the robot control apparatus <NUM> can operate a plurality of robots (some or all of which may be the robot <NUM>, but do not need to be the robot <NUM>) by the operation of a single user. At this time, the robot control apparatus <NUM> may cause the plurality of robots to operate substantially simultaneously by the operation, or may cause some or all of the plurality of robots to operate at different timings.

In this instance, the robot control apparatus <NUM> accepts login to one robot <NUM> from a plurality of regions, and allows users who have logged in to operate the robot <NUM>. As a result, users in different regions can operate the robot <NUM> for a long time without a break by utilizing the time difference. Consequently, the robot control apparatus <NUM> can increase the efficiency of work of the robot <NUM>.

Though the explanation is omitted in the above, the various types of information (the pre-change posture information, visual information, tactile information, and auditory information) included in the pre-change robot information acquired by the control apparatus <NUM> must be synchronized with each other. Synchronization of the various types of information may be performed in the user apparatus <NUM>, in the robot control apparatus <NUM>, or in other device connected to the network N. The synchronization of the various types of information is realized by, for example, a time stamp. It should be noted that the synchronization may be realized by a method other than the time stamp.

The control apparatus <NUM> may acquire the user information not including the user posture information as the teacher data, or may acquire the post-change robot information not including the post-change posture information as the teacher data. In this case, the user information includes, for example, information detected by various sensors (for example, an image capturing part, a tactile sensor, a sound detection part, etc.) provided in the user apparatus <NUM> and synchronized by the time stamp or the like. In this case, the post-change robot information includes, for example, the detection information synchronized by the time stamp or the like. That is, the control apparatus <NUM> constructs the big data of the teacher data not including at least one piece of the user posture information or the post-change posture information. In this instance, the control apparatus <NUM> may or may not perform the process of the flowchart shown in <FIG> on the basis of, for example, the reference information generated by another device.

The robot system <NUM> described above is an example of telexistence including the control apparatus <NUM>. Here, the detailed description of telexistence is omitted because it is described in <CIT> and the above-mentioned Non-Patent Document <NUM>.

The big data constructed in the robot system <NUM> may be a set of the user information and the post-change robot information for each execution in a case where one user repeatedly causes the robot <NUM> to execute a task many times, or may be a set of the user information and the post-change robot information for each user in a case where each of a plurality of users causes the robot <NUM> to execute the task.

As described above, the robot control apparatus <NUM> (or the control apparatus <NUM>) acquires the third information (in this example, the user information), and acquires fourth information (in this example, the post-change robot information) including at least the post-change posture information from among (i) the post-change posture information indicating the post-change posture, which is the posture of the third robot after the posture of the third robot (in this example, the robot <NUM>) has changed to the target posture corresponding to the posture indicated by the third user posture information included in the acquired third information, and (ii) the third detection information (in this example, the detection information). On the basis of the acquired third information and the acquired fourth information, the robot control apparatus <NUM> (or the control apparatus <NUM>) generates, for each piece of the third user posture information included in the third information, the reference information including information in which the third user posture information is associated with the reference posture information indicating the posture serving as the reference. Consequently, the robot control apparatus <NUM> can assist the user in operating the robot on the basis of the generated reference information.

In a case where the robot control apparatus <NUM> and the control apparatus <NUM> are separate entities, the control apparatus <NUM> is communicably connected to each of the user apparatus <NUM>, the robot <NUM>, the robot control apparatus <NUM>, and the server SB. Then, the control apparatus <NUM> outputs the information indicating the target posture determined by the determination part <NUM> to the robot control part <NUM> provided in the robot control apparatus <NUM>, and outputs the reference information generated by the generation part <NUM> to the database DB of the server SB to store the reference information. In this instance, the control apparatus <NUM> may be provided in the user apparatus <NUM>, may be provided in the robot <NUM>, or may be provided in the server SB.

In the above explanation, the robot control apparatus <NUM> includes the control apparatus <NUM>, but the server SB may include the control apparatus <NUM>. <FIG> is a diagram for explaining procedures for generating the reference posture information in a case where the server SB includes the control apparatus <NUM>. <FIG> is a diagram for explaining procedures for controlling the robot <NUM> in a case where the server SB includes the control apparatus <NUM>. The configuration and operation of the control apparatus <NUM> in the following descriptions may be applied in the case where the robot control apparatus <NUM> includes the control apparatus <NUM>.

<FIG> shows a process in which the server SB generates the reference information on the basis of the second user posture information, which is the reference posture information indicating the posture of the second user (the user indicated by the reference numeral U2 in <FIG>) who is familiar with the operation of the robot <NUM>. The server SB periodically acquires the posture information indicating the posture of the robot <NUM> via the robot control apparatus <NUM>.

After the user information acquisition part <NUM> of the server SB acquires the second user posture information via the user apparatus <NUM> (A1), the pre-change robot information acquisition part <NUM> acquires the most recent posture information of the robot <NUM> at the time when the user information acquisition part <NUM> acquires the second user posture information as the pre-change posture information (A2).

Since the second user U2 is familiar with the operation of the robot <NUM>, it is considered that the operation performed by the second user U2 for the robot <NUM> with the posture indicated by the pre-change posture information is a desirable operation. Accordingly, the generation part <NUM> generates a plurality of pieces of the reference information by associating the pre-change posture information acquired by the pre-change robot information acquisition part <NUM> with the second user posture information as the reference posture information acquired by the user information acquisition part <NUM> (A3).

Incidentally, just because the robots <NUM> are in the same posture, the posture that the user wants the robots <NUM> to take next is not necessarily the same. For example, even if the robots <NUM> are in the same posture, there may be a robot <NUM> being operated to grasp an object, or a robot <NUM> being operated to tilt the object. Therefore, the generation part <NUM> may generate a plurality of pieces of the reference information by associating (i) the plurality of pieces of the pre-change posture information acquired immediately before the user information acquisition part <NUM> acquires the second user posture information with (ii) the second user posture information. If the generation part <NUM> generates the reference information in this manner, the probability that the first user U1 who is not familiar with the operation of the robot <NUM> can, by using the reference information, cause the robot <NUM> to perform a desired action is improved.

Further, the generation part <NUM> may generate a plurality of pieces of the reference information in which the pre-change posture information is associated with the second user posture information for each piece of operation content of the second user. For example, the generation part <NUM> estimates the operation content on the basis of a plurality of pieces of consecutive second user posture information, generates a plurality of pieces of the reference information corresponding to the estimated operation content, and stores the reference information in the database DB. The generation part <NUM> may store the plurality of pieces of the reference information in the database DB in association with text information indicating the operation content inputted by the second user. Specifically, the generation part <NUM> stores, for example, the reference posture information in the database DB in association with each of the operation content of having the robot <NUM> grab the object at the pre-change posture indicated by the pre-change posture information and the operation content of having the robot <NUM> tilt the object. The generation part <NUM> may store the plurality of pieces of reference information in the database in association with the attributes of the second user U2 (such as the length of the operation experience of the robot <NUM>, the size of the body, the sex, and the like).

The generation part <NUM> may generate a machine-learning model capable of outputting the reference posture information suitable for the inputted pre-change posture information and the inputted first user posture information, using the pre-change posture information and the second user posture information as the teacher data. For example, the generation part <NUM> can update the machine-learning model by inputting the pre-change posture information and the second user posture information to the previously constructed machine-learning model, and can output the reference posture information suitable for the first user posture information inputted to the machine-learning model.

Instead of directly using the second user posture information, the generation part <NUM> may generate the reference posture information on the basis of the post-change posture information of the robot <NUM> that acted on the basis of the second user posture information. For this purpose, the generation part <NUM> acquires the post-change posture information from the post-change robot information acquisition part <NUM> serving as the fourth information acquisition part for acquiring the post-change posture information indicating the post-change posture after the robot changes the posture on the basis of the second user posture information. The generation part <NUM> specifies the second user posture information on the basis of the pre-change posture information and the post-change posture information, and associates the pre-change posture information with the specified second user posture information, thereby generating the plurality of pieces of the reference posture information.

Next, the operation of the server SB in a case where the first user U1 who is not familiar with the operation of the robot <NUM> operates the robot <NUM> will be described while referencing <FIG>. Also in the example shown in <FIG>, the server SB periodically acquires the posture information indicating the posture of the robot <NUM> via the robot control apparatus <NUM>.

After the user information acquisition part <NUM> of the server SB acquires the first user U1's posture information via the user apparatus <NUM> (B <NUM>), the pre-change robot information acquisition part <NUM> acquires the most recent posture information of the robot <NUM> at the time when the user information acquisition part <NUM> acquires the first user posture information as the pre-change posture information (B2). Subsequently, the determination part <NUM> of the server SB determines, as the posture of the robot, a target posture different from the posture of the first user U1 on the basis of the pre-change posture information and the first user posture information acquired by the user information acquisition part <NUM> as the first information acquisition part at the time when the robot is in the pre-change posture indicated by the pre-change posture information.

Specifically, the specification part <NUM> refers to the database DB and selects one piece of reference posture information from the plurality of pieces of reference posture information (B3). The specification part <NUM> notifies the determination part <NUM> about the selected reference posture. The determination part <NUM> determines the posture indicated by the notified reference posture as the target posture. The determination part <NUM> transmits, as the subject reference posture information, the reference posture information corresponding to the determined target posture to the robot control apparatus <NUM> (B4). Hereinafter, the operation of the specification part <NUM> will be described in detail.

The specification part <NUM> specifies the subject reference posture information corresponding to the pre-change posture information and the first user posture information acquired by the user information acquisition part <NUM> at the time when the robot <NUM> is in the pre-change posture indicated by the pre-change posture information, from among the plurality of pieces of the reference posture information used to change the posture of the robot <NUM>. That is, the specification part <NUM> selects, as the subject reference posture information, the reference posture information corresponding to one piece of reference information selected from the plurality of pieces of reference information in which the pre-change posture information is associated with the reference posture information indicating the posture of the second user U2 acquired at a previous point in time when the robot <NUM> is in the pre-change posture indicated by the pre-change posture information.

Specifically, the specification part <NUM> estimates the posture of the second user U2 corresponding to the posture of the first user U1 indicated by the first user posture information on the basis of the pre-change posture information and the first user posture information. Then, the specification part <NUM> specifies, as the subject reference posture information, the reference posture information stored in the database DB in association with the second user posture information indicating the estimated posture of the second user U2. By having the specification part <NUM> select the subject reference posture information in this manner, the determination part <NUM> can determine the target posture that best matches the posture of the first user U1, so that the first user U1 can appropriately operate the robot <NUM>.

On the basis of the first user posture information acquired by the user information acquisition part <NUM> serving as the first information acquisition part and the pre-change posture information acquired by the pre-change robot information acquisition part <NUM> serving as the second information acquisition part, the specification part <NUM> selects one piece of reference information having a relatively high probability of changing the posture of the robot to the target posture intended by the first user, from the plurality of pieces of reference information.

The specification part <NUM> may estimate the target posture intended by the first user U1 on the basis of the first user posture information and the pre-change posture information, and may select one piece of reference information having a relatively high probability of changing the posture of the robot to the estimated target posture. As a specific process, the specification part <NUM> generates the temporary target posture information indicating the rotation angle, the angular velocity, the angular acceleration, and the torque of each of one or more motors of the robot <NUM>, on the basis of the first user posture information acquired by the user information acquisition part <NUM>. The specification part <NUM> specifies the subject reference posture information on the basis of the generated temporary target posture information and the pre-change posture information.

The specification part <NUM> may estimate the target posture by specifying the operation content of the first user U1 on the basis of the first user posture information, for example. Specifically, in a case where the generation part <NUM> generates the plurality of pieces of the reference information in which the pre-change posture information is associated with the second user posture information for each piece of operation content of the second user, the specification part <NUM> specifies the operation content that the first user U1 intends to perform, and selects one piece of reference information corresponding to the pre-change posture information from the plurality of pieces of reference information corresponding to the specified operation content. The specification part <NUM> may specify the operation content to be performed by the first user U1 on the basis of a plurality of pieces of consecutive first user posture information, or may specify the operation content on the basis of text information indicating the operation content inputted by the first user U1. The specification part <NUM> can increase the possibility that the target posture intended by the first user U1 can be correctly specified by using the result of specifying the operation content.

Incidentally, it is considered that the posture of the first user U1 having a relatively long experience of operating the robot <NUM> is closer to the posture of the second user U2 having performed the same operation in the state where the pre-change posture of the robot <NUM> is the same than the posture of the first user U1 having a relatively short experience of operating the robot <NUM>. Therefore, the specification part <NUM> may acquire information indicating the length of experience the first user U1 has in operating the robot <NUM>, and may estimate the second user posture information corresponding to the first user posture information on the basis of the acquired information.

Specifically, the specification part <NUM> selects one piece of reference posture information from the plurality of pieces of reference posture information assumed to correspond to the pre-change posture information and the first user posture information on the basis of the length of experience that the first user U1 has in operating the robot <NUM>. In this manner, the determination part <NUM> can determine the target posture on the basis of the reference posture information most suitable for the operation to be performed by the first user U1. It should be noted that the specification part <NUM> may select one piece of reference posture information from the plurality of pieces of reference posture information associated with attributes such as the size of the body or sex of the first user U1, in addition to the length of the operating experience of the first user U1.

The specification part <NUM> may specify the subject reference posture information by inputting the pre-change posture information and the first user posture information to the machine-learning model created by using the plurality of pieces of pre-change posture information and the plurality of pieces of second user posture information as the teacher data. The specification part <NUM> specifies the reference posture information outputted from the machine-learning model as the subject reference posture information.

In order to enable the determination part <NUM> to determine the target posture using the first user posture information and the pre-change posture information acquired at the same time, the user information acquisition part <NUM> may acquire the first user posture information in association with a time, and the pre-change robot information acquisition part <NUM> may acquire the pre-change posture information in association with a time. In this instance, the determination part <NUM> determines the target posture as the posture of the robot <NUM> on the basis of the pre-change posture information and the first user posture information associated with the same time. With the determination part <NUM> having such a configuration, the target posture is determined on the basis of the posture of the robot <NUM> at the time when the first user U1 changes the posture, so that the determination part <NUM> is more likely to determine the target posture according to the intent of the first user U1.

While the embodiment of the present invention has been described above in detail with reference to the drawings, specific configurations are not limited to the embodiment, and may be changed, substituted, deleted, or the like without departing from the scope of the invention as defined in the claims.

Further, a program for realizing the function of an optional component of the above-described apparatus (for example, the robot control apparatus <NUM> or the control apparatus <NUM>) may be recorded on a computer-readable recording medium, and the program may be read and executed by the computer system. Here, the term "computer system" includes hardware such as an operating system (OS) and peripheral devices. A "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a compact disk (CD)-ROM, or a storage device such as a hard disk incorporated in a computer system. Further, the "computer-readable recording medium" includes a recording medium that holds a program for a predetermined time, such as a volatile memory (RAM) in a computer system serving as a server or a client when the program is transmitted via a network such as the Internet or a communication line such as a telephone line.

The program may be transmitted from a computer system in which the program is stored in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the "transmission medium" for transmitting a program refers to a medium having a function of transmitting information, for example, a network (communication network) such as the Internet or a telecommunication line (communication line) such as a telephone line.

Claim 1:
A control apparatus (<NUM>) comprising:
a first information acquisition part (<NUM>) that is configured to acquire first user posture information indicating a posture of a first user operating a robot (<NUM>);
a second information acquisition part (<NUM>) that is configured to acquire pre-change posture information indicating a pre-change posture, which is a posture of the robot before changing the posture of the robot (<NUM>) on the basis of the first user posture information;
a specification part (<NUM>) that is configured to select, as a subject reference posture information corresponding to the pre-change posture information and the first user posture information, a reference posture information corresponding to one piece of reference information selected from among a plurality of pieces of the reference information in which the pre-change posture information is associated with the reference posture information indicating the posture of a second user differing from the first user acquired at a previous point in time when the robot (<NUM>) is in the pre-change posture indicated by the pre-change posture information; and
a determination part (<NUM>) that is configured to determine, as the posture of the robot (<NUM>), a posture corresponding to the subject reference posture information selected by the specification part (<NUM>).