Patent Description:
Telexistence is known as a subset of virtual reality. Telexistence is also said to be a technique that allows a person to have a sense of remote reality or remote presence. Telexistence provides an environment in which a user manipulates an object or person at a remote location in real time while making the user feel as if the object or person is nearby.

In order to operate a robot in real time through a network by telexistence, a highspeed network is required, and the network is required to have a very low latency and not to lose data. However, depending on the network in which data is transmitted, transmission delay or loss of data may occur.

In a case where data to be transferred is one piece of data transmitted in one direction, an influence of the transmission delay and the loss of data can be reduced by temporarily storing the data and stream in a buffer using a streaming algorithm that dynamically controls the rate. Also, by using streaming protocols such as Web Real-Time Communication (WebRTC) or HTTP Live Streaming (HLS), high-quality audio and video media can be transmitted synchronously in one direction.

<CIT> discloses a surgical robot system using augmented reality or history information and a control method thereof. A master interface for a surgical robot is provided, where the master interface is configured to be mounted on a master robot, which is configured to control a slave robot having a robot arm. The interface includes: a screen display unit configured to display an endoscope picture corresponding to a picture signal provided from a surgical endoscope; one or more arm manipulation unit for respectively controlling the robot arm; and an augmented reality implementer unit configured to generate virtual surgical tool information according to a user manipulation on the arm manipulation unit for displaying a virtual surgical tool through the screen display unit. This makes it possible to display an actual surgical tool and a virtual surgical tool together using augmented reality and thus enables surgery in a facilitated manner.

In telexistence, it is necessary to transmit or receive multiple media streams bidirectionally and synchronously among a plurality of endpoints on the Internet in order to enable a user to operate a robot as if he/she were in the robot. The plurality of pieces of media may be, for example, motion data, audio data, video data, or tactile data.

In the related art, in a case where a delay occurs between an operation of a robot operator and an action of a robot in telexistence, the control of the robot may become difficult. For example, if the delay is large, the situation assumed by the operator does not coincide with the situation visually recognized by the operator through an image capturing device provided in the robot, which causes dizziness in the operator, making it difficult for the operator to continue controlling the robot. The delay is, for example, a delay generated in data communication via networks, a delay generated by performing data encoding and decoding processing, a delay generated by performing Analog to Digital (A/D) transform processing on data (for example, a delay due to sampling), a mechanical delay, or the like.

This invention focuses on these points, and an object of the present invention is to provide a sensation imparting apparatus, a robot control system, and a robot control method capable of improving operability in telexistence.

A sensation imparting apparatus according to the first aspect of the present invention comprises a transmission part that transmits operator state information indicating a state of an operator operating a robot to the robot, a reception part that receives robot state information indicating a state of the robot from the robot, a sensation imparting part that imparts a predetermined sensation to the operator, and a control part that controls the sensation imparting part to impart a sensation based on the robot state information to the operator if a delay time required from when the transmission part transmits the operator state information to when the reception part receives the robot state information is equal to or shorter than a predetermined time period, and controls the sensation imparting part to impart a sensation based on virtual state information indicating an estimated state of the robot to the operator if the delay time is longer than the predetermined time period.

The control part may control the sensation imparting part to impart the sensation based on the robot state information received by the reception part to the operator after imparting the sensation based on the virtual state information to the operator.

The control part controls the sensation imparting part to impart, to the operator, a sensation interpolated between a first sensation based on the virtual state information and a second sensation based on the robot state information received by the reception part after imparting the sensation based on the virtual state information to the operator, if there is a difference of a predetermined magnitude or more between the first sensation and the second sensation.

The sensation imparting apparatus may further comprises a buffer part that temporarily stores one or more pieces of the robot state information received by the reception part and then sequentially outputs the one or more pieces of the robot state information to the sensation imparting part, wherein the control part may control the sensation imparting part to impart a sensation based on virtual state information to the operator if an amount of the one or more pieces of the robot state information stored in the buffer part becomes less than a threshold value.

The sensation imparting apparatus may further comprises a buffer part that temporarily stores one or more pieces of the robot state information received by the reception part and then sequentially outputs the one or more pieces of the robot state information to the sensation imparting part, wherein the control part may control an amount of the robot state information temporarily stored in the buffer part so that a variation amount of a delay time required from when the transmission part transmits the operator state information to when the reception part receives the robot state information falls within a predetermined range.

The control part may include a storage part that stores a plurality of pieces of the virtual state information corresponding to the robots in different states, and control the sensation imparting part to impart a sensation based on virtual state information selected from the plurality of pieces of the virtual state information to the operator on the basis of the operator state information.

The storage part may store the plurality of pieces of the virtual state information including virtual images corresponding to images captured by the robots in different states, and the control part may control the sensation imparting part to impart, to the operator, a sensation based on virtual state information including the virtual image corresponding to a position of the robot specified on the basis of an operation content indicated by the operator state information among the plurality of pieces of the virtual state information.

The control part may estimate a state of the robot after the operator operates, on the basis of the state of the operator indicated by the immediately preceding operator state information and the state of the operator indicated by the most recent operator state information, in order to select one piece of virtual state information from the plurality of pieces of the virtual state information, and select one piece of virtual state information corresponding to the estimated state of the robot.

The control part may store sensation history information indicating content of the sensation based on the robot state information imparted by the sensation imparting part in the storage part, and select virtual state information to be used for imparting a sensation to the operator from the plurality of pieces of the virtual state information on the basis of the content of the sensation indicated by the sensation history information stored immediately before.

The control part may specify a factor causing a delay from when the transmission part transmits the operator state information to when the reception part receives the robot state information, and determine, on the basis of the specified factor, whether to control the sensation imparting part to impart the sensation based on the virtual state information stored in the storage part to the operator or to control the sensation imparting part to impart the sensation based on the virtual state information received via a network to the operator.

A robot control system according to the second aspect of the present invention comprises a sensation imparting apparatus that transmits, to a network, operator state information indicating a state of an operator operating a robot, a robot control apparatus that controls the robot on the basis of the operator state information received via the network, and a management apparatus that is capable of communicating with the robot control apparatus and the sensation imparting apparatus via the network. The sensation imparting apparatus includes a transmission part that transmits the operator state information indicating the state of the operator operating the robot to the robot, a reception part that receives robot state information indicating a state of the robot from the robot, a sensation imparting part that imparts a predetermined sensation to the operator, and a control part that controls the sensation imparting part to impart a sensation based on the robot state information to the operator if a delay time required from when the transmission part transmits the operator state information to when the reception part receives the robot state information is equal to or shorter than a predetermined time period, and controls the sensation imparting part to impart a sensation based on virtual state information indicating an estimated state of the robot to the operator if the delay time is longer than the predetermined time period. The management apparatus includes a delay specification part that specifies the delay time, and a notification part that notifies the control part about the delay time specified by the delay specification part.

The delay specification part may specify a factor causing the delay time, the notification part may instruct the sensation imparting apparatus to impart the sensation based on the virtual state information to the operator when the factor specified by the delay specification part is a transmission delay of the network, and the control part of the sensation imparting apparatus may control the sensation imparting part to impart the sensation based on the virtual state information to the operator.

The notification part may instruct the robot control apparatus to transmit virtual state information to the sensation imparting apparatus if the factor specified by the delay specification part is an operation delay of the robot, and the control part of the sensation imparting apparatus may control the sensation imparting part to impart the sensation based on the virtual state information received from the robot control apparatus to the operator.

A robot control method according to the third aspect of the present invention, the method comprises the computer-implemented steps of transmitting operator state information indicating a state of an operator operating a robot to the robot, determining whether or not a delay time required from transmitting the operator state information to receiving robot state information indicating a state of the robot is equal to or less than a predetermined time period, and imparting a sensation based on the robot state information to the operator if the delay time is equal to or less than the predetermined time period, and imparting a sensation based on virtual state information indicating an estimated state of the robot to the operator if the delay time is longer than the predetermined time period.

A program according to the fourth aspect of the present invention, the program for making a computer perform transmitting operator state information indicating a state of an operator operating a robot to the robot, determining whether or not a delay time required from transmitting the operator state information to receiving robot state information indicating a state of the robot is equal to or less than a predetermined time period, imparting a sensation based on the robot state information to the operator if the delay time is equal to or less than the predetermined time period, and imparting a sensation based on virtual state information indicating an estimated state of the robot to the operator if the delay time is longer than the predetermined time period.

According to the present invention, it is possible to improve the operability in telexistence.

Embodiments of the present invention will be described below while referencing the drawings.

<FIG> is a block diagram illustrating a schematic configuration of a robot system <NUM> according to an embodiment of the present invention. The robot system <NUM> includes n (n is an integer of <NUM> or more) terminals <NUM>-<NUM> to <NUM>-n, n robots <NUM>-<NUM> to <NUM>-n, n robot control apparatuses <NUM>-<NUM> to <NUM>-n, and a network <NUM>. The network <NUM> is, for example, the Internet, or may be any other network. The network <NUM> may be, for example, a wired network, a wireless network, or a network including both wired and wireless networks.

<FIG> shows n users <NUM>-<NUM> to <NUM>-n and n robots <NUM>-<NUM> to <NUM>-n (n is an integer of <NUM> or more). The robots <NUM>-<NUM> to <NUM>-n include, for example, a totally humanoid robot or a partially humanoid robot. A partially humanoid robot is a robot in which a part of the entire body is humanoid and another part is not humanoid, the part being, for example, an upper body part, a lower body part, or an arm part. Each of the robots <NUM>-<NUM> to <NUM>-n may be any other form of robot. The robots <NUM>-<NUM> to <NUM>-n may have, for example, a robot operating system (ROS) function.

Here, in the example of <FIG>, the respective users <NUM>-<NUM> to <NUM>-n and the respective terminal apparatuses <NUM>-<NUM> to <NUM>-n are associated with each other. Further, in the example of <FIG>, the respective robots <NUM>-<NUM> to <NUM>-n and the respective robot control apparatuses <NUM>-<NUM> to <NUM>-n are associated with each other. Furthermore, in the example of <FIG>, the respective terminal apparatuses <NUM>-<NUM> to <NUM>-n and the respective robot control apparatuses <NUM>-<NUM> to <NUM>-n are associated with each other. Thus, in the example of <FIG>, one user <NUM>-i (i is any integer of <NUM>≤i≤n), one terminal apparatus <NUM>-i, one robot <NUM>-i, and one robot control apparatus <NUM>-i are associated with each other.

In the embodiment, the n terminal apparatuses <NUM>-<NUM> to <NUM>-n have the same function, the n robot control apparatuses <NUM>-<NUM> to <NUM>-n have the same function, and the n robots <NUM>-<NUM> to <NUM>-n have the same function. Also, the n users <NUM>-<NUM> to <NUM>-n are different persons, but have the same function in that they operate the corresponding robots <NUM>-<NUM> to <NUM>-n. Therefore, a combination of a user <NUM>-<NUM>, a terminal apparatus <NUM>-<NUM>, a robot <NUM>-<NUM>, and a robot control apparatus <NUM>-<NUM> will be described as an example below.

<FIG> is a block diagram illustrating a schematic configuration of the terminal apparatus <NUM>-<NUM> according to the embodiment of the present invention. The terminal apparatus <NUM>-<NUM> detects a state of the user <NUM>-<NUM> and also functions as a sensation imparting apparatus for imparting various types of sensations to the user <NUM>-<NUM>. The sensation imparted by the terminal apparatus <NUM>-<NUM> to the user <NUM>-<NUM> is arbitrary, but is, for example, a visual, auditory, tactile, or force sensation. The terminal apparatus <NUM>-<NUM> imparts a sensation based on a state of the robot <NUM>-<NUM> acquired via the robot control apparatus <NUM>-<NUM> to the user <NUM>-<NUM>.

The terminal apparatus <NUM>-<NUM> includes a detection part <NUM>, a buffer <NUM>, a communication part <NUM>, a sensation imparting part <NUM>, and a control part <NUM>. The communication part <NUM> includes a transmission part <NUM> and a reception part <NUM>. The control part <NUM> includes a storage part <NUM> including a read only memory (ROM) and a random access memory (RAM).

The detection part <NUM> detects the state of the user <NUM>-<NUM>. The state detected by the detection part <NUM> is, for example, a motion of fingers of the user <NUM>-<NUM>, a motion of a torso of the user <NUM>-<NUM>, or a motion of eyes (for example, a line of sight) of the user <NUM>-<NUM>.

The detection part <NUM> includes, for example, a sensor for detecting a state of a detection target. The detection part <NUM> includes, for example, a sensor for a head, a sensor for fingers, and a sensor for a torso. The sensor for the head is provided in, for example, a head-mounted display that provides a video image and sound to the user <NUM>-<NUM>. The head-mounted display includes, for example, a sensor for detecting the motion of the user's eyes (for example, a line of sight). The sensor may detect the motion of the left eye and the motion of the right eye separately.

The sensor may be, for example, a sensor in the form of a sensor attached to the user <NUM>-<NUM>, or may be a sensor not attached to the user <NUM>-<NUM>. The sensor not attached to the user <NUM>-<NUM> is, for example, an image capturing device (for example, a camera) that captures an image (for example, a video image) of all or a part of the user <NUM>-<NUM>.

The buffer <NUM> is a memory for temporarily storing data. In the embodiment, the buffer <NUM> has a function of temporarily storing data to be transmitted by the transmission part <NUM> and a function of temporarily storing data received by the reception part <NUM>. A different buffer may be provided for each of these two functions. The buffer <NUM> temporarily stores one or more pieces of robot state information received by the reception part <NUM>, and then sequentially outputs the one or more pieces of robot state information to the sensation imparting part <NUM>. The robot state information is information indicating the state of the robot <NUM>-<NUM>, and is information, for example, indicating an image captured by the robot <NUM>-<NUM>, a pressure detected by the robot <NUM>-<NUM>, or a temperature detected by the robot <NUM>-<NUM>.

The buffer <NUM> temporarily stores operator state information indicating a state of an operator detected by the detection part <NUM>, and then sequentially outputs one or more pieces of the operator state information to the transmission part <NUM>. The operator state information is, for example, information indicating the position of the body of the user <NUM>-<NUM> detected by using a sensor attached by the user <NUM>-<NUM> to his/her body.

The communication part <NUM> communicates with the robot control apparatus <NUM>-<NUM> corresponding to the terminal apparatus <NUM>-<NUM>. The communication part <NUM> communicates with the robot control apparatus <NUM>-<NUM> via the network <NUM>. Also, the communication part <NUM> communicates via WebRTC. The communication part <NUM> can communicate at a high frame rate and with a low delay, for example, via WebRTC.

The transmission part <NUM> acquires the operator state information indicating the state of the user <NUM>-<NUM> operating the robot <NUM>-<NUM> from the buffer <NUM>, and transmits the acquired operator state information to the robot <NUM>-<NUM>. Specifically, the transmission part <NUM> transmits data to be transmitted including the operator state information to the robot control apparatus <NUM>-<NUM>.

The reception part <NUM> receives the robot state information indicating the state of the robot <NUM>-<NUM> as data transmitted from the robot control apparatus <NUM>-<NUM>. The reception part <NUM> inputs the received robot state information to the buffer <NUM>.

The sensation imparting part <NUM> imparts a sensation based on the robot state information received by the reception part <NUM> to the user <NUM>-<NUM>. That is, the sensation imparting part <NUM> imparts the sensation corresponding to the robot state information indicating the state of the robot <NUM>-<NUM> received by the reception part <NUM> to the user <NUM>-<NUM>.

The sensation imparting part <NUM> includes, for example, a device for imparting a sensation to the user <NUM>-<NUM>. For example, the sensation imparting part <NUM> may include a video display device for displaying an image (video) to impart a visual sensation to the user <NUM>-<NUM>, a speaker for outputting sound (audio) to impart an auditory sensation to the user <NUM>-<NUM>, and a motion generator for generating a pressure or a vibration (haptics) to impart a tactile sensation to the user <NUM>-<NUM>. The video display device and the speaker are provided, for example, in a head device (such as a head-mounted display). The device for imparting the sensation to the user <NUM>-<NUM> may be a device attached to the user <NUM>-<NUM> or a device not attached to the user <NUM>-<NUM>. All or a part of the detection part <NUM> and the sensation imparting part <NUM> may be included in the same device.

The control part <NUM> performs various controls in the terminal apparatus <NUM>-<NUM>. The control part <NUM> includes, for example, a central processing unit (CPU) and a storage part <NUM> for storing data. The control part <NUM> performs the various controls by executing programs stored in the storage part <NUM> using, for example, parameters stored in the storage part <NUM>.

In the embodiment, the control part <NUM> can adjust the length of time for temporarily storing the data to be transmitted with the buffer <NUM>, and can adjust the length of time for temporarily storing the received data with the buffer <NUM>.

Here, the terminal apparatus <NUM>-<NUM> may be composed of a plurality of devices, and may include, for example, the detection part <NUM>, the buffer <NUM>, the communication part <NUM>, the sensation imparting part <NUM>, and the control part <NUM> shown in <FIG> in a distributed manner in two or more devices. Further, the transmission part <NUM> and the reception part <NUM> may be provided in different devices. For example, the terminal apparatus <NUM>-<NUM> may be configured by using a device including the detection part <NUM>, a device including the sensation imparting part <NUM>, and a device including the buffer <NUM>, the communication part <NUM>, and the control part <NUM>. In addition, the device including the detection part <NUM> and the device including the sensation imparting part <NUM> may be partially or entirely integrated with each other.

<FIG> is a block diagram illustrating a schematic configuration of the robot control apparatus <NUM>-<NUM> according to the embodiment of the present invention. The robot control apparatus <NUM>-<NUM> includes a communication part <NUM>, a buffer <NUM>, a robot driving part <NUM>, a detection part <NUM>, and a control part <NUM>. The communication part <NUM> includes a reception part <NUM> and a transmission part <NUM>. The control part <NUM> includes a CPU and a storage part <NUM> including a ROM and a RAM.

The communication part <NUM> communicates with the terminal apparatus <NUM>-<NUM> corresponding to the robot control apparatus <NUM>-<NUM>. In the embodiment, the communication part <NUM> communicates via the network <NUM>. The communication part <NUM> communicates via WebRTC, for example. The reception part <NUM> receives data such as the operator state information transmitted from the terminal apparatus <NUM>-<NUM>. The transmission part <NUM> transmits data such as the robot state information to be transmitted to the terminal apparatus <NUM>-<NUM>.

The buffer <NUM> is a memory for temporarily storing data. In the embodiment, the buffer <NUM> has a function of temporarily storing the robot state information to be transmitted by the transmission part <NUM>, and a function of temporarily storing the operator state information received by the reception part <NUM>. It should be noted that the buffer <NUM> may have different buffers for each of these two functions.

The robot driving part <NUM> drives a driving part of the robot <NUM>-<NUM> corresponding to the robot control apparatus <NUM>-<NUM> on the basis of the operator state information received from the terminal apparatus <NUM>-<NUM>. Here, the driving part of the robot <NUM>-<NUM> is, for example, a driving part for driving a head, a driving part for driving fingers, or a driving part for driving a torso.

The detection part <NUM> detects motion of the robot <NUM>-<NUM>. The motion detected by the detection part <NUM> is, for example, the motion of the fingers of the robot <NUM>-<NUM>, the motion of the torso of the robot <NUM>-<NUM>, or the motion of the eyes (e.g., a line of sight) of the robot <NUM>-<NUM>.

The detection part <NUM> includes, for example, the sensor for detecting the state of the detection target. As an example, the detection part <NUM> includes the sensor for the head, the sensor for the fingers, and the sensor for the torso. The sensor for the head is, for example, an image capturing device (a camera) for capturing a video image or a microphone for inputting a sound. For example, the detection part <NUM> may separately process the video image corresponding to the left eye and the video image corresponding to the right eye.

The sensor may be a sensor attached to the robot <NUM>-<NUM> or a sensor not attached to the robot <NUM>-<NUM>. The sensor not attached to the robot <NUM>-<NUM> is, for example, an image capturing device which is installed in the vicinity of the robot <NUM>-<NUM> and captures an image (for example, a video image). Various types of data are captured by sampling using, for example, A/D conversion. In the embodiment, the data detected by the detection part <NUM> is the robot state information to be transmitted. It should be noted that all or a part of the robot driving part <NUM> and the detection part <NUM> may be, for example, included in the same device.

The control part <NUM> performs various controls in the robot control apparatus <NUM>-<NUM>. The control part <NUM> includes, for example, a CPU and a storage part <NUM> for storing data. The control part <NUM> performs the various controls by, for example, executing programs stored in the storage part <NUM> using parameters stored in the storage part <NUM>. In the embodiment, the control part <NUM> can adjust the length of time for temporarily storing data such as the robot state information to be transmitted in the buffer <NUM>, and can adjust the length of time for temporarily storing data such as the received operator state information in the buffer <NUM>.

Here, the robot control apparatus <NUM>-<NUM> may be composed of a plurality of devices, and may include, for example, the communication part <NUM>, the buffer <NUM>, the robot driving part <NUM>, the detection part <NUM>, and the control part <NUM> shown in <FIG> in a distributed manner in two or more devices. Further, the reception part <NUM> and the transmission part <NUM> may be provided in different devices. For example, the robot control apparatus <NUM>-<NUM> may be configured using a device including the robot driving part <NUM>, a device including the detection part <NUM>, and a device including the communication part <NUM>, the buffer <NUM>, and the control part <NUM>. In addition, the device including the robot driving part <NUM> and the device including the detection part <NUM> may be partially or entirely integrated with each other.

The terminal apparatus <NUM>-<NUM> detects the state of the user <NUM>-<NUM> with the detection part <NUM>, and transmits the operator state information including the detected data to the robot control apparatus <NUM>-<NUM> via the network <NUM>. The robot control apparatus <NUM>-<NUM> receives the operator state information transmitted from the terminal apparatus <NUM>-<NUM> via the network <NUM>, and drives the robot <NUM>-<NUM> using the robot driving part <NUM> on the basis of the received operator state information. The robot <NUM>-<NUM> is driven and operated by the robot driving part <NUM> of the robot control apparatus <NUM>-<NUM>.

The robot control apparatus <NUM>-<NUM> detects the state of the robot <NUM>-<NUM> by the detection part <NUM>, and transmits the robot state information including the detected data to the terminal apparatus <NUM>-<NUM> via the network <NUM>. The terminal apparatus <NUM>-<NUM> receives the robot state information transmitted from the robot control apparatus <NUM>-<NUM> via the network <NUM>, and imparts the sensation to the user <NUM>-<NUM> with the sensation imparting part <NUM> on the basis of the received robot state information.

Here, in the embodiment, the terminal apparatus <NUM>-<NUM> and the robot control apparatus <NUM>-<NUM> are associated with each other before being used by the user <NUM>-<NUM> or when being used by the user <NUM>-<NUM>. The terminal apparatus <NUM>-<NUM> stores information of the address of the robot control apparatus <NUM>-<NUM> corresponding to its own apparatus (the terminal apparatus <NUM>-<NUM>), and the robot control apparatus <NUM>-<NUM> stores information of the address of the terminal apparatus <NUM>-<NUM> corresponding to its own apparatus (the robot control apparatus <NUM>-<NUM>). Then, the terminal apparatus <NUM>-<NUM> and the robot control apparatus <NUM>-<NUM>, which correspond to each other, communicate with each other as communication partners by using the stored address information, for example, by including the address information in a signal to be transmitted.

<FIG> is a diagram for explaining how to adjust delay time using buffers in the terminal apparatus <NUM>-<NUM> and the robot control apparatus <NUM>-<NUM> according to the embodiment of the present invention. <FIG> shows the user <NUM>-<NUM>, a head device <NUM>, finger devices <NUM>-<NUM> to <NUM>-<NUM>, and torso devices <NUM>-<NUM> to <NUM>-<NUM>, which are constituent parts of the terminal apparatus <NUM>-<NUM>, as well as the buffer <NUM>, which is a constituent part of the terminal apparatus <NUM>-<NUM>. Further, <FIG> shows the robot <NUM>-<NUM>, a visual device <NUM>, audio devices <NUM>-<NUM> to <NUM>-<NUM>, and finger devices <NUM>-<NUM> to <NUM>-<NUM>, which are constituent parts of the robot control apparatus <NUM>-<NUM>, as well as the buffer <NUM>, which is a constituent part of the robot control apparatus <NUM>-<NUM>.

The user <NUM>-<NUM> wears the head device <NUM> on the head, the finger device <NUM>-<NUM> on the fingers of his/her right hand, the finger device <NUM>-<NUM> on the fingers of his/her left hand, the torso device <NUM>-<NUM> on the right of his/her torso, and the torso device <NUM>-<NUM> on the left of his/her torso. The head device <NUM> includes a visual part <NUM> at a position corresponding to both eyes of the user <NUM>-<NUM>, an audio part <NUM>-<NUM> at a position corresponding to the right ear of the user <NUM>-<NUM>, and an audio part <NUM>-<NUM> at a position corresponding to the left ear of the user <NUM>-<NUM>. The head device <NUM> is, for example, a head-mounted display. The visual part <NUM> processes data for each of the right eye and the left eye.

Here, the head device <NUM>, the finger devices <NUM>-<NUM> to <NUM>-<NUM>, and the torso devices <NUM>-<NUM> to <NUM>-<NUM> have the function of the detection part <NUM> of the terminal apparatus <NUM>-<NUM>. The head device <NUM> detects the motion of the head. The finger devices <NUM>-<NUM> to <NUM>-<NUM> detect the motion of the fingers of the right and left hands, respectively. The torso devices <NUM>-<NUM> to <NUM>-<NUM> detect the motion of the torso. The terminal apparatus <NUM>-<NUM> realizes a motion capture function for detecting the motion (change in posture) of the user <NUM>-<NUM> on the basis of the motion of each part detected by the head device <NUM>, the finger devices <NUM>-<NUM> to <NUM>-<NUM>, and the torso devices <NUM>-<NUM> to <NUM>-<NUM>.

In the example of <FIG>, the user <NUM>-<NUM> has the right and left torso devices <NUM>-<NUM> to <NUM>-<NUM> separately, but the user <NUM>-<NUM> may have one torso device instead of the right and left torso devices <NUM>-<NUM> to <NUM>-<NUM>. It should be noted that <FIG> is an example, and any other device for detecting the state of the user may be used as the detection part <NUM>. For example, a touch panel for detecting the content of an operation using fingers or the like of a person may be used.

The finger devices <NUM>-<NUM> to <NUM>-<NUM>, the torso devices <NUM>-<NUM> to <NUM>-<NUM>, the visual part <NUM>, and the audio parts <NUM>-<NUM> to <NUM>-<NUM> have the function of the sensation imparting part <NUM> of the terminal apparatus <NUM>-<NUM>. The finger devices <NUM>-<NUM> to <NUM>-<NUM> each have a function of imparting the tactile sensation to each of the fingers. The torso devices <NUM>-<NUM> to <NUM>-<NUM> have a function of imparting the tactile sensation to the torso. The visual part <NUM> has the function of imparting the visual sensation to both eyes (each of the left and right eyes) of the user <NUM>-<NUM>, and displays, for example, a video image. The audio parts <NUM>-<NUM> to <NUM>-<NUM> have the function of imparting the auditory sensation to both ears (each of the left and right ears) of the user <NUM>-<NUM>, and output sound, for example. <FIG> is an example, and the sensation imparting part <NUM> may be any device that imparts other sensations to the user.

The robot <NUM>-<NUM> is equipped with a visual device <NUM> in a position corresponding to both eyes of a person, an audio device <NUM>-<NUM> in a position corresponding to a right ear of a person, an audio device <NUM>-<NUM> in a position corresponding to a left ear of a person, a finger device <NUM>-<NUM> in a position corresponding to fingers of a right hand of a person, and a finger device <NUM>-<NUM> in a position corresponding to fingers of a left hand of a person.

Here, the visual device <NUM>, the audio devices <NUM>-<NUM> to <NUM>-<NUM>, and the finger devices <NUM>-<NUM> to <NUM>-<NUM> have the function of the detection part <NUM> of the robot control apparatus <NUM>-<NUM>. The visual device <NUM> includes, for example, an image capturing device, and generates video data as data relating to the visual sensation. In the embodiment, the visual device <NUM> processes the video data for each of the right eye and the left eye. The audio devices <NUM>-<NUM> to <NUM>-<NUM> generate audio data as data relating to the auditory sensation.

The finger devices <NUM>-<NUM> to <NUM>-<NUM> generate data relating to the tactile sensation for the respective fingers. The tactile sensation data is data indicating the degree of hardness and roughness of a place touched by the finger devices <NUM>-<NUM> to <NUM>-<NUM>. The detection part <NUM> may include a device detachable from the robot <NUM>-<NUM>. Further, as another configuration example, the detection part <NUM> may include a device that is integrated and fixed to the robot <NUM>-<NUM>. In this case, the device may be regarded as a part of the robot <NUM>-<NUM>.

It should be noted that illustration of the communication part <NUM> and the control part <NUM> of the terminal apparatus <NUM>-<NUM> is omitted in the example shown in <FIG>. Further, illustration of the communication part <NUM>, the robot driving part <NUM>, and the control part <NUM> of the robot control apparatus <NUM>-<NUM> is omitted in the example shown in <FIG>. Furthermore, for convenience of description, the buffer <NUM> and the buffer <NUM> are shown as a single unit in the example shown in <FIG>, but the configuration is not necessarily limited to this, and the buffer <NUM> is provided in the terminal apparatus <NUM>-<NUM> while the buffer <NUM> is provided in the robot control apparatus <NUM>-<NUM>.

In the example shown in <FIG>, the operator state information indicating the state of the user <NUM>-<NUM> detected by the detection part <NUM> of the terminal apparatus <NUM>-<NUM> is transmitted to the control part <NUM> or the robot driving part <NUM> of the robot control apparatus <NUM>-<NUM> by a transmission session P1, a transmission session P2, and a transmission session P3. The transmission session P1 is a session from the control part <NUM> or the detection part <NUM> of the terminal apparatus <NUM>-<NUM> to the buffer <NUM> of the terminal apparatus <NUM>-<NUM>. The transmission session P2 is a session from the buffer <NUM> of the terminal apparatus <NUM>-<NUM> to the buffer <NUM> of the robot control apparatus <NUM>-<NUM>. The transmission session P3 is a session from the buffer <NUM> of the robot control apparatus <NUM>-<NUM> to the control part <NUM> or the robot driving part <NUM> of the robot control apparatus <NUM>-<NUM>.

The robot state information indicating the states of the robots <NUM>-<NUM> to <NUM>-n detected by the detection part <NUM> of the robot control apparatus <NUM>-<NUM> is transmitted to the control part <NUM> or the sensation imparting part <NUM> of the terminal apparatus <NUM>-<NUM> by a transmission session P11, a transmission session P12, and a transmission session P13. The transmission session P11 is a session to the buffer <NUM> of the robot control apparatus <NUM>-<NUM>. The transmission session P12 is a session from the buffer <NUM> of the robot control apparatus <NUM>-<NUM> to the buffer <NUM> of the terminal apparatus <NUM>-<NUM>. The transmission session P13 is a session from the buffer <NUM> of the terminal apparatus <NUM>-<NUM> to the control part <NUM> or the sensation imparting part <NUM> of the terminal apparatus <NUM>-<NUM>.

The data transmission between the buffer <NUM> of the terminal apparatus <NUM>-<NUM> and the buffer <NUM> of the robot control apparatus <NUM>-<NUM> is performed by the communication part <NUM> of the terminal apparatus <NUM>-<NUM> and the communication part <NUM> of the robot control apparatus <NUM>-<NUM> via the network <NUM>.

As shown in <FIG>, in the embodiment, there are mainly two sessions serving as bi-directional data communication between the terminal apparatus <NUM>-<NUM> and the robot control apparatus <NUM>-<NUM>. The first session is a session including the transmission sessions P1 to P3 from the terminal apparatus <NUM>-<NUM> to the robot control apparatus <NUM>-<NUM>. In the first session, for example, the operator state information indicating the state of the user <NUM>-<NUM> is transmitted. The operator state information is, for example, information including data indicating the motion of the user's eyes, the motion of the user's fingers, or the motion of the user's torso.

The second session is a session including the transmission sessions P11 to P13 from the robot control apparatus <NUM>-<NUM> to the terminal apparatus <NUM>-<NUM>. In the second session, the robot state information including, for example, the video data, the audio data, and the tactile data indicating the state of the robot <NUM> -<NUM> is transmitted.

The robot <NUM>-<NUM> operates in accordance with the motion of the user <NUM>-<NUM>. The user <NUM>-<NUM> sees an image, hears a sound, and feels a tactile sensation at the position of the robot <NUM>-<NUM>, and performs an operation. The robot <NUM>-<NUM> and the user <NUM>-<NUM> repeat these operations.

The robot system <NUM> of the embodiment compensates for the delay of bi-directional communication in the first session and the second session. For example, the control part <NUM> controls the amount of the robot state information temporarily stored in the buffer <NUM> so that the variation amount in the delay time required from the transmission of the user <NUM>-<NUM> state information by the transmission part <NUM> to the reception of the robot state information by the reception part <NUM> falls within a predetermined range. Here, for example, the user <NUM>-<NUM> and the robot <NUM>-<NUM> may have different delay times for their respective sessions. In such a case, the robot system <NUM> compensates for a different amount of the delay time for each of the first session and the second session in order to achieve synchronized motion on the side of the user <NUM>-<NUM> and on the side of the robot <NUM>-<NUM>.

In this example, the total delay time of the transmission sessions P1 to P3 from the user <NUM>-<NUM> side (the terminal apparatus <NUM>-<NUM>) to the robot <NUM>-<NUM> side (the robot control apparatus <NUM>-<NUM>) is represented by LatM. Further, the total delay time of the transmission sessions P1 <NUM> to P13 from the robot <NUM>-<NUM> side (the robot control apparatus <NUM>-<NUM>) to the user <NUM>-<NUM> side (the terminal apparatus <NUM>-<NUM>) is represented by LatN. In addition, the delay time of the transmission session P1 is represented by LatMu, the delay time of the transmission session P3 is represented by LatMr, the delay time of the transmission session P11 is represented by LatNr, and the delay time of the transmission session P13 is represented by LatNu.

The delay time in a mechanism (in the embodiment, the terminal apparatus <NUM>-<NUM>) for detecting the motion of the user <NUM>-<NUM> is represented by LatMm, and the delay time in the mechanism (in the present embodiment, the robot control apparatus <NUM>-<NUM>) for driving the robot <NUM>-<NUM> is represented by LatNm. In this case, the equations (<NUM>) and (<NUM>) are established. [Equation <NUM>] <MAT> [Equation <NUM>] <MAT>.

The equation (<NUM>) is established when a total round-trip delay time (the delay time of a round-trip) including the transmission sessions P1 to P3 and the transmission sessions P11 to P13 is expressed by RTT in a case where the side of the robot <NUM>-<NUM> is seen from the side of the user <NUM>-<NUM> (for example, the side of the control part <NUM> of the terminal apparatus <NUM>-<NUM>). [Equation <NUM>] <MAT>.

If RTT becomes large, the user <NUM>-<NUM> may feel dizziness and feel ill. Also, if the variation amount of RTT over time is large, the user <NUM>-<NUM> may feel ill.

Therefore, the robot system <NUM> of the embodiment is configured such that the buffer <NUM> is provided on the side of the user <NUM>-<NUM> that is on one side of the network <NUM>, and the buffer <NUM> is provided on the side of the robot <NUM>-<NUM> that is on the other side of the network <NUM>, so that the delay time of the data in each of the buffer <NUM> and the buffer <NUM> can be adjusted. That is, the robot system <NUM> can adjust the delay time on each of the side of the user <NUM>-<NUM> and the side of the robot <NUM>-<NUM>.

On the side of the user <NUM>-<NUM>, the control part <NUM> of the terminal apparatus <NUM>-<NUM> controls the delay time of the data in the buffer <NUM> in real time so that the equation (<NUM>) is established. On the side of the robot <NUM>-<NUM>, the control part <NUM> of the robot control apparatus <NUM>-<NUM> controls the delay time of the data in the buffer <NUM> in real-time so that the equation (<NUM>) is established. [Equation <NUM>] <MAT> [Equation <NUM>] <MAT>.

Here, in the equations (<NUM>) and (<NUM>), "~" represents the same or close. That is, the control part <NUM> of the terminal apparatus <NUM>-<NUM> controls the delay time so that the difference between the delay time LatMu of the transmission session P1 and the delay time LatNu of the transmission session P13 is equal to or less than a predetermined first time period so that the equation (<NUM>) is established. Also, the control part <NUM> of the robot control apparatus <NUM>-<NUM> controls the delay time so that the difference between the delay time LatMr of the transmission session P3 and the delay time LatNr of the transmission session P <NUM> is equal to or less than a predetermined second time period so that the equation (<NUM>) is established. The predetermined first time period and the predetermined second time period are, for example, the maximum values of a time period that does not cause a hindrance in the operation of the robot <NUM>-<NUM> by the user <NUM>-<NUM>.

Here, in this example, it is assumed that the video data transmitted from the robot control apparatus <NUM>-<NUM> to the terminal apparatus <NUM>-<NUM> requires a longer transmission time than that of the data of other media (such as the audio data and the tactile data). Regardless of the type of media, the control part <NUM> and the control part <NUM> control the delay times in the buffer <NUM> and the buffer <NUM> so that the difference in the round-trip delay time (in this example, RTT) between the side of the user <NUM>-<NUM> and the side of the robot <NUM>-<NUM> falls within the predetermined range (within the predetermined time period). The predetermined time period is determined on the basis of the video data requiring the longest transmission time, and is, for example, <NUM>. The control part <NUM> and the control part <NUM> may allow the data of media whose difference between the delay times is within the predetermined time period to pass through the buffer <NUM> and the buffer <NUM> without adding the delay time.

Even if the operability is improved by suppressing the variation of the round-trip delay time between the side of the user <NUM>-<NUM> and the side of the robot <NUM>-<NUM>, the user <NUM>-<NUM> may feel uncomfortable if the delay time is long. In particular, if there is a long delay time from the operation of the user <NUM>-<NUM> until the user <NUM>-<NUM> visually recognizes the image captured by the robot <NUM>-<NUM> in the posture after being changed in accordance with the operation of the user <NUM>-<NUM>, the user <NUM>-<NUM> is likely to feel dizziness. Therefore, the terminal apparatus <NUM> is configured to be able to impart a virtual sensation to the user <NUM>-<NUM>. Hereinafter, an embodiment will be exemplified in which the terminal apparatus <NUM> causes the user <NUM>-<NUM> to visually recognize a virtual visual image generated by estimating an image visually recognized by the robot <NUM>-<NUM> as a virtual sensation. The virtual sensation is not limited to the virtual visual sensation, and may be other sensations such as a virtual auditory sensation, a virtual tactile sensation, or a virtual force sensation.

<FIG> is a block diagram showing an image of a configuration for providing the virtual sensation in the robot system <NUM> according to the embodiment of the present invention. <FIG> shows the user <NUM>-<NUM> wearing the head device <NUM> or the like, and a virtual robot <NUM>. The virtual robot <NUM> is a robot capable of providing the virtual sensation using data stored in the buffer <NUM> of the terminal apparatus <NUM>-<NUM>, and includes a computer that virtually reproduces the state of the robot <NUM>-<NUM>. In this example, the control part <NUM> of the terminal apparatus <NUM>-<NUM> operates as if it is transmitting/receiving data to/from the virtual robot <NUM>.

If the delay time (for example, RTT) required from the transmission of the operator state information by the transmission part <NUM> to the reception of the robot state information by the reception part <NUM> is equal to or less than the predetermined time period, the control part <NUM> controls the sensation imparting part <NUM> to impart the sensation based on the robot state information to the user <NUM>-<NUM> as an operator. On the other hand, if the delay time is longer than the predetermined time period, the control part <NUM> controls the sensation imparting part <NUM> to impart the sensation based on virtual state information indicating the estimated state of the robot <NUM>-<NUM> to the user <NUM>-<NUM>. The virtual state information includes data indicating visual, auditory, tactile, and force sensations, in the same manner as the robot state information.

<FIG> is a diagram for explaining the operation of the robot system <NUM> if the delay time is equal to or less than the predetermined time period. <FIG> is a diagram for explaining the operation of the robot system <NUM> if the delay time is longer than the predetermined time period. As shown in <FIG>, if the delay time is small enough not to cause trouble in the operation of the user <NUM>-<NUM>, the robot state information is imparted to the user <NUM>-<NUM> via the buffer <NUM> of the terminal apparatus <NUM>-<NUM>. On the other hand, as shown in <FIG>, if the delay time is long enough to cause trouble in the operation of the user <NUM>-<NUM>, for example, the virtual state information stored in the storage part <NUM> is imparted to the user <NUM>-<NUM> from the terminal apparatus <NUM>-<NUM>. Hereinafter, the details of the operation of the terminal apparatus <NUM>-<NUM> for providing the virtual state information will be described.

The control part <NUM> controls the sensation imparting part <NUM> so as to impart, to the user <NUM>-<NUM>, at least some pieces of the virtual state information among a plurality of pieces of the virtual state information corresponding to virtual robots in different states, which are stored in the storage part <NUM>, for example. Specifically, the control part <NUM> controls the sensation imparting part <NUM> to impart, to the user <NUM>-<NUM>, the sensation based on the virtual state information selected from a plurality of pieces of the virtual state information on the basis of the operator state information.

More specifically, the storage part <NUM> stores the plurality of pieces of the virtual state information including virtual images corresponding to images captured by the robots <NUM>-<NUM> in different states. The virtual image is a captured image generated by the visual device <NUM> already included in the robot <NUM>-<NUM> or an image capturing device mounted on another equivalent robot, and the image is associated with information indicating the state of the robot (for example, the position of each part of the robot) at the time of capturing the image. The control part <NUM> controls the sensation imparting part <NUM> so as to impart, to the operator, the sensation based on the virtual state information, among the plurality of pieces of the virtual state information, including the virtual images corresponding to the position of the robot <NUM>-<NUM> specified on the basis of operation content indicated by the operator state information.

In order to select one piece of the virtual state information from the plurality of pieces of the virtual state information, the control part <NUM> estimates the state of the robot <NUM>-<NUM> after the operation of the user <NUM>-<NUM> on the basis of the state of the user <NUM>-<NUM> indicated by the immediately preceding operator state information and the state of the user <NUM>-<NUM> indicated by the most recent operator state information. The control part <NUM> selects one piece of the virtual state information corresponding to the estimated state of the robot <NUM>-<NUM>. In this manner, even if the robot state information transmitted from the robot control apparatus <NUM>-<NUM> is delayed, the control part <NUM> can impart the sensation based on the virtual state information to the operator before the reception part <NUM> receives the robot state information. This allows the user <NUM>-<NUM> to easily operate the robot <NUM>-<NUM>.

The control part <NUM> may extract an image of the robot <NUM>-<NUM> included in an image as the robot state information that was provided to the user <NUM>-<NUM> immediately before, and may provide the user <NUM>-<NUM> with an image obtained by replacing the extracted image with an image of the robot indicated by the virtual state information. In this manner, the control part <NUM> can provide the user <NUM>-<NUM> with an image including an image of a virtual robot attempting to grasp an object when, for example, the robot <NUM>-<NUM> is operating to grasp the object.

The control part <NUM> may store sensation history information indicating content of the sensation based on the robot state information imparted by the sensation imparting part <NUM> in the storage part <NUM>. In this instance, the control part <NUM> selects the virtual state information, from the plurality of pieces of the virtual state information, to be used for imparting the sensation to the operator on the basis of the content of the sensation indicated by the sensation history information stored in the storage part <NUM> immediately before. For example, the control part <NUM> selects the virtual state information in which the variation amount from the sensation indicated by the sensation history information stored immediately before falls within the predetermined range. By doing so, since the sensation imparted to the user <NUM>-<NUM> does not change abruptly, the terminal apparatus <NUM>-<NUM> can prevent the user <NUM>-<NUM> from feeling discomfort.

After imparting the sensation based on the virtual state information to the user <NUM>-<NUM>, the control part <NUM> controls the sensation imparting part <NUM> to impart the sensation based on the robot state information received by the reception part <NUM> to the user <NUM>-<NUM>. Specifically, for example, if it is determined that the motion of the robot <NUM>-<NUM> within the delay time required from the transmission of the operator state information by the transmission part <NUM> to the reception of the robot state information by the reception part <NUM> is smaller than an amount perceivable by the user <NUM>-<NUM>, the control part <NUM> imparts the sensation based on the actual state of the robot <NUM>-<NUM> to the user <NUM>-<NUM>. More specifically, the control part <NUM> stops the impartation of a first sensation based on the virtual state information and switches to the impartation of a second sensation based on the robot state information when it is determined that the variation amount of the robot state information within the delay time is less than a threshold value. In such a way, the user <NUM>-<NUM> can grasp the actual state of the robot <NUM>-<NUM> at the point in time when the motion of the robot <NUM>-<NUM> stops.

Here, the state indicated by the virtual state information does not necessarily coincide completely with the actual state of the robot <NUM>-<NUM>. Therefore, after the impartation of the first sensation based on the virtual state information is stopped and the impartation of the second sensation based on the robot state information is started, the user <NUM>-<NUM> may have a sense of discomfort because the difference between the first sensation and the second sensation is large. To solve this problem, when there is a difference of a predetermined magnitude or more between the first sensation based on the virtual state information and the second sensation based on the robot state information received by the reception part <NUM> after the sensation based on the virtual state information is imparted to the user <NUM>-<NUM>, the control part <NUM> may control the sensation imparting part <NUM> so as to impart a sensation interpolated between the first sensation and the second sensation to the user <NUM>-<NUM>.

The control part <NUM>, for example, generates the plurality of pieces of the virtual state information for interpolating between the first sensation and the second sensation and controls the sensation imparting part <NUM> so that the user gradually reaches the second sensation from the first sensation. In a case where the first sensation and the second sensation are visual sensations, the control part <NUM> provides a plurality of virtual images to the user <NUM>-<NUM> so that the fingertip of the robot <NUM>-<NUM> at the position corresponding to the first sensation included in the image provided to the user <NUM>-<NUM> appears to move gradually to the position corresponding to the second sensation. This allows the control part <NUM> to prevent the user <NUM>-<NUM> from feeling discomfort due to a sudden change in the imparted sensation.

The control part <NUM> may control the sensation imparting part <NUM> to impart the sensation based on the robot state information to the user <NUM>-<NUM> if the amount of the one or more pieces of the robot state information stored in the buffer <NUM> is equal to or greater than the threshold value, and to impart the sensation based on the virtual state information to the user <NUM>-<NUM> if the amount of the one or more pieces of the robot state information stored in the buffer <NUM> becomes less than the threshold value. By doing so, even if a transmission delay of the robot state information occurs in a state where the amount of the robot state information temporarily stored in the buffer <NUM> is reduced, it is possible to impart the sensation based on the virtual state information to the user <NUM>-<NUM>. Accordingly, the delay time required from the transmission of the operator state information by the transmission part <NUM> to the reception of the robot state information by the reception part <NUM> can be reduced, and the user <NUM>-<NUM> can easily control the robot <NUM>-<NUM>.

The control part <NUM> may specify a factor of the delay from the transmission of the user <NUM>-<NUM> state information by the transmission part <NUM> to the reception of the robot state information by the reception part <NUM>. On the basis of the specified factor, the control part <NUM> determines whether to control the sensation imparting part <NUM> to impart the sensation based on the virtual state information stored in the storage part <NUM> to the user <NUM>-<NUM>, or to control the sensation imparting part <NUM> to impart the sensation based on the virtual state information received via the network <NUM> to the user <NUM>-<NUM>. The virtual state information received via the network <NUM> is, for example, the virtual state information provided from the robot control apparatus <NUM>-<NUM> or a server on a cloud that monitors the state of the robot <NUM>-<NUM> and generates the virtual state information on the basis of the state of the robot <NUM>-<NUM>.

If a throughput or storage capacity of the robot control apparatus <NUM>-<NUM> or the server on the cloud is better than that of the terminal <NUM>-<NUM>, it is considered that the robot control apparatus <NUM>-<NUM> or the server on the cloud can generate the virtual state information with higher accuracy (that is, closer to the actual state of the robot <NUM>-<NUM>) than the virtual state information stored in the storage part <NUM>. Therefore, the control part <NUM> can impart a sensation close to the actual state of the robot <NUM>-<NUM> to the user <NUM>-<NUM> by using the virtual state information provided from the robot control apparatus <NUM>-<NUM> or the server on the cloud when the delay time in the network <NUM> is equal to or less than the threshold value.

It should be noted that the control part <NUM> of the terminal apparatus <NUM>-<NUM> and the control part <NUM> of the robot control apparatus <NUM>-<NUM> communicate information about a processing delay time on their own side with each other via the network <NUM>. The processing delay time is (i) a time from when the user <NUM>-<NUM> operates to when the terminal apparatus <NUM>-<NUM> transmits the operator state information, and (ii) a time from when the robot <NUM>-<NUM> changes the state to when the robot control apparatus <NUM>-<NUM> transmits the robot state information. The control part <NUM> and the control part <NUM> specify a value of the round-trip delay time RTT between the side of the user <NUM>-<NUM> and the side of the robot <NUM>-<NUM> on the basis of the acquired delay time of the other side. The control part <NUM> determines whether to impart the sensation based on the virtual state information to the user <NUM>-<NUM> or to impart the sensation based on the robot state information to the user <NUM>-<NUM> on the basis of the specified delay time value.

The robot system <NUM> may control the robot <NUM>-<NUM> on the basis of virtual operator state information generated by estimating the state of the user <NUM>-<NUM>. For example, if the delay time from when the transmission part <NUM> of the terminal apparatus <NUM>-<NUM> transmits the operator state information to when the operator state information reaches the reception part <NUM> of the robot control apparatus <NUM>-<NUM> is equal to or longer than a predetermined threshold value, the control part <NUM> of the robot control apparatus <NUM>-<NUM> controls the robot <NUM>-<NUM> on the basis of the virtual operator state information. The virtual operator state information is, for example, information generated by estimating the most recent state of the operator on the basis of one or more pieces of the immediately preceding operator state information. This makes it possible to accelerate the response of the robot <NUM>-<NUM> to the operation performed by the user <NUM>-<NUM>, thereby further improving the operability.

The virtual operator state information used by the control part <NUM> may be virtual operator state information selected from a plurality of pieces of the virtual operator state information stored in the storage part <NUM>, or may be virtual operator state information generated on the basis of a model of the user <NUM>-<NUM> generated by computer graphics (CG).

In the embodiment, the second session from the robot control apparatus <NUM>-<NUM> to the terminal apparatus <NUM>-<NUM> includes the video data, audio data, and tactile data. Depending on the processing speed of the data of respective media, the delay times of the data of respective media may be different. Among these media, the delay time of the video data is the largest. Therefore, in the embodiment, the control part <NUM> of the terminal apparatus <NUM>-<NUM> and the control part <NUM> of the robot control apparatus <NUM>-<NUM> use time information assigned to respective image frames included in the video data as a reference value of the delay time.

In one example, the control part <NUM> of the robot control apparatus <NUM>-<NUM> includes the audio data and the tactile data in each of the image frames that transmit the video data. This makes it possible to associate the time information with the audio data and the tactile data to which the time information is not assigned.

Alternatively, the control part <NUM> of the robot control apparatus <NUM>-<NUM> may synchronize the video data, the audio data, and the tactile data using time stamps. In this instance, the control part <NUM> of the robot control apparatus <NUM>-<NUM> assigns a time stamp indicating the time to the data or the communication frame of the respective media. The control part <NUM> of the terminal apparatus <NUM>-<NUM> and the control part <NUM> of the robot control apparatus <NUM>-<NUM> may use a clock signal supplied from a shared clock source as a clock used for referencing the time stamp. As the shared clock source, for example, a source provided by a third party may be used, or a source provided by any device included in the robot system <NUM> (for example, a server that is a management apparatus connected to the network <NUM>) may be used.

<FIG> is a block diagram illustrating a schematic configuration of communication frames F1 to Fj (j is an integer of <NUM> or more) according to the embodiment of the present invention. <FIG> shows a plurality (j pieces) of the consecutive communication frames F1 to Fj transmitted from the robot control apparatus <NUM>-<NUM>.

In the embodiment, it is assumed that the communication speed of the communication frames F1 to Fj is <NUM> [fps (frames/second)]. Assuming that the transmission time T1 of the communication frame F1 is <NUM> [ms], the transmission time Tj of the communication frame Fj is approximately {<NUM>*(j-<NUM>)}[ms], Each of the communication frames F1 to Fj includes the video data, imaged audio data, and imaged tactile data. The video data includes, for example, the left-eye video data and the right-eye video data. The audio data includes, for example, the audio data of the left ear and the audio data of the right ear. The tactile data includes, for example, the fingertip data.

Each of the communication frames F1 to Fj includes information of consecutive serial numbers (frame numbers) in the order of transmission. The terminal apparatus <NUM>-<NUM> and the robot control apparatus <NUM>-<NUM> can specify the timings of the respective communication frames F1 to Fj on the basis of such serial numbers (frame numbers). It should be noted that communication packets may be used instead of the communication frames.

Even if some the communication frames F1 to Fj are lost in the second session from the robot control apparatus <NUM>-<NUM> to the terminal apparatus <NUM>-<NUM>, the control part <NUM> and the control part <NUM> can discard data or perform retransmission control in units of the communication frames including the data of all media (video, audio, and tactile) in response to the elapse of a time-out period. The control part <NUM> and the control part <NUM> may retransmit the corresponding data if the delay time is equal to or less than the predetermined threshold value, and may discard the corresponding data if the delay time is greater than the predetermined threshold value.

The data transmitted and received between the terminal apparatus <NUM>-<NUM> and the robot control apparatus <NUM>-<NUM> may be lost during data transmission. In such cases, the control part <NUM> of the terminal apparatus <NUM>-<NUM> may interpolate the lost data using the data stored in the buffer <NUM> so that the robot <NUM>-<NUM> operates as intended by the user <NUM>-<NUM> and the operability of the user <NUM>-<NUM> improves. Thus, for example, even if some data is lost, the control part <NUM> of the terminal apparatus <NUM>-<NUM> can estimate and reproduce the lost data on the basis of other data that has not been lost.

Similarly, the control part <NUM> of the robot control apparatus <NUM>-<NUM> can compensate for any intermediate data by interpolating the data using the data stored in the buffer <NUM>. As a result, the control part <NUM> of the robot control apparatus <NUM>-<NUM> can estimate and reproduce the lost data on the basis of other data that has not been lost even if some data has been lost, for example. Such an interpolation may be applied, for example, to any one or both of the data of the first session and the data of the second session.

As described above, the robot system <NUM> according to the embodiment can realize high-quality robot control by compensating for the delay by using the buffer <NUM> and the buffer <NUM> in telexistence. For example, the robot system <NUM> temporarily stores the operator state information and the robot state information in the buffer <NUM> and the buffer <NUM>, so that the variation amount of the delay time required from when the transmission part <NUM> transmits the operator state information to when the reception part <NUM> receives the robot state information can be suppressed within the predetermined range. This makes it easier for the user <NUM>-<NUM> using the robot system <NUM> to operate the robot <NUM>-<NUM>.

If the delay time required from when the transmission part <NUM> transmits the operator state information to when the reception part <NUM> receives the robot state information is longer than the predetermined time period, the robot system <NUM> imparts the sensation based on the virtual state information to the user <NUM>-<NUM>. With such a configuration of the robot system <NUM>, even if the delay time of data transmission in the network <NUM> is long, it is possible to shorten the time from the operation by the user <NUM>-<NUM> to the acquisition of the sensation corresponding to the state of the robot <NUM>-<NUM>. As a result, the user <NUM>-<NUM> can comfortably control the robot <NUM>-<NUM>.

It should be noted that the robot system <NUM> may control the delay time in each of the buffer <NUM> on the user <NUM>-<NUM> side and the buffer <NUM> on the robot <NUM>-<NUM> side in another manner. As an example, the robot system <NUM> may be controlled to hold bi-directional data (data of the first session and data of the second session) for a predetermined period of time in each of the buffer <NUM> on the user <NUM>-<NUM> side and the buffer <NUM> on the robot <NUM>-<NUM> side, and output the data every predetermined period of time. The predetermined period of time is determined in accordance with, for example, the medium having the longest delay time, and in this example, the predetermined period of time is determined in accordance with the delay time of the video data transmitted from the robot control apparatus <NUM>-<NUM> to the terminal apparatus <NUM>-<NUM>.

Here, the number of users <NUM>-<NUM> to <NUM>-n, the number of terminal apparatuses <NUM>-<NUM> to <NUM>-n, the number of robot control apparatuses <NUM>-<NUM> to <NUM>-n, and the number of robots <NUM>-<NUM> to <NUM>-n may be any number. For example, the number of users <NUM>-<NUM> to <NUM>-n and the number of robots <NUM>-<NUM> to <NUM>-n may be the same or different.

In addition, in the embodiment, a configuration is shown in which the terminal apparatus <NUM>-<NUM> is provided with the buffer <NUM> and the control part <NUM> for controlling the delay time in the buffer <NUM> on the user <NUM>-<NUM> side, but as another configuration, one or both of the buffer <NUM> and the delay time control function in the buffer <NUM> may be provided in a control apparatus different from the terminal apparatus <NUM>-<NUM>.

Similarly, in the embodiment, a configuration is shown in which the robot control apparatus <NUM>-<NUM> is provided with the buffer <NUM> and the control part <NUM> for controlling the delay time in the buffer <NUM> on the robot <NUM>-<NUM> side, but the present invention is not limited to such a configuration. As another configuration, one or both of the buffer <NUM> and the delay time control function in the buffer <NUM> may be provided in a control apparatus different from the robot control apparatus <NUM>-<NUM>.

Here, in the robot system <NUM>, one or more pieces of data of the visual, auditory, and tactile sensations and the like may be detected and transmitted to the side of the robot <NUM>-<NUM> by the terminal apparatus <NUM>-<NUM> on the side of the user <NUM>-<NUM>, and the robot control apparatus <NUM>-<NUM> may receive the data on the side of the robot <NUM>-<NUM> to reproduce the sensation (such as, the visual, auditory, or tactile sensation) based on the data. In this case, the terminal apparatus <NUM>-<NUM> includes, for example, a sensor (for example, an image capturing device (a camera)) for detecting a video image, a sensor (for example, a microphone) for detecting sound, a sensor (for example, a pressure sensor) for detecting the tactile sensation, or the like as the detection part <NUM>.

Further, the robot control apparatus <NUM>-<NUM> includes, for example, a sensation reproduction part that reproduces a sensation based on data such as the video, audio, and tactile data received from the terminal apparatus <NUM>-<NUM>. The function of the sensation reproduction part may be provided in, for example, the robot driving part <NUM>, or may be provided separately from the robot driving part <NUM>. A part or all of the function of the sensation reproduction part may be mounted on the robot <NUM>-<NUM>. The sensation reproduction part in this case is, for example, a video display device for reproducing a video image, a speaker for reproducing sound, a pressure generator (a motion generator) for reproducing the tactile sensation, or the like.

In the robot system <NUM>, a plurality of users may control one shared robot. A case where three users <NUM>-<NUM> to <NUM>-<NUM> control one robot <NUM>-<NUM> is shown as an example. In this example, the terminal apparatuses <NUM>-<NUM> to <NUM>-<NUM> of the three users <NUM>-<NUM> to <NUM>-<NUM> communicate with the robot control apparatus <NUM>-<NUM> in a time-sharing manner to control the robot <NUM>-<NUM>. For example, the robot <NUM>-<NUM> includes a plurality of movable parts, and the three users <NUM>-<NUM> to <NUM>-<NUM> respectively control different movable parts. The plurality of movable parts is, for example, a plurality of arms.

<FIG> is a block diagram illustrating a schematic configuration of time-sharing communication frames Z1 to Z6 according to the embodiment of the present invention. <FIG> shows a plurality of the consecutive communication frames Z1 to Z6. In this example, the terminal apparatus <NUM>-<NUM> of the first user <NUM>-<NUM> communicates with the robot control apparatus <NUM>-<NUM> using the first communication frame Z1, for example. The terminal apparatus <NUM>-<NUM> of the second user <NUM>-<NUM> communicates with the robot control apparatus <NUM>-<NUM> using the second communication frame Z2. The terminal apparatus <NUM>-<NUM> of the third user <NUM>-<NUM> communicates with the robot control apparatus <NUM>-<NUM> using the third communication frame Z3. In this manner, the plurality of terminal apparatuses <NUM>-<NUM> to <NUM>-<NUM> communicate with the shared robot control apparatus <NUM>-<NUM> in a time-sharing manner, thereby controlling the shared robot <NUM>-<NUM>.

In the robot system <NUM>, one user may control a plurality of robots. A case where the one user <NUM>-<NUM> controls three robots <NUM>-<NUM> to <NUM>-<NUM> is shown as an example. In this embodiment, the terminal apparatus <NUM>-<NUM> of the one user <NUM>-<NUM> controls the three robots <NUM>-<NUM> to <NUM>-<NUM> by communicating with three robot control apparatuses <NUM>-<NUM> to <NUM>-<NUM> in a time-sharing manner.

The description will be made using the example of <FIG>. In this example, the terminal apparatus <NUM>-<NUM> of the one user <NUM>-<NUM> communicates with the robot control apparatus <NUM>-<NUM> of the first robot <NUM>-<NUM> using the first communication frame Z1. The terminal apparatus <NUM>-<NUM> communicates with the robot control apparatus <NUM>-<NUM> of the second robot <NUM>-<NUM> using the second communication frame Z2. The terminal apparatus <NUM>-<NUM> communicates with the robot control apparatus <NUM>-<NUM> of the third robot <NUM>-<NUM> using the third communication frame Z3. In this manner, the one terminal apparatus <NUM>-<NUM> communicates with the plurality of robot control apparatuses <NUM>-<NUM> to <NUM>-<NUM> in a time-sharing manner, thereby controlling the plurality of robots <NUM>-<NUM> to <NUM>-<NUM>.

The robot system <NUM> may further include a management apparatus connected to the terminal apparatuses <NUM>-<NUM> to <NUM>-n and the robot control apparatuses <NUM>-<NUM> to <NUM>-n via the network <NUM>. The management apparatus is, for example, a server.

A management apparatus <NUM> is connected to the network <NUM>, and relays the communication between the terminal apparatuses <NUM>-<NUM> to <NUM>-n and the robot control apparatuses <NUM>-<NUM> to <NUM>-n. That is, the management apparatus <NUM> receives a signal transmitted from the terminal apparatuses <NUM>-<NUM> to <NUM>-n to the network <NUM> and transmits the signal to the robot control apparatus <NUM>-<NUM> to <NUM>-n of the destination, and further, the management apparatus <NUM> receives a signal transmitted from the robot control apparatus <NUM>-<NUM> to <NUM>-n to the network <NUM> and transmits the signals to the terminal apparatuses <NUM>-<NUM> to <NUM>-n of the destination.

As described above, the robot system <NUM> may include the management apparatus <NUM> on the cloud to manage the communication performed between the terminal apparatuses <NUM>-<NUM> to <NUM>-n and the robot control apparatuses <NUM>-<NUM> to <NUM>-n. For example, the management apparatus <NUM> stores information of each address of the terminal apparatuses <NUM>-<NUM> to <NUM>-n and the robot control apparatuses <NUM>-<NUM> to <NUM>-n corresponding to each other, and relays the signals communicated between them on the basis of the information of the addresses. The management apparatus <NUM> may have a machine learning function, and may correct, on the basis of a result of the machine learning, an operation in which the users <NUM>-<NUM> to <NUM>-n control the robots <NUM>-<NUM> to <NUM>-n.

In the embodiment described while referencing <FIG>, the terminal apparatus <NUM>-<NUM> includes the buffer <NUM> and the function of adjusting the delay time of the data in the buffer <NUM>. Further, the robot control apparatus <NUM>-<NUM> includes the buffer <NUM> and the function of adjusting the delay time of the data in the buffer <NUM>. Any one or both of these two functions may be provided in the management apparatus <NUM>.

As an example, the management apparatus <NUM> may include the buffer <NUM> and the function of adjusting the delay time of data in the buffer <NUM> (referred to as a function C1 in this example for convenience of description) instead of the terminal apparatus <NUM>-<NUM>. In this case, the terminal apparatus <NUM>-<NUM> does not need to have such a function C1.

As another example, the management apparatus <NUM> may include the buffer <NUM> and the function of adjusting the delay time of the data in the buffer <NUM> (referred to as a function C2 in this example for convenience of description) instead of the robot control apparatus <NUM>-<NUM>. In this case, the robot control apparatus <NUM>-<NUM> does not need to have such a function C2.

Further, as another example, the management apparatus <NUM> may include both the function C1 and the function C2. In particular, if the communication delay is small, it is considered preferable that the management apparatus <NUM> has the function C1 or the function C2. For example, if the delay of the data communication from the terminal apparatus <NUM>-<NUM> to the robot control apparatus <NUM>-<NUM> via the network <NUM> is smaller than the delay of the data communication from the robot control apparatus <NUM>-<NUM> to the terminal apparatus <NUM>-<NUM> via the network <NUM>, the configuration in which the terminal apparatus <NUM>-<NUM> includes the function C1 and the management apparatus <NUM> includes the function C2 may be used.

<FIG> is a diagram illustrating the management apparatus <NUM> capable of communicating with the terminal apparatus <NUM>-<NUM> and the robot control apparatus <NUM>-<NUM> via the network <NUM>. The management apparatus <NUM> includes a delay specification part <NUM> and a notification part <NUM>. The delay specification part <NUM> specifies the delay time from when the terminal apparatus <NUM>-<NUM> transmits the operator state information to when the terminal apparatus <NUM>-<NUM> receives the robot state information, on the basis of the time at which the operator state information is received from the terminal apparatus <NUM>-<NUM> and the time at which the robot state information is received from the robot control apparatus <NUM>-<NUM>. The notification part <NUM> notifies the control part <NUM> of the terminal apparatus <NUM>-<NUM> about the delay time specified by the delay specification part <NUM>.

The control part <NUM> may decide to impart the sensation based on the virtual state information to the user <NUM>-<NUM> on the basis of an instruction from the management apparatus. In this case, the delay specification part <NUM> of the management apparatus specifies the factor causing the delay, and the notification part <NUM> instructs the control part <NUM> of the terminal apparatus <NUM>-<NUM> to impart the sensation based on the virtual state information to the user <NUM>-<NUM> if the factor specified by the delay specification part <NUM> is the transmission delay of the network <NUM>. On the basis of the instruction from the notification part <NUM>, the control part <NUM> controls the sensation imparting part <NUM> to impart the sensation based on the virtual state information to the user <NUM>-<NUM>. In this manner, the control part <NUM> imparts the sensation based on the virtual state information on the basis of the instruction from the management apparatus, and therefore the control part <NUM> can impart the sensation based on whichever information, either the virtual state information or the robot state information, is more suitable for the delay state of the networks <NUM>.

The notification part <NUM> may instruct the robot control apparatus <NUM>-<NUM> to transmit the virtual state information to the terminal apparatus <NUM>-<NUM> if the factor specified by the delay specification part <NUM> is the operation delay of the robot. <FIG> is a diagram illustrating the robot system <NUM> in which the robot control apparatus <NUM>-<NUM> provides the virtual state information to the user <NUM>-<NUM>. The robot control apparatus <NUM>-<NUM> provides the terminal apparatus <NUM>-<NUM> with, for example, the virtual state information selected from the plurality of pieces of the virtual state information stored in the storage part <NUM>. With such a configuration of the robot system <NUM>, even if the terminal apparatus <NUM>-<NUM> does not store the virtual state information, the terminal apparatus <NUM>-<NUM> can impart the sensation based on the virtual state information to the user <NUM>-<NUM>.

In the robot system <NUM>, the terminal apparatus (in the example of <FIG>, the terminal apparatuses <NUM>-<NUM> to <NUM>-n) transmits the data relating to the user (in the example of <FIG>, the users <NUM>-<NUM> to <NUM>-n) to the robot control apparatus (in the example of <FIG>, the robot control apparatuses <NUM>-<NUM> to <NUM>-n) via the network (in the example of <FIG>, the network <NUM>). The robot control apparatus controls the robot (in <FIG>, robots <NUM>-<NUM> to <NUM>-n) on the basis of the data.

In addition, the robot control apparatus transmits the data relating to the robot to the terminal apparatus via the network, and the terminal apparatus imparts the sensation to the user on the basis of the data. At least one of the terminal apparatus and the robot control apparatus includes (i) the buffer (the buffers <NUM> and <NUM> in the examples of <FIG> and <FIG>) for storing at least one piece of data from among the data to be transmitted and the data received, and (ii) the control part (the control parts <NUM> and <NUM> in the examples of <FIG> and <FIG>) for controlling the delay time of the data stored in the buffer. Since the robot system <NUM> has such a configuration, the robot system <NUM> can adjust the transmission delay of data so as not to cause trouble in the operation of the robot by the user thereby improving the operability of the robot in telexistence.

In one configuration, when transmitting the data to be transmitted and the received data, the control part controls the delay time of a predetermined transmission between the network and the user or between the network and the robot to be within the predetermined range.

In one configuration, the data related to the user includes the data related to the motion of the user, the data related to the robot includes the video data, the audio data, and the tactile data in the robot, and the robot control apparatus communicates the video data, the audio data, and the tactile data in a shared communication frame. In one configuration, the data is interpolated using the data stored in the buffer for at least one of the terminal apparatus and the robot control apparatus.

In one configuration, the terminal apparatus imparts the sensation based on the robot state information to the user if the delay time required from transmitting the operator state information to receiving the robot state information is equal to or less than the predetermined time period, and imparts the sensation based on the virtual state information indicating the estimated state of the robot to the user if the delay time is longer than the predetermined time period. Because the terminal apparatus works in this manner, even if the delay time is long, the user can quickly feel the sensation corresponding to the state of the robot, thereby improving the operability.

A program for realizing the functions of the respective apparatuses (for example, the terminal apparatuses <NUM>-<NUM> to <NUM>-n or the robot control apparatus <NUM>-<NUM> to <NUM>-n) according to the embodiment described above may be recorded in a computer-readable recording medium (a storage medium), and the program recorded in the recording medium may be read into a computer system and executed to perform the process.

The "computer system" herein may include an operating system (OS) or hardware such as a peripheral device. The "computer-readable recording medium" refers to a writable non-volatile memory such as a flexible disk, a magneto-optical disk, a ROM, a flash memory, a portable medium such as a digital versatile disc (DVD), or a storage device such as a hard disk incorporated in a computer system. It should be noted that the recording medium may be, for example, a recording medium detachable from a computer. Further, the recording medium may be, for example, a recording medium that temporarily records data.

Furthermore, the "computer-readable recording medium" includes a medium that holds a program for a predetermined period of time, such as a volatile memory (for example, a dynamic random access memory (DRAM)) in a computer system serving as a server or a client when the program is transmitted via a network such as the Internet or a communication line such as a telephone line.

The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the "transmission medium" for transmitting a program refers to a medium having a function of transmitting information, for example, a network (communication network) such as the Internet or a telecommunication line (communication line) such as a telephone line.

Further, the above-mentioned program may be a program for realizing a part of the above-mentioned functions. Furthermore, the above-mentioned program may be a so-called difference file (difference program) capable of realizing the above-mentioned functions in combination with a program already recorded in the computer system.

The embodiment of the present invention is explained above in detail while referencing the drawings. The specific configuration is not limited to above embodiment and it is possible to make various modifications within the scope of the invention as defined in the claims.

Claim 1:
A sensation imparting apparatus (<NUM>) comprising:
a transmission part (<NUM>) that transmits operator state information indicating a state of an operator operating a robot (<NUM>) to the robot (<NUM>);
a reception part (<NUM>) that receives robot state information indicating a state of the robot (<NUM>) from the robot (<NUM>);
a sensation imparting part (<NUM>) that imparts a predetermined sensation to the operator; and
a control part (<NUM>) that controls the sensation imparting part to impart a sensation based on the robot state information to the operator if a delay time required from when the transmission part transmits the operator state information to when the reception part receives the robot state information is equal to or shorter than a predetermined time period, and controls the sensation imparting part to impart a sensation based on virtual state information indicating an estimated state of the robot to the operator if the delay time is longer than the predetermined time period, wherein
the control part (<NUM>) controls the sensation imparting part (<NUM>) to impart, to the operator, a sensation interpolated between a first sensation based on the virtual state information and a second sensation based on the robot state information received by the reception part after imparting the sensation based on the virtual state information to the operator, if there is a difference of a predetermined magnitude or more between the first sensation and the second sensation.