Remote robot system and method of controlling remote robot system

A remote robot system and a method of controlling a remote robot system capable of responding appropriately according to a user are provided. A remote robot system includes: a robot configured to perform a predetermined operation including collection of local information near the robot; a guide terminal capable of remotely operating the operation of the robot; and a participant terminal capable of remotely communicating with the guide terminal, in which the participant terminal includes: a display panel configured to output the local information acquired from the robot to the user; and an interest detection unit configured to detect user's interest in the output local information.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority from Japanese patent application No. 2021-068229, filed on Apr. 14, 2021, the disclosure of which is incorporated herein in its entirety by reference.

BACKGROUND

The present disclosure relates to a remote robot system and a method of controlling the remote robot system.

Recent techniques have enabled autonomous travelling robots to be remotely operated by users using operation terminals. Further, a telepresence robot including, for example, a camera, a microphone, and a monitor, which is one example of the above robots, enables users to communicate with each other remotely or enjoy virtual experiences remotely. Further, in view of a recent spread of remote services such as Web meeting services, it is expected that robots that can be remotely operated or can be used for remote communication will be used in a broad range.

With regard to operations performed by users, for example, Japanese Unexamined Patent Application Publication No. 2016-224583 discloses a technique in which an analysis server apparatus analyzes the behavior of a user viewing Web pages by a heat map. Further, Japanese Unexamined Patent Application Publication No. 2014-50465 discloses a technique in which an image processing apparatus that displays a medical image such as an X-ray CT image designates an area that a user is interested in to make a diagnosis.

SUMMARY

By using a robot such as a telepresence robot, a remote sightseeing tour which enables users to remotely join and virtually experience this tour can, for example, be provided. In a remote robot system for a remote sightseeing tour, for example, a tour guide (an operator) remotely operates a robot and users (participants) who join this tour remotely communicate with the guide. In this remote robot system, however, it is difficult to respond appropriately according to the users since the guide and the users are present in places away from each other. The technique disclosed in Japanese Unexamined Patent Application Publication No. 2016-224583 is to analyze Web page views and the technique disclosed in Japanese Unexamined Patent Application Publication No. 2014-50465 is to specify a diagnosis area of a medical image. Neither of them solves the above-described problem.

The present disclosure has been made in order to solve the aforementioned problem and provides a remote robot system and a method of controlling the remote robot system capable of responding appropriately according to a user.

A remote robot system according to one aspect of the present disclosure includes: a robot configured to perform a predetermined operation including collection of local information near the robot; a first remote terminal capable of remotely operating the operation of the robot; and a second remote terminal capable of remotely communicating with the first remote terminal, in which the second remote terminal includes: an information output unit configured to output the local information acquired from the robot to a user; and an interest detection unit configured to detect user's interest in the output local information. Accordingly, the user's interest detected by the second remote terminal can be detected, whereby it becomes possible to recognize the user's interest and respond appropriately according to the user even when the first remote terminal is away from the second remote terminal.

Further, the interest detection unit may detect an operation state by the user in response to the output local information and detect the user's interest based on the operation state that has been detected. Further, the operation state may include an operation position, the number of times of operations, or an operation time in the user's operation performed on a display screen that displays the local information. Accordingly, it is possible to accurately detect the interest in accordance with the state in which the user has operated.

The interest detection unit may further detect an utterance state of the user in response to the output local information and detect the user's interest based on the utterance state that has been detected. Further, the utterance state may include a sound pressure of a voice emitted by the user, an utterance time of the voice, or results of recognizing the voice. Accordingly, it is possible to accurately detect the interest in accordance with the state of the voice emitted by the user.

The interest detection unit may further detect an attention state by the user in response to the output local information and detect the user's interest based on the attention state that has been detected. Further, the attention state may include the orientation of the face of the user, the orientation of the line of sight of the user, an attention position of the user, or an attention time of the user. Accordingly, it is possible to accurately detect the interest in accordance with the state in which the user is focusing on.

The second remote terminal may further include a notification unit configured to notify the first remote terminal of interest information indicating the detected interest. Accordingly, it is possible to definitely send feedback on the user's interest detected by the second remote terminal to the first remote terminal.

The first remote terminal may include an interest output unit configured to output the interest information that has been sent. Further, the interest output unit may highlight a position that corresponds to the interest information on a display screen that displays the local information. The interest output unit may further display a temporal change of the interest information on a display screen that displays the local information. This enables the first remote terminal to instantly recognize the user's interest detected by the second remote terminal.

Further, the first remote terminal may include a remote control determination unit configured to determine a method of remotely operating the robot based on the interest information that has been sent. Further, the remote control determination unit may determine a method of moving a robot including a moving direction, a moving path, or a moving speed of the robot based on the interest information. The remote control determination unit may further determine an information acquisition method of the robot including an imaging direction of an image of the robot or a voice collection direction based on the interest information. Accordingly, it is possible to remotely control a robot appropriately in accordance with the user's interest detected by the second remote terminal.

A method of controlling a remote robot system according to one aspect of the present disclosure is a method of controlling a remote robot system including: a robot configured to perform a predetermined operation including collection of local information near the robot; a first remote terminal capable of remotely operating the operation of the robot; and a second remote terminal capable of remotely communicating with the first remote terminal, in which the second remote terminal outputs the local information acquired from the robot to a user, and detects user's interest in the output local information. Accordingly, the user's interest detected by the second remote terminal can be detected, whereby it is possible to recognize the user's interest and respond appropriately according to the user even when the first remote terminal is away from the second remote terminal.

According to the present disclosure, it is possible to provide a remote robot system and a method of controlling the remote robot system capable of responding appropriately according to a user.

DESCRIPTION OF EMBODIMENTS

Hereinafter, with reference to the drawings, embodiments of the present disclosure will be described. Throughout the drawings, the same elements are denoted by the same reference symbols and overlapping descriptions are omitted as appropriate. While the present disclosure will be described with reference to the embodiments, the present disclosure set forth in the claims is not limited to the following embodiments. Further, not all the components described in the following embodiments are necessary for solving problems.

First Embodiment

First, a first embodiment according to the present disclosure will be described.FIG.1shows a configuration example of a remote robot system10according to this embodiment. As shown inFIG.1, the remote robot system10according to this embodiment includes a robot100, a guide terminal (first remote terminal)200, a plurality of participant terminals (second remote terminals)300(e.g.,300-1to300-3), and a system server400.

The remote robot system10is a system in which a guide (tour guide) G remotely operates the robot100located in a tourist spot (a first environment) by operating the guide terminal200and participants P are able to virtually experience a sightseeing tour via the robot100that has been remotely operated. In the remote robot system10, a plurality of participants P (e.g., P1to P3) and the guide G are connected to the robot100, which is, for example, a telepresence robot, via terminal apparatuses, and the plurality of participants P are able to enjoy the remote sightseeing tour via the robot100while sharing video images/voices acquired by the robot100. One of the users connected to the robot100is the guide G, who gives a tour of the tourist spot while changing the travelling direction or the angle of view of the robot by remotely operating the robot100. The participants P enjoy sightseeing from remote places while watching and listening to the video images/voices of the robot100and the guide G. In this tour, the guide G needs to give a tour of the tourist spot in accordance with interest of the participants P, but it is difficult for the guide G to appropriately give the tour on his/her connection terminal in accordance with the interest of the participants P because the guide G cannot see the reactions of the participants P due to the distance between their respective positions. In order to solve this problem, in this embodiment, interest of the participants P is detected, and the detected interest is further fed back to the guide G, which enables the guide G to guide the tour in accordance with the interest of the participants P.

Note that a plurality of robots100may be installed so that the remote operation and the virtual experience can be provided for the respective participants. Further, a plurality of guide terminals200may be included so that a plurality of guides G may be able to remotely operate the plurality of robots100.

As shown inFIG.1, the robot100, the guide terminal200, and the plurality of participant terminals300are located in environments away from one another (a first environment, a second environment, and a third environment) and are connected to the system server400via wireless routers600and the Internet500. For example, the system server400provides data, processing, services and the like that are necessary for the remote operation or remote communication for the robot100, the guide terminal200, and the participant terminals300. For example, while the wireless routers600, the robot100, the guide terminal200, and the participant terminals300can communicate with one another via a wireless LAN, they may communicate with one another by mobile phone communication such as Long Term Evolution (LTE) or 5th Generation Mobile Communication System (5G). By using mobile phone communication for the communication of the robot100, the robot100can be used in a large number of places, whereby it becomes possible to remotely operate robots100installed in various places. In this case, the wireless routers600may be base stations of mobile phone communication.

The robot100is able to execute a predetermined operation including collection of local information (including, for example, images or voices) near the robot100in the first environment. The robot100is remotely operated by the guide G, who is a remote operator and is in the second environment located away from the first environment, operating the guide terminal200(operation terminal) via the system server400connected to the Internet500. The robot100receives various kinds of operation signals from the guide terminal200and executes, for example, a moving operation and/or an imaging operation. Further, the guide G and the participants P are able to remotely communicate with each other via the robot100.

The robot100captures images of a space near the robot100by a camera131(an imaging unit) and transmits the captured images (video images) to the guide terminal200and the participant terminals300via the Internet500. Further, a microphone132collects voices emitted near the robot100, and transmits the collected voices to the guide terminal200and the participant terminals300via the Internet500. Further, the robot100receives voices from the guide terminal200and the participant terminals300via the Internet500and outputs the received voices from the speaker133. The robot100is able to execute a grasping motion or the like performed by a hand124in accordance with an operation performed by the guide terminal200. It is therefore possible to provide various kinds of virtual experience in a sightseeing tour.

While the Internet500is used as a network to which the robot100, the guide terminal200, the participant terminals300, etc. are connected in this embodiment, the network may be other types of network such as an intranet. Alternatively, the robot100, the guide terminal200, and the participant terminals300may be directly connected to each other through near-field communication such as Bluetooth (Registered Trademark) without using any network interposed therebetween. That is, the remote control and the remote communication may be performed without using the Internet500and the system server400interposed therebetween.

The guide terminal200, which is a terminal operated by the guide G in the second environment, remotely operates the robot100and performs remote communication with the participant terminals300and the robot100. The participant terminals300(e.g.,300-1to300-3), which are terminals operated by participants P (e.g., participants P1to P3) in the third environment (e.g., environment3-1to environment3-3), perform the remote operation with the guide terminal200, other participant terminals300, and the robot100.

The guide terminal200and the participant terminals300are, for example, tablet terminals and include display panels201and301on which touch panels are superimposed thereon. The display panels201and301are able to display the captured images received from the robot100and the guide G and the participants P can visually recognize a situation in the spatial area near the robot100in an indirect manner. In other words, the display panel201is an information output unit configured to output local information of the robot100to the guide G and the display panel301is an information output unit configured to output local information of the robot100to the participants P (user).

Further, the guide terminal200and the participant terminals300are able to output voices received from the robot100, and the guide G and the participants P can listen to the voices emitted near the robot100in an indirect manner. The guide G is able to cause various kinds of operation signals for operating the robot100to be generated by operating the guide terminal200via the touch panel. The generated operation signal is transmitted to the robot100via the Internet500.

Further, the participant terminal300includes an interest detection unit321that detects interest of the participants P in the local information of the robot100that has been output. The interest detection unit321generates interest information indicating the interest of a participant P in accordance with the operation of the display screen of the display panel301by the participant P via the touch panel. The generated interest information is transmitted to the guide terminal200via the Internet500. Further, the guide terminal200includes the interest analysis unit221that analyzes interest information received from the participant terminal300. The interest of the participant P that has been analyzed is displayed on the display panel201. The display panel201also serves as an interest output unit that outputs interest of the participant P.

FIG.2is an external perspective view showing an example of an external configuration of the robot100according to this embodiment. As shown inFIG.2, the robot100according to this embodiment mainly includes a movable base part110and a main-body part120.

The movable base part110supports two driving wheels111and a caster112, each of which is in contact with a traveling surface, inside its cylindrical housing. The two driving wheels111are arranged so that the centers of their rotation axes coincide with each other. Each of the driving wheels111is rotationally driven by a motor (not shown) independently of each other. The caster112is a driven wheel and is disposed so that its pivotal axis extending from the movable base part110in the vertical direction axially supports the wheel at a place away from its rotation axis. Further, the caster112follows the movement of the movable base part110so as to move in the moving direction of the movable base part110.

Further, the movable base part110includes a laser scanner134in a peripheral part of its top surface. The laser scanner134scans a certain range on the horizontal plane at intervals of a certain stepping angle and outputs information as to whether or not there is an obstacle in each direction. Further, when there is an obstacle, the laser scanner134outputs a distance to the obstacle.

The main-body part120includes, mainly, a body part121mounted on the top surface of the movable base part110, a head part122placed on the top surface of the body part121, an arm123supported on the side surface of the body part121, and the hand124disposed at the tip of the arm123. The arm123and the hand124are driven by motors (not shown) and grasp an object to be grasped. The body part121is able to rotate around a vertical axis with respect to the movable base part110by a driving force of a motor (not shown).

The head part122mainly includes the camera131, the microphone132, a speaker133, and a display panel141. The camera131is disposed, for example, on the front surface of the head part122in such a way that the front direction of the head part122is an imaging direction. The camera131may either be a two-dimensional camera or a stereo camera. When, for example, the camera131is a stereo camera, this camera131has a configuration in which two camera units having the same angle of view are arranged away from each other, and it outputs imaging signals taken by the respective camera units.

The microphone132is disposed, for example, on the front surface of the head part122in such a way that the front direction of the head part122is a collect direction. The microphone132may either be a unidirectional microphone or a non-directional microphone. The speaker133is disposed, for example, in front or side of the head part122in such a way that the front direction of the head part122is an output direction. The speaker133may either be a monaural speaker or a stereo speaker.

The display panel141is disposed on the front surface of the head part122in such a way that the front direction of the head part is a display direction. The display panel141is, for example, a liquid crystal panel, and displays an animated face of a pre-defined character and displays information about the robot100in the form of text or by using icons. By displaying the face of the character on the display panel141, it is possible to impart an impression that the display panel141is a pseudo face part to people around the robot100. The display panel141is able to display images of the guide G and the participants P received from the guide terminal200and the participant terminals300.

The head part122is able to rotate around a vertical axis with respect to the body part121by a driving force of a motor (not shown). Thus, the camera131can shoot an image in any direction, the microphone132can collect a sound in any direction, the speaker133can output a sound in any direction, and the display panel141can show the displayed content in any direction.

FIG.3is a block diagram showing a functional configuration example of the robot100according to this embodiment. As shown inFIG.3, the robot100includes, as functional blocks, besides the display panel141, the camera131, the microphone132, the speaker133, and the laser scanner134described above, a control unit150, a movable-base drive unit142, an upper-body drive unit143, a memory160, a communication unit170, and a GPS unit135.

Main elements related to the remote operation and the remote communication by the guide terminal200and the participant terminals300will be described. However, the robot100may include elements in its configuration other than the above ones and may include additional elements that contribute to the remote operation and the remote communication.

The control unit150, which is, for example, a Central Processing Unit (CPU), executes control of the whole robot100and various calculation processes by executing a control program read from the memory160. Further, the control unit150also serves as a function execution unit that executes various calculations and controls related to the control.

The control unit150is included in, for example, a control unit disposed in the body part121. The movable-base drive unit142includes the driving wheels111, and a driving circuit and motors for driving the driving wheels111. The control unit150executes rotation control of the driving wheels by sending a driving signal to the movable-base drive unit142. Further, the control unit150receives a feedback signal such as an encoder signal from the movable-base drive unit142and recognizes a moving direction and a moving speed of the movable base part110.

The upper-body drive unit143includes the arm123and the hand124, the body part121, the head part122, and driving circuits and motors for driving these components. The control unit150performs a grasping motion and a gesture by sending a driving signal to the upper-body drive unit143. Further, the control unit150receives a feedback signal such as an encoder signal from the upper-body drive unit143and recognizes positions and moving speeds of the arm123and the hand124, and orientations and rotation speeds of the body part121and the head part122.

The display panel141receives an image signal generated by the control unit150and displays an image thereof. The control unit150generates an image signal of the character or the like based on information in the memory160, received information or the like and causes the display panel141to display an image thereof. When, for example, the display panel141includes a touch panel, the display panel141outputs a touch signal to the control unit150.

The camera131shoots, for example, the front part of the robot100in accordance with a request from the control unit150and outputs an imaging signal generated by imaging to the output unit150. The control unit150executes image processing using the imaging signal from the camera131and converts the imaging signal into a captured image in accordance with a predetermined format.

The microphone132collects voices emitted near the robot100in accordance with a request from the control unit150, and outputs a voice signal generated by sound collection to the control unit150. The control unit150performs voice processing by using the voice signal from the microphone132and converts the imaging signal in accordance with a predetermined format.

The speaker133outputs, for example, voices sent from the guide terminal200and the participant terminals300. The control unit150outputs voice signals received from the guide terminal200and the participant terminals300to the speaker133via the communication unit170, and the speaker133outputs the voice signals from the control unit150to the surroundings.

The laser scanner134detects whether there is an obstacle in the moving direction of the robot100in accordance with a request from the control unit150, and outputs a detection signal, which indicates the results of the detection, to the control unit150. The GPS unit135detects the position of the robot100based on a GPS signal received from a Global Positioning System (GPS) satellite in accordance with a request from the control unit150, and outputs a positional signal (positional information), which is the result of the detection, to the control unit150. The position of the robot100may be detected by a method other than the GPS.

The memory160, which is a non-volatile storage medium, is, for example, a solid-state drive. The memory160stores, besides a control program for controlling the robot100, various parameter values, functions, lookup tables and the like used for the control and the calculation. The memory160stores, for example, a map DB161, which is a database of map information that describes the space of the first environment where the robot100is planned to autonomously move. In the map DB161, target map information which has been acquired from, for example, the system server400according to the range in which the robot100moves is accumulated.

The communication unit170, which is, for example, a wireless LAN unit, performs radio communication with the wireless router600. The communication unit170transmits or receives control signals, data or the like to or from the guide terminal200and the participant terminals300via the wireless router600. For example, the communication unit170receives a designation signal and an operation instruction sent from the guide terminal200and outputs the designation signal or the operation instruction to the control unit150. The designation signal is a signal for requesting the captured image or the voice signal generated in the guide terminal200by designating a spatial area that the user wants to observe. Further, the communication unit170transmits captured images acquired by the camera131and voices acquired by the microphone132to the guide terminal200and the participant terminals300in accordance with control by the control unit150. The communication unit170may transmit positional information detected by the GPS unit135to the guide terminal200and the participant terminals300and transmit map information accumulated in the map DB161to the guide terminal200and the participant terminals300. The communication unit170further receives images or voices sent from the guide terminal200and the participant terminals300and outputs these images or voices to the control unit150.

FIG.4is a block diagram showing a functional configuration example of the guide terminal200according to this embodiment. As shown inFIG.4, the guide terminal200includes, as functional blocks, a calculation unit220, the display panel201, an input unit202, a microphone203, a speaker204, a camera205, a memory230, and a communication unit240.

Main elements related to the remote operation of the robot100or the remote communication with the robot100and the participant terminals300will be described. However, the guide terminal200may include elements in its configuration other than the above one and may include additional elements that contribute to the processing for performing remote operation and remote communication.

The calculation unit (control unit)220, which is, for example, a CPU, executes control of the whole guide terminal200and various calculation processes by executing a control program read from the memory230. The display panel201, which is, for example, a liquid crystal panel, displays, for example, a captured image or map information sent from the robot100and/or the participant terminal300and images captured by the camera205.

The input unit202includes a touch panel disposed so as to be superimposed on the display panel201and a push button provided on a peripheral part of the display panel201. The input unit202generates an operation signal in response to an operation performed by the guide G, and outputs the generated operation signal to the calculation unit220.

The microphone203collects voices emitted near the guide terminal200in accordance with a request from the calculation unit220, and outputs the voice signal generated by voice collection to the calculation unit220. The speaker204outputs, for example, voices sent from the robot100and/or the participant terminal300. The camera205captures images of, for example, the guide G who is in front of the guide terminal200(in front of the display panel201) in accordance with a request from the calculation unit220, and outputs the imaging signal generated by imaging to the calculation unit220.

The memory230, which is a non-volatile storage medium, may be, for example, a solid-state drive. The memory230stores, besides a control program for controlling the guide terminal200, various parameter values, functions, lookup tables, and the like used for the control and the calculation.

The communication unit240, which is, for example, a wireless LAN unit, performs radio communication with the wireless router600. The communication unit240transmits or receives control signals, data or the like to or from the robot100and/or the participant terminals300via the wireless router600. For example, the communication unit240receives captured images or voice signals sent from the robot100and/or the participant terminal300, map information, or the like and outputs the received data to the calculation unit220. Further, the communication unit240cooperates with the calculation unit220and transmits designation signals, operation signals, captured images, voice signals or the like to the robot100and/or the participant terminal300.

The calculation unit220also serves as a functional calculation unit that executes various processes and calculation. The calculation unit220includes, for example, an interest analysis unit221and a display control unit222. The interest analysis unit221analyzes interest of participants P based on interest information sent from the participant terminals300. The display control unit222displays, for example, the result of analyzing interest of the participants P on the display panel201. The display control unit222displays information indicating the interest of the participants P superimposed on the captured image etc. displayed on the display panel201. It can also be said that the display control unit222and the display panel201, which cooperate with each other, configure an output unit that outputs the interest.

FIG.5shows a configuration example of the guide screen210displayed on the display panel201of the guide terminal200according to this embodiment. As shown inFIG.5, for example, the guide screen210includes a robot image display area211, a map display area212, a guide image display area213, a participant image display area214, a manipulation panel display area215, and an interest display area216. Note that the display form of each area shown inFIG.5is merely one example, and is not limited thereto.

The robot image display area211displays captured images around the robot100that have been captured and transmitted by the robot100. Further, the robot image display area211displays, for example, a heat map in accordance with the interest of the participants P in such a way that the heat map is superimposed on an image captured by the robot100.

The map display area212displays map information and positional information near the robot100transmitted by the robot100. The map display area212displays, for example, information on, for example, a tourist spot near the robot100, the current location, the moving path, the destination, and the orientation of the robot100, and the like.

The guide image display area213displays images (guide images) that have been captured by the camera205and mainly includes the face of the guide G (manipulator). The participant image display area214displays captured images (participant images) that have been captured and transmitted by the plurality of participant terminals300and mainly include the faces of the plurality of participants P.

The manipulation panel display area215displays a manipulation panel for manipulating (remotely operating) the robot100. The manipulation panel includes a button or the like for adjusting the moving direction or the moving speed of the robot100, the orientation of the camera and the like.

The interest display area216displays information regarding the interest of the participants P. The interest display area216displays, for example, a level meter indicating the level of interest and a graph indicating a temporal change of interest.

FIG.6is a block diagram showing a block configuration example of the participant terminal300according to this embodiment. As shown inFIG.6, the participant terminal300includes, as functional blocks, a calculation unit320, the display panel301, an input unit302, a microphone303, a speaker304, a camera305, a memory330, and a communication unit340.

Main elements related to the remote communication with the robot100, the guide terminal200, and other participant terminals300will be described below. However, the participant terminal300may include elements in its configuration other than the above ones and may include additional elements that contribute to the processing for performing remote communication.

The calculation unit (control unit)320is, for example, a CPU, and executes control of the whole participant terminal300and various calculation processes by executing a control program loaded from the memory330. The display panel301is, for example, a liquid crystal panel, and displays, for example, captured images sent from the robot100, the guide terminal200, or other participant terminals300, map information, and images captured by the camera305.

The input unit302includes a touch panel disposed so as to be superimposed on the display panel301and a push button provided on a peripheral part of the display panel301. The input unit302generates an operation signal in response to an operation by the participant P and outputs the operation signal to the calculation unit320.

The microphone303collects voices emitted near the participant terminal300in accordance with a request from the calculation unit320, and outputs the voice signal generated by voice sound collection to the calculation unit320. The speaker304outputs, for example, voices sent from the robot100, the guide terminal200, and other participant terminals300. The camera305captures images of, for example, a participant P who is in front of the participant terminal300(in front of the display panel301) in accordance with a request from the calculation unit320, and outputs the imaging signal generated by imaging to the calculation unit320.

The memory330, which is a non-volatile storage medium, may be, for example, a solid-state drive. The memory330stores, besides a control program for controlling the participant terminal300, various parameter values, functions, lookup tables and the like used for the control and the calculation.

The communication unit340, which is, for example, a wireless LAN unit, performs radio communication with the wireless router600. The communication unit340transmits or receives control signals, data or the like to or from the robot100, the guide terminal200, and other participant terminals300via the wireless router600. The communication unit340receives, for example, captured images or voice signals sent from the robot100, the guide terminal200, and the other participant terminals300, or the map information, and outputs the received data to the calculation unit320. Further, the communication unit340, which cooperates with the calculation unit320, transmits the captured images, the voice signals and the like to the guide terminal200and the other participant terminals300.

The calculation unit320also serves as a functional calculation unit that executes various processes and calculation. For example, the calculation unit320includes an interest detection unit321and an interest notification unit322. The interest detection unit321detects the interest of a participant P in accordance with an input operation or the like performed by the participant P. The interest notification unit322notifies the guide terminal200of the interest information indicating the detected interest via the communication unit340.

FIG.7shows a configuration example of a participant screen310displayed on the display panel301of the participant terminal300according to this embodiment. As shown inFIG.7, the participant screen310includes, for example, a robot image display area311, a map display area312, an own participant image display area313, an other participant image display area314, and a guide image display area315. Note that the display form in each area shown inFIG.7is merely one example, and is not limited to them.

The robot image display area311displays captured images near the robot100that have been captured and transmitted by the robot100. The map display area312displays the map information and the positional information near the robot100that have been transmitted from the robot100. The map display area312displays, for example, information on, for example, a tourist spot near the robot100, the current location, the moving path, the destination, the orientation of the robot100and the like.

The own participant image display area313displays an image (an own participant image) that has been captured by the camera305and mainly includes the face of the own participant P who operates the participant terminal300. The other participant image display area314displays images (other participant images) that have been captured and transmitted by the other participant terminals300and mainly include the faces of the other participants P who operate the other participant terminals300. The guide image display area315displays an image (a guide image) that has been captured and transmitted by the guide terminal200and mainly includes the face of the guide G.

FIG.8shows an operation example of the remote robot system10according to this embodiment. As shown inFIG.8, the guide terminal200transmits an operation signal for remotely operating the robot100to the robot100in accordance with the operation by the guide G (S101). In the guide terminal200, when the guide G operates the manipulation panel displayed in the manipulation panel display area215of the guide screen210, the calculation unit220generates an operation signal indicating control of moving, imaging, etc. in accordance with the operation by the guide G, and transmits the operation signal to the robot100via the communication unit240.

Next, the robot100performs a moving operation and an imaging operation in accordance with the operation signal (S102) and transmits the positional information and the captured images to the guide terminal200and the plurality of participant terminals300(S103). In the robot100, after the communication unit170receives the operation signal from the guide terminal200, the control unit150controls driving of the movable-base drive unit142and the upper-body drive unit143in accordance with the operation signal that has been received, and controls an imaging operation and the like of the camera131. The control unit150transmits the images captured by the camera131and the positional information (e.g., the latitude and the longitude) detected by the GPS unit135to the guide terminal200and the plurality of participant terminals300via the communication unit170. Further, the control unit150transmits map information, voices and the like near the robot100to the guide terminal200and the plurality of participant terminals300as necessary.

Note that the positional information and the captured images may be transmitted from the robot100to the guide terminal200and each of the plurality of participant terminals300, or the positional information and the captured images may be transmitted from the robot100to the guide terminal200and further forwarded from the guide terminal200to the plurality of participant terminals300.

Next, the guide terminal200and the plurality of participant terminals300output the information received from the robot100(S104). In the guide terminal200, when the communication unit240receives the positional information and the captured images from the robot100, the calculation unit220(the display control unit222) causes the robot image display area211of the guide screen210to display the received images captured by the robot100, and causes the map display area212to display positional information of the robot100that has been received. Further, when the guide terminal200has received map information from the robot100, the guide terminal200updates the map information in the map display area212. When the guide terminal200has received a voice signal, the guide terminal200causes the speaker204to output the voice.

Like in the guide terminal200, in the participant terminal300, when the communication unit340receives the positional information and the captured images from the robot100, the calculation unit320causes the robot image display area311of the participant screen310to display the received images captured by the robot100, and causes the map display area312to display positional information of the robot100that has been received. Further, when the participant terminal300has received map information from the robot100, the participant terminal300updates the map information in the map display area312. When the participant terminal300has received a voice signal, the participant terminal300causes the speaker304to output the voice.

Next, the plurality of participant terminals300detect the interest of each of the participants P in accordance with, for example, the operation by each participant P (S105), and transmits interest information indicating the interest that has been detected to the guide terminal200(S106). The interest detection unit321of the participant terminal300detects the interest of a participant P based on, for example, an operation state, an utterance state, an operation attention or the like of the participant P. The interest detection unit321of the participant terminal300may detect the interest of the participant P based on, for example, one of the operation state, the utterance state, the attention state and the like or based on any given states. After the interest detection unit321detects interest, the interest notification unit322transmits information on the interest (each state) that have been detected to the guide terminal200via the communication unit340.

For example, the interest detection unit321detects the operation state of the participant P in response to the display (output) of the participant screen310, and detects the interest of the participant P based on the operation state that has been detected. For example, the interest notification unit322may transmit interest information including the operation state that has been detected. The operation state to be detected includes operation elements such as an operation position, the number of times of operations, an operation time and the like in the operation by the participant P on the participant screen310that displays information on the robot100. One of the operation elements or any given operation elements may be used. The operation to be detected may be a touch operation on the touch panel of the display panel301or may be a mouse pointer movement or a clicking operation by a mouse operation.

When the operation position is detected, a specific position operated on the participant screen310may be detected, or a predetermined area including the operated position may be detected. For example, the participant screen310may be divided into some areas and a divided area including the operated position may be detected. The interest detection unit321detects the position or the area operated in the robot image display area311or the map display area312.

Further, when the number of times of operations is detected, the number of touches or the number of mouse clicks or the like in a predetermined period may be detected. When the operation time is detected, a touch duration time (long press time), a click duration time or the like may be detected. When, for example, the number of touches is larger or the touch duration time is longer than a predetermined threshold in a predetermined area, the interest detection unit321may detect that this area is of great interest (that the participant is excited) and transmit information indicating that the area of great interest has been detected.

Further, the interest detection unit321detects the utterance state of a participant P in response to a display on the participant screen310, and detects the interest of the participant P based on the utterance state that has been detected. The interest notification unit322may transmit interest information including the utterance state that has been detected. The utterance state to be detected includes utterance elements such as a sound pressure of the voice emitted by the participant P, an utterance time, and results of recognizing the voice (keyword). One of the utterance elements may be used or any given utterance elements may be used.

The interest detection unit321detects, for example, the sound pressure of the voice of a participant P by the microphone303. Then, the interest detection unit321may detect that the participant P is excited by detecting phrases like “Wow!” or “Terrific!”. When the utterance time is detected, a time during which a predetermined sound pressure continues may be detected. For example, the interest detection unit321may detect that the level of interest of the participant P has been increasing when the sound pressure of the participant P is larger than a predetermined threshold or a predetermined sound pressure continues for a predetermined period of time.

Further, the interest detection unit321may detect a predetermined voice by voice recognition. For example, voice recognition may be performed on a voice of a participant P detected by the microphone303. When the interest detection unit321has recognized predetermined phrases like “Wow!” or “Terrific!”, which indicate that the participant P is excited, the interest detection unit321may detect that the level of interest of the participant P has been increasing. In some embodiments, a predetermined keyword related to interest is recognized. This keyword may be the name of a building which is located near the robot100or a keyword related to the tourist spot. Further, the words to be detected regarding the interest (indicating that the participant is excited) may be negative keywords, not positive keywords. When the participant terminal300performs remote communication, the utterance state for the interest may be detected in a state in which the remote communication audio is muted.

Further, the interest detection unit321may detect the attention state of a participant P in response to a display on the participant screen310, and detect the interest of the participant P based on the attention state that has been detected. The interest notification unit322may transmit interest information including the attention state that has been detected. The attention state that has been detected includes attention elements such as the orientation of the face of the participant P, the orientation of the line of sight of the participant P, the attention position, and the attention time. One of the attention elements may be used or any given attention elements may be used.

The interest detection unit321detects, for example, the orientation of the face of a participant P by an image of the camera305, or detects the orientation of the line of sight of the participant P by a line-of-sight sensor provided in the participant terminal300. The interest detection unit321estimates the attention position (it may be an attention area) on the participant screen310from the orientation of the face of the participant P or the orientation of the line of sight of the participant P. When the attention time is detected, the interest detection unit321may detect time during which a predetermined orientation of the face, a predetermined orientation of the line of sight, or a predetermined attention position continues. The interest detection unit321may detect that the level of interest of the participant P in this position (orientation) has been increasing when, for example, the same orientation of face, the same orientation of the line of sight, or the same attention position continues for a predetermined period of time. Further, the interest detection unit321may detect, in addition to or in place of the orientation of the face of the participant P, facial expressions of the participant P by image recognition. For example, positive and negative facial expressions may be detected.

Next, the guide terminal200displays the interest information received from the plurality of participant terminals300(S107). In the guide terminal200, when the communication unit240receives the interest information from the plurality of participant terminals300, the interest analysis unit221analyzes the interest of the plurality of participants P based on the received interest information and the display control unit222displays the results of the analysis on the guide screen210. The interest analysis unit221aggregates or statistically processes the operation state, the utterance state, the attention state or the like of a participant P, detects the part of the screen which the participant P is focusing on, or detects a situation in which the participant P is excited, and displays the results of the detection in such a way that the guide G can recognize them. The statistical processing includes processing of obtaining a total value, an average value, a median value, a maximum value, a minimum value or the like, and includes statistics for each position (area) of the screen or statistics for each time. For example, the position of the guide screen200that corresponds to the interest information may be highlighted or a temporal change of the interest information may be displayed on the guide screen210that displays information on the robot100. As an example of highlighting the position, a heat map may be superimposed on the robot image display area211of the guide screen210. Further, as an example of displaying the temporal change, a level meter or a graph may be displayed in the interest display area216of the guide screen210. In some embodiments, they are displayed in an aspect in accordance with the degree (level) of interest indicated by the interest information. The degree of interest is, for example, a value obtained based on the number of times of operations or the operation time in the state of operation by a plurality of participants, the sound pressure or an utterance time in the utterance state, and an attention time in the attention state, and each element may be weighted. The results of statistics of all the participants may be collectively displayed or they may be displayed in such a manner that the results of statistics for each participant can be recognized. They may be displayed in such a way that the number of participants who show interest can be recognized.

FIG.9is an example of displaying results of analyzing the interest information by a heat map. As shown inFIG.9, the display control unit222displays, for example, the heat map in accordance with the interest information in the robot image display area211of the guide screen210. The heat map may be displayed on the captured image in the robot image display area211or may be displayed on the map information in the map display area212. Distribution of the interest is displayed in such a way that it is superimposed on parts of the captured image or the map information where the attention of the participant P is focused on (parts where the participant P who sees these parts is excited) so that the distribution can be visually recognized. For example, the area of great interest (interest is focused on) is displayed in red, and the heat map is displayed by using different shades and colors according to the level of attention (interest).

FIG.10is an example of displaying the results of analyzing the interest information by shapes such as circles. As shown inFIG.10, the display control unit222causes, for example, the robot image display area211of the guide screen210to display circular frames in accordance with the interest information. The circular frame may be displayed on a captured image in the robot image display area211or may be displayed on the map information in the map display area212. The circular frames are displayed so as to surround parts on the captured image or the map information where the attention of a participant P is focused on (parts where the participant P who sees these parts is excited). For example, an area of great interest (interest is focused on) is displayed by a red frame, and the frames are displayed by using different colors, sizes, or the thicknesses according to the level of attention (the level of interest). Note that the shape of the frames is not limited to circle and may be any other shape.

FIGS.11A and11Bare examples of displaying the results of analyzing the interest information by a level meter. As shown inFIGS.11A and11B, the display control unit222displays, for example, the level meter of the level in accordance with the interest information in the interest display area216of the guide screen210. The level meter displays the level (high/low) of the current interest (how excited a participant is). As shown inFIG.11A, when the current interest level is low, the level meter shows a low level. As shown inFIG.11B, when the current interest level is high, the level meter shows a high level. The level of the current interest may be displayed in the level meter in such a way that different levels are shown by different colors.

FIG.12is an example of displaying results of analyzing the interest information by a graph. As shown inFIG.12, the display control unit222displays, for example, the graph showing the level of the current interest in accordance with the interest information in the interest display area216of the guide screen210. The graph shows a temporal change regarding how excited a participant is. As shown inFIG.12, the level of interest is displayed in a line graph in time order to display a relative change regarding how excited the participant is. The graph is not limited to a line graph and may be a bar graph or the like. The graph may be displayed by using different colors according to the level and temporal change.

As shown inFIG.8, next, the guide terminal200transmits the operation signal for remotely operating the robot100to the robot100in accordance with the operation by the guide G, like in S101(S108). The guide G knows the interest of the participant P from the interest information displayed on the guide screen210and operates the manipulation panel of the manipulation panel display area215of the guide screen210in such a way that the robot100moves, for example, toward the place of great interest. The calculation unit220generates an operation signal in accordance with the operation by the guide G, and transmits the operation signal to the robot100via the communication unit240. After that, the robot100performs the moving operation, the imaging operation and the like in accordance with the operation signal, like in S102.

As described above, in this embodiment, a remote robot system that implements a remote sightseeing tour or the like includes a function of detecting interest of participants and a function of further presenting the detected results to the connection terminal of the guide. Accordingly, even when the guide and the participants are remotely away from each other, it is possible to send feedback information indicating which part the participants are focusing on and which part the participants are interested in to the guide. This enables the guide to explain the part the participants are interested in and enables to obtain detailed information by moving the robot. Accordingly, it is possible to provide a tour that meet the needs of participants while recognizing reactions of the participants (users), and thus respond appropriately according to participants.

Second Embodiment

Next, a second embodiment according to the present disclosure will be described. In this embodiment, an example of automatically controlling the operation of the robot in accordance with participants' interest in the remote robot system according to the first embodiment will be described.

FIG.13is a block diagram showing a functional configuration example of the guide terminal200according to this embodiment. As shown inFIG.13, the guide terminal200includes a control method determination unit223in the calculation unit220. The other configurations are similar to those in the first embodiment. The control method determination unit (remote control determination unit)223determines a method of controlling the robot100(remote operation method) based on interest information sent from the participant terminal300, that is, in accordance with interest of a participant P analyzed by the interest analysis unit221.

FIG.14shows an operation example of the remote robot system10according to this embodiment. InFIGS.14, S101to S107are similar to those in the first embodiment. After S107, the guide terminal200determines the method of controlling the robot100in accordance with interest information of a participant P (S110), and transmits an operation signal of the control method that has been determined to the robot100(S108). When the interest analysis unit221analyzes interest information of the participant P and the display control unit222displays the results of the analysis, the control method determination unit223determines the method of controlling the robot100in accordance with the interest information of the participant P that has been analyzed. Note that the display of the interest information may be omitted and the control method may be determined in S110.

The control method determination unit223determines, for example, a moving method of the robot100including the moving direction, the moving path, the moving speed and the like of the robot100. The control method determination unit223may automatically select a path of great interest from a plurality of tour paths in accordance with the interest information that has been analyzed. For example, when there are many participants P who are looking at a specific building, a path that passes near this building is selected. Further, the control method determination unit223may control the moving speed of the robot100in accordance with the interest information that has been analyzed. For example, the speed of the robot100may be adjusted depending on the magnitude of the degree of interest in such a way that the viewing time for areas of interest becomes longer and the viewing time for areas of less interest becomes shorter. The speed at which the robot100moves may be reduced or the robot100may stop moving in areas of interest.

Further, the control method determination unit223may determine an information acquisition method of the robot100including, for example, an imaging direction in which the robot100captures images or a voice collecting direction. The control method determination unit223may control the imaging direction of the camera131of the robot100in such a way that the camera131automatically faces the direction of interest in accordance with the interest information that has been analyzed, or may zoom in the imaging range of the camera131. Likewise, the control method determination unit223may control the collection direction of the microphone132in such a way that the microphone132faces the direction of interest.

The robot100may be automatically controlled by transmitting an operation signal in accordance with the control method that has been determined or may display the control method that has been determined on the guide screen210to recommend the moving path and the like of the robot100to the guide G. Further, the determination of the control method and the execution of the control may not be performed by the guide terminal200and may be performed by another apparatus.

As described above, in this embodiment, the method of controlling the robot is determined in accordance with participant's interest in a remote robot system. Accordingly, the robot can be automatically controlled in accordance with the participant's interest, whereby it is possible to respond appropriately according to participants.

Note that the present disclosure is not limited to the above embodiments and may be changed as appropriate without departing from the spirit of the present disclosure. For example, while the example in which the remote robot system is used for a remote sightseeing tour has been described in the above embodiments, the remote robot system may be used for applications other than the remote sightseeing tour. This remote robot system may be used, for example, to remotely search for missing persons such as lost children or to remotely check the states of family members. The above embodiments can be efficiently used when multiple remote users remotely operate a robot and don't know where to focus on when remotely controlling the robot.

Each of the configurations in the above embodiments may be constructed by software, hardware, or both of them. Further, each of the configurations may be formed by one hardware device or one software program, or a plurality of hardware devices or a plurality of software programs. The function (the process) of each apparatus may be implemented by a computer including a CPU, a memory and the like. For example, a program for performing a method (control method) according to the embodiments may be stored in a storage device, and each function may be implemented by having the CPU execute the program stored in the storage device.

The program includes instructions (or software codes) that, when loaded into a computer, cause the computer to perform one or more of the functions described in the embodiments. The program may be stored in a non-transitory computer readable medium or a tangible storage medium. By way of example, and not limitation, non-transitory computer readable media or tangible storage media can include a random-access memory (RAM), a read-only memory (ROM), a flash memory, a solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray (Registered Trademark) disc or other optical disc storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may be transmitted on a transitory computer readable medium or a communication medium. By way of example, and not limitation, transitory computer readable media or communication media can include electrical, optical, acoustical, or other form of propagated signals.