Robot device, method of controlling robot device, and program

There is provided a robot device including an instruction acquisition unit that acquires an order for encouraging a robot device to establish joint attention on a target from a user, a position/posture estimation unit that estimates a position and posture of an optical indication device, which is operated by the user to indicate the target by irradiation of a beam, in response to acquisition of the order, and a target specifying unit that specifies a direction of the target indicated by irradiation of the beam based on an estimation result of the position and posture and specifies the target on an environment map representing a surrounding environment based on a specifying result of the direction.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a robot device, a method of controlling the robot device, and a program.

2. Description of the Related Art

In recent years, robot devices (hereinafter, referred to as “robots”) that behave together with users and act according to the users' orders have been in widespread use. In order for the robot to pay attention to the same target (an object or position) as the user does, it may be required to establish joint attention.

For example, when the user indicates a target and orders an action to the target, the target indicated by the user must be identical to a target that the robot specifies according to the user's order. In this case, although it is necessary to establish joint attention on the target between the user and the robot, it may be difficult to establish joint attention with certainty.

In the past, laser pointer recognition, gesture recognition, a touch panel operation, an acceleration sensor, and the like have been used for establishing joint attention on the target.

For example, Japanese Patent Application Laid-Open (JP-A) No. 2005-349497 discloses a control technique for controlling a robot using a remote controller in association with an acceleration sensor. In the control technique, if the user indicates a target through a pointer of the remote controller, the remote controller calculates the position indicated by the pointer based on a detection result of the acceleration sensor and transmits the calculation result to the robot. As a result, the robot can specify the position indicated by the pointer as the next movement position based on the calculation result.

SUMMARY OF THE INVENTION

In the case of using the laser pointer recognition, the user indicates a target through a high intensity laser and the robot performs recognition by detecting a high brightness point or area. In this case, if a dedicated filter is not mounted in a camera, detection robustness deteriorates due to influence of a lighting environment. However, if the dedicated filter is mounted, the camera cannot be used for any purpose other than detection of the high intensity laser. Further, in the case of using only the laser pointer recognition, if any high brightness point or area other than the target indicated by the user is present, a degree of accuracy of recognition cannot be sufficiently secured.

Further, in the case of using the gesture recognition, although the camera can be used for purposes other than the gesture recognition, the gesture recognition cannot sufficiently secure a degree of accuracy of recognition. Further, a target considered to be indicated by the user may be different from a target objectively recognized as being indicated by the user, leading to a difference in intention.

Further, in the case of using the touch panel operation, mapping between an actual space around the robot and a display space on a touch panel is necessary, and an interface is unnatural. Further, if the display resolution is low, a degree of accuracy for indicating a target may decrease, whereas if the resolution is high, it may become difficult to perform mapping.

Further, in the case of using the acceleration sensor, in the control technique disclosed in JP-A No. 2005-349497, if the acceleration sensor is not frequently reset, a detection error may be accumulated, and thus it may be difficult to specify the next movement position with a high degree of accuracy.

As described above, it was difficult to establish joint attention on a target between the robot and the user with certainty. Thus, in light of the foregoing, it is desirable to provide a robot device, a method of controlling the robot device, and a program in which establishment of joint attention on a target between the user and the robot can be supported.

According to an embodiment of the present invention, there is provide a robot device, including an instruction acquisition unit that acquires an order for encouraging a robot device to establish joint attention on a target from a user, a position/posture estimation unit that estimates a position and posture of an optical indication device, which is operated by the user to indicate the target by irradiation of a beam, in response to acquisition of the order, and a target specifying unit that specifies a direction of the target indicated by irradiation of the beam based on an estimation result of the position and posture and specifies the target on an environment map representing a surrounding environment based on a specifying result of the direction.

The robot device may further include an interaction processing unit that confirms establishment of interaction performed on establishment of joint attention with the user.

The interaction processing unit may confirm establishment of the interaction after the order is acquired.

The interaction processing unit may confirm establishment of the interaction before the order is acquired.

The position/posture estimation unit may estimate the position and posture of the optical indication device by recognizing the optical indication device gripped by the user.

According to an embodiment of the present invention, there is provide a method of controlling a robot device, including the steps of acquiring an order for encouraging establishment of joint attention with a robot device on a target from a user, estimating a position and posture of an optical indication device, which is operated by the user to indicate the target by irradiation of a beam, in response to acquisition of the order, and specifying a direction of the target indicated by irradiation of the beam based on an estimation result of the position and posture and specifying the target on an environment map representing a surrounding environment based on a specifying result of the direction.

According to another embodiment of the present invention, there is provided a program for causing a computer to execute the method of controlling the robot device. Here, the program may be provided by using a computer-readable recording medium and may be provided via communication means.

As described above, according to the present invention, it is possible to provide a robot device, a method of controlling the robot device, and a program in which establishment of joint attention on a target between the user and the robot can be supported.

DETAILED DESCRIPTION OF THE EMBODIMENT(S)

In this disclosure, exemplary embodiments of the present invention will be described according to the following items:

1. A configuration of a robot system

2. A configuration of a robot

3. A configuration of a pointer

4. A control method of a robot

5. An example of interaction

1. A CONFIGURATION OF A ROBOT SYSTEM

First, a configuration of a robot system according to an exemplary embodiment of the present invention will be described with reference toFIG. 1.FIG. 1illustrates a configuration of a robot system.

As illustrated inFIG. 1, in the robot system, establishment of joint attention on a target O between a user10and a robot device100(hereinafter, also referred to as a “robot100”) is supported using a pointer200(an optical indication device) such as a laser pointer.

For example, the robot100is a two-legged robot that supports human activities in various scenes in daily life. The robot100can act corresponding to an internal state (happiness, anger, sorrow, pleasure, etc.) and express a basic action done by a human being.

The robot100includes an image input device121, a voice input device122, and a voice output device123which are built therein as illustrated inFIG. 3. The robot100receives surrounding image data through the image input device121, receives voice data through the voice input device122, and outputs voice data through the voice output device123. Further, the robot100further includes a communication interface (I/F)124built therein as illustrated inFIG. 3. Thus, the robot100receives an order command C output from the pointer200or the like in response to the user's operation and transmits a response to the pointer200or the like.

The pointer200is used to indicate the target O by irradiation of a beam L with high directivity. The pointer200preferably has sufficient light intensity for the user10to view a bright point P of the beam L irradiated to the target O, which needs not be detectable by the robot100. The light of the pointer200is not used for the robot100to recognize the target O in which joint attention is to be established, but rather for the user10to confirm the target O.

If the user10desires to establish joint attention on the target O with the robot100, the user10gives an order (a joint attention order) command C for encouraging establishment of joint attention to the robot100and operates the pointer200to indicate the target O by irradiating the beam L. The robot100acquires the order command C and estimates the position and posture of the pointer200operated by the user10in response to the order command C. The robot100specifies a direction of the target O (an indication direction) indicated by irradiation of the beam L based on the estimation result of the position and posture and specifies the target O on an environment map EM representing a surrounding environment based on the estimation result of the indication direction.

Here, establishment of interaction I, which is made with regard to establishment of joint attention based on a recognition result of an image or a voice or a detection result of the order command C, may be confirmed between the user10and the robot100.

2. A CONFIGURATION OF THE ROBOT100

Next, the robot100according to an exemplary embodiment of the present invention will be described with reference toFIGS. 2 to 4.FIG. 2illustrates an appearance of the robot100, andFIG. 3illustrates a configuration of a control system of the robot100.FIG. 4illustrates a major functional configuration of the robot100.

As illustrated inFIG. 2, the robot100is constituted such that a head section101, a pair of right and left arm sections103L and103R, and a pair of right and left leg sections104L and104R are coupled to predetermined positions of the body trunk unit102, respectively.

As illustrated inFIG. 3, the robot100includes a control system disposed, for example, in the body trunk unit102. The control system includes a thought control module110that deals with an emotional judgment or the expression of feelings in response to the user's input actively and a motion control module130that controls whole-body cooperative motion of the robot100such as driving of an actuator140.

The thought control module110includes a central processing unit (CPU)111that executes arithmetic processing related to the emotional judgment or the expression of feelings, a random access memory (RAM)112, a read only memory (ROM)113, an external storage device114, and the like. The thought control module110decides a current feeling or thought of the robot100in response to a stimulus from the outside world such as the image data input from the image input device121, the voice data input from the voice input device122, and the order command C input from the communication I/F124. The thought control module110transfers the order to the motion control module130to execute an action or a behavior that is based on decision-making.

The motion control module130includes a CPU131that controls whole-body cooperative motion of the robot100, a RAM132, a ROM133, an external storage device134, and the like. For example, a walking pattern calculated off line, a target zero moment point (ZMP) orbit, and other action patterns are accumulated in the external storage device134.

Various kinds of devices such as the actuator140, a distance measurement sensor (not shown), a posture sensor141, ground confirmation sensors142L and142R, a load sensor (not shown), and a power control device143are connected to the motion control module130through a bus interface (I/F)135. The actuator140implements movement of joints of the robot100. The distance measurement sensor measures the distance between the user10and the target O. The posture sensor141is used to measure the posture or gradient of the body trunk unit102. The ground confirmation sensors142L and142R detect the soles of the right and left feet departing from the floor or landing on the floor, respectively. The load sensor detects a load that acts on the soles of the right and left feet. The power control device143is used to manage power, for example, battery power.

The thought control module110and the motion control module130are constructed on a common platform and are connected with each other through the bus I/Fs115and135.

The motion control module130controls the whole-body cooperative motion through each actuator140so that a behavior instructed from the thought control module110can be implemented. That is, the CPU131reads out an action pattern corresponding to a behavior instructed from the thought control module110from the external storage device134or generates an action pattern internally. The CPU131sets motion of the foot, a ZMP orbit, motion of the body trunk, motion of the upper limb, and the horizontal position or height of the lumbar part according to the instructed action pattern. The CPU131transfers orders for instructing actions according to the set content to each actuator140.

Further, by detecting the posture or gradient of the body trunk unit102of the robot100based on an output signal of the posture sensor141and detecting in which state of a standing leg or an idling leg each of the leg sections104L and104R is based on an output signal from each of the ground confirmation sensors142L and142R, the CPU131adaptively controls the whole-body cooperative motion of the robot100. Further, the CPU131controls the posture or motion of the robot100so that the ZMP position can be directed to the center of a ZMP stable region.

Further, the motion control module130provides feedback on to what degree an intended behavior decided by the thought control module110has been implemented, that is, a processing status, back to the thought control module110. As described above, the robot100can autonomously act by judging its own status and a surrounding status based on a control program. Further, the robot100executes a control method for supporting establishment of joint attention, which will be described later, based on a control program.

As illustrated inFIG. 4, the robot100includes a variety of functional configurations for establishing joint attention on the target O with the user10. A major functional configuration of the robot100includes a command reception unit151, an image input unit152, a voice input unit153, a voice output unit154, an image recognition unit155, a voice recognition unit156, an environment map creation unit157, an interaction processing unit158, a position/posture estimation unit159, a target specifying unit160, an environment map storage unit161, and a feature quantity storage unit162.

Further, the command reception unit151, the image input unit152, the voice input unit153, and the voice output unit154correspond to the communication I/F124, the image input unit121, the voice input unit122, and the voice output unit123, respectively as shown inFIG. 3. The other functional configurations are implemented, for example, by the CPU111, the RAM112, the external storage device114, and the like of the thought control module110.

The command reception unit151receives the order command C including the joint attention order from the pointer200in response to an operation on the pointer200in a wired or wireless manner. The command reception unit151includes, for example, one or more receiving devices (not shown) disposed in the head section101. If the order command C is a radio signal, a plurality of receiving devices are disposed in different directions, and a rough direction in which the user10is positioned may be estimated based on a difference of a received signal intensity between the receiving devices. The command reception unit151functions as an instruction acquisition unit.

The image input unit152receives an image around the robot100as image data and supplies the image data to the image recognition unit155and the environment map creation unit157. The image data includes data representing the user10or an object positioned around the robot100and data representing the position and posture of the pointer200. Further, the image data includes data representing the user's expression or gesture used to confirm establishment of the interaction I.

The voice input unit153receives a voice around the robot100as voice data and supplies the voice data to the voice recognition unit156. The voice data includes data of a voice uttered by the user10or the object positioned around the robot100. Further, the voice data includes data representing a voice order from the user10used to confirm establishment of the interaction I.

The voice output unit154outputs a voice to the outside in response to predetermined voice data. The voice data includes data representing a voice response used to confirm establishment of the interaction I. Further, the voice response is generated based on data stored in the external storage device114or the like.

The image recognition unit155recognizes the user10or the object positioned around the robot100, the position and posture of the pointer200, and the user's expression or gesture based on a feature quantity included in the image data. Further, the feature quantities of the user10or the object, the pointer200, the user's expression or gesture, and the like are previously stored in the feature quantity storage unit162.

The voice recognition unit156recognizes the voice uttered by the user100or the object positioned around the robot100and the voice order from the user100based on a feature quantity included in the voice data. Further, the feature quantity of the voice order is also previously stored in the feature quantity storage unit162.

The environment map creation unit157recognizes an environment of an object, a step difference, or the like through a plurality of planes extracted from distance information obtained by a stereo vision or the like to create the environment map EM and stores the environment map EM in the environment map storage unit161. The environment map creation unit157updates the environment map EM and supplies the environment map EM to the position/posture estimation unit159. The stereo vision is obtained by subjecting images, which are input from two cameras (not shown) that constitute the image input unit152, to stereo image processing.

The environment map creation unit157obverses the outside world through a stereo vision system and outputs stereo data that is three-dimensional distance information calculated by a parallax image of a stereoscopic camera as an image. That is, image inputs from two left and right cameras corresponding to both eyes of a human being are compared near each pixel, the distance to the target O from the parallax is estimated, and three-dimensional distance information is output as the image.

The environment map creation unit157recognizes a plurality of planes present inside the environment by detecting a plane from stereo data through a plane detector (not shown). Further, the environment map creation unit157creates the environment map EM in which the surrounding environment acquired from plane data by the stereo vision system is divided in a horizontal direction or a vertical direction in the form of a grid of a predetermined size. The environment map creation unit157further updates the environment map EM based on the stereo data and plane data and creates the environment map EM by referring to an obstacle presence possibility of a three-dimensional grid.

The interaction processing unit158confirms establishment of the interaction I made with regard to establishment of joint attention with the user based on the recognition result of the image or the voice or the detection result of the order command C. Further, the details of an interaction process will be described later. The interaction processing unit158functions as an interaction processing unit in cooperation with the image recognition unit155, the voice recognition unit156, and the like.

The position/posture estimation unit159estimates the position and posture of the pointer200operated by the user10and supplies the estimation result to the target specifying unit160. Based on the estimation result of the position and posture of the pointer200, the target specifying unit160specifies a direction of the target O indicated by the pointer200and specifies the target O on the environment map EM. Further, the details of a process of estimating the position and posture of the pointer200and a process of specifying the target O will be described later.

3. A CONFIGURATION OF THE POINTER200

Next, the pointer200according to an exemplary embodiment of the present invention will be described with reference toFIGS. 5 and 6.FIG. 5illustrates an appearance of the pointer200, andFIG. 6illustrates a major function configuration of the pointer200.

As illustrated inFIG. 5, the pointer200is an optical indication device such as a laser pointer that indicates the target O by irradiation of the beam L. The pointer200includes an irradiation hole201which is disposed a front end thereof and through which the beam L such as laser light is irradiated, a beam irradiation change-over switch202, an order command C operation switch203, a lighting device204such as a light emitting device. The pointer200has a size that is grippable by the user10. InFIG. 5, the pointer200has a flat rectangular shape but may have any other shape.

As illustrated inFIG. 6, the pointer200includes an irradiation operation unit251, a beam irradiation unit252, a lighting unit253, a command operation unit254, a command transmission unit255, and an antenna256.

The irradiation operation unit251is an operation unit such as the change-over switch202and switches ON/OFF of beam irradiation in response to the user's operation. The beam irradiation unit252irradiates the beam L in a certain direction from the irradiation hole201of the pointer200when beam irradiation is in an ON state. The lighting unit253is a light emitting device such as a light emitting diode (LED) and lights up when beam irradiation is in an on state. The lighting unit253may be configured to light up according to a predetermined blinking pattern. The lighting unit253may be disposed in the center of the pointer200or may be dispersedly disposed in each end section

The command operation unit254is an operation unit such as the operation switch203and transmits a predetermined order command C to the robot100in response to the user's operation. The order command C includes an order command C for encouraging establishment of joint attention or an order command C used for confirming establishment of the interaction I. The command transmission unit255transmits the order command C corresponding to a command input to the robot100through the antenna256. Hereinafter, it is assumed that the order command C is transmitted in a radio manner, but the order command C may be transmitted in a wired manner.

The user10turns on beam irradiation while turning the irradiation hole201of the pointer200in a direction of the target O for which joint attention is established. The pointer200irradiates the beam L to the target O and indicates the target O through the bright point P of the beam L. Further, the pointer200lights up according to a predetermined blinking pattern or the like.

Here, the beam L preferably has the light intensity in which the bright point P of the beam L is visible by the user10, and need not be detected by the robot100. Meanwhile, the light emitting diode preferably has the light intensity in which the predetermined blinking pattern can be detected by the robot100. The beam L is used for the user10to confirm the target O for which joint attention is established, and the light emitting device is used for the robot100to estimate the position and posture of the pointer200.

Further, the lighting unit253is disposed to improve a degree of accuracy of estimation of the spatial position and posture of the pointer200. For this reason, in order to improve the degree of accuracy of estimation, instead of the lighting unit253or together with the lighting unit253, the point200may be configured to have a characteristic shape, pattern, or color. Alternatively, random dot patterns may be formed on the entire pointer200. Here, by detecting features of three or more points dispersed in various positions of the pointer200, the spatial position and posture of the pointer200can be estimated.

4. A CONTROL METHOD OF THE ROBOT100

Next, a procedure of a first control method of the robot100will be described with reference toFIG. 7andFIGS. 8A to 8F.FIG. 7illustrates the procedure of the first control method of the robot100, andFIGS. 8A to 8Fillustrate the control method illustrated inFIG. 7.

As illustrated inFIG. 7, the robot100performs an autonomous action or a standby action under control of the thought control module110and the motion control module130(step S101). The autonomous action refers to an action performed autonomously in response to a surrounding external state or an internal state of the robot100. The standby action refers to an action for waiting for an order from the user10.

As illustrated inFIG. 8A, the user10gives the joint attention order for encouraging establishment of joint attention to the robot100and visualizes the robot100(step S103and step S105). Further, the action for giving the joint attention order (step S103) and the action for visualizing the robot100(step S105) may be performed in reverse order or in parallel.

The joint attention order is performed by a predetermined operation on the pointer200. In this case, the order command C representing the joint attention order is transmitted from the pointer200to the robot100in response to the predetermined operation. Further, the joint attention order may be performed, for example, by a predetermined operation on an operation input device other than the pointer200or a predetermined voice order on the voice input unit153. Further, the joint attention order may be performed at the same time as indication of the target O which will be described later.

Meanwhile, the robot100confirms whether or not the joint attention order is present based on a reception status of the order command C by the command reception unit151during the autonomous action or the standby action (step S107). Then, when the joint attention order is confirmed, as illustrated inFIG. 8B, the robot100turns in a direction of the user10by an action M1(step S109).

Specifically, the robot100adjusts a shooting direction (for example, face portion) the input unit152to a direction of the user10, particularly, a direction capable of detecting the pointer200operated by the user's operation. As a result, the robot100turns in the direction of the user10, a sense of interaction is expressed, and subsequent establishment of joint attention can be performed in a natural status.

The shooting direction is specified by detecting the feature quantity of the user10or the pointer200from the image data input to the image input unit152and recognizing the user10or the pointer200. The shooting direction is adjusted by rotating the whole body of the robot100or only the head section101through the motion control module130. Further, the feature quantity of the user10or the pointer200may be detected while rotating the position of the whole body of the robot100or the head section101.

If the user10and the robot100face each other, as illustrated inFIG. 8C, the robot100confirms whether or not establishment of the interaction I is present (step S111). Establishment of the interaction I will be described in detail later, but is confirmed based on the recognition result of the image or the voice or the detection result of the order command C. The robot100performs a subsequent process (step S113) when establishment of the interaction I is confirmed within a predetermined time, but returns to the autonomous action or the standby action (step S101) when establishment of the interaction I is not confirmed within a predetermined time.

Here, since the user10and the robot100face each other, the robot100can confirm that the joint attention order has been given by confirming establishment of the interaction I. For this reason, compared to a case in which there is no confirmation for establishment of the interaction I is, an erroneous action caused by error detection of the joint attention order can be prevented.

The robot100estimates the position and posture of the pointer200operated by the user10(step S113). Here, it is assumed that at a point in time when the position and posture of the pointer200are estimated, as illustrated inFIG. 8D, the user10operates the pointer200to irradiate the beam L and indicates the target O.

The target O may be indicated as a point represented by the bright point P of the stopped beam L or as an area represented by the bright point P of the beam L that moves in a predetermined range. In the former case, the point represented by the bright point P is specified as the target O, and in the latter case, the range represented by the bright point P is specified as the target O. The user10can confirm the indicated target O by visualizing the bright point P of the beam L. The robot100specifies the target O based on the position and posture of the pointer200other than the bright point P of the beam L.

The position and posture of the pointer200are estimated by detecting the feature quantity of the pointer200from the image data input to the image input unit152. Particularly, the position and posture of the pointer200may be estimated with a high degree of accuracy by detecting the lighting status of the light emitting device disposed in the pointer200, and the shape, the color, or the pattern of the pointer200as the feature quantity. Here, it is assumed that the feature quantity of the pointer200is previously stored in the feature quantity storage unit162or the like.

Further, the position and posture of the pointer200may be estimated using an object recognition technique proposed by the applicant in Japanese Patent Application Laid-Open (JP-A) No. 2004-326693 or a technique such as stereo recognition or Hough transform. Further, the position and posture of the pointer200may be estimated using detection data of an acceleration sensor or a magnetic sensor (not shown) disposed in the pointer200instead of the image data or together with the image data.

Particularly, when the robot100is near the user10, although the target O in which joint attention is established is positioned in the distance, the position and posture of the pointer200may be estimated with a high degree of accuracy and thus the target O can be specified with a high degree of accuracy. Further, when the robot100is near the user10, since the position and posture of the pointer200may be estimated with a high degree of accuracy, the user10can easily perform a means for improving a degree of accuracy of estimation in response to a request from the robot100or voluntarily. For example, the user10can change the posture of the pointer200so that the pointer200can be easily recognized in a state in which a direction of the target O is indicated.

When the position and posture of the pointer200are estimated, the robot100specifies a direction of the target O (the indication direction) indicated by irradiation of the beam L based on the estimation result (step S115). That is, the indication direction is specified based on recognition of the position and posture of the pointer200other than recognition of the bright point P of the beam L irradiated to the target O. The indication direction is specified by estimating the position and direction of the irradiation hole201based on the position and posture of the pointer200. Further, the position and direction of the irradiation hole201on the pointer200are set as information previously known to the robot100.

When the direction of the target O is specified, the robot100specifies the target O on the environment map EM based on the specifying result of the indication direction (step S117). The target O is specified, for example, as a specific object or position by specifying the position and posture of the pointer200on the environment map EM and further checking the indication direction and data on the environment map EM.

FIG. 8Eillustrates the specifying result of the target O on the environment map EM. The target O is specified on the environment map EM using three-dimensional coordinates. The position and posture of the pointer200are estimated, for example, by detecting three or more feature quantities present in the pointer200from the image data. As a result, the position and posture of the pointer200are estimated using the three-dimensional coordinates on the environment map EM.

The indication direction is specified by estimating the position and direction of the irradiation hole201from the position and posture of the pointer200. As a result, the position and direction of the irradiation hole201are estimated using the three-dimensional coordinates on the environment map EM. Further, the indication direction is specified as a parameter (three-dimensional angle information, etc.) that defines a straight line extending in a direction that the irradiation hole201faces starting from the position Pp of the irradiation hole201as a basic point. The target O is specified as the object or the position present on the straight line in the front side from the pointer200side on the environment map EM.

When the target O is specified, as illustrated inFIG. 8F, the robot100starts a predetermined action on the specified target O (step S119). For example, the robot100turns in the direction of the target O by an action M2illustrated inFIG. 8Dand moves in the direction of the target O by an action M3illustrated inFIG. 8F. Then, when the predetermined action is completed (step S121), the robot100returns to the autonomous action or the standby action (step S101).

As a result, establishment of joint attention on the target O between the user10and the robot100can be supported. Further, the process of confirming establishment of the interaction I (step S111), the process of estimating the position and posture of the pointer200(step S113), and the process of specifying the direction of the target O (step S115) may be performed in reverse order or in parallel. At this time, when establishment of the interaction I is confirmed after the position and posture of the pointer200are estimated, since estimation of the position and posture has been completed, establishment of joint attention can be supported with certainty.

Next, a second control method of the robot100will be described with reference toFIG. 9. Further, duplicate description with the first control method will be omitted.

The robot100performs the autonomous action or the standby action (step S151). When the user10visualizes the robot100(step S153), it is confirmed whether or not establishment of the interaction I is present (step S157). When the robot100turns in the direction of the user10(step S155), establishment of the interaction I is confirmed (Yes in step S157). Further, when the robot100turns in the direction of the user10(step S155), it is confirmed whether or not establishment of the interaction I is present (step S159). When the user10visualizes the robot100(step S153), establishment of the interaction I is confirmed (Yes in step S159).

If at least one of the robot100and the user10confirms establishment of the interaction I (step S157and step S159), the robot100confirms whether or not the joint attention order is present (step S163). The robot100performs a subsequent process (step S165) when the joint attention order is confirmed but returns to the autonomous action or the standby action (step S151) when the joint attention order is not confirmed.

Then, when the joint attention order is confirmed, the robot100estimates the position and posture of the pointer200operated by the user10(step S165). Here, it is assumed that at a point in time when the position and posture of the pointer200are estimated, the user10operates the pointer200to irradiate the beam L and indicates the target O.

When the position and posture of the pointer200are estimated, the robot100specifies the direction of the target O (the indication direction) indicated by irradiation of the beam L based on the estimation result (step S167). When the direction of the target O is specified, the robot100specifies the target O on the environment map EM based on the specifying result of the indication direction (step S169).

Then, the robot100starts a predetermined action on the specified target O (step S171). When the predetermined action is completed (Yes in step S173), the robot100returns to the autonomous action or the standby action (step S151).

As a result, establishment of joint attention on the target O between the user10and the robot100can be supported. Further, the process of confirming establishment of the interaction I (step S159), the process of estimating the position and posture of the pointer200(step S165), and the process of specifying the direction of the target O (step S167) may be performed in reverse order or in parallel.

In the second control method, since the presence of the joint attention order is confirmed after establishment of the interaction I is confirmed, the joint attention order can be confirmed with a higher degree of certainty than in the first control method.

5. AN EXAMPLE OF THE INTERACTION I

Next, an example of the interaction I performed on establishment of joint attention between the user10and the robot100will be described with reference toFIGS. 10A to 10G. For example, the interaction I is performed using the recognition result of the image or the voice, the detection result of the order command C, and a combination thereof.

In the case of using user face recognition, as illustrated inFIG. 10A, the robot100may detect the user's facial feature C1from the image data of the image input unit152and recognize the facial feature C1as the user's face. In this case, when the user's face is recognized, the robot100performs the action M1for adjusting the shooting direction of the image input unit152to a direction of the user's face. The user10can confirm establishment of the interaction I by confirming the action M1. Further, the user's face may be recognized by detecting a feature of the front or the side of the user's face or may be recognized by detecting a feature of the user's whole body and then estimating the position of the face.

Further, in order to ensure establishment of the interaction I, as illustrated inFIG. 10B, the robot100may detect the user's facial feature C2from the image data and estimate a direction of the face or a direction of a sight line. Further, when the user's facial feature C1is difficult to detect from the image data, the robot100may encourage the user10to turn his or her face toward itself using a visual or auditory means (for example, a voice V1“look this way”).

Further, in order to ensure establishment of the interaction I, the robot100may confirm establishment of the interaction I by detecting that the user10has performed a predetermined action in a state facing the user10.

For example, as illustrated inFIG. 10C, the robot100may detect the user's facial features C3and C4from the image data and detect the user's nod M3or wink. Further, as illustrated inFIG. 10D, the robot100may detect an issue of a predetermined voice order (for example, a voice V2“go” and “fetch”) or a predetermined gesture (for example, an action “go” and “fetch”). Further, the robot100may detect a response “OK” from the user10, as illustrated inFIG. 10E, after the robot100outputs a voice “I will get it” or detect that there is no response within a predetermined time (tacit consent).

In the case in which the user face recognition is not used, the robot100outputs a voice V3“I will go” and “fetch” after specifying the target O as illustrated inFIG. 10F. The robot100confirms whether or not there is a response from the user10within a predetermined time. If a voice V4“go” and “fetch” is confirmed within a predetermined time, the robot100may start a predetermined action on the specified target O. In this case, the robot100may detect a predetermined operation on the pointer200or a predetermined voice order as a response.

Further, as illustrated inFIG. 10Gthe robot100may encourage the user10to perform an action M6turning a sight line toward itself by lighting up M5the light emitting device or the like disposed in a part of the body after specifying the target O. The robot100confirms whether or not there is a response from the user10within a predetermined time. If a response is confirmed, the robot100may start a predetermined action on the specified target O. In this case, the robot100may detect a predetermined operation on the pointer200, a predetermined voice order, or the like as a response.

Further, in order to improve a degree of accuracy for specifying the target O, the processing result may be fed back from the robot100to the user10, thereby changing an operation of the pointer200in response to the processing result by the robot100.

For example, the robot100estimates the position and posture of the pointer200, calculates an error accompanied with the estimation result, and transmits the error to the pointer200. The pointer200judges that a degree of accuracy of estimation is sufficient when the estimation error is less than a threshold value but judges that the degree of accuracy of estimation is not sufficient when the estimation error is equal to or more than the threshold value. Further, a judgment on the degree of accuracy of estimation may be performed by the robot100instead of the pointer200.

The pointer200changes an aspect of the beam L irradiated to the target O such as directivity or color according to the judgment result. For example, as the aspect of the beam L, as the degree of accuracy of estimation is higher, the directivity may be stronger, the color may be deeper, and/or the beam L may be continuously irradiated. However, as the degree of accuracy of estimation is lower, the directivity may be weaker, the color may be thinner, and/or the beam L may be intermittently irradiated.

The user10can confirm the error accompanied with the estimation result by visualizing an aspect of the beam L irradiated to the target O. When it is judged that the degree of accuracy of estimation is sufficient, the user10performs, for example, a response “OK.” As a result, the robot100can specify the target O in a state in which the sufficient degree of accuracy of estimation is confirmed by the user10.

However, when it is judged that the degree of accuracy of estimation is not sufficient, the user10adjusts, for example, the posture of the pointer200in a state in which the target O is indicated. At this time, when, for example, the posture of the pointer200is adjusted, the robot100estimates the position and posture of the pointer200, and the pointer200changes an aspect of the beam L according to the judgment result. As a result, the robot100can improve the degree of accuracy for specifying the target O based on the feedback of the processing result.

As described above, according to the control method of the robot100according to an exemplary embodiment of the present invention, since the target O is indicated by irradiation of the beam L using the pointer200, the user10can confirm the indicated target O. Further, since the target O is specified by estimating the position and posture of the pointer200, the robot100can specify the indicated target O without detecting irradiation of the beam L. Further, since the robot100turns in the direction of the user10in order to recognize the pointer200operated by the user10, a sense of interaction can be expressed between the user10and the robot100. As a result, establishment of joint attention can be supported between the user10and the robot100on the target O.

As described above, the exemplary embodiments of the present invention have been described hereinbefore in detail with reference to the accompanying drawings, but the present invention is not limited to the embodiments. A person having ordinary skill in the art would understand that various modifications or variations can be made within the scope of the technical spirit defined in the claims and included within the technical scope of the invention.

The present application contains subject matter related to that disclosed in Japanese Priority Patent Application JP 2010-82047 filed in the Japan Patent Office on Mar. 31, 2010, the entire content of which is hereby incorporated by reference.