Collaborative operation support device

The collaborative operation support device includes a display device including a display area; and a processor configured to detect, based on an image in which the operator or the robot is represented, a position of a section of the robot in the display area when the operator looks at the robot through the display area, the section associated with an operation mode of the robot specified by means of an input device; select, in accordance with the specified operation mode of the robot, display data corresponding to the specified mode among display data stored in a memory; and display the selected display data in the display area of the display device in such a way that the selected display data is displayed at a position that satisfies a certain positional relationship with the position of the section of the robot in the display area.

RELATED APPLICATIONS

The present application claims priority of Japanese Application Number 2018-146907, filed Aug. 3, 2018, the disclosure of which is hereby incorporated by reference herein in its entirety.

FIELD

The present invention relates to, for example, a collaborative operation support device for supporting a collaborative operation with a robot.

BACKGROUND

When an operator teaches a robot which includes a movable part driven by a servo motor such as an arm about an operation to be implemented by the robot, or when such a robot and the operator work in cooperation, the operator may cause the robot to implement a lead-through operation by manually changing a position or posture of the robot.

When the operator teaches the robot about the operation by means of a lead-through operation, the operator typically operates an input device called a teach pendant in order to confirm settings for movements of the robot or input control parameters for setting movements of the robot. However, to operate the teach pendant, the operator needs to temporarily stop the ongoing task, thereby reduces an operation efficiency. In view of this problem, there has been proposed a technique in which a keyword or phrase for instructing a robot about an operation are extracted from an operator's speech, and using the keyword or phrase, data for instructing the robot, about the operation are created (see, for example, Japanese Unexamined Patent Publication (Kokai) No. 2006-289543).

SUMMARY

However, even with the technique described above, it is not easy for the operator to intuitively understand how the robot will actually move for the control parameters that have been set. For example, even if a certain value of viscosity is set for a predetermined direction that is defined with a tool attached to the robot at the origin, it is difficult for the operator to intuitively understand which direction the predetermined direction is oriented in a real space and what extent movements of the robot is changed by the value of viscosity that has been set.

In one aspect, it is an object to provide a collaborative operation support device that facilitates an operator to intuitively understand settings for movements of a robot.

According to one embodiment, the collaborative operation support device is provided. The collaborative operation support device includes: a memory configured to store, for each operation mode of a robot when the robot and an operator collaboratively operate with each other, display data representing movements of the robot for the mode; a display device configured to be worn by the operator, wherein the operator can view the robot through a display area of the display device; a camera configured to image the robot or the operator and generate an image in which the robot or the operator is represented; and a processor configured to detect, based on the image, a position of a section of the robot in the display area when the operator looks at the robot through the display area, the section associated with an operation mode of the robot specified by means of an input device; select, according to the specified operation mode of the robot, display data corresponding to the specified mode among the display data stored in the memory; and display the selected display data in the display area of the display device in such a way that the selected display data is displayed at a position that satisfies a certain positional relationship with the position of the section of the robot in the display area.

DESCRIPTION OF EMBODIMENTS

With reference to the drawings, a collaborative operation support device will be described below. The collaborative operation support device displays, on a transparent display worn by an operator, display data with respect to a robot that collaboratively operates with the operator, the display data representing movements of the robot in a specified operation mode, in association with a section of the robot associated with the specified operation mode, thereby facilitating the operator to intuitively understand settings for the movements of the robot.

FIG. 1is a schematic configuration diagram of the collaborative operation support device. The collaborative operation support device1includes an input device2, a head-mounted display3, a camera4, a wireless communication interface5, a communication interface6, a memory7, and a processor8. Among these units, the wireless communication interface5, the communication interface6, the memory7, and the processor8are mounted in a controller12of a robot11, which is a target for teaching or works in cooperation with the operator, for example. The wireless communication interface5, the communication interface6, and the memory7are connected to the processor8via a signal line in a bi-directionally communicative fashion. In addition, each of the input device2, the head-mounted display3, and the camera4is communicatively connected to the processor8via the wireless communication interface5by means of wireless communication compliant with a predetermined wireless communication standard such as Bluetooth (registered trademark).

In this example, the robot11is an articulated robot including three shafts11ato11cand a tool11dis attached to an end of the robot11. Each of the shafts11ato11cof the robot11is driven by a servo motor (not illustrated) to change a posture of the robot11. Note that the robot11that collaboratively operates with the operator is not limited to this example, and the robot11may be a robot including at least one shaft driven by a servo motor.

The input device2is an input device that can be carried by the operator such as a so-called teach pendant, and is used for inputting information on settings for movements of the robot11. For this purpose, the input device2includes, for example, a plurality of operation buttons, a control circuit for generating an operation signal corresponding to the operation button that has been pressed among the plurality of operation buttons, a wireless communication circuit for generating a radio signal containing the operation signal generated by the control circuit and transmitting the radio signal to the controller12. The input device2generates, in response to an operation by the operator, an operation signal indicating whether or not to cause the robot11to implement a lead-through operation, a mode of the lead-through operation, and values of various types of control parameters for controlling the servo motors for driving the respective shafts in the lead-through operation implemented by the robot11, and transmits a radio signal containing the operation signal to the controller12.

The head-mounted display3is an example of a display unit, and displays, for the operator, display data received from the controller12and representing movements of the robot11in the lead-through operation implemented by the robot11by superimposing the display data on a field of view of the operator in a state in which the head-mounted display3is worn on the operator's head. For this purpose, the head-mounted display3includes a transparent display such as, for example, a liquid crystal display; a mounting member for arranging the transparent display in such a way that the transparent display overlaps the field of view of the operator when the head-mounted display3is worn on the operator's head; a wireless communication circuit for wirelessly communicating with the controller12; and a control circuit for displaying the display data contained in the radio signal received from the controller12on the transparent display and transmitting an image received from the camera4to the controller12via the wireless communication circuit.

The camera4is an example of an imaging unit, and is attached to the head-mounted display3in such a way that, for example, the camera4can image the field of view of the operator when the operator wears the head-mounted display3. The camera4is attached to the head-mounted display3, for example, laterally to or above the transparent display of the head-mounted display3in such a way that the camera4is oriented in the front direction of the operator who is wearing the head-mounted display3. The camera4is communicatively connected, for example, to the control circuit of the head-mounted display3. While the collaborative operation support device1is performing a collaborative operation support process, the camera4images a shooting range including the field of view of the operator who is wearing the head-mounted display3at predetermined imaging periods to generate images, and outputs the generated images to the control circuit of the head-mounted display3.

The wireless communication interface5includes a circuit for performing a process of transmitting/receiving a radio signal compliant with the predetermined wireless communication standard, and the like. The wireless communication interface5extracts an operation signal or information such as images contained in the radio signals received from the input device2and the head-mounted display3, and passes the signal or information to the processor8. In addition, the wireless communication interface5generates a radio signal containing the data to be displayed on the head-mounted display3, which have been received from the processor8, and transmits the generated radio signal to the head-mounted display3.

The communication interface6includes, for example, a communication interface for connecting the controller12to a communication line13and a circuit for performing a process of transmitting/receiving a signal via the communication line13, and the like. The communication interface6outputs, for example, a torque command value for the servo motor of the robot11, which has been received from the processor8, and the like to the robot11via the communication line13. In addition, the communication interface6receives, from the robot11via the communication line13, information indicating operation statuses of the respective servo motors such as values of feedback current at the respective servo motors, encoder signals from respective encoders indicating rotation amounts of the respective shafts11ato11cdriven by the servo motors, and passes the information to the processor8.

The memory7is an example of a storage unit, and includes, for example, a readable/writable semiconductor memory and a read-only semiconductor memory. The memory7may further include a storage medium such as a semiconductor memory card, a hard disk, or an optical storage medium and a device for accessing the storage medium.

The memory7stores various types of computer programs for controlling the robot11, a computer program for the collaborative operation support process, and the like, which are executed by the processor8. The memory7also stores display data which is displayed on the head-mounted display3and represents movements of the robot11for each lead-through operation mode, and the like, in association with the corresponding lead-through operation mode. In addition, the memory7stores information indicating operation statuses of the respective servo motors, which can be obtained from the robot11while the robot11is operating, various types of data generated in the collaborative operation support process, images obtained from the camera4, and the like.

The processor8is an example of a control unit, and includes a Central Processing Unit (CPU) and a peripheral circuit thereof. The processor8may further include a processor for numeric operations. The processor8controls the entire collaborative operation support device1. In addition, the processor8controls the movements of the robot11and performs the collaborative operation support process.

FIG. 2is a functional block diagram of the processor8relating to the collaborative operation support process. The processor8includes a lead-through control, unit21, a display data selection unit22, a display position setting unit23, and a display control unit24. Each of these units included in the processor8is, for example, a functional module achieved by a computer program executed by the processor8. Alternatively, each of these units may be achieved as a dedicated arithmetic circuit implemented as part of the processor8.

The lead-through control unit21is an example of an operation control unit and controls the movements of the robot11based on whether or not to cause the robot11to implement a lead-through operation, the mode of the lead-through operation, or the values of the control parameters specified in the operation signal contained in the radio signal received from the input device2via the wireless communication interface5.

The lead-through operation will be described hereafter.

FIG. 3AandFIG. 3Bare diagrams each illustrating an example of modes of lead-through operations. In the operation mode illustrated inFIG. 3A, the robot11operates in such a way that the shafts11ato11cof the respective joints of the robot11move in accordance with external forces applied to the shafts11ato11cof respective joints. For example, inFIG. 3A, as indicated by an arrow301, in response to pushing an end of the robot11by operator300, the shafts11ato11cof the respective joints of the robot11rotates as indicated by arrows302to304. Hereinafter, for the sake of convenience, the operation mode illustrated inFIG. 3Ais referred to as the each shaft lead-through mode.

In contrast, in the operation mode illustrated inFIG. 3B, the robot11operates in such a way that, in accordance with an external force applied to any position of the robot11, an orthogonal coordinate system set for the position moves. For example, as illustrated inFIG. 3B, by applying a force to the tool11dattached to the end of the robot11by the operator300, any shaft of the shafts11ato11cof the respective joints, e.g. the shaft11cto which the tool11dis attached rotates in such a way that an orthogonal coordinate system311set for the tool11dmoves in accordance with the direction of the force. Hereinafter, for the sake of convenience, the operation mode illustrated inFIG. 3Bis referred to as the orthogonal lead-through mode.

Note that, in the each shaft lead-through mode, any shaft of the robot11may be fixed. In the orthogonal lead-through mode, movements in any axial direction in the orthogonal coordinate system or movements about any axis in the orthogonal coordinate system may be fixed.

In addition, in a lead-through operation, in order to set the movements of the robot11by means of the input device2, virtual viscosity, elasticity, and/or inertia for the respective shafts11ato11cof the robot11or rotation speeds for the respective shafts11ato11cto be applied in a situation in which the robot11is moving may be set as the control parameters for the servo motors.

The lead-through control unit21determines a shaft to be fixed in accordance with the lead-through operation mode specified in the operation signal, designation of a shaft to be fixed, and the like. When the operator applies an external force to the robot11, the lead-through control unit21rotates, in response to the external force, shafts other than the one to be fixed in accordance with the specified lead-through operation mode. In the process, the lead-through control unit21determines, in accordance with the values of virtual viscosity, elasticity, and/or inertia for the respective shafts contained in the operation signal (when the values of viscosity and the like are not contained in the operation signal, a value set in the past or a default value), the torque command value for the servo motor that drives the shaft. In this case, for example, the lead-through control unit21refers to a table representing a relationship between combinations of the values of the virtual viscosity, elasticity, and/or inertia and torque command values to determine the torque command value. Such a table is stored in advance, for example, in the memory7. Alternatively, the lead-through control unit21may calculate the torque command value in accordance with an equation representing the relationship between combinations of the values of the virtual viscosity, elasticity, and/or inertia and torque command values. The lead-through control unit21then outputs the torque command value to the robot11via the communication interface6and the communication line13and causes the servo motors that drive the respective shafts other than the one to be fixed to generate torques having magnitude equivalent to the values of the virtual viscosity, elasticity, and/or inertia. The servo motors may be controlled by means of a Proportional-Integral-Derivative (PID) controller in accordance with the actual torque generated by the external force that the operator applied to the robot11and the torque specified by the torque command value. Furthermore, the lead-through control unit21outputs control values for the servo motors indicating rotation speeds that have been set (e.g., a duty cycle in pulse width modulation) to the robot11via the communication interface6and the communication line13.

The lead-through control unit21may calculate rotation amounts of the respective shafts by counting the number of times that the encoder signals from the encoders attached to the respective shafts or the servo motors that drive the respective shafts are received. The lead-through control unit21can calculate a posture of the robot11at a time of interest based on the posture of the robot11at the time when the robot11started to move and the rotation amounts of the respective shafts at the time of interest.

The display data selection unit22selects, among display data for the respective operation modes stored in the memory7, display data to be displayed on the head-mounted display3in accordance with the lead-through operation mode specified by the operation signal. For this purpose, for example, the display data selection unit22selects, by referring to a table representing a relationship between the lead-through operation modes and display data, display data corresponding to the lead-through operation mode specified in the operation signal. For example, in the each shaft lead-through mode, display data representing a symbol (e.g., an arc-like arrow) indicating the direction of rotation of a shaft is selected for each shaft while in the orthogonal lead-through mode, display data representing a symbol (e.g., a combination of a straight line arrow representing each axis of the orthogonal coordinate system and an arc-like arrow around the straight line arrow for each axis) indicating the orthogonal coordinate system is selected. Subsequently, the display data selection unit22reads out the selected display data from the memory7.

The display data selection unit22may change the display data in accordance with the values of the control parameters (e.g., the values of viscosity, elasticity, inertia, the rotation speed, or the like) contained in the operation signal. For example, the display data selection unit22may set luminance of the arrow specified in the display data to a higher value as the values of virtual viscosity, elasticity, and/or inertia that have been set are higher. Alternatively, the display data selection unit22may set a color, a thickness, a length, or transparency of the arrow specified in the display data to a higher value in accordance with the values of virtual viscosity, elasticity, and/or inertia that have been set. When the control parameters for the respective shafts of the robot11are set to different values, the display data may be changed for each shaft in accordance with the values of the control parameters set for the shaft.

Furthermore, when any shaft of the shafts11ato11cincluded in the robot11is to be fixed, the display data selection unit22may read out display data indicating the shaft to be fixed from the memory7.

The display data selection unit22passes the selected display data read out from the memory7to the display control unit24.

The display position setting unit23sets a position for displaying the display data selected by the display data selection unit22in a display area of the transparent display of the head-mounted display3. For example, in the each shaft lead-through mode, it is preferable to display a symbol representing a direction of rotation of a shaft of the robot11that is not fixed around the shaft. In the orthogonal lead-through mode, it is preferable to display a symbol representing an orthogonal coordinate system at a specified position at which the orthogonal coordinate system is to be located. Thus, the display data representing the movements of the robot11is preferably displayed in association with the section of the robot11associated with the specified lead-through operation mode in such a way as to facilitate understanding of movable portions of the robot11in the lead-through operation. Therefore, when the collaborative operation support process is started, the display position setting unit23detects, every time an image is obtained from the camera4after the process is started, a section or sections of the robot11associated with the specified lead-through operation mode in the obtained image. For example, when the specified lead-through operation mode is the each shaft lead-through mode, the display position setting unit23detects the respective shafts of the robot11on the image. When the specified lead-through operation mode is the orthogonal lead-through mode, the display position setting unit23detects a position of the robot11at which an orthogonal coordinate system is to be located, for example, the tool11dattached to an end of the robot11. For this purpose, the display position setting unit23detects, by performing template matching, for each section to be detected of the robot11, on the image using a template representing the section, an area in which the section to be detected is represented on the image. Mote that such a template is stored in advance, for example, in the memory7. In this case, the display position setting unit23may use different templates in the template matching according to the posture of the robot11viewed from the operator. The display position setting unit23may determine, for each section to be detected, a posture of the section represented in a template that best matches the image among the templates for the section as the posture of the section viewed from the operator. Alternatively, the display position setting unit23may input an image to a classifier that has been learned in advance in such a way as to detect the section to be detected of the robot11, to detect the area in which the section of the robot11is represented on the image. The display position setting unit23may use, as such a classifier, a classifier such as a deep neural network or AdaBoost.

In the present embodiment, the camera4is attached to the head-mounted display3in such a way that the shooting range thereof covers the field of view of the operator who wears the head-mounted display3. The position at which a certain object is captured in the image acquired by the camera4is associated in a one-to-one manner with a position at which the certain object is to be superimposed on the display area of the transparent display when the operator sees the certain object through the transparent display of the head-mounted display3. Therefore, the display position setting unit23can obtain, for each detected section of the robot11, by referring to a table representing a relationship between positions in the image acquired by the camera4and positions in the display area of the transparent display of the head-mounted display3, the position in the display area corresponding to a centroid position of the area in which the detected section is captured in the image acquired by the camera4as the position of the section in the display area. Such a table is stored in advance, for example, in the memory7. Alternatively, the display position setting unit23may obtain the position of the detected section in the display area, instead of using such a table, by inputting coordinates of the centroid of the area in which the detected section is captured in the image to a position conversion equation representing a correspondence relationship between positions in the image acquired by the camera4and positions in the display area of the transparent display of the head-mounted display3.

The display position setting unit23performs the process described above every time an image is obtained from the camera4, to update the positions of respective sections associated with the specified lead-through operation mode in the display area. In this manner, even if the operator moves or changes an orientation of his/her face during the lead-through operation, the display position setting unit23can display, for each of sections of the robot11associated with the specified lead-through operation mode, the display data representing the movements of the robot11in the specified lead-through operation mode with respect to the section on the head-mounted display3in such a way that the display data is displayed at a position that satisfies certain positional relationship with the section.

The display position setting unit23informs the display control unit24, every time an image is obtained from the camera4, of the positions of the respective sections of the robot11associated with the specified lead-through operation mode.

The display control unit24displays, for each of sections of the robot11associated with the specified lead-through operation mode, the display data representing the movements in the specified lead-through operation mode with respect to the section on the head-mounted display3in such a way that the display data is displayed at a position that satisfies the certain positional relationship with the section. For example, the display control unit24set, for each of the sections of the robot11associated with the specified lead-through operation mode, the position of the display data corresponding to the section in the display area in such a way that the position of the display data satisfies the certain positional relationship with the position of the section in the display area. The display control unit24then transmits, for each of the sections of the robot11associated with the specified lead-through operation mode, a signal containing the display data corresponding to the section and the display position of the display data to the head-mounted display3via the wireless communication interface5.

FIG. 4Ais a diagram illustrating an example of the display data for the each shaft lead-through mode, displayed on a head-mounted display3. In the example illustrated inFIG. 4A, the respective shafts11ato11cof the robot11are movable portions in a display area400of the head-mounted display3, and therefore, arc-like arrows401to403representing directions of rotation of the respective shafts are displayed for the respective shafts11ato11cin such a way as to encircle the shafts at the center.

Note that, when any shaft of the shafts11ato11cis to be fixed, the display data representing the movements of the robot11need not be displayed for the shaft to be fixed. For example, when the shaft11ais fixed, the arc-like arrow401around the shaft11aneed not be displayed. In this manner, the operator can intuitively understand the shaft to be fixed and the shafts not to be fixed in the lead-through operation.

Alternatively, the display control unit24may display, with respect to the shaft to be fixed, display data indicating that the shaft does not move. For example, when the shaft11ais fixed, the display control unit24may display an X-shaped symbol at the position in the display area corresponding to the shaft11a. In this case, when an absolute value of the actual torque of the servo motor for the shaft to be fixed, which is obtained via the communication interface6, is greater than a predetermined value indicating that an external force is applied, the display control unit24may display the display data indicating that the shaft does not move. In this manner, the display control unit24decreases the number of display data to be superimposed on the field of view of the operator to keep the operator free from feeling complexity, and the display control unit24further facilitates the operator to intuitively understand the shaft to be fixed when the operator moves the robot11.

FIG. 4Bis a diagram illustrating an example of display data for the orthogonal lead-through mode, displayed on the head-mounted display3. In the example illustrated inFIG. 4B, an orthogonal coordinate system is set at the position of the tool11aattached to the robot11, and therefore, in the display area400of the head-mounted display3, an orthogonal coordinate system410is displayed with the position in the display area corresponding to the tool11dat the origin and arc-like arrows411to413are displayed about the respective axes of the orthogonal coordinate system.

In this example, the display control unit24need not display, with respect to the axis to be fixed of the orthogonal coordinate system, an arrow indicating the axis. Alternatively, the display control unit24may set luminance for the axis to be fixed of the orthogonal coordinate system to be lower than luminance for the axes not to be fixed of the orthogonal coordinate system. Alternatively, the display control unit24may set transparency for the axis to be fixed of the orthogonal coordinate system to be higher than transparency for the axes not to be fixed of the orthogonal coordinate system.

FIG. 4Cis a diagram illustrating an example of the display data, in the each shaft lead-through mode, displayed on the head-mounted display3when the values of the control parameters for the servo motors for the respective shafts such as virtual viscosity, elasticity and/or inertia are different. In the example illustrated inFIG. 4C, the respective shafts11ato11cof the robot11are movable portions, and therefore in the display area400of the head-mounted display3arc-like arrows421to423representing directions of rotation of the respective shafts11ato11care displayed with respect to each of the shafts11ato11cin such a way as to encircle the shafts at the center. In addition, in this example, thickness of each arrow is changed in accordance with the values of the virtual viscosity, elasticity and/or inertia that have been set. For example, when the value of viscosity set for the shaft11bis greater than those set for the shafts11aand11c, the arrow422displayed around the shaft11bis displayed with a thickness greater than that of the arrow421displayed around the shaft11aand that of the arrow423displayed around the shaft11c. In this manner, the operator can intuitively understand that the rotation movement of the shaft11bis more elastic than those of the shafts11aand11c. Thus, by changing the display data in response to the values of the control parameters set for the servo motors for the respective shafts, the operator can intuitively understand the control parameters to be set.

FIG. 5is an operation flowchart of the collaborative operation support process. The processor8performs the collaborative operation support process in accordance with the operation flowchart described below, for example, every time the processor8receives the operation signal from the input device2.

The lead-through control unit21controls the lead-through operation implemented by the robot11in accordance with the operation signal from the input device2(step S101). The display data selection unit22selects, among the display data stored in the memory7, display data to be displayed on the head-mounted display3in accordance with the lead-through operation mode specified in the operation signal (step S102).

The display position setting unit23detect, in the image obtained from the camera4, a section of the robot11associated with the specified lead-through operation mode (step S103). The display position setting unit23then identifies, from the position of the detected section of the robot11in the image, a position of the section in the display area of the head-mounted display3(step S104).

The display control unit24displays, for the section of the robot11associated with the specified lead-through operation mode, selected display data on the head-mounted display3in such a way that the display data is displayed at a position that satisfies a certain positional relationship with the position of the section in the display area of the head-mounted display3(step S105).

After step S105, the processor8determines whether or not the processor8has received, from the input device2, an operation signal indicating that the lead-through operation is to be finished (step S106). When the processor8has not received the operation signal indicating that the lead-through operation is to be finished (step S106—No), the processor8repeats the processing steps from step S103. When the processor8has received the operation signal indicating that the lead-through operation is to be finished (step S106—Yes), the processor8ends the collaborative operation support process.

As described above, when the collaborative operation support device causes the robot to implement the lead-through operation, the collaborative operation support device displays the display data representing the movements of the robot in the specified lead-through operation mode on the head-mounted display worn by the operator in association with a section of the robot associated with the specified lead-through operation mode. Therefore, the collaborative operation support device can facilitate the operator to intuitively understand the settings for the movements of the robot that implements the lead-through operation.

According to a variation, the display control unit24may use, as the display data, an image of the section of the robot11, which is captured in the image obtained from the camera4and associated with the specified lead-through operation mode, for example, a contour image of the section. In this case, the display control unit24crops the image by removing an area other than the area in which the section is captured, the section being detected by the display position setting unit23and being associated with the specified lead-through operation mode. The display control unit24performs an edge detection process on the remaining area after cropping using an edge detection filter such as a Sobel filter, to detect an edge pixel having an edge strength of a predetermined value or greater. The display control unit24then detects, for each row of pixels in the remaining area, the leftmost edge pixel and the rightmost edge pixel as contour pixels representing a contour of the section associated with the specified lead-through operation mode. Subsequently, the display control unit24may generate a contour image in which each contour pixel has a pixel value different from those of other pixels as display data.

In this case, the display control unit24may change the size of the contour image in accordance with the values of the control parameters for the servo motors for the respective shafts of the robot11. For example, the display control unit24may enlarge the contour image in such a way that the size of the contour image is larger as the value of virtual viscosity or elasticity is higher. In this manner, the operator can visually know, from the size of the section of the robot11associated with the specified lead-through operation mode, the control parameters that have been set for the section, and thus, the operator can intuitively understand the control parameters that have been set.

According to another variation, the lead-through control unit21of the processor8may change the values of the control parameters that have been set for the section associated with the display data when the operator points a finger at a position that overlaps the display data displayed on the head-mounted display3over a certain period of time (e.g., one to two seconds). For example, when the operator points a finger at an end of the arrow401illustrated inFIG. 4Ain a clockwise direction over a certain period of time, the lead-through control unit21may increase the virtual viscosity for the shaft11aby a certain value; on the other hand, when the operator points a finger at an end of the arrow401illustrated inFIG. 4Ain a counterclockwise direction over a certain period of time, the lead-through control unit21may decrease the value of virtual viscosity for the shaft11aby a certain value.

For this purpose, an area is set in the display data for changing the values of the control parameters by a certain value when the operator points a finger at a point within the area (e.g., each end of the arrow401inFIG. 4A). In addition, the lead-through control unit21detects, in the image acquired by the camera4, the operator's finger. For example, the lead-through control unit21detects, by inputting the image to a classifier that has been learned for detecting a finger in advance, an area in the image in which the finger is captured, and determines a centroid of the area as a position of the finger. The lead-through control unit21may use, as such a classifier, a classifier such as a deep neural network or AdaBoost. As described with respect to the display position setting unit23, positions in the image acquired by the camera4correspond to positions in the display area of the head-mounted display3in a one-to-one manner. Therefore, the lead-through control unit21may perform a process similar to the process performed by the display position setting unit23to calculate a position in the display area corresponding to the position of the finger detected in the image, and when the position in the display area is within the area set for changing the control parameters, the lead-through control unit21may change the values of the control parameters by a value set for the area.

According to this variation, the operator need not operate the input device2to change the control parameters, and thus, an operation efficiency is improved.

According to another variation, the display control unit24may display a movement trajectory of the section of the robot11associated with the specified lead-through operation mode on the head-mounted display3. In this case, while a rotation amount of any shaft of the robot11, which is calculated on the basis of the encoder signal received from the robot11via the communication interface6with respect to the shaft is changing, i.e., while the posture of the robot11is changing, every time the display control unit24obtains an image from the camera4, the display control unit24obtains, from the display position setting unit23, position in the display area of the section of the robot11associated with the specified lead-through operation mode. The display control unit24then displays a line obtained by connecting the obtained positions in time order as the movement trajectory on the head-mounted display3.

Note that a sensor that can measure an amount indicating a change in the orientation of the operator's face, such as a gyroscope sensor, may be attached to the head-mounted display3in such a way that an accurate movement trajectory can be displayed even if the operator changes the orientation of his/her face during a series of tasks. In this case, the head-mounted display3measures the amount indicating the change in the orientation of the operator's face by the sensor at every a predetermined period, and transmits a radio signal containing the amount to the controller12. The display control unit24calculates, every time the display control obtains an image from the camera4, a displacement of the orientation of the operator's face between the time of the previous image acquisition and the time of the most recent image acquisition on the basis of the amount indicating the change in the orientation of the operator's face, which is contained in the radio signal received via the wireless communication interface5. For example, when the amount indicating the change in the orientation of the operator's face is represented by an angular acceleration, the display control unit24can calculate the displacement of the orientation of the operator's face by calculating double integral of the angular acceleration. The display control unit24may move the positions in the display area of the section of the robot11associated with the specified lead-through operation mode at time when respective images were acquired, by an amount equivalent to the displacement of the orientation of the operator's face using, as the reference, the position at the time when the most recent image was acquired.

According to still another variation, the camera4may be attached, for example, facing downward on a ceiling of a room where the robot11is installed in such a way that a position of the operator relative to the robot11can be detected. A sensor that can measure an amount indicating the change in the orientation of the operator's face, such as a gyroscope sensor, may be attached to the head-mounted display3. In addition, the position of the robot11in a real space is stored in the memory7in advance. In this case, for example, when the operator performs an input operation by operating the input device2to cause the robot11to start implementing a lead-through operation, the operator gazes at a certain position of the robot11(e.g., the tool11dattached to the robot11) over a certain period of time (e.g., three to five seconds) for initialization. When the lead-through control unit21of the processor8receives, from the input device2, an operation signal indicating that implementation of the lead-through operation by the robot11is to be started, the lead-through control unit21controls the servo motor in such a way that the robot11has a predetermined posture set in advance. When the display position setting unit23of the processor8receives, from the input device2, the operation signal indicating that implementation of the lead-through operation by the robot11is to be started, the display position setting unit23detects the operator in the image obtained by imaging made by the camera4during the certain period of time. For example, the display position setting unit23detects, by inputting the image to a classifier that has been learned for detecting an operator in advance, a centroid of an area in which the operator is captured in the image. The display position setting unit23may use, as such a classifier, a classifier such as a deep neural network or AdaBoost. The position in the image corresponds to a direction from the camera4in a one-to-one manner. In addition, since a distance from the camera4to a floor on which the robot11and the operator are located is known, the display position setting unit23can identify, on the basis of the position of the centroid of the area in the image, in which the operator is captured, a position of the operator in a real space (hereinafter, referred to as an initial position). Therefore, the display position setting unit23can identify a relative positional relationship between the robot11and the operator. In addition, since the posture of the robot11during the period in which the operator is gazing at the certain position of the robot11is known, a gaze direction of the operator to the certain position of the robot11in a case where the operator is located at the initial position is also known. Therefore, the display position setting unit23sets an intersection point between the gaze direction and the display area of the head-mounted display as a reference point.

After the initialization, the head-mounted display3measures the amount indicating the change in the orientation of the operator's face by the sensor at every a predetermined period, and transmits a radio signal containing the amount to the controller12. The display position setting unit23of the processor8calculates a displacement of the orientation of the operator's face from the time when the operator is gazing at the certain position on the basis of the amount indicating the change in the orientation of the operator's face, which is contained in the radio signal received via the wireless communication interface5. For example, when the amount indicating the change in the orientation of the operator's face is represented by an angular acceleration, the display position setting unit23can calculate the displacement of the orientation of the operator's face by calculating double integral of the angular acceleration. The display position setting unit23can detect, by detecting the position of the operator in the image acquired by the camera4in a similar manner as described above, a movement direction and an amount of movement of the operator from the initial position. In addition, the display position setting unit23can calculate a change in the posture of the robot11from the predetermined posture from the rotation amounts of the respective shafts. Therefore, the display position setting unit23can calculate, on the basis of the movement direction and the amount of movement of the operator from the initial position, the displacement of the orientation of the operator's face from the orientation of the operator's face at the time point when the operator located at the initial position is gazing at the certain position of the robot11, and the amount of the change in the posture of the robot11from the predetermined posture, angles of directions from the operator toward the respective movable portions of the robot11relative to the front direction of the operator. The display position setting unit23may identify, on the basis of the angles, positions of the respective movable portions in the display area of the head-mounted display3.

According to still another variation, the display data selection unit22may change the selected display data in accordance with movements of the robot11generated by the external force applied by the operator to the robot11. For example, the display data selection unit22may change, in response to values of the actual torque, received from the robot11via the communication interface6, of the servo motors of the respective shafts11ato11cof the robot11, a color or luminance of the arrow in the display data with respect to each shaft. Alternatively, the display data selection unit22may calculate, for each of the shafts11ato11cof the robot11, based on the rotation amount for the shaft, which is calculated on the basis of the encoder signals received from the robot11via the communication interface6, and a rotation range for the shaft a remaining rotation amount for which the shaft can be still rotated, and shorten a length of the arrow in the display data as the remaining rotation amount is smaller. Alternatively, the display data selection unit22may blink the arrow in the display data with respect to a shaft for which the absolute value of the actual torque of the servo motor received from the robot11via the communication interface6is equal to or greater than a predetermined value, among the shafts11ato11cof the robot11. In this manner, the operator can more easily understand movements of the robot11.

According to still another variation, the input device2may include a speech input unit such as a microphone in such a way that the operator can change the values of the control parameters with speech. The input device2transmits a radio signal containing a speech signal representing speech of the operator obtained by the speech input unit to the controller12. Alternatively, the speech input unit may be attached to the head-mounted display3, and the radio signal containing the speech signal representing the speech of the operator obtained by the speech input unit may be transmitted from the head-mounted display3to the controller12. The lead-through control unit21of the processor8of the controller12may perform a certain speech recognition process such as word spotting techniques based on hidden Markov models on the speech signal contained in the radio signal received via the wireless communication interface5to recognize the speech of the operator, and according to a result of the speech recognition, the lead-through control unit21may change the values of the control parameters or the lead-through operation mode to be specified. In this manner, the operator can reduce time and effort for operating the input device2and improve the operation efficiency.

All examples and specific terms used herein are intended for an instructive purpose to facilitate readers to understand the present invention and a concept contributed by the inventor of the present invention for promoting the technique, and should not be construed as limited to such specific examples and conditions described herein and any configurations in such examples, which are related to describing superiority and inferiority of the present invention. Although the embodiments of the present invention have been described in detail, it should be understood that various changes, substitutions, and alterations may be made without departing from the spirit and scope of the present invention.