Robot system

A robot system includes a robot, a robot controller, a video acquisition device configured to acquire a real video of a work space, and a head-mounted type video display device provided with a visual line tracking section configured to acquire visual line information. A robot controller includes an information storage section configured to store information used for controlling the robot while associating the information with a type of an object, a gaze target identification section configured to identify, in the video, a gaze target viewed by a wearer based on the visual line information, and a display processing section configured to cause the video display device to display the information associated with the object corresponding to the identified gaze target, side by side with the gaze target in the form of one image through which the wearer can visually grasp, select, or set contents of the information.

RELATED APPLICATIONS

The present application claims priority of Japanese Application Number 2019-013403 filed Jan. 29, 2019, the disclosure of which is hereby incorporated by reference herein in its entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a robot system.

2. Description of the Related Art

Recently, Augmented Reality (AR) technology is expected to be applied to industrial robots. For example, Japanese Unexamined Patent Application Publication No. 2016-107379A describes a robot system in which an operation guide indicating a direction of a jog operation at a tool tip point is superimposed on an image of a robot in a real environment displayed on a head-mounted display equipped with an imaging device. Japanese Unexamined Patent Application Publication (Translation of PCT Application) No. 2017-538218A describes a system in which a user operates an application launcher superimposed on a real environment video by using a visual line as a mouse cursor while recognizing a gaze target of the user by a visual line tracking device provided in a head-mounted display.

SUMMARY OF THE INVENTION

In general, in order for a user to access desired information through a user interface screen displayed on a display section of a teach pendant or the like in a robot system, it is necessary to perform operation for finding desired information by sequentially selecting hierarchically presented selection menus. There is a need for a robot system having a configuration that allows a user to access desired information more efficiently.

One aspect of the present disclosure is a robot system which includes a robot, a robot controller configured to control the robot, a video acquisition device configured to acquire a real video of a work space including the robot and the robot controller, and a head-mounted type video display device provided with a visual line tracking section configured to acquire visual line information of a wearer, where the robot controller includes an information storage section configured to store information used for controlling the robot while associating the information with a type of an object existing in the work space, a gaze target identification section configured to identify, in the video, a gaze target viewed by a wearer of the video display device in the work space, based on the visual line information, and a display processing section configured to cause the video display device to display the information associated with the object corresponding to the identified gaze target, side by side with the gaze target in the form of one image through which the wearer can visually grasp, select, or set contents of the information.

DETAILED DESCRIPTION

Embodiments of the present disclosure will be described below with reference to the accompanying drawings. Throughout the drawings, corresponding components are denoted by common reference numerals. For ease of understanding, these drawings are scaled as appropriate. The embodiments illustrated in the drawings are examples for implementing the present invention, and the present invention is not limited to the embodiments illustrated in the drawings.

FIG. 1is a diagram illustrating a configuration of a robot system100according to an embodiment of the present disclosure.FIG. 2is a functional block diagram of the robot system100. As illustrated inFIG. 1toFIG. 2, the robot system100includes a robot10, a robot controller50configured to control the robot10, a video acquisition device91configured to acquire a real video of a work space including the robot10and the robot controller50, and a head-mounted type video display device90provided with a visual line tracking section92configured to acquire visual line information of a wearer. The robot controller50includes an information storage section80configured to store information used for controlling the robot10while associating the information with a type of an object (robot10, robot controller50, peripheral devices, etc.) existing in a work space, a gaze target identification section51configured to identify, in the video, a gaze target viewed by a wearer of the video display device90in the work space, based on the visual line information, and a display processing section53configured to cause the video display device90to display the information associated with the object corresponding to the identified gaze target, side by side with the gaze target in the form of one image through which the wearer can visually grasp, select, or set contents of the information. In the present embodiment, the video acquisition device91is integrated with the video display device90, but this is an example, and the video acquisition device91may be provided separately from the video display device90.

The robot10is a vertical articulated robot in the present embodiment, and a tool12is mounted on the tip portion of the arm. The robot10may be another type of robot. For example, when the robot10is a 6 axis robot, the robot controller50acquires the position of each axis (J1to J6) from the robot10as information representing the current position of the robot10. The robot controller50may have a configuration as a general computer including a CPU, a ROM, a RAM, a storage device, etc.

The video display device90includes a display94disposed in front of both eyes of the wearer, and displays the video acquired by the video acquisition device91on the display94. As the visual line tracking section92, a device having a general configuration known in the art for measuring the visual line can be used. As an example, the visual line tracking section92acquires the positional relationship between the corneal reflex (reference point) and the pupil (moving point) of both eyes of the wearer by an infrared LED and an infrared camera which are disposed at positions in front of both eyes of the wearer of the video display device90, and thereby obtains the visual line direction (indicated by a dashed arrow E inFIG. 1). The video acquired by the video acquisition device91and the visual line information acquired by the visual line tracking section92are transmitted to the robot controller50. An image processing section93executes video processing in which information relating to a gaze target provided from the robot controller50is superimposed and displayed on the video from the video acquisition device91so as to be disposed side by side with the gaze target.

In the present embodiment, the video acquisition device91and the visual line tracking section92are integrally provided in the video display device90, and their positions are fixed to the head of the wearer. Therefore, by using the visual line information (eye direction) measured by the visual line tracking section92, it is possible to identify the gaze target of the wearer in the video. The robot controller50includes an object model storage section81configured to store object models which model their respective objects (robot10, robot controller50, and peripheral device, etc.) in the work space. As an example, the gaze target identification section51may identify the position of the object in the video and the portion of a gaze target on the object by image recognition processing using the object model stored in the object model storage section81.

The robot controller50further includes a feature identification section52and a tool coordinate system setting section54. In order to enable the video display device90to display information specific to each feature portion predetermined for each object, the information storage section80stores the information specific to each feature portion while associating the information with each feature portion predetermined for each object. Table 1 below is an example of the information stored in the information storage section80. As shown in Table 1 below, the predetermined feature portion for the robot10is, for example, a tool and a joint portion, and the predetermined feature portion for the robot controller50is, for example, a power switch and a mode selection switch. The information specific to each feature portion includes internal information and setting information of the feature portion.

The feature identification section52identifies a feature portion that the wearer is gazing at by using the object model and the video and the visual line information. The feature identification section52uses, for example, an object model of the robot10and a current position of each axis of the robot10, causes the object model to assume a posture corresponding to the current posture of the robot10, identifies a position of the robot10in the video by pattern matching, and further identifies which of a plurality of feature portions previously set for the robot10is a target of the wearer's gaze.

The display processing section53acquires the specific information of the feature portion identified by the feature identification section52from the information storage section80, and supplies the acquired information to the video display device90so as to image and display the information on the video displayed on the display94in a superimposed manner. InFIG. 1, a reference numeral190denotes an example of a screen displayed on the display94of the video display device90.

FIG. 3is a flowchart illustrating a process of displaying information relating to a gaze target on the video display device90in the form of one image through which the wearer can visually grasp, select or set the contents (hereinafter referred to as information display processing). The information display processing ofFIG. 3is executed under control of the CPU of the robot controller50. Specifically, the display processing section53displays information relating to the gaze target (feature portion) while disposing the information side by side with the gaze target in the video by using the results of identifying the gaze target (feature portion) by the gaze target identification section51and the feature identification section52. The user wears the video display device90, performs a predetermined operation on the robot controller50, and activates the information display processing. First, it is determined whether or not the gaze target is identified by the gaze target identification section51and the feature identification section52(step S1). When the gaze target is not identified, the process waits until the gaze target is identified (step S1: NO). When the gaze target is identified (S1: YES), it is determined whether or not the gaze target is the tool12of the robot10(step S2).

When it is determined that the gaze point A according to the visual line direction E of the wearer is on the tool12and the gaze target is the tool12of the robot10(S2: YES), the display processing section53causes the video display device90to display an image which allows the wearer to visually perform at least one of grasping the setting state of the tool coordinate system, new setting or changing setting of the tool coordinate system, grasping of the setting state of the load, and new setting or changing of the load, depending on whether or not the tool coordinate system of the tool12is already-set and whether or not the load corresponding to the tool12is already-set (steps S6to S8).

Specifically, when the wearer watches the tool12(S2: YES), the robot10is stopped (S3: YES), and the tool coordinate system is not set (S4: NO), the display processing section53, as illustrated inFIG. 4, superimposes and displays a message301indicating that the tool coordinate system is not set and a selection menu302for selecting whether or not to newly set the tool coordinate system on the video on the display94(Step S8). InFIG. 4, a reference numeral201denotes a display screen on the display94in which the message301and the selection menu302are superimposed on a video including the robot10.

The wearer can select the selection menu302by the visual line on the display screen201(FIG. 5). For example, when the gaze point A of the wearer stays near “YES” on the selection menu302for several seconds, the robot controller50can determine that the wearer has selected “YES”. The tool coordinate system setting process when the wearer selects “YES” will be described later.

When the tool coordinate system is already-set (S4: YES) and the load is not set (S5: NO), the display processing section53superimposes and displays a message311indicating that the load is not set on the video on the display94in step S7(FIG. 6). At this time, the display processing section53also displays a selection menu312for selecting, by the visual line, whether or not to newly set the load information. InFIG. 6, a reference numeral202denotes a display screen on the display94in which the message311and the selection menu312are superimposed on a video including the robot10. Here, when the wearer selects “YES” on the selection menu312by the wearer's visual line, the display processing section53may further display a user interface screen for inputting the load setting value.

When the tool coordinate system is set (S4: YES) and the load is also set (S5: YES), the display processing section53superimposes and displays a current setting value321of the tool coordinate system or the load information on the video on the display94in step S6(FIG. 7). InFIG. 7, a reference numeral203denotes a display screen on the display94in which the setting values321of the tool coordinate system are superimposed and displayed on a video including the robot10. The display screen203also displays a selection menu322for instructing transition to processing for changing setting values or displaying of load information. Here, when the wearer selects “CHANGE” on the selection menu322by the wearer's visual line, the process shifts to a tool coordinate system setting process described later. When the wearer selects “TO THE LOAD INFORMATION” on the selection menu322by the visual line, the setting value of the load information is displayed instead of the setting value of the tool coordinate system. In the display image of the setting value of the load information, a menu for shifting to a screen for changing the setting value of the load information is displayed.

When it is determined in step S3that the robot10is in operation (S3: NO), the display processing section53causes the video display device90to display an image that enables the wearer to visually grasp the operation information of the robot10that moves the tool12. Specifically, when the robot10is in operation (S3: NO) and the robot10is executing an operation program (S9: YES), the display processing section53superimposes and displays information relating to the speed and the trajectory of the robot10on a video in step S10(FIG. 8). InFIG. 8, a reference numeral204denotes a display screen on the display94in which information331relating to the speed and trajectory of the robot10is superimposed and displayed on a video including the robot10. On the display screen204, character information indicating a name of the operation program, the operation form of the robot10, the speed override, the teaching speed, and the tool center point speed (TCP speed) is superimposed and displayed, and an operation trajectory T and teaching points P of the tool tip are superimposed and displayed graphically.

When the robot10is in operation (S3: NO) and the robot10is not executing an operation program (S9: NO), the robot controller50determines that the robot10is executing the operation by a jog operation using a teach pendant (not illustrated) connected to the robot controller50. In this case, in step S11, the display processing section53superimposes and displays information332relating to the jog operation on the video (FIG. 9). InFIG. 9, a reference numeral205denotes a display screen on the display94in which the operation speed, the operation range, and the traveling direction of the robot10by the jog operation are superimposed and displayed as characters and images on a video including the robot10.

When the gaze target of the wearer is a joint portion of the robot10(S2: NO, S12: YES), the display processing section53superimposes and displays information333relating to the current position of the joint portion and the movable range of the joint portion on the image in step13(FIG. 10). Here, it is assumed that the gaze point A of the wearer's visual line E is located at the 5th joint position (J5). InFIG. 10, a reference numeral206denotes a display screen on the display94in which information relating to the current position and the movable range of the 5th joint portion is superimposed and displayed as characters and images on a video including the robot10.

In order for an industrial robot to avoid interference with the environment surrounding the robot, an area in which the robot cannot enter may be defined, and the robot may have a function of stopping when the robot tries to enter the area. Hereinafter, such an area is referred to as an “entry prohibited area”. Information relating to the position of the entry prohibited area is stored in the information storage section80of the robot controller50, for example, as the position on a base coordinate system of the robot10. In step S14, when the gaze point A of the wearer is not the robot10but the surrounding environment of the robot10(S12: NO, S14: YES), the display processing section53superimposes and displays the information relating to the entry prohibited area on the video in the case where the entry prohibited area is defined in the surrounding area (Step S15;FIG. 11). InFIG. 11, a reference numeral207denotes a display screen on the display94in which a broken line H representing an entry prohibited area is superimposed and displayed as an image on a video including the robot10.

In steps S16to S21, when the gaze target identification section51identifies a selection switch provided on the robot controller50as the gaze target, the display processing section53causes the video display device90to display an image enabling the wearer to visually select the selection switch. Specifically, when the gaze target of the wearer is the robot controller50(S16: YES) and the robot10is stopped (S17: YES), it is further determined at which portion of the robot controller50the gaze point A of the wearer is (steps S18and S20). When the gaze point A by the wearer's visual line E is near a power switch55and it is determined that the feature portion of the gaze target is the power switch55(S18: YES), the display processing section53superimposes and displays, on the video, a selection menu341for selecting whether or not to perform power-on again in step S19(FIG. 12). InFIG. 12, a reference numeral208denotes a display screen on the display94in which the selection menu341is superimposed and displayed as character information on the video including the robot controller50. When the wearer selects “YES” on the selection menu341by the visual line, the robot controller50performs power-on again of the robot10.

When the gaze point A of the wearer's visual line E is near a mode selection switch56and it is determined that the feature portion of the gaze target is the mode selection switch56(S18: NO, S20: YES), the display processing section53superimposes and displays a message351indicating the current operation mode of the robot controller50and a selection menu352for switching the operation mode on a video in step S21(FIG. 13). InFIG. 13, a reference numeral209denotes a display screen on the display94in which the message351and the selection menu352are superimposed and displayed as character information on the video including the robot controller50. When the wearer selects either the T1 mode or the T2 mode by the visual line, the robot controller50switches the operation mode to the selected mode.

When the wearer watches an arbitrary point on the robot controller50(S16: YES) and the robot10is not stopped (S17: NO), it is determined that the operation program is being executed (S22: YES), and the display processing section53superimposes and displays information361indicating the execution state of the operation program and a selection menu362for selecting the pause or resume of the program on a video (Step S23;FIG. 14). InFIG. 14, a reference numeral210denotes a display screen on the display94in which information361indicating the state of the operation program and the selection menu362are superimposed and displayed as character information on the video including the robot controller50. When the wearer selects “YES” by the visual line on the selection menu362, the robot controller50pauses the operation program.

Next, a tool coordinate system setting process executed when “YES” of the selection menu302is selected in step S8(FIG. 4) or “CHANGE” of the selection menu322is selected in step S6(FIG. 7) will be described with reference toFIGS. 15 to 16. The tool coordinate system setting process is executed by the tool coordinate system setting section54included in the robot controller50. Here, the tool coordinate system setting process is a process of setting a tool coordinate system having a predetermined relationship with the flange coordinate system of the arm tip of the robot10. As illustrated inFIG. 15, the position of the tool tip is estimated by the intersection of visual lines of the wearer defined when the tool tip is viewed in a plurality of directions. As an example, as illustrated inFIG. 15, the wearer is made to observe the tool tip from 2 positions, and the position of the tool tip is obtained by the intersection of the two visual line directions E1and E2at that time.

FIG. 16is a diagram for explaining a specific method of obtaining the tool tip position. The flange coordinate system is defined as ΣF, and the base coordinate system of the robot10is defined as ER. The relationship between ΣF and ΣR is known from the current position of the robot10. Here, the coordinates (x, y, z) of the tool tip on the flange coordinate system ΣF will be obtained. The head coordinate systems that represent positions and postures of the head when the wearer watches the tool tip from two different positions are defined as ΣH and ΣH′, respectively. The ΣH and ΣH′ may be a coordinate system fixed to the video display device90. The head coordinate systems ΣH and ΣH′ can be obtained by using various position detection methods known in the art, such as a position detection method by using an acceleration sensor and a gyro sensor mounted on the video display device90, or a method for measuring by using an AR marker. Thus, the conversion between ΣR, ΣF, ΣH, and ΣH′ is known. Next, the starting points of the two visual line directions E1and E2measured by the visual line tracking section92are set to the origins of ΣH and ΣH′ respectively, and intersections of the visual line directions E1and E2with an arbitrary plane (e.g., the XY plane) defined on the flange coordinate system ΣF are defined as points P and P′ respectively. In this case, the coordinate (x, y, z) of the tool tip can be obtained as the point of intersection M between the line segment OP and the line segment O′P′ or as the point where the distance between the line segment OP and the line segment O′P′ is minimized. Thus, a tool coordinate system having a predetermined relationship with the flange coordinate system can be set. For example, the origin of the tool coordinate system may be set to the tool tip.

After steps S6to S8, S10, S11, S13, S15, S19, S20(NO), S21, and S23, the information display processing ends. After these steps, the process may be returned to step S1to continue the process.

As described above, according to the above embodiments, the user can grasp, select, or set information relating to a portion of interest for each object only by looking at the portion of interest. That is, according to the above embodiments, the user can access the desired information more efficiently.

While the disclosure has been described with reference to specific embodiments, it will be understood, by those skilled in the art, that various changes or modifications may be made thereto without departing from the scope of the following claims.

In the embodiment described above, the video display device90has a so-called closed type configuration including the display94configured to display video acquired by the video acquisition device91, but the video display device90may have a configuration including transmissive type display. The transmissive type display is constituted by a half mirror, and the information regarding the gaze target provided from the robot controller50is synthesized and displayed in the field of view of the wearer wearing the display.

When the video display device90has a configuration including a transmissive type display, the video acquisition device91may be connected to the robot controller50as a device separately provided from the video display device90. In such a configuration, the video display device90is provided with a sensor (acceleration sensor, gyro sensor, etc.) configured to detect the position and posture of the head of the wearer. The positions of the objects in the work space are known. In this case, the gaze target identification section obtains a visual line vector of the wearer toward a target object by grasping the relative position between the video display device90and the target object such as the robot10, the posture of the wearer's head, and the visual line information (eye direction), and identifies the gaze target in the video acquired by the video acquisition device. The position of the target object in the video acquired by the video acquisition device may be identified by using image recognition processing, a relative positional relationship between the video acquisition device and the target object, or the like.

The arrangement of the functional blocks illustrated inFIG. 2is an example, and the distribution of the functions to the robot controller50and the video display device90is not limited to the example illustrated inFIG. 2. For example, the robot controller50may be provided with the image processing section93configured to perform image processing for superimposing various kinds of information side by side with the gaze target in a video. In this case, a video signal on which various information is superimposed is transmitted from the robot controller50to the video display device90. The feature identification section52may be provided in the video display device90. In this case, the feature identification section52uses an object model acquired from the side of the robot controller50or retained in the feature identification section52to identify at which feature portion on the object the gaze point in the video is located.

When the gaze target identification section51identifies a gaze target portion on an object, the gaze target identification section51may set a predetermined range including the gaze point of the wearer as the gaze range and identify a portion included in the gaze range as the gaze target portion. In this case, for example, when the wearer looks at the vicinity of the tool of the robot, the tool is identified as the gaze target portion, thereby improving the user convenience. Further, the robot controller may be configured to allow a user to set the gaze range.

The program for executing the information display processing illustrated inFIG. 3can be recorded on various recording media (e.g., a semiconductor memory such as a ROM, an EEPROM and a flash memory, a magnetic recording medium, and an optical disk such as a CD-ROM and a DVD-ROM) readable by a computer.