Patent Publication Number: US-11040451-B2

Title: Teaching device and teaching method

Description:
BACKGROUND 
     1. Technical Field 
     The present invention relates to a teaching device and a teaching method. 
     2. Related Art 
     When causing a robot to perform a task, often control is performed such that the task is broken down into a plurality of processes and a manipulation target object moves between a plurality of teaching positions in each step. Ordinarily, it is necessary for the teaching positions to be taught before starting the operation of the robot. A technique of using a teaching pendant (for example, refer to JP-A-2003-236784), and a technique of a teacher personally moving a robot to perform direct teaching (for example, refer to JP-A-2016-124065) are known as techniques for performing the teaching. 
     For the teaching using the teaching pendant, it is necessary to have an extensive knowledge of the manipulation of the teaching pendant and the movement and rotation of the robot caused by the manipulation and it is difficult for a person other than an expert to perform the teaching. For the teaching using direct teaching, it is necessary for the teacher personally to move the robot and it is difficult to perform the teaching of a heavy robot, the teaching which accompanies minute positional adjustments, and the like. In other words, in a configuration in which the user directly or indirectly moves the robot to perform the teaching, the difficulty of the teaching is high. 
     SUMMARY 
     A teaching device according to an aspect of the invention includes an image acquisition unit which acquires an image including a manipulation target object which is linked to operations of an arm of a robot and a teaching position of the manipulation target object, a movement control unit which controls the arm to move the manipulation target object in the image to the teaching position, and a teaching information acquisition unit which acquires a state of the arm in a state in which the manipulation target object in the image is present at the teaching position as teaching information. 
     That is, the state of the arm in a state in which movement of the manipulation target object to the teaching position is performed based on an image and the movement of the manipulation target object to the teaching position is completed is acquired as teaching information. According to this configuration, it is not necessary for the user to perform the movement of the arm through direct touching or the movement of the arm through indirect manipulation which uses a teaching pendant, and it is possible to perform the teaching automatically. Therefore, it is possible to perform the teaching in an extremely easy manner. 
     The teaching device may be configured such that the teaching position is defined by a position relative to a reference target object which is included in the image. According to this configuration, it is possible to specify the teaching position based on the image. 
     The teaching device may be configured such that the teaching device further includes an input reception unit which receives an input of a position relative to the reference target object, and the movement control unit acquires a position of the manipulation target object relative to the reference target object based on the image of the manipulation target object and the reference target object and controls the arm such that the relative position becomes the teaching position. According to this configuration, the user is capable of easily performing the teaching by inputting a position relative to the reference target object. 
     The teaching device may be configured such that the movement control unit detects the reference target object based on the image and causes a display to display an image in which the reference target object is emphasized. According to this configuration, by visually recognizing the display, the user is capable of determining whether or not the reference target object which serves as a reference when specifying the teaching position is accurately detected in the image. 
     The teaching device may be configured such that the movement control unit detects the manipulation target object based on the image and causes a display to display an image in which the manipulation target object is emphasized. According to this configuration, by visually recognizing the display, the user is capable of determining whether or not the manipulation target object which serves as a reference when determining whether or not the arm is in the state to be taught is accurately detected in the image. 
     The teaching device may be configured such that the image acquisition unit acquires an image in which at least a portion of a background is removed excluding the manipulation target object and the teaching position. According to this configuration, it is possible to remove parts which may serve as noise for moving the manipulation target object from the image. 
     The teaching device may be configured such that the movement control unit controls the arm to align the manipulation target object in the image with a target image which is present at the teaching position. According to this configuration, it is possible to move the manipulation target object to the teaching position without specifying a virtual position based on the image. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The invention will be described with reference to the accompanying drawings, wherein like numbers reference like elements. 
         FIG. 1  is a perspective view of a robot. 
         FIG. 2  is a functional block diagram of a teaching device. 
         FIG. 3  is a flowchart of a teaching process. 
         FIG. 4  is a diagram illustrating an example of a user interface. 
         FIG. 5  is a diagram illustrating an example of a user interface. 
         FIG. 6  is a flowchart of a visual servo process. 
         FIG. 7  is a diagram illustrating an example of the visual servo. 
         FIG. 8  is a diagram illustrating an example of a user interface. 
         FIG. 9  is a diagram illustrating an example of a user interface. 
     
    
    
     DESCRIPTION OF EXEMPLARY EMBODIMENTS 
     Hereinafter, a description will be given of the embodiments of the invention in the following order with reference to attached drawings. Corresponding constituent elements in the drawings will be given the same reference numerals and duplicated description thereof will be omitted. 
     (1) Configuration of Robot: 
     (2) Configuration of Teaching Device: 
     (3) Teaching Process: 
     (4) Visual Servo Process: 
     (5) Other Embodiments: 
     (1) Configuration of Robot 
       FIG. 1  is a perspective view of a robot  1  of an embodiment of the invention. As illustrated in  FIG. 1 , the robot  1  is provided with an arm  10  and an end effector  20 . The arm  10  is a six axes arm including three bending joints B 1  to B 3  and three torsion joints R 1  to R 3 . The bending joints B 1  to B 3  are joints at which the members which configure the arm  10  rotate relative to each other centered on axes perpendicular to the length direction of the arm  10 . The torsion joints R 1  to R 3  are joints at which the members which configure the arm  10  rotate relative to each other centered on axes of the length direction of the arm  10 . The arm  10  is provided with motors (not illustrated) as drive sources for driving the bending joints B 1  to B 3  and the torsion joints R 1  to R 3 . The arm  10  is provided with encoders (not illustrated) which detect values corresponding to rotation angles of each motor. 
     The end effector  20  is mounted to a distal end of the arm  10 . By driving the six axes arm  10 , the robot  1  is capable of disposing the end effector  20  at an arbitrary position in a movable range and assuming an arbitrary posture (angle). In a case in which the term “position” is simply denoted in the present specification, this may include the posture. The end effector  20  is provided with a force sensor P and a gripper  21 . The force sensor P is a sensor which measures the translation force of the three axes which act on the end effector  20  and the rotational force (torque) which acts around the three axes. 
     The robot  1  is a general-use robot that becomes capable of various tasks through performing teaching. In the present embodiment, teaching for performing an assembly task in which a manipulation target object W 1  is inserted into a reference target object W 2  is performed, and a description will be given using a case in which the assembly task is performed as an example. In other words, the robot  1  moves the gripper  21  to a stock position of the manipulation target object W 1 , picks up the manipulation target object W 1  using the gripper  21 , and carries the manipulation target object W 1  to the space above the reference target object W 2 . The robot  1  performs a task of inserting the manipulation target object W 1  into a recessed portion of the reference target object W 2 . 
     In the present embodiment, the robot  1  and cameras  30   a  and  30   b  are connected to a robot control device  40 . The robot  1  is controlled by the robot control device  40 . The robot control device  40  is capable of executing visual servos which use the cameras  30   a  and  30   b . In the present embodiment, it is possible to use a teaching device  50  to teach the robot  1 . 
     The robot control device  40  is provided with an execution unit (CPU, RAM, ROM, and the like) of a program (not illustrated). The robot control device  40  is capable of performing input and output of signals via an interface (not illustrated) and is capable of acquiring the output of the force sensor P which is mounted to the end effector of the robot  1  and the output of an encoder E (not illustrated) which is embedded in the arm  10  and outputting control signals for controlling each of the motors which are provided in the arm  10 . The robot control device  40  is capable of acquiring images which are output by the cameras  30   a  and  30   b  via the interface (not illustrated). 
     In other words, based on the images which are output by the cameras  30   a  and  30   b , the robot control device  40  determines the target operation and is capable of performing a visual servo which controls the robot  1  to execute the target operation based on the output of the force sensor P and the encoder E. The robot control device  40  is capable of storing teaching information in a storage medium such as a ROM. In the present embodiment, the teaching information is the control information which is necessary for moving the manipulation target object W 1  to a teaching position which is taught in advance and the robot control device  40  is capable of controlling the robot  1  and moving the manipulation target object W 1  to the teaching position based on the teaching information. 
     In this controlling, the positions of each portion and the forces which act on each portion are defined by a plurality of coordinate systems. For example, it is possible to define the positions of and the forces acting on the arm  10  of the robot  1 , the end effector  20 , the gripper  21 , the manipulation target object W 1 , and the reference target object W 2  using a robot coordinate system (the orthogonal xyz coordinate system illustrated in  FIG. 1 ). 
     When the visual servo is performed, the relative posture of the cameras  30   a  and  30   b  with respect to the robot  1  is defined. In the present embodiment, the posture of the cameras  30   a  and  30   b  with respect to the robot  1  is defined by the orientation of a camera coordinate system with respect to the robot coordinate system. Specifically, the cameras  30   a  and  30   b  are provided with an area sensor in which a plurality of lines, each including a plurality of pixels, are lined up. In the present embodiment, an orthogonal coordinate system is formed by two axes which are parallel to the line direction of the area sensor and the direction in which the lines are lined up, respectively, and one axis which is parallel to an optical axis direction of an optical system which guides light to the area sensor, and a coordinate system which uses a predetermined position (for example, a middle point or the like between the optical axes of the cameras  30   a  and  30   b , or the like) as an origin point is defined as the camera coordinate system. In  FIG. 1 , the axes of the camera coordinate system are illustrated as an i axis, a j axis, and a k axis. 
     The camera coordinate system is fixed with respect to the cameras  30   a  and  30   b  and the origin point and the orientation of the axes change according to changes in the positions and postures of the cameras  30   a  and  30   b . The orientation of the camera coordinate system is defined by the orientation of the axes (the i axis, the j axis, and the k axis) of the camera coordinate system with respect to the axes (the x axis, the y axis, and the z axis) of the robot coordinate system. Specifically, a state in which the x axis, the y axis, and the z axis are parallel to the i axis, the j axis, and the k axis, respectively is used as a reference, and the orientation of the cameras  30   a  and  30   b  with respect to the robot  1  is defined due to the posture of the camera coordinate system being specified by an angle u by which the camera coordinate system is rotated with respect to the z axis, an angle v by which the camera coordinate system is rotated with respect to the y axis, and an angle w by which the camera coordinate system is rotated with respect to the x axis. 
     In the present embodiment, the positions of the cameras  30   a  and  30   b  are also defined from the perspective of the robot  1 . In other words, the position of the origin point of the camera coordinate system is defined as a coordinate in the robot coordinate system. However, in a case in which the positions of the cameras  30   a  and  30   b  from the perspective of the robot  1  are not considered in the visual servo, it is not essential to define the positions of the cameras  30   a  and  30   b  from the perspective of the robot  1  in the visual servo. As described above, in the present embodiment, the relationship between the coordinate systems is defined in advance and is recorded in the storage medium of the robot control device  40 . 
     (2) Configuration of Teaching Device 
     The teaching device  50  is provided with an execution unit (CPU, RAM, ROM, and the like) of a program (not illustrated). The teaching device  50  is capable of performing input and output of signals with the robot control device  40  via an interface (not illustrated). In other words, the teaching device  50  is capable of acquiring the output images and the like from the cameras  30   a  and  30   b  from the robot control device  40 , outputting instructions to the robot control device  40 , and executing the visual servo of the robot  1 . In the present embodiment, the teaching device  50  is provided with a function for teaching the robot  1  using the visual servo. 
       FIG. 2  is a block diagram illustrating the configuration of the teaching device  50 . The teaching device  50  is capable of acquiring the output images of the camera  30   a  and  30   b  and the output of the encoder E and the force sensor P and of controlling a motor M which is provided in the arm  10  by instructing the robot control device  40 . In  FIG. 2 , the input and output of signals via the robot control device  40  is illustrated using dashed lines inside the robot control device  40 . A display  50   a  and an input unit  50   b  are connected to the teaching device  50  and the teaching device  50  is capable of causing the display  50   a  to display an arbitrary image. The input unit  50   b  is provided with a keyboard, a pointing device, and the like and the teaching device  50  is capable of receiving input of a user. 
     The teaching device  50  functions as an image acquisition unit  51 , an input reception unit  52 , a movement control unit  53 , and a teaching information acquisition unit  54  due to a program (not illustrated). The image acquisition unit  51  is a function of acquiring an image which includes the manipulation target object W 1  which is linked to the operations of the arm  10  of the robot  1  and a teaching position of the manipulation target object W 1 . In other words, through the function of the image acquisition unit  51 , the teaching device  50  instructs the robot control device  40  and acquires the images which are acquired by the cameras  30   a  and  30   b  in a state in which the manipulation target object W 1  and the reference target object W 2  are included in the visual fields of the cameras  30   a  and  30   b.    
     In the present embodiment, the teaching position is a position that serves as a movement target of the manipulation target object W 1  and is defined by a position relative to the reference target object W 2 . For example, in a task in which the manipulation target object W 1  illustrated in  FIG. 1  is inserted into a recessed portion Cv of the reference target object W 2 , a position Po to be overlapped with a center of gravity Cg of the manipulation target object W 1  is defined as a position viewed from the reference target object W 2 . Therefore, when performing the teaching, the teaching device  50  uses the function of the image acquisition unit  51  to acquire the images which are obtained by the cameras  30   a  and  30   b  performing photography in a state in which a position relative to the reference target object W 2  (the position Po in the example illustrated in  FIG. 1 ) and the manipulation target object W 1  are included in the visual field. 
     Using the function of the image acquisition unit  51 , the teaching device  50  outputs the acquired images to the display  50   a  and causes the display  50   a  to display the images. Therefore, by visually recognizing the display  50   a , the user is capable of checking whether or not a state in which the manipulation target object W 1  and the reference target object W 2  are imaged by the cameras  30   a  and  30   b  is attained. 
     The input reception unit  52  is a function of receiving the input of a position relative to the reference target object W 2 . In other words, using the function of the input reception unit  52 , the teaching device  50  controls the display  50   a  and causes the display  50   a  to display a user interface screen for inputting the teaching position. Using the function of the input reception unit  52 , the teaching device  50  receives the input of the user to the input unit  50   b  and acquires the teaching position. 
     The teaching position may be defined by a position relative to the reference target object W 2 , and in the present embodiment, the teaching position is specified by inputting the position and the posture of a manipulation target object coordinate system which is fixed to the manipulation target object W 1  in the reference target object coordinate system which is fixed to the reference target object W 2 . Examples of the reference target object coordinate system which is fixed to the reference target object W 2  include an orthogonal coordinate system in which the origin point is present at the center of gravity of the reference target object W 2  and which includes axes parallel to the edges of the reference target object W 2 , and examples of the manipulation target object coordinate system which is fixed to the manipulation target object W 1  include an orthogonal coordinate system in which the origin point is present at the center of gravity of the manipulation target object W 1  and which includes axes parallel to the edges of the manipulation target object W 1 . 
     The teaching device  50  receives the teaching position by receiving the input of the position of the origin point of the manipulation target object coordinate system and the angles of the axes using the function of the input reception unit  52  and treating them as positions and angles in the reference target object coordinate system. According to this configuration, it is possible to easily and clearly define the teaching position. 
     The movement control unit  53  is a function of controlling the arm  10  to move the manipulation target object W 1  in the images to the teaching position. In other words, using the function of the movement control unit  53 , the teaching device  50  acquires the relative position of the manipulation target object W 1  with respect to the reference target object W 2  based on the images of the manipulation target object W 1  and the reference target object W 2  and controls the arm  10  such that the relative position becomes the teaching position. In other words, the teaching device  50  specifies an image of the manipulation target object W 1  and an image of the reference target object W 2  using template matching or the like from the images which are acquired using the function of the image acquisition unit  51 . The teaching device  50  specifies the positions (and postures) of the manipulation target object W 1  and the reference target object W 2  in the camera coordinate system based on the specified images. It is possible to realize the specification of the positions (and postures) using a configuration or the like which is associated with the positions (and postures) with respect to a template, for example. 
     When the positions (and postures) of the manipulation target object W 1  and the reference target object W 2  are specified in the camera coordinate system, the teaching device  50  specifies the relative position of the manipulation target object W 1  with respect to the reference target object W 2  based on the relationship with the coordinate system and compares the relative position with the teaching position. In a case in which both differ, the teaching device  50  moves the robot  1 . In other words, the teaching device  50  specifies the current position (and posture) of the arm  10  based on the output of the encoder E and specifies the control target for micro-movement to cause the manipulation target object W 1  to approach the teaching position. The teaching device  50  outputs the control target to the robot control device  40  and controls the robot  1  to realize the control target. When the teaching device  50  repeats the above-described process, eventually, the manipulation target object W 1  in the image will reach the teaching position. 
     According to this configuration, it is possible to control the robot  1  such that the manipulation target object W 1  moves to the teaching position which is instructed through a user using the input unit  50   b  and it is possible to set the robot  1  to a state to be taught without using a teaching pendant or performing direct teaching. Since the user is capable of designating the teaching position and performing the teaching by only inputting a position relative to the reference target object W 2  using the input unit  50   b , it is possible to easily perform trial and error. 
     The teaching information acquisition unit  54  is a function of acquiring the state of the arm  10  as teaching information in a state in which the manipulation target object W 1  in the images is present at the teaching position. In other words, when the manipulation target object W 1  in the images reaches the teaching position, the teaching device  50  acquires the output value of the encoder E via the robot control device  40  and acquires the angle of each arm  10  to realize the state in which the manipulation target object W 1  is present at the teaching position. The teaching device  50  outputs information indicating the angle of each arm  10  to the robot control device  40  as the teaching information. The robot control device  40  records the teaching information in the ROM (not illustrated) or the like. When the teaching information is recorded, the robot control device  40  is capable of causing the robot  1  to execute the task of moving the manipulation target object W 1  to the teaching position indicated by the teaching information. The teaching information may be saved automatically in the ROM (not illustrated) or the like, may be saved after confirmation of the positions and postures of the robot  1  by the user or the like, and it is possible to adopt various configurations. 
     According to this configuration, it is not necessary for the user to perform the movement of the arm  10  through the user personally directly touching the arm  10  or the movement of the arm  10  through indirect manipulation which uses a teaching pendant. The user is capable of automatically performing the teaching by only inputting the teaching position using the input unit  50   b . Therefore, it is possible to perform the teaching in an extremely easy manner. 
     (3) Teaching Process 
     Next a detailed description will be given of the teaching process which is executed by the teaching device  50  following the flowchart illustrated in  FIG. 3 . When the teaching device  50  is started up, the teaching device  50  controls the display  50   a  using the function of the input reception unit  52  and causes the user interface for the teaching process to be displayed on the display  50   a.    
       FIG. 4  is a diagram illustrating an example of a user interface. In the user interface illustrated in  FIG. 4 , designation buttons B 1  to B 4  of a task number, cameras, a manipulation target object, and a reference target object are provided and it is possible to designate the task number, the cameras, the manipulation target object, and the reference target object using the designation boxes. In other words, by manipulating the input unit  50   b , the user is capable of designating the task number with a numerical value, is capable of designating at least one of the cameras  30   a  and  30   b  as the cameras to be used in the teaching, and is capable of designating the manipulation target object and the reference target object. In the example illustrated in  FIG. 4 , the manipulation target object W 1  and the reference target object W 2  are designated in  FIG. 1 . 
     In the user interface illustrated in  FIG. 4 , buttons B 5  to B 7  for instructing image acquisition, target object detection, and teaching execution are further provided. In the user interface illustrated in  FIG. 4 , a display portion B 8  of the image which is captured by the camera  30   a  and a display portion B 9  of the image which is captured by the camera  30   b  are provided. 
     When the user instructs the image acquisition using the button B 5  in a state in which the task number, the cameras, the manipulation target object, and the reference target object are designated using the designation boxes, the teaching device  50  acquires images through the process of the image acquisition unit  51  (step S 100 ). In other words, captured images of the cameras which are designated by the designation box B 2  are acquired. When the images are acquired, the teaching device  50  controls the display  50   a  and causes the captured images of the designated cameras to be displayed on the display portions B 8  and B 9  of the user interface. 
     If the user visually recognizes the display portions B 8  and B 9  in this state, the user is capable of checking whether or not the manipulation target object W 1  and the reference target object W 2  are included in the visual fields of the cameras  30   a  and  30   b . If the manipulation target object W 1  and the reference target object W 2  are not included in the visual fields of the cameras  30   a  and  30   b , the user is capable of redoing until the manipulation target object W 1  and the reference target object W 2  are included in the visual fields of the cameras  30   a  and  30   b  by performing control of the robot  1 , movement of the reference target object W 2 , and the like. Naturally, the state in which the manipulation target object W 1  and the reference target object W 2  are included in the visual fields of the cameras  30   a  and  30   b  may be automatically achieved through the control of the robot  1  and the analysis of the images. 
     When the user instructs the detection of the target object using the button B 6  in a state in which the manipulation target object W 1  and the reference target object W 2  are included in the visual fields of the cameras  30   a  and  30   b , the teaching device  50  detects the manipulation target object W 1  and the reference target object W 2  using the function of the movement control unit  53  (step S 105 ). In other words, the teaching device  50  detects the manipulation target object W 1  and the reference target object W 2  in the images by acquiring a template image which is recorded on a ROM (not illustrated) or the like and performing a template matching process. 
     When the detection is appropriately performed, the teaching device  50  specifies the positions and the postures of the manipulation target object W 1  and the reference target object W 2  based on a template image. When the reference target object W 2  is detected, the teaching device  50  causes the display  50   a  to display an image in which the reference target object W 2  is emphasized using a process of the movement control unit  53 . In the present embodiment, a configuration is adopted in which an outline (an outermost circumferential edge) of the image of the reference target object W 2  is emphasized and illustrated, and in  FIG. 4 , a state is illustrated in which the outline of the reference target object W 2  is emphasized by a thick line. Naturally, it is possible to adopt various configurations in addition to that in which emphasis is used, and a configuration in which the outline is displayed using a specific color or the like may be adopted. 
     The emphasis is not displayed at an appropriate position in a state in which the detection of the reference target object W 2  fails or in a state in which the reference target object W 2  is not appropriately detected. Therefore, according to this configuration, by visually recognizing the display  50   a , the user is capable of determining whether or not the reference target object W 2  which serves as a reference when specifying the teaching position is accurately detected in the image. 
     Next, the teaching device  50  sets the operating conditions using the function of the movement control unit  53  (step S 110 ). In the present embodiment, the operating conditions are a normal end condition and an abnormal end condition of the visual servo operation and the teaching position, and the information indicating the operating conditions is stored in a ROM or the like. In the present embodiment, the end condition is predetermined. However, the teaching position can be input by the user. Naturally, the end condition may be possible to define by the user. 
     In the inputting of the teaching position, the teaching device  50  controls the display  50   a  using the function of the input reception unit  52  and causes the user interface for inputting the teaching position to be displayed on the display  50   a .  FIG. 5  illustrates an example of the user interface for inputting the teaching position. In the example illustrated in  FIG. 5 , the display portions B 8  and B 9  of the user interface illustrated in  FIG. 4  are replaced by a display portion B 10  and a designation box B 11 . 
     A three-dimensional model of the manipulation target object W 1  and the reference target object W 2  in the reference target object coordinate system is displayed in the display portion B 10  (in  FIG. 5 , the target objects are schematically illustrated). In other words, the teaching device  50  stores data depicting a three-dimensional model of the manipulation target object W 1  and the reference target object W 2  in a recording medium such as a ROM. The origin point and the directions of the axes of the reference target object coordinate system are fixed with respect to the reference target object W 2  and the axes of the reference target object coordinate system are displayed together with the three-dimensional model of the reference target object W 2  on the display portion B 10 . In  FIG. 5 , two axes (the X axis and the Z axis) of the reference target object coordinate system are in a visually recognizable state and the two axes are illustrated using dashed lines. 
     The origin point and the directions of the axes of the manipulation target object coordinate system are fixed with respect to the manipulation target object W 1  and the axes of the manipulation target object coordinate system are displayed together with the three-dimensional model of the manipulation target object W 1  on the display portion B 10 . In  FIG. 5 , two axes (the A axis and the B axis) of the manipulation target object coordinate system are in a visually recognizable state and the two axes are illustrated using dot-dash lines. The manipulation target object coordinate system in the present embodiment is the coordinate system in which the center of gravity of the manipulation target object W 1  is present. However, another position may be the origin point. 
     In the designation box B 11 , the position and the posture of the manipulation target object coordinate system can be designated as a position and a posture in the reference target object coordinate system. In other words, the positions X, Y, and Z which are indicated in the designation box B 11  are values representing the position of the origin point of the manipulation target object coordinate system in the reference target object coordinate system and are defined as positions on each axial direction of the X, Y, and Z axes which configure the reference target object coordinate system. The postures U, V, and W which are indicated in the designation box B 11  are values which represent a posture of the axes (the A axis, the B axis, and the C axis) of the manipulation target object coordinate system as angles with respect to the axes (the X axis, the Y axis, and the Z axis) of the reference target object coordinate system. 
     In the present embodiment, the teaching device  50  changes the image of the manipulation target object W 1  and the manipulation target object coordinate system according to the values which are input in the designation box B 11  using the function of the input reception unit  52 . In other words, in the example illustrated in  FIG. 5 , since only the position Z is 40 and the other values are 0 in the designation box B 11 , the center of gravity Cg of the manipulation target object W 1  is a position which is separated from the origin point of the reference target object coordinate system by a distance of 40 in the Z axis direction, and the center of gravity Cg and the origin point of the reference target object coordinate system align in the X axis direction and the Y axis direction. The axes of the reference target object coordinate system are parallel to the axes of the manipulation target object coordinate system. When another value, for example, the posture U is input, an image of a state in which the A axis is inclined with respect to the X axis according to an input value is rendered. 
     According to this configuration, the user is capable of inputting numerical values to the designation box B 11  and inputting the teaching position while moving the image of the manipulation target object W 1  on the display portion B 10 . The display content of the display portion B 10  illustrated in  FIG. 5  is an example and the inclination states of the target objects may be displayed by the manipulation of a pointing device or the like, and the position and the posture of the manipulation target object W 1  may be input by the manipulation of a pointing device or the like. 
     The end condition which constitutes one of the operating conditions may be various conditions, and examples of a normal end condition include a difference between the relative position of the manipulation target object W 1  with respect to the reference target object W 2  which is specified based on the image and the teaching position which is input by the user being less than or equal to a threshold. Examples of an abnormal end condition include any one or combination of a cumulative movement amount of the manipulation target object W 1  after the starting of the teaching process being greater than or equal to a threshold, an output value of the force sensor P being greater than or equal to a threshold, and detection failure or timing out of the detection of either or both of the manipulation target object W 1  and the reference target object W 2 . 
     Next, the teaching device  50  executes the visual servo process using the function of the movement control unit  53  (step S 115 ). In other words, the teaching device  50  acquires captured images of the cameras which are designated by the designation box B 2 , and based on the captured image, performs a process which instructs the robot control device  40 , causes the robot control device  40  to control the arm  10 , and causes the manipulation target object W 1  to move to the teaching position. The details of the visual servo process will be described later. 
     In a case in which the visual servo ends normally, or in a case in which the visual servo process ends abnormally, the teaching device  50  determines whether the end condition is the abnormal end condition or the normal end condition using the function of the movement control unit  53  (step S 120 ). In a case in which it is determined that the end condition is the abnormal end condition in step S 120 , the teaching device  50  repeats the processes of step S 115  onward. 
     In a case in which it is determined that the end condition is the normal end condition in step S 120 , the teaching device  50  acquires the teaching information using the function of the teaching information acquisition unit  54  (step S 125 ). In other words, the teaching device  50  acquires the output value of the encoder E via the robot control device  40 , acquires the information indicating the angle of each of the arms  10  as the teaching information, and outputs the teaching information to the robot control device  40 . 
     (4) Visual Servo Process 
     Next, a detailed description will be given of the visual servo process of step S 115 .  FIG. 6  is a flowchart of a visual servo process. In the present embodiment, the visual servo process is executed in a state in which the manipulation target object W 1  and the reference target object W 2  are included in the visual fields of the cameras  30   a  and  30   b . In the visual servo process, the teaching device  50  acquires the images of the cameras and acquires the position of the manipulation target object W 1  (step S 200 ). In other words, the teaching device  50  acquires the images which are output by the cameras  30   a  and  30   b.    
     The teaching device  50  performs feature extraction and analysis such as pattern matching on the images and specifies the position and the posture of the manipulation target object W 1 . As in the present embodiment, in a case in which there are greater than or equal to two cameras  30   a  and  30   b , the distance between the manipulation target object W 1  and the cameras may be specified using the parallax between the cameras and the position and the posture of the manipulation target object W 1  may be specified based on the distance. As a result of the analysis, a state is attained in which the position and the posture of the manipulation target object W 1  in the camera coordinate system are specified.  FIG. 7  is a diagram schematically illustrating the target objects in an image which is captured by the camera  30   a  or  30   b , and when step S 200  is performed, the position and the posture of the center of gravity Cg are specified as the position and the posture of the manipulation target object W 1 . 
     In step S 200 , the teaching device  50  detects the manipulation target object W 1  based on the images and causes the display  50   a  to display images in which the manipulation target object W 1  is emphasized. In the present embodiment, a configuration is adopted in which the outline (the outermost circumferential edge) of the image of the manipulation target object W 1  is emphasized and illustrated.  FIG. 8  illustrates an example of the user interface during the execution of the visual servo and the emphasized display is performed on the display portions B 8  and B 9  of the user interface. In this example, a state is illustrated in which the outline of the manipulation target object W 1  is emphasized with a thick line. Naturally, it is possible to adopt various configurations in addition to that in which emphasis is used, and a configuration in which the outline is displayed using a specific color or the like may be adopted. 
     The emphasis is not displayed at an appropriate position in a state in which the detection of the manipulation target object W 1  fails or a state in which the manipulation target object W 1  is not appropriately detected. Therefore, according to this configuration, by visually recognizing the display  50   a , the user is capable of determining whether or not the manipulation target object W 1  which serves as a reference of whether or not a teachable state is attained is accurately detected in the image. 
     Next, the teaching device  50  acquires the difference between the teaching position and the current position (step S 205 ). In other words, the teaching device  50  performs the feature extraction and the analysis such as pattern matching on the images and specifies the position and the posture of the reference target object W 2 . Even in this case, in a case in which there are greater than or equal to two cameras  30   a  and  30   b , the distance between the manipulation target object W 1  and the cameras may be specified using the parallax between the cameras and the position and the posture of the manipulation target object W 1  may be specified based on the distance. As a result of the analysis, a state is attained in which the position and the posture of the reference target object W 2  in the camera coordinate system are specified. 
     The teaching device  50  specifies the positions and the postures of the reference target object W 2  and the manipulation target object W 1  in the reference target object coordinate system based on the position and the posture of the reference target object W 2 . In other words, the teaching device  50  converts the positions and the postures in the camera coordinate system to positions and postures in the reference target object coordinate system. The teaching position is defined as a position relative to the reference target object W 2 , that is, an origin position and axial directions of the manipulation target object coordinate system in the reference target object coordinate system which is fixed to the reference target object W 2 . 
     The teaching device  50  compares the position and the posture of the center of gravity Cg of the current manipulation target object W 1  which is specified in step S 200  to the position and the posture of the teaching position (the origin position and the axial directions in the manipulation target object coordinate system) and acquires the difference. In  FIG. 7 , the position and posture of the manipulation target object W 1  in a state in which the manipulation target object W 1  is disposed at the teaching position is illustrated using the dashed line (reference numeral W 11 ) and the teaching position is the position Po and the posture of the center of gravity Cg in the state (where the difference in the posture in the example illustrated in  FIG. 7  is 0 (that is, it is not necessary to change the posture)). The difference in the position is a vector dP. Hereinafter, a description will be given of this example. However, there is no fundamental difference in the control even if the difference in the posture is limited. 
     Next, the teaching device  50  determines whether or not the end condition is satisfied (step S 210 ). In other words, in a case in which the difference which is acquired in step S 205  is less than or equal to a threshold, for example, in a case in which the difference is 0, the teaching device  50  determines that the normal end condition is satisfied. When a predetermined abnormal end condition such as the cumulative movement amount of the manipulation target object W 1  after the starting of the teaching process being greater than or equal to a threshold is satisfied, the teaching device  50  determines that the abnormal end condition is satisfied. In step S 210 , in a case in which the end condition is satisfied, the teaching device  50  ends the visual servo process. The threshold which is compared to the difference may be various values, may be a fixed value, and may be set by the user. If the value of the difference is set to a large value, the time until convergence is shortened. 
     Meanwhile, in step S 210 , in a case in which it is not determined that the end condition is satisfied, the teaching device  50  converts the difference which is acquired in step S 205  from an expression in the reference target object coordinate system to an expression in the robot target object coordinate system (step S 215 ). In other words, in the present embodiment, the posture of the cameras  30   a  and  30   b  with respect to the robot  1  is defined by the orientation of the camera coordinate system with respect to the robot coordinate system, and the correspondence relationship between the camera coordinate system and the reference target object coordinate system is acquired by the position and the posture of the reference target object W 2  being specified in step S 205 . Therefore, the teaching device  50  converts the difference to an expression in the robot coordinate system based on the correspondence relationship of the coordinate systems. If the example illustrated in  FIG. 7  is adopted, a positional change corresponding to the vector dP which is the difference in the reference target object coordinate system is specified in the robot coordinate system (the xyz coordinate system illustrated in  FIG. 1 ). 
     Next, the teaching device  50  multiplies the difference which is acquired in step S 215  by a gain and acquires the movement amount (step S 220 ). In other words, in the present embodiment, a configuration is adopted in which control of moving the center of gravity Cg of the manipulation target object W 1  gradually toward the teaching position by repeating minute movements is repeated. Therefore, the robot  1  is configured to gradually solve the difference by multiplying the difference by a gain (&lt;1) and performing a task of inserting the manipulation target object W 1  into the recessed portion of the reference target object W 2 . 
     When the movement amount is acquired by multiplying the difference by a gain, the teaching device  50  instructs the robot control device  40  and moves the robot  1  by the movement amount (step S 225 ). In other words, the teaching device  50  specifies the position of the center of gravity Cg in the robot coordinate system after the robot  1  is moved by the movement amount and causes the robot control device  40  to execute positional control in which the position of the center of gravity Cg is set to the target position. When the movement is performed, the teaching device  50  repeats the processes of step S 200  onward. As a result, the manipulation target object W 1  gradually approaches the teaching position and, finally, the manipulation target object W 1  is inserted into the recessed portion of the reference target object W 2 . 
     (5) Other Embodiments 
     The embodiment is an example for embodying the invention and it is possible to adopt various other embodiments. For example, the teaching device may be embedded in a robot control device or a robot, and the teaching device may be provided at a different location from the installation location of the robot, for example, in an external server or the like. The teaching device may be configured to control a plurality of robots. The teaching device may be distributed and disposed in a plurality of apparatuses. For example, a portion of the teaching device may be embedded in the robot and another portion of the teaching device may be disposed in a server or the like which is external to the robot. A portion of the configuration of the embodiment may be omitted, and the procedure of the process may be changed or omitted. 
     The arm may move and the posture may be configured to change relative to the installation position of the robot, and the number of degrees of freedom (the number of movable axes or the like) is optional. Various configurations may be adopted for the robot, and the robot may be an orthogonal robot, a horizontal multi-jointed robot, a vertical multi-jointed robot, a double-armed robot, or the like. Naturally, various configurations may be adopted for the number of axes, the number of arms, the configuration of the end effector, and the like. 
     In the embodiment, the robot is capable of executing the visual servo based on the captured images of the cameras  30   a  and  30   b . However, during the operation of the robot, a configuration may be adopted in which the cameras  30   a  and  30   b  are not used. In other words, a configuration may be adopted in which although the captured images of the cameras  30   a  and  30   b  are used in the teaching, the cameras  30   a  and  30   b  are not used during the operation of the robot after the teaching. 
     The teaching device  50  acquires the images and the like which are output by the cameras  30   a  and  30   b  via the robot control device  40 . However, naturally, the teaching device  50  may be connected to the cameras  30   a  and  30   b  and the robot  1  and the acquisition of the output images, the control of the robot  1 , and the like may be directly performed. 
     The image acquisition unit may be capable of acquiring an image which includes the manipulation target object which is linked to the operations of the arm of the robot and a teaching position of the manipulation target object. In other words, it is sufficient to be able to acquire the images which are the basis for teaching the position of the manipulation target object using the visual servo. With respect to the cameras which capture the images, two cameras may be used as in the embodiment, one camera may be used, and various other configurations may be used. The fixed positions of the cameras are not limited to the configuration of the embodiment, and, for example, the cameras may be attached to the robot. 
     The manipulation target object may be an object which is linked to the operations of the arm and with which the state of the arm is taught in a case in which the object is present at a specific position. Therefore, as described above, the target object which is ascertained by the robot may be the manipulation target object, a portion of the end effector of the robot (for example, the distal end portion of a manipulator) may be the manipulation target object, and various grids may be adopted. 
     The teaching position of the manipulation target object is a position at which the manipulation target object is to be disposed when teaching the state of the arm. As described above, the teaching position may be a state which is unrelated to the reference target object (for example, a specific position of the working space is specified in the robot coordinate system or the like). In a case in which the teaching position is defined as a position at a specified distance from the object or the like instead of the surface of the object, there is a case in which the manipulation target object is not present and only space is present at the teaching position before the teaching. Even in this case, an image including at least the teaching position is acquired such that the image acquisition unit can determine whether or not the manipulation target object moves to the teaching position. 
     The movement control unit may be capable of controlling the arm to move the manipulation target object in the image to the teaching position. In other words, the movement control unit may be capable of specifying the correspondence relationship between the manipulation target object and the teaching position based on the image and executing the visual servo which performs feedback control such that the relative positional difference between the manipulation target object which is specified from the image and the teaching position is effectively lost. 
     It is possible to adopt various configurations for the configuration for moving the manipulation target object to the teaching position. For example, the manipulation target object may be moved to the teaching position by specifying the manipulation target object and the teaching position in a virtual three-dimensional space based on an image and specifying the operations of the arm which changes position in the three-dimensional space. For example, a configuration may be adopted in which the arm is moved based on the image and the arm is moved until it is possible to confirm that the manipulation target object reaches the teaching position using an image. The movement direction of the arm may be specified based on the image, and it may be learned which direction in which to move the arm based on an image using machine learning or the like. 
     The teaching information acquisition unit may be capable of acquiring the state of the arm as teaching information in a state in which the manipulation target object in the image is present at the teaching position. In other words, in a case in which the fact that the manipulation target object reaches the teaching position is specified based on the image, as long as the state of the arm is reproduced, it is possible to move the manipulation target object from another position to the teaching position. Therefore, it is possible to use the state of the arm as the teaching information. 
     The teaching information may be defined in various manners. In other words, the teaching information may be defined according to elements to be reproduced in the process of operating the robot later. For example, in a case in which the state of the arm in a state in which the manipulation target object is present at the teaching position is to be reproduced together with the position and the posture, information reproducing the state of the arm (for example, parameters of movable portions of the entire arm (angles of joints and the like)) is included in the teaching information. In a configuration in which at least the position of the manipulation target object may be reproduced, the minimum information to be provided to the arm for disposing the manipulation target object at the position may become the teaching information. For example, in a case in which the position of the manipulation target object is designated, in a configuration in which the conditions of the arm for moving the manipulation target object to the position are specified, the position of the manipulation target object may become the teaching information. One or a combination of a plurality of parameters which may define the state of the arm may become the teaching information. 
     For a configuration in which the teaching position is defined as a position relative to the reference target object, it is possible to adopt various definitions other than that of the embodiment. For example, a position in a specific relationship (a specific distance, being parallel, being perpendicular, a specific angle, or the like) with respect to a specific part (a surface, a side, or the like) of the reference target object may be defined as the teaching position. 
     The input reception unit may be capable of receiving the input of a position relative to the reference target object, and other than the configuration in which a numerical value is input as in the embodiment, various configurations may be adopted. For example, it is possible to execute input which changes the position or the posture of a three-dimensional model of the manipulation target object on the screen, and a configuration in which a relative position is specified at the stage at which the input is confirmed or the like may be adopted. 
     Furthermore, the teaching device  50  may be configured to acquire an image in which at least a portion of the background is removed excluding the manipulation target object and the teaching position using the function of the image acquisition unit  51 . In other words, the manipulation target object and the teaching position (the reference target object in a case in which the reference target object is used) which are necessary for the teaching may be included in the captured images of the cameras  30   a  and  30   b . When an image of an object which is different from the manipulation target object and the teaching position is included, for example, time is necessary for the template matching and the acquisition of the difference between the current position and the teaching position, and there is a case in which these processes are difficult. 
     Therefore, if a configuration is adopted in which the background is removed from the image, it is possible to realize an increase in the speed and a reduction in the difficulty of the processes relating to the teaching. The removal of the background may be carried out using various methods, for example, it is possible to realize the removal of the background using processes such as capturing images in which the manipulation target object and the reference target object are not present using the cameras  30   a  and  30   b , comparing these images with images in which the manipulation target object and the reference target object are present, and correcting the gradation values of the pixels of portions at which there is no change to a fixed value. In a case in which the removal of the background is performed, the images of the cameras  30   a  and  30   b  attain a state such as that illustrated on the display portions B 8  and B 9  of  FIG. 4  (if the removal of the background is not performed, the background may be included in the images illustrated on the display portions B 8  and B 9 ). 
     The teaching device  50  may be configured to control the arm so as to align the manipulation target object in the image with the target image which is present at the teaching position using the function of the movement control unit  53 . In other words, for the method adopted when performing the visual servo, the method is not limited to a configuration in which the position and the posture are calculated in various coordinate systems based on the images as described above and the target is approached, and a configuration may be adopted in which the target is approached through the comparison of the images. 
       FIG. 9  is an example of the user interface in a configuration in which a target image is set. Similarly, as illustrated in  FIG. 9 , target images I 1  and I 2  are overlaid on the images which are captured by the cameras  30   a  and  30   b . In this state, the teaching device  50  executes control by the robot control device  40  such that the images of the manipulation target object W 1  overlap the target images. According to this configuration, it is possible to move the manipulation target object W 1  to the teaching position using a simple process. The target image may be created using various methods, for example, a configuration in which the user interface is configured such that the user is capable of moving the three-dimensional model of the manipulation target object W 1  to an arbitrary position and posture on the display, or the like may be adopted. In this configuration, if the three-dimensional model of the manipulation target object W 1  is disposed at the teaching position, it becomes possible to define the position and the posture of the target image. 
     The entire disclosure of Japanese Patent Application No. 2017-064541, filed Mar. 29, 2017 is expressly incorporated by reference herein.