Patent ID: 12186918

DETAILED DESCRIPTION

The following detailed description is merely exemplary in nature and is not intended to limit the invention or the application and uses of the invention. Furthermore, there is no intention to be bound by any expressed or implied theory presented in the preceding technical field, background, brief summary or the following detailed description.

Embodiments described herein relate to updating calibration of a camera used to control a robot, such as a robot used in a warehouse, a manufacturing plant, or in some other environment. The calibration may be referred to as camera calibration, and may be performed by, e.g., a robot control system (also referred to as a robot controller) to generate camera calibration information that facilitates an ability of the robot control system to control of the robot based on images captured (e.g., photographed) by the camera. For instance, the robot may be used to pick up a package in a warehouse, wherein placement of a robot arm or other component of the robot may be based on images of the package captured by the camera. In that instance, the camera calibration information may be used along with the images of the package to determine, for example, a location and orientation of the package relative to the robot arm of the robot. The camera calibration may involve, e.g., determining respective values of calibration parameters, such as respective estimates of intrinsic parameters of the camera (which may also be referred to as internal parameters), and/or an estimate of a relationship between the camera and its external environment. An intrinsic parameter of the camera may have one or more parameter values such as a matrix, a vector, or a scalar value. Further, examples of an intrinsic parameter include a projection matrix and a distortion parameter. In an instance, the camera calibration may involve determining the camera's position with respect to a fixed position in an external environment, which may be expressed as a transformation function representing the relationship between the camera and the fixed position in the external environment. In some cases, the camera calibration may be performed with the aid of a calibration pattern, which may have pattern elements disposed at defined locations on the calibration pattern. The camera may capture an image of the pattern elements of the calibration pattern (also referred to as a calibration image), and the camera calibration may be performed based on comparing an image of the pattern elements with the defined locations of the pattern elements. Camera calibration is discussed in more detail in U.S. application Ser. No. 16/295,940, filed on Mar. 7, 2019 and titled “METHOD AND DEVICE FOR PERFORMING AUTOMATIC CAMERA CALIBRATION TO CONTROL A ROBOT BASED ON IMAGES FROM A CAMERA”, hereinafter referred to as “Application MJ0021US1”, the entire content of which is incorporated herein by reference.

As stated above, one aspect of the present disclosure relates to updating a camera calibration that was performed at an earlier point in time so as to obtain camera calibration information that is still accurate at a later point in time. The camera calibration performed at the earlier point in time may generate camera calibration information that reflects a property of the camera at that earlier point in time, such as an intrinsic parameter of the camera or a relationship between the camera and its external environment at that point in time. In some cases, an earlier camera calibration may lose accuracy over time because the property of the camera may change over time. In a first example, an intrinsic parameter of the camera may change over time. Such a change may be caused by, e.g., a temperature change that alters a shape of a housing and/or a lens of the camera. In a second example, a relationship between the camera and its external environment may change over time. For instance, the camera may shift in position or orientation relative to, e.g., a base of the robot or a location in a warehouse. Such a change may be caused by, e.g., a temperature change that expands or contracts any component used to mount the camera, by a person or other object bumping into the camera, by a vibration in the camera's external environment (e.g., a warehouse), by a force from the camera's own weight (i.e., by gravity), or by some other factor. These changes may render the camera calibration information outdated as time progresses, and using this camera calibration information to position a robot arm or other component of the robot at a later point in time may lead to errors. In other words, if a property associated with the camera has changed over time but the camera calibration information is not updated to reflect such a change, the robot may operate based on outdated or otherwise incorrect camera calibration information, thereby causing undesirable errors in the robot's operation. To address the possibility that changes in one or more properties of the camera may occur, a robot control system may automatically determine updated camera calibration information based on more recent calibration images. In some cases, the robot control system may maintain a captured image set that acts as a sliding window identifying a certain number of the most recent calibration images, wherein the calibration images in the sliding window are used to update the camera calibration and to generate updated camera calibration information. In an embodiment, the updated camera calibration information may be used to perform a verification that detects a change in a property of the camera or a change in a relationship between the camera and its external environment. This change may be detected by, e.g., detecting a significant change in the calibration information over a short period of time. In some implementations, this detection may generate a notification signal if the property of the camera changes too quickly and/or too drastically. Such a change may in some cases reflect an undesirable condition in the camera or the camera's external environment. The notification signal may, e.g., be communicated to a user interface device (e.g., a laptop), which may alert a robot operator or other user of the potential existence of such an undesirable condition.

One aspect of the embodiments herein relates to updating a first camera calibration by performing a subsequent camera calibration (also referred to as a later camera calibration) based on more recently captured calibration images, which may be images of a calibration pattern. In some cases, the subsequent camera calibration may be performed if a sufficient number of calibration images that are more recent than the first camera calibration have been captured by a camera. For instance, the number of calibration images may have to reach a target count before the subsequent camera calibration is performed. The target count may be a defined value that was provided to the robot control system, or may be a defined value that is dynamically determined by the robot control system. In some cases, the subsequent camera calibration may use a captured image set that retains a defined number of the most recently captured images. In such cases, the captured image set, which may be stored on the robot control system or elsewhere, may act as a sliding window that adds the most recently captured calibration images and discards an equal number of the oldest calibration images, so that the subsequent camera calibration uses a defined number of the most recently captured calibration images, so as to provide updated camera calibration information.

In an embodiment, detecting a change in a property of the camera or in its external environment may be performed by comparing camera calibration information from the first camera calibration with camera calibration information from the subsequent camera calibration. The comparison may be performed by, e.g., a robot control system (also referred to as a robot controller), which may output a notification signal when camera calibration information from the subsequent camera calibration deviates from camera calibration information of the first camera calibration by an undesirable amount, and/or at an undesirable rate. In some cases, the first camera calibration may be an initial camera calibration performed before beginning a robot operation to determine camera calibration information for the camera. In an instance, the first camera calibration may be performed by controlling the robot to move the calibration pattern to reference locations within the camera's field of view (also referred to as a camera field of view of the camera) via an initial movement command, capturing initial calibration images of the calibration pattern at respective reference locations, and performing the first camera calibration based on the initial calibration images. The locations of the reference locations may be randomly selected or may be defined (e.g., defined manually, or dynamically calculated by the robot control system).

After the first camera calibration, the robot may perform the robot operation by moving the robot based on the camera calibration information determined from the first camera calibration. During the robot operation, more calibration images are captured by the camera at respective locations within the camera field of view and added to the captured image set. As stated above, the subsequent camera calibration may in some cases be performed only when a total number of calibration images in the captured image set has reached a defined target count, which may indicate that a sufficient number of recent calibration images are available for reliable camera calibration. In some cases, the target count may indicate a minimum number of calibration images for reliably updating the camera calibration. In an embodiment, after the robot control system performs an initial camera calibration, or more generally a first camera calibration, the robot control system may subsequently receive calibration images and place them in a captured image set. The calibration images in the captured image set may thus be used to perform a subsequent camera calibration, or more generally a second camera calibration, which outputs updated camera calibration information. In an embodiment, an amount of deviation between the camera calibration information from the first camera calibration and the updated camera calibration information from the second camera calibration may be determined. If the amount of deviation exceeds a defined threshold, a notification signal is output to indicate that the amount of deviation exceeds the defined threshold (which may also be referred to as a deviation threshold). In an embodiment, the amount of deviation exceeding the defined threshold may indicate that the robot operation may not be reliably performed, and/or that an operator or other user needs to be notified. Thus, if the amount of deviation exceeds a defined threshold, the robot operation may be stopped or paused, to prevent undesirable errors in robot tasks during the robot operation, and/or a notification may be output to a user interface device. On the other hand, if the amount of deviation does not exceed the defined threshold, the robot operation may be continued, while capturing new calibration images of the calibration pattern.

Depending on the robot's characteristics or configurations, the calibration images may be captured only during an idle period of a robot operation, or may be captured without the use of an idle period (e.g., during a period in which the robot is picking up objects). In an embodiment, the idle period may be a time period during which the robot is free from a robot task during the robot operation. In one example, an idle period may be used for situations in which a calibration pattern is disposed on a portion of the robot that does not regularly face the camera while a robot task is being performed. In such an example, the idle period may be utilized to move the calibration pattern to face the camera and capture the calibration pattern via the camera. In another example, an idle period may be skipped for situations in which the calibration pattern may be, e.g., disposed on a portion of the robot that may regularly face the camera while the robot is performing a robot task.

In an embodiment, if the calibration images are to be captured only during an idle period, the second camera calibration may also be completed only during an idle period. Completing the second camera calibration may involve, e.g., performing calculations to determine intrinsic parameters based on the calibration images, and/or to determine transformation functions based on the calibration images that relate the camera to its external environment, as discussed in more detail in U.S. application Ser. No. 16/295,940, filed on Mar. 7, 2019 and titled “METHOD AND DEVICE FOR PERFORMING AUTOMATIC CAMERA CALIBRATION TO CONTROL A ROBOT BASED ON IMAGES FROM A CAMERA”, the entire content of which is incorporated herein by reference. In some instances, the calculations or other steps for completing the second camera calibration may be performed only during an idle period that is at least as long as a defined calibration time period. The defined calibration time period may indicate a time necessary to complete the second camera calibration (e.g., a time necessary to perform the calculations). If an amount of time in the idle period is shorter than the defined calibration time period, then the calculations or other steps involved in completing the second camera calibration may be postponed to a subsequent idle period. In some cases, if the calculations are postponed to a subsequent idle period, a current idle period may still be used to control the camera to capture more calibration images, so that when the calculations are performed in the subsequent time period, they are performed with more recently captured calibration images from the current idle period.

FIG.1Aillustrates a block diagram of a robot operation system100(also referred to as a system100) for performing automatic camera calibration and automatic update of the camera calibration. The robot operation system100includes a robot150, a robot control system110(also referred to as a robot controller), and a camera170. In an embodiment, the system100may be located within a warehouse, a manufacturing plant, or other premises. The robot control system110may be configured to perform camera calibration, which is discussed in more detail below, to determine camera calibration information that is later used to control the robot150to perform a robot operation, such as picking up packages in the warehouse. The robot control system110may further be configured to update camera calibration, which is also discussed in more detail below, and to detect a situation in which camera calibration information is changing too much or too quickly, which may indicate an undesirable condition in the camera170or in its external environment. In some cases, the robot control system110is configured to perform the camera calibration and to control the robot150to perform robot operation based on the camera calibration information. In some cases, the robot control system110may form a single device (e.g., a single console or a single computer) that communicates with the robot150and the camera170. In some cases, the robot control system110may include multiple devices.

In some cases, the robot control system110may be dedicated to performing the camera calibration and/or update of the camera calibration, and may communicate the most current camera calibration information to another control system (also referred to as another controller, not shown) that then controls the robot150to perform a robot operation based on the most current camera calibration information. The robot150may be positioned based on images captured by the camera170and on the camera calibration information. More specifically, the robot control system110may, in an embodiment, be configured to generate movement commands based on the images and based on the camera calibration information, and to communicate the movement commands to the robot150to control movement of its robot arm. In some cases, the robot control system110is configured to perform an update of the camera calibration during an idle period in the robot operation. In some cases, the robot control system110is configured to perform the update while performing a robot operation with the robot150.

In an embodiment, the robot control system110may be configured to communicate via a wired or wireless communication with the robot150and the camera170. For instance, the robot control system110may be configured to communicate with the robot150and/or the camera170via a RS-232 interface, a universal serial bus (USB) interface, an Ethernet interface, a Bluetooth® interface, an IEEE 802.11 interface, or any combination thereof. In an embodiment, the robot control system110may be configured to communicate with the robot150and/or the camera170via a local computer bus, such as a peripheral component interconnect (PCI) bus.

In an embodiment, the robot control system110may be separate from the robot150, and may communicate with the robot via the wireless or wired connection discussed above. For instance, the robot control system110may be a standalone computer that is configured to communicate with the robot150and the camera170via a wired connection or wireless connection. In an embodiment, the robot control system110may be an integral component of the robot150, and may communicate with other components of the robot150via the local computer bus discussed above. In some cases, the robot control system110may be a dedicated control system (also referred to as a dedicated controller) that controls only the robot150. In other cases, the robot control system110may be configured to control multiple robots, including the robot150. In an embodiment, the robot control system110, the robot150, and the camera170are located at the same premises (e.g., warehouse). In an embodiment, the robot control system110may be remote from the robot150and the camera170, and may be configured to communicate with the robot150and the camera170via a network connection (e.g., local area network (LAN) connection).

In an embodiment, the robot control system110may be configured to retrieve or otherwise receive images of a calibration pattern160disposed on the robot150(e.g., on a robot arm of the robot) from the camera170. In some instances, the robot control system110may be configured to control the camera170to capture such images. For example, the robot control system110may be configured to generate a camera command that causes the camera170to capture an image of a field of view of the camera170(also referred to as a camera field of view), and to communicate the camera command to the camera170via the wired or wireless connection. The same command may cause the camera170to also communicate the image to the robot control system110, or more generally to a storage device accessible by the robot control system110. Alternatively, the robot control system110may generate another camera command that causes the camera170, upon receiving the camera command, to communicate an image(s) it has captured to the robot control system110. In an embodiment, the camera170may automatically capture an image in its camera field of view, either periodically or in response to a defined triggering condition, without needing a camera command from the robot control system110. In such an embodiment, the camera170may also be configured to automatically, without a camera command from the robot control system110, communicate the image to the robot control system110or, more generally, to a storage device accessible by the robot control system110.

In an embodiment, the robot control system110may be configured to control movement of the robot150via movement commands that are generated by the robot control system110and communicated over the wired or wireless connection to the robot150. The robot150may be configured to have the calibration pattern160on the robot150. For example, the calibration pattern160may be permanently disposed on the robot150, or may be a separate component that can be attached to and detached from the robot150.

In an embodiment, the only images used in the system100to control the robot150may be those captured by the camera170. In another embodiment, the system100may include multiple cameras, and the robot150may be controlled by images from the multiple cameras.

FIG.1Billustrates a robot operation system100A that includes the robot150, the robot control system110, the camera170, and a user interface device180. The robot150may have the calibration pattern disposed on the robot150. In some cases, the robot operation system100A may perform an initial camera calibration, or more generally a first camera calibration, to obtain initial camera calibration information, which may act as baseline camera calibration information. The robot operation system100A may subsequently perform additional camera calibrations over time to obtain updated camera calibration information. The robot operation system100A may monitor whether an amount of deviation between the updated camera calibration information and the baseline camera calibration information exceeds a defined threshold. If the amount of deviation is determined to exceed the defined threshold during the robot operation, then the robot control system110may output a notification signal to the user interface device180. The user interface device180may be configured to interface with an operator of the robot150, such as an employee at a warehouse in which the robot150is located. The user interface device180may include, e.g., a tablet computer or desktop computer that provides a user interface displaying information relating to operation of the robot150. In an instance, the user interface device180may provide an alarm or other alert to notify the operator of the amount of deviation exceeding the defined threshold. The notification signal may reflect, e.g., an undesirable condition in which a property of the camera170is changing too much or too quickly, or in which a relationship between the camera170and the robot150is changing too much or too quickly.

FIG.1Cdepicts a block diagram of the robot control system110. As illustrated in the block diagram, the robot control system110includes a control circuit111, a communication interface113, and a non-transitory computer-readable medium115(e.g., memory). In an embodiment, the control circuit111may include one or more processors, a programmable logic circuit (PLC) or a programmable logic array (PLA), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or any other control circuit.

In an embodiment, the communication interface113may include one or more components that are configured to communicate with the camera170ofFIG.1A or1Band the robot150ofFIG.1A or1B. For instance, the communication interface113may include a communication circuit configured to perform communication over a wired or wireless protocol. As an example, the communication circuit may include a RS-232 port controller, a USB controller, an Ethernet controller, a Bluetooth® controller, a PCI bus controller, any other communication circuit, or a combination thereof.

In an embodiment, the non-transitory computer-readable medium115may include computer memory. The computer memory may comprise, e.g., dynamic random access memory (DRAM), solid state integrated memory, and/or a hard disk drive (HDD). In some cases, the camera calibration may be implemented through computer-executable instructions (e.g., computer code) stored on the non-transitory computer-readable medium115. In such cases, the control circuit111may include one or more processors configured to perform the computer-executable instructions to perform updating of camera calibration (e.g., the steps illustrated inFIGS.4A,4B,9, and11).

FIG.1Ddepicts a block diagram of the camera170that includes one or more lenses171, an image sensor173, and a communication interface175. The communication interface175may be configured to communicate with the robot control system110ofFIG.1A,1, or1C, and may be similar to the communication interface113ofFIG.1Cof the robot control system110. In an embodiment, the one or more lenses171may focus light that is coming from outside the camera170onto the image sensor173. In an embodiment, the image sensor173may include an array of pixels configured to represent an image via respective pixel intensity values. The image sensor173may include a charge-coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor, a quanta image sensor (QIS), or any other image sensor.

As stated above, the camera calibration may be performed in order to facilitate the control of a robot based on images captured by a camera. For instance,FIG.2depicts a robot operation system200(also referred to as the system200) in which the images are used to control a robot250to perform a robot operation, such as an operation to pick up an object292in a warehouse. More specifically, the system200may be an embodiment of system100ofFIG.1A, and includes a camera270, the robot250, and the robot control system110. The camera270may be an embodiment of the camera170ofFIG.1A,1B, or1D, and the robot250may be an embodiment of the robot150ofFIG.1A or1B. The camera270may be configured to capture an image of the object292(e.g., a package for shipping) disposed on a conveyor belt293in the warehouse, and the robot control system110may be configured to control the robot250to pick up the object292. When there are one or more objects on the conveyor belt293, the robot control system110may be configured to schedule movement of the robot250to pickup the objects. The robot control system110may in some cases be configured to detect an idle period for the robot operation by detecting when there are no objects on the conveyor belt293, or when there are no objects on the conveyor belt293that are within reach of the robot250.

In the embodiment ofFIG.2, the robot250may have a base252and a robot arm that is movable relative to the base252. More specifically, the robot arm may comprise a plurality of links254A through254E, and a robot hand255attached to the link254E. The plurality of links254A through254E may be rotatable relative to each other, and/or may be prismatic links that are movable linearly with respect to each other. BecauseFIG.2involves the robot250that is used to pick up objects, the robot hand255may include grippers255A and255B used to grab the object292. In an embodiment, the robot control system110may be configured to communicate a movement command to rotate one or more of the links254A through254E. The movement command may be a low-level command, such as motor movement commands, or a high-level command. If the movement command from the robot control system110is a high-level command, the robot150may be configured to convert the high-level command to a low-level command.

In an embodiment, the camera calibration information determined from the camera calibration describes a relationship between the camera270and the robot250, or more specifically a relationship between the camera270and a world point294that is stationary relative to the base252of the robot250. The world point294may represent a world or other environment in which the robot250is located, and may be any imaginary point that is stationary relative to the base252. In other words, the camera calibration information may include information describing a relationship between the camera270and the world point294. In an embodiment, this relationship may refer to a location of the camera270relative to the world point294, as well as an orientation of the camera270relative to a reference orientation for the robot250. The above relationship between the camera270and the world point294may be referred to as a camera-to-world relationship, and may be used to represent a relationship between the camera270and the robot250. In some cases, the camera-to-world relationship may be used to determine a relationship between the camera270and the object292(also referred to as a camera-to-object relationship), and a relationship between the object292and the world point294(also referred to as an object-to-world relationship). The camera-to-object relationship and the object-to-world relationship may be used to control the robot250to pick up the object292.

In an embodiment, the camera calibration information may describe an intrinsic parameter of the camera270, where the intrinsic parameter may be any parameter whose value is independent of a location and an orientation of the camera270. The intrinsic parameters may characterize a property of the camera270, such as its focal length, a size of its image sensor, or an effect of lens distortion introduced by the camera270.

An example showing a detailed structure of an example robot350is depicted inFIG.3, which depicts a robot operation system300that includes the robot control system110in communication with a camera370and the robot350. The camera370may be an embodiment of the camera170/270ofFIG.1A,1B,1D, or2, respectively, and the robot350may be an embodiment of the robot150/250ofFIG.1A,1B, or2respectively. The camera370may be capable of capturing images (e.g., calibration images) within a field of view of the camera370, which may also be referred to as a camera field of view330. The robot350may include a base352and a robot arm movable relative to the base352. The robot arm includes one or more links, such as links354A through link354E. In an embodiment, the links354A-354E may be rotatably attached to each other. For instance, the link354A may be rotatably attached to the robot base352via a joint356A. The remaining links354B through354E may be rotatably attached to each other via joints356B through356E. In an embodiment, the base352may be used to mount the robot350to, e.g., a mounting frame or mounting surface (e.g., floor of a warehouse). In an embodiment, the robot350may include a plurality of motors that are configured to move the robot arm by rotating the links354A-354E. For instance, one of the motors may be configured to rotate the first link354A with respect to the joint356A and the base302, as shown with the dotted arrow inFIG.3. Similarly, other motors of the plurality of motors may be configured to rotate the links354B through354E. The plurality of motors may be controlled by the robot control system110.FIG.3further depicts the robot hand355disposed in a fixed manner on the fifth link354E. The robot hand355may have a calibration pattern320thereon, such that the robot control system110may capture images of the calibration pattern320via the camera370and perform camera calibration based on the captured images of the calibration pattern320. For example, the robot control system110may move the robot arm such that the calibration pattern320may be within the camera field of view330and visible to the camera370when capturing the images of the calibration pattern320. The robot hand355may be removable and replaceable with another robot hand.

As stated above, one aspect of the present application relates to continuing to capture images of the calibration pattern (which may be referred to as calibration images) after a first camera calibration (e.g., an initial camera calibration), and performing an additional camera calibration (e.g., a subsequent camera calibration) based on the more recently captured calibration images. In some cases, the more recently captured calibration images may be added to a captured image set, which may act as a sliding window that maps to a defined number of the most recently captured calibration images, so that the additional camera calibration may be based on the defined number of most recently captured calibration images.

FIGS.4A and4Bprovide a flow diagram that depicts example steps of a method400for updating camera calibration. In an embodiment, the method400may be performed by the control circuit111ofFIG.1Cof the robot control system110ofFIG.1A,1, or1C. As stated above, the robot control system110may include the control circuit111and the communication interface113ofFIG.1C, which is configured to communicate with the robot150ofFIG.1A or1B, and the camera170ofFIG.1A,1B, or1D, which has a camera field of view. The robot150may have a base (e.g. the base252ofFIG.2or the base352ofFIG.3) and a robot arm (e.g. the robot arm ofFIG.2orFIG.3) movable relative to the base, and the robot arm may have a calibration pattern (e.g. the calibration pattern160ofFIG.1A or1Bor the calibration pattern320ofFIG.3) disposed thereon.

In an embodiment, method400may begin with step401, in which the control circuit111performs a first camera calibration to determine camera calibration information for the camera (e.g.,170). The control circuit111may perform the first camera calibration based on images of a calibration pattern (also referred to as calibration images). In an aspect, the first camera calibration may be performed by: controlling the robot arm to move the calibration pattern to a first set of one or more locations within the camera field of view, receiving a first set of one or more calibration images from the camera (e.g., camera170) via the communication interface (e.g., communication interface113), where the camera (e.g.,170) is configured to capture the first set of one or more calibration images while the calibration pattern is or was at the first set of one or more locations, respectively, and performing the first camera calibration based on the first set of one or more calibration images. The first set of one or more locations may be randomly selected or may be defined (e.g., defined manually, or dynamically calculated by the robot control system). Camera calibration is discussed in more detail in in U.S. application Ser. No. 16/295,940, filed on Mar. 7, 2019 and titled “METHOD AND DEVICE FOR PERFORMING AUTOMATIC CAMERA CALIBRATION TO CONTROL A ROBOT BASED ON IMAGES FROM A CAMERA”, the entire content of which is incorporated herein by reference.

An example environment in which step401and other steps of method400are performed is depicted inFIGS.5A and5B, which depicts a robot operation system500that includes the robot control system110in communication with a camera570and a robot550, according to one aspect. The camera570may be an embodiment of the camera170/270/370ofFIG.1,2, or3, respectively, and the robot550may be an embodiment of the robot150/250/350ofFIG.1,2, or3, respectively. The robot550may include a base552and a robot arm movable relative to the base552. The robot arm includes one or more links, such as links554A through link554E. In an embodiment, the links554A-554E may be rotatably attached to each other. For instance, the link554A may be rotatably attached to the robot base552. The remaining links554B through554E may be rotatably attached to each other via a plurality of joints. In an embodiment, the base552may be used to mount the robot550to, e.g., a mounting frame or mounting surface (e.g., floor of a warehouse). The robot550may operate in a similar manner to the robot350. For instance, the robot550may include multiple motors configured to move the robot arm by rotating the links554A-554E. A robot hand555may be disposed in a fixed manner on the fifth link554E and may have a calibration pattern520disposed on the robot hand555.FIG.5Aillustrates an embodiment in which the calibration pattern520may not regularly face the camera570while the robot550is performing a robot task (e.g., picking up a first object582A) during a robot operation because the robot hand555may move in various directions and may rotate away from the camera570to perform the robot task. As discussed in more detail below, such an embodiment may wait until an idle period (e.g., when the robot550is not required to pick up an object) to rotate the calibration pattern520to face the camera570so as to capture an image of the calibration pattern520(also referred to as a calibration image of the calibration pattern520). For instance,FIG.5Billustrates a situation in which the robot550may, during an idle period in which the robot550is not performing a robot task, move the calibration pattern520to face the camera570to allow the camera570to capture a calibration image of the calibration pattern520. InFIG.5B, the robot550may have already finished picking up objects582A and582B, while a distance between the object582C and the robot hand555of the robot550may be more than a defined threshold.

As stated above, in some cases the control circuit111may control the camera170/670ofFIG.1or6, respectively to capture the calibration images without the use of an idle period. An example of such a situation is depicted inFIGS.6A and6B, which provide another example environment in which step401and other steps of method400may be performed.FIGS.6A and6Bdepict a robot operation system600(also referred to as the system600) that includes the robot control system110in communication with the camera670and a robot650.FIG.6Ais a side view of the system600andFIG.6Bis a top view of the system600. The camera670may be an embodiment of the camera170/270/370ofFIG.1,2, or3, respectively, and the robot650may be an embodiment of the robot150/250/350ofFIG.1,2, or3, respectively. The robot650may include a base652and a robot arm movable relative to the base652. The robot arm includes one or more links, such as links654A through link654E. In an embodiment, the links654A-654E may be rotatably attached to each other. For instance, the link654A may be rotatably attached to the robot base652. The remaining links654B through654E may be rotatably attached to each other via a plurality of joints. In an embodiment, the base652may be used to mount the robot650to, e.g., a mounting frame or mounting surface. The robot650may operate in a similar manner to the robot350.

In the embodiment ofFIGS.6A and6B, a calibration pattern620is disposed on a portion of the link654E, which is constantly facing upward, toward the camera670, even while the robot650is performing a robot task by interacting with the objects682. In one example, the objects may be packages to be loaded onto or off of a pallet (i.e., packages to be palletized or de-palletized). The robot650may be a depalletizer robot configured to stack or unstack the objects682, and thus may have the calibration pattern620facing the camera670while performing the robot task by interacting with the objects682, as shown inFIG.6B. Because the calibration pattern620is facing the camera670even when the robot650is engaged in a robot operation and is not idle, the robot control system110may control the camera670to capture the calibration images via the camera670without the use of the idle periods.

In an embodiment, the first camera calibration of step401may be performed using, e.g. the calibration pattern520/620ofFIG.1,5A,5B6A, or6B. In some cases, the first camera calibration may be an initial camera calibration performed before starting a robot operation. Alternatively, the first camera calibration may be performed during a robot operation. The robot control system110may control the robot arm of, e.g., the robot550/650to move (e.g., via movement commands) the calibration pattern (e.g.,520/620) to various locations within a field of view510/610of the camera570/670(also referred to as camera field of view) ofFIG.5A,5B,6A, or6B, respectively, and to capture images of the calibration pattern520/620(which may be referred to as calibration images of the calibration pattern520/620) at such locations. Subsequently, the robot control system110may perform the first camera calibration to determine the camera calibration information for the camera (e.g.,570/670) based on the calibration images of the calibration pattern520/620. The robot control system110may control the robot550/650to move based on the camera570/670and the camera calibration information of the camera570/670. The accuracy of the camera calibration information may affect the accuracy of the movement of the robot550/650. Hence, the robot control system110may control the camera570/670to capture, via the camera570/670, new calibration images over time, wherein the new calibration images are more recent than the calibration images used to perform the first camera calibration. The new calibration images may be used to perform additional camera calibration, which may yield camera calibration information that is up to date.

Returning toFIG.4A, the method400may further include step403, in which the control circuit111controls, based on the camera calibration information, movement of the robot arm to perform a robot operation, e.g., by outputting a first movement command that is based on the camera calibration information to the robot (e.g.,550/650) via the communication interface (e.g.,113). For instance, the control circuit111may output the first movement command that is based on the camera calibration information to the communication interface (e.g.,113), wherein the communication interface (e.g.,113) is configured to communicate the first movement command to the robot to cause the robot arm to move to perform a robot operation (e.g., picking up an object on a conveyor belt).

In step405, the control circuit111controls the robot arm, after the first camera calibration, to move the calibration pattern (e.g.,520/620) to one or more locations within the camera field of view (e.g.,510/610), such as by outputting a second movement command to the robot via the communication interface. For instance, after the first camera calibration, the control circuit111may output the second movement command to the communication interface113, where the communication interface113is configured to communicate the second movement command to the robot to cause the robot arm to move the calibration pattern to one or more locations within the camera field of view (e.g.,510/610). The one or more locations may be randomly selected, or may be defined locations. The control circuit111may further control the robot arm to orient the calibration pattern (e.g.,520/620) at each of the one or more locations so that the calibration pattern is visible to the camera (e.g.,570/670).

In step407, the control circuit111receives one or more calibration images from the camera (e.g.,170/570/670) via the communication interface (e.g.,113), where the one or more calibration images are respectively captured by the camera (e.g.,170/570/670) when the calibration pattern (e.g.,520/620) is at the one or more respective locations (this may also be referred to as the one or more calibration images being respectively captured at the one or more locations). For instance, the control circuit111may receive, from the communication interface113, the one or more calibration images, where the communication interface113is configured to receive the one or more calibration images from the camera170, wherein the one or more calibration images are respectively captured when the calibration pattern is at the one or more locations, respectively, within the camera field of view. Each image of the one or more calibration images may be captured at a respective location of the one or more locations. In some instances, the control circuit111may use the communication interface113to retrieve the one or more calibration images from the camera170. In some instances, step407may involve the control circuit111generating a camera command and communicating the camera command to the camera170via the communication interface113, wherein the camera command causes the camera170to capture the one or more calibration images, and/or to communicate the captured one or more images back to the communication interface113. In other instances, the camera may automatically capture the one or more calibration images, either periodically or in response to triggering conditions, and automatically communicate the one or more calibration images to the communication interface113or to a storage device that is accessible by the communication interface113.

For instance, as illustrated inFIG.5B, a control circuit of the robot control system110may control the robot arm of the robot550to move the calibration pattern520to one or more locations within the camera field of view510, such that the robot control system110may capture, via the camera570, one or more respective calibration images corresponding to the calibration pattern520being at the one or more locations, respectively. In some cases, the one or more locations may be a plurality of locations. In such cases, after a calibration image has been captured by a camera while the calibration pattern520is at a first location, the robot control system110may move the calibration pattern520to a second location and capture another calibration image while the calibration pattern520is at the second location. When a calibration image is being captured, the robot control system110may control the robot550to move the calibration pattern520such that the calibration pattern520faces the camera570. For instance, the robot control system110may control the robot550to orient the calibration pattern520such that the calibration pattern520is visible to the camera570.

Returning toFIG.4A, the method400may further include step409, in which the control circuit111adds the one or more calibration images to a captured image set which includes the calibration images. In an embodiment, the captured image set may be a set of images stored on, e.g., the non-transitory computer-readable medium115or any other storage device.

In some cases, the captured image set may have a fixed size and may, for example, include a specific maximum number of the calibration images. For instance, the fixed size may be 27, such that the captured image set holds exactly 27 calibration images, such as the 27 most recently captured calibration images. It is understood that the fixed size for the captured image set may be set to a fixed value other than 27. A camera calibration may be performed using some or all of the images in the captured image set. In such a scenario, the captured image set may act as a sliding window that slides to cover a defined number of the most recently captured images.

In some cases, the captured image set may start from a size of zero, and may increase in size as calibration images are added to the set. For instance, the captured image set may be reset after the first camera calibration is performed in step401. The captured image set may be reset by removing (e.g., deleting) all of the calibration images from the set, such that it has a size of zero at a beginning of step409. In this case, the captured image set may increase in size as the calibration images are added thereto, and may have a defined maximum size (e.g., 27 calibration images) that limits a total number of the calibration images in the captured image set. The defined maximum size may, in an embodiment, be equal to a target count, which is discussed below. For example, the target count may indicate a minimum number of calibration images for performing reliable camera calibration. The target count may be a value that is defined manually or dynamically, and thus may be referred to as a defined target count. In some situations, the captured image set is not used to perform the camera calibration until the captured image set has reached the defined maximum size. Once the captured image set reaches the maximum size, the control circuit111may discard a number of the oldest calibration images in the set when an equal number of the most recent calibration images are added, such that the captured image set acts as a sliding window that captures a defined number of the most recently captured images. In an embodiment, the control circuit111may determine whether a total number of the calibration images in the captured image set equals the defined target count every time a calibration image is added to the captured image set, such as during step409.

The method400may further include step411, in which the control circuit111performs a second camera calibration based on the calibration images in the captured image set, wherein the second camera calibration outputs updated camera calibration information. For instance, after the robot control system110performs the first camera calibration, the robot control system110may control the camera (e.g.,570/670) to continue capturing calibration images over a period of time, and may add these captured calibration images to the captured image set, which may include a number of the most recently captured calibration images. The robot control system110may then perform a second camera calibration based on the calibration images in the captured image set. Because the second camera calibration is subsequent to the first camera calibration, it may be referred to as a subsequent camera calibration.

In an embodiment, the second camera calibration may be performed only when a total number of calibration images in the captured image set has reached the defined target count (e.g. 27 calibration images). For instance, the defined target count may be defined to ensure that a sufficient number of the calibration images may be used for each camera calibration following an initial camera calibration, or more generally for each camera calibration following the first camera calibration. For example, the defined target count may represent a minimum number of the calibration images to be used for each camera calibration to achieve high accuracy.

In an embodiment, the control circuit111is configured to update the captured image set by adding a number of one or more calibration images that are most recently captured by the camera (e.g.,170/570/670) to the captured image set, and removing from the captured image set an equal number of one or more calibration images that were least recently captured by the camera (e.g.,170/570/670). This may occur, for instance, when the number of the calibration images in the captured image set is equal to its maximum size or fixed size, e.g., as determined by the target count, such that the newest calibration images replace the oldest calibration images in the set. In such an embodiment, when the second camera calibration is performed, a total number of the calibration images in the captured image set is equal to the defined target count. In an embodiment, the control circuit111may store a list that identifies which calibration images are in the captured image set. For instance, the list may include image identifiers (e.g., file names) of respective calibration images in the captured image set. When a calibration image is added to the captured image set, the control circuit111may be configured to add an image identifier for the calibration image to the list, which may effectively add the calibration image to the captured image set. In this example, the control circuit111may also remove from the list another image identifier corresponding to an oldest calibration image in the captured image set, which may effectively remove the oldest calibration image from the captured image set.

As stated above, the control circuit111may be configured to detect an idle period during the robot operation, and to output the second movement command during the idle period such that the robot arm is controlled to move the calibration pattern (e.g.,160/520/620) to the one or more locations during the idle period and the one or more respective calibration images are captured at the one or more locations during the idle period. For instance, as illustrated inFIG.5A, the robot control system110may detect an idle period between a robot task associated with the second object582B and a robot task associated with the third object582C because a large gap exists between the second object582B and the third object582C. In one example, the robot control system110may detect the idle period when no object upstream on the conveyor belt573is reachable by the robot550and/or when the robot control system110determines that a distance between the robot550and the closest object (e.g., third object582C) upstream on the conveyor belt573exceeds a certain threshold. As illustrated inFIG.5B, the robot control system110may control the robot arm of the robot550to move the calibration pattern520to a location during the idle period and capture the calibration image (e.g., a first calibration image) at the location during the idle period. If more time is remaining in the idle period, the robot control system110may control the robot arm of the robot550to move the calibration pattern520to another location and control the camera570to capture another calibration image (e.g., a second calibration image). The robot arm may be controlled to orient the calibration pattern520at each of the locations to be visible to the camera570.

In an embodiment, the above idle period may be a first idle period, and the control circuit111may be configured to determine that a total number of calibration images in the captured image set has reached the defined target count. In such an embodiment, the control circuit111may be configured to detect, after the first idle period, a second idle period during the robot operation, and may be further configured to determine whether the second idle period is longer than or equal to a defined calibration time period. In an embodiment, the defined calibration time period may indicate an amount of time necessary to complete the second camera calibration. As stated above, completing the second camera calibration may involve performing calculations to determine the intrinsic parameters of the camera, and/or transformation functions (e.g., matrices) that describe a relationship between the camera and the robot. The defined calibration time period may indicate an amount of time necessary to perform such calculations. In response to a determination that the second idle period is longer than or equal to the defined calibration time period, the control circuit111may be configured to complete the second camera calibration in the second idle period.

On the other hand, in response to a determination that the second idle period is shorter than the defined calibration time period, the control circuit111may be configured to postpone the completion of the second camera calibration to a subsequent (e.g., third) idle period. In some cases, no more calibration images are captured during the second idle period. In other cases, the control circuit111may receive (e.g., retrieve) one or more additional calibration images from the camera (e.g.,170/570/670) via the communication interface113during a remaining amount of time in the second idle period, wherein the camera (e.g.,170/570/670) is configured to capture (e.g., in response to a camera command) the one or more additional calibration images respectively at one or more additional locations within the camera field of view (e.g.,510/610), so as to provide recent calibration images from the second idle period for the second camera calibration. The control circuit111may update the captured image set by adding the one or more additional images to the captured image set and by removing an equal number of one or more calibration images that were least recently captured by the camera (e.g.,170/570/670) so as to generate an updated captured image set. The control circuit111may wait until a subsequent idle period that is longer than or equal to the defined calibration time period to complete the second camera calibration with the updated captured image set.

As stated above, as the calibration images are captured and added to the captured image set, a size of the captured image set may increase. If the size of the captured image set indicates that a number of captured calibration images in the captured image set equals the defined target count discussed above, then a sufficient number of calibration images are available in the captured image set for the second camera calibration. In an embodiment, completing the second camera calibration may involve performing the calculation discussed above or other steps for the second camera calibration, and these calculations or other steps may need at least an amount of time equal to the defined calibration time period. Hence, in this embodiment, an idle period that is at least as long as the defined calibration time period is needed to complete the second camera calibration. Thus, after determining that the number of captured calibration images in the captured image set has reached the defined target count, if the subsequent (e.g., third) idle period is longer than or equal to the defined calibration time period, then the robot control system110may perform the second camera calibration during the subsequent idle period.

On the other hand, if the subsequent (e.g., third) idle period is shorter than the defined calibration time period, the robot control system110may decide not to complete the second camera calibration during the subsequent idle period, but may instead capture additional calibration images during the subsequent (e.g., third) idle period. Then, the captured image set is updated by adding the additional calibration images to the captured image set and by removing an equal number of the oldest calibration images (i.e., least recently captured calibration images) from the captured image set, so that the updated captured image set maintains the same size, which is equal to the defined target count. When the robot control system110detects another (e.g., fourth) idle period having a duration that is longer than or equal to the defined calibration time period, the robot control system110may complete the second camera calibration during that (e.g., fourth) idle period.

As an example, assuming that the defined target count is 10, a size of a captured image set S may be limited to 10 calibration images or less. In this example, 10 calibration images I1-I10are captured by a camera, where I1is captured earliest in time and I10is captured most recently in time. The 10 images may be added to the captured image set S. If two additional calibration images I11and I12are captured, the two additional calibration I11and I12are added to the captured image set S, and the two calibration images I1and I2captured earliest in time are removed from the captured image set S to generate an updated captured image set S, as shown below:
Captured image setS=[I1,I2,I3,I4,I5,I6,I7,I8,I9,I10]
Additional calibration images=[I11,I12]
Updated captured image setS=[I3,I4,I5,I6,I7,I8,I9,I10,I11,I12]

In an embodiment, the idle period discussed above may be the first idle period, and the control circuit111may be configured to determine, after the first idle period, that a total number of calibration images in the captured image set has not reached the defined target count. In such an embodiment, the control circuit111may wait for the second idle period, so that more calibration images may be captured via the camera (e.g.,170/570/670) and communicated to the control circuit111via the communication interface113. More specifically, the control circuit111may be configured to detect the second idle period that follows the first idle period, and to receive one or more additional calibration images from the camera170/570/670via the communication interface113during the second idle period, where the camera170/570/670is configured to capture the one or more additional calibration images when the calibration pattern is placed at one or more additional locations, respectively, within the camera field of view510/610. The control circuit111may be configured to add the one or more additional calibration images to the captured image set so as to generate the updated captured image set. After the one or more additional calibration images have been added to the captured image set in the second idle period to generate the updated captured image set, the control circuit111may be further configured to determine whether a total number of calibration images in the updated captured image set has reached the defined target count. In response to a determination that the total number of calibration images in the updated captured image set has reached the defined target count, the control circuit111may be configured to determine whether a remaining time in the second idle period is longer than or equal to the defined calibration time period which indicates an amount of time necessary to complete the second camera calibration. The control circuit111may perform the second camera calibration in response to a determination that the remaining time in the second idle period is longer than or equal to the defined calibration time period. If the remaining time in the second idle period is shorter than the defined calibration time period, the control circuit111may postpone the completion of the second camera calibration to a subsequent idle period, and may use the remaining time in the second idle period to control the camera to capture additional calibration images, or more generally to receive additional calibration images from the camera170/570/670.

In an embodiment, when the control circuit111of the robot control system110determines that a total number of calibration images in the captured image set has not reached the defined target count, it may continue to control the camera (e.g.,170/570/670) to capture more calibration images (or, more generally, to receive more calibration images from the camera170/570/670) and to add them to the captured image set, either during an idle period or a subsequent idle period(s), until the total number of calibration images in the captured image set reaches the defined target count.

In one instance, inFIG.5A or5B, when the robot control system110captures additional calibration images via the camera570during the second idle period and adds the additional calibration images to the captured image set to generate the updated captured image set, the total number of calibration images in the updated captured image set may reach the defined target count before the second idle period ends. In such a scenario, the robot control system110may determine whether a sufficient time is remaining in the second idle period to perform the second camera calibration by determining whether the remaining time in the second idle period is longer than or equal to the defined calibration time period. If the remaining time in the second idle period is longer than or equal to the defined calibration time period, the robot control system110may perform the second camera calibration during the remaining time in the second idle period.

On the other hand, in response to a determination that the remaining time in the second idle period is shorter than the defined calibration time period, the control circuit111may be further configured to receive one or more subsequent additional calibration images form the camera570via the communication interface113during the remaining time in the second idle period, where the camera570is configured to capture the one or more subsequent additional calibration images respectively at one or more subsequent additional locations within the camera field of view510. The control circuit111may be further configured to update the captured image set by adding the one or more subsequent additional calibration images and by removing an equal number of calibration images in the captured image set that were least recently captured. The control circuit111may be configured to wait until a subsequent idle period to complete the second camera calibration with the updated captured image set. For instance, when the additional calibration images are captured to bring the total number of calibration images in the updated captured image set to the defined target count before the second idle period ends, the remaining time in the second idle period may be shorter than the defined calibration time period. Then, the robot control system110may postpone the completion of the second camera calibration to a subsequent (e.g., third) idle period, and may instead use the remaining time of the second idle period to capture more recent calibration images. The captured image set is updated by adding the more recent calibration images to the captured image set and by removing an equal number of the oldest calibration images. When the robot control system110detects, after the second idle period, a subsequent idle period that is longer than or equal to the defined calibration time period, the robot control system110may perform the second camera calibration during this subsequent idle period.

In an embodiment, the control circuit111may be configured, each time a new calibration image is added to the captured image set, whether the total number of calibration images in the captured image set exceeds the defined target count. In response to a determination that the total number of calibration images in the captured image set exceeds the defined target count, the control circuit111may be configured to remove a least recently captured calibration image from the captured image set to maintain the total number of calibration images in the captured image set to the defined target count.

Returning toFIG.4B, the method400may further include step451, in which the control circuit111determines an amount of deviation between the camera calibration information (generated from the first camera calibration) and the updated camera calibration information (generated from the second camera calibration). For instance, the amount of deviation may indicate how much deviation the camera570/670has experienced between the first camera calibration and the second camera calibration. In an embodiment, the camera calibration information and the updated camera calibration information include respective values for a calibration parameter. For instance, the amount of deviation may be based on a difference between a first set of calibration parameter values obtained from the first camera calibration and a second set of calibration parameter values obtained from the second camera calibration. As an example, the calibration parameter may be, e.g., a distortion parameter, a projection matrix, and/or a transformation function that describes a relationship between a camera (e.g.,570/670) and a robot (e.g.,550/650). In such an example, the camera calibration information may indicate a first calibration parameter value or first set of calibration parameter values for the calibration parameter, and the updated camera calibration information may indicate a second calibration parameter value or second set of calibration parameter values for the calibration parameter. The amount of deviation between the camera calibration information and the updated camera calibration information may be based on a difference between the first calibration parameter value and the second calibration parameter value, or based on respective differences between the first set of calibration parameter values and the second set of calibration parameter values. In another example, the camera calibration information and the updated camera calibration information may be based on respective coordinates of a pattern element of the calibration pattern520/620. In such an example, the amount of deviation may be based on an overall difference between a set of coordinates at which the pattern elements appear in the calibration image obtained for the first camera calibration and a set of coordinates at which the pattern elements appear in the calibration image obtained by the second camera calibration.

Returning toFIG.4B, the method400may further include step453, in which the control circuit111determines whether the amount of deviation exceeds a defined threshold (which may also be referred to as a defined deviation threshold). For example, the control circuit111may compare the amount of deviation with the defined deviation threshold to determine whether the deviation exceeds a defined threshold. In an embodiment, the defined deviation threshold may indicate when the updated camera calibration information discussed above has changed too much relative to the camera calibration information (e.g., baseline camera calibration information) discussed above. For instance, a large change between the baseline camera calibration information and the updated calibration information may reflect an undesirable condition in an external environment of the camera (e.g.,170/570/670), such as an external environment in which temperature is unregulated and temperature change is causing a significant change in the camera calibration information, or such as a condition in which the camera (e.g.,170/570/670) is not securely mounted to a mounting structure and movement of the camera caused by external forces is causing a significant change in the camera calibration information. In an embodiment, the defined deviation threshold may indicate when the updated camera calibration information has changed too quickly from the camera calibration information (e.g., baseline camera calibration information). For instance, if a user bumps into the camera (e.g.,170/570/670) or the robot (e.g.,150/550/650) and causes either the camera or the robot to suddenly shift in position, the sudden shift may be reflected in a sudden and significant change in camera calibration information. In such an embodiment, the defined deviation threshold may have a value that is a defined rate of change.

The method400may further include step455, in which the control circuit111outputs, in response to a determination that the amount of deviation exceeds the defined threshold, a notification signal that indicates the amount of deviation exceeds the defined threshold. In an embodiment, the notification may be output to a user interface device, such as user interface device180. In an embodiment, the defined threshold may be adjusted based on a temperature in an environment surrounding the camera. For example, when the temperature is within a defined normal operating temperature range (e.g., within 10 degrees of the room temperature), then the defined threshold may be defined to have a first value. When the temperature is outside the normal operating temperature range, then the defined threshold may be defined to a second value lower than the first value.

In an embodiment, the amount of deviation exceeding the defined threshold may indicate that undesirable errors may result if the robot operation continues, because the properties associated with the camera570/670may have deviated too much from an initial state. Therefore, the control circuit111outputs the notification signal to indicate the amount of deviation exceeding the defined threshold, such that an operator of the robot550/650may address the deviation before the robot operation continues. On the other hand, if the amount of deviation does not exceed the defined threshold, the control circuit111may continue with the robot operation without outputting the notification signal. Further, if the amount of deviation does not exceed the defined threshold, the control circuit111may reset the captured image set (e.g., by removing all images from the captured image set) and start capturing new calibration images to be used for another (e.g., third) camera calibration.

In an embodiment, one or more locations to which the robot arm moves the calibration pattern (e.g.,520/620) to capture calibration images may include a plurality of locations, where each of the plurality of locations is a location disposed on a surface of an imaginary sphere that is concave with respect to the camera (e.g.,570/670). In such an embodiment, the control circuit111may be further configured to control the robot arm to move the calibration pattern (e.g.,520/620) to be tangent to the surface of the imaginary sphere at each location of the plurality of locations. For instance, as illustrated inFIGS.7A and7B, the robot control system110may control the robot arm of the robot550to move the calibration pattern520to locations710A-710I, and capture a respective calibration image at each of the locations710A-710I. The locations710A-710I may be divided among a plurality of imaginary spheres within the field of view510of the camera570. For example, the locations710A and710B may be disposed on a first spherical surface721of a first imaginary sphere720, where the first spherical surface721that include locations710A and710B is (partially or completely) within the field of view510of the camera570. As another example, the locations710C,710D, and710E may be disposed on a second spherical surface731of a second imaginary sphere730, where the second spherical surface731that include locations710C,710D, and710E is within the field of view510. In a further example, the locations710F,710G,710H, and710I may be disposed on a third spherical surface741of a third imaginary sphere740, where the third spherical surface741that includes the locations710F,710G,710H, and710I is within the field of view510. As illustrated inFIGS.7A and7B, the first, second, and third spherical surfaces721,731, and741, respectively, are concave with respect to the camera570. Although the examples inFIGS.7A and7Bshow three spherical surfaces based on three spheres, a number of different spherical surfaces on which locations may be disposed may be greater than three or less than three. In an embodiment, the camera570may be a center of each of the imaginary spheres720,730, and740.

In an embodiment, as illustrated inFIGS.7A and7B, when the calibration pattern520is moved to a location, the robot control system110may control the robot arm of the robot550to position the calibration pattern520to be tangent to the spherical surface on which the location is disposed. For example,FIG.7Billustrates that the calibration pattern520is tangent to the second spherical surface731at the location710D. More particularly, the calibration pattern520may be disposed on a flat plane (e.g., on a sticker), and the flat plane of the calibration pattern520may be tangent to the second spherical surface731at the location710D.

In an embodiment, the control circuit111is configured to control the robot arm to move the calibration pattern (e.g.,520) to directly face the camera when the calibration pattern (e.g.,520) is moved to a location. For instance, as illustrated inFIG.7A, the robot control system110may control the robot arm of the robot550to move the calibration pattern520to directly face the camera570when the calibration pattern520is moved to the location710D. In this example, the robot control system110may control the robot hand555to be rotated such that the calibration pattern520directly faces the camera570. In some cases, the calibration pattern520may directly face the camera570by being tangent to the spherical surface (e.g. the first spherical surface721, the second spherical surface731, the third spherical surface741) at the camera's field of view510. When the calibration pattern520directly faces the camera570, the camera570may be able to photograph the calibration pattern520head-on, so that there is no perspective effect or a reduced perspective effect in a resulting image of the calibration pattern520.

FIG.8depicts an example time line800where the camera calibration update process is performed. In an embodiment, the time line800may relate to operations performed by the system500ofFIGS.5A and5B. The time line800may begin with a calibration period811, which may occur before a robot operation begins. For instance, the robot control system110ofFIG.1A,1i, or1C may perform the first camera calibration to determine the camera calibration information for the camera570. After the first camera calibration is complete, the robot operation begins with a robot task period813.

During the robot task period813, the robot control system110may control the robot550(fromFIGS.5A and5B) to perform one or more robot tasks. As discussed above with respect toFIGS.5A and5B, the system500may be unable to capture any calibration image during the robot task period813, and may have to wait until an idle period is detected to do so.

As depicted inFIG.8, the robot control system110may, after the task period813, detect an idle period815, during which the robot550is free from performing a robot task. During the idle period815, the robot control system110captures, via the camera570, one or more calibration images of the calibration symbol520ofFIGS.5A and5Bat one or more locations (e.g., at a first set of six locations), respectively, and adds these calibration images to the captured image set. In the example ofFIG.8, the calibration images I1-I6are captured during idle period815and are added to the captured image set S.

After the idle period815ends, during a task period817, the robot control system110resumes controlling the robot550to perform one or more robot tasks. During the task period817, the system500may be unable to capture any calibration image. After the idle period817, the robot control system110in this example detects an idle period819during which the robot550is free from performing a robot task. During the idle period819, the robot control system110captures the calibration images I7-I10of the calibration pattern520at a second set of four other locations, respectively. The captured images I7-I10are added to the captured image set S. As a result, the captured image set S includes I1-I10after the second idle period819. In an embodiment, the defined target count may be equal to ten. Because the captured image set S now has ten calibration images, the robot control system110may be ready to perform a second camera calibration. However, this second camera calibration may be postponed to a subsequent idle period (e.g.,827), because the second idle period819may have already elapsed or nearly elapsed.

After the idle period819, the robot control system110resumes controlling the robot550to perform one or more robot tasks during a robot task period821. After the robot ask period, the robot control system110may detect an idle period823during which the robot550is free from performing a robot task. The robot control system110further determines that the idle period823is shorter than the defined calibration time period, and thus determines not to perform the calculations or other steps for completing the second camera calibration during the idle period823. Instead, the robot control system110may wait for a subsequent idle period (e.g., idle period827) to complete the second camera calibration. In some cases, the robot control system110may allow the idle period823to elapse without capturing any more calibration images. Alternatively, as depicted inFIG.8, the robot control system110may capture additional calibration images I11-I12during the idle period823, and add them to the captured image set S. Because the total number of calibration images in the captured image set S has already reached the defined target count of 10 of the calibration images, the robot control system110may remove an equal number of the oldest calibration images in the captured image set S from the captured image set S. More specifically, in this embodiment, the robot control system110can remove the two oldest instances of the calibration images I1-I2from the captured image set S since two newly captured instances of the calibration images I11-I12were added to the captured image set S. As a result, the captured image set S now includes the calibration images I3-I12.

As illustrated inFIG.8, the idle period823may be followed by a robot task period825and then another idle period827. The robot control system110may determine that this idle period827has a duration that is longer than or equal to the defined calibration time period, and thus determines to perform the calculation or other steps necessary to complete the second camera calibration during the idle period827. During the idle period827, the robot control system110performs the second camera calibration to determine updated camera calibration information based on the calibration images in the captured image set. In this example, the second camera calibration is performed based on the calibration images I3-I12included in the captured image set. After the second camera calibration is complete, the robot control system110determines the amount of deviation between the camera calibration information from the first camera calibration and the updated camera calibration information from the second camera calibration. In this example, because the amount of deviation does not exceed the defined threshold, the robot control system110may continue with the robot operation by performing robot tasks during the task period829.

FIG.9depicts an example flow diagram900that shows a camera calibration update process that reflects the features discussed above forFIGS.4A,4B,5A,5B, and8. The steps of the flow diagram900may be performed by, e.g., the system500ofFIGS.5A and5B. At step901, the robot control system110ofFIG.1A,1, or1C performs the first camera calibration of the camera570ofFIGS.5A and5Bto determine camera calibration information of the camera570. After the first camera calibration, the robot control system110may begin a robot operation after the first camera calibration.

At step903, the robot control system110detects an idle period during the robot operation, as discussed above. At step905, during the idle period, the robot control system110controls the robot550ofFIGS.5A and5Bto move the calibration pattern520ofFIGS.5A and5Bto one or more locations within a camera field of view510ofFIGS.5A and5Bof the camera570, and captures one or more respective calibration images via the camera570at the one or more locations. The captured calibration images may be added to the captured image set. In an embodiment, the captured image set may have zero instances of the captured calibration images at the beginning of step905and thus may have a size zero at the beginning of step905.

At step907, the robot control system110determines whether a total number of the captured calibration images within the captured image set has reached the defined target count. If the total number of the captured calibration images within the captured image set has reached the defined target count, the robot control system110determines whether a remaining time in the idle period is longer than or equal to the defined calibration time period, at step909. The defined calibration time period may indicate an amount of time necessary to complete an additional camera calibration (e.g., a second camera calibration). If the remaining time in the idle period is longer than or equal to the defined calibration time period, the robot control system110performs the additional camera calibration, at step919.

Returning to step909, if the remaining time in the idle period is shorter than the defined calibration time period, the robot control system110does not complete the additional camera calibration in that time period. Instead, at step911, during the remaining time in the idle period, the robot control system110captures, via the camera570, new calibration images and updates the captured image set by adding these new calibration images to the captured image set and by removing an equal number of calibration images least recently captured by the camera570to maintain the total number of the calibration images in the captured image set at the defined target count. The robot control system110may resume performing robot tasks after the idle period is over after911.

At step913, the robot control system110attempts to detect an additional idle period during the robot operation. When the additional idle period is detected, the robot control system110may at step915determine whether the additional idle period is longer than or equal to the defined calibration time period. If the additional idle period is shorter than the defined calibration time period, the robot control system110at step917captures, via the camera570, new calibration images and updates the captured image set by adding these new calibration images to the captured image set and by removing an equal number of calibration images least recently captured by the camera570to maintain the total number of the calibration images in the captured image set at the defined target count. The robot control system110may return to step913and wait for a subsequent idle period that has a duration which is longer than or equal to the calibration time period.

Returning to step907, if the robot control system110determines that a total number of the captured calibration images within the captured image set has not reached the target count, the robot control system110attempts to detect an additional idle period during the robot operation at step921. When the additional idle period is detected, the robot control system110captures one or more calibration images at one or more locations within the camera field of view510of the camera570, and may add the captured calibration images to the captured image set.

At step925, the robot control system110may again determine whether a total number of the captured calibration images within the captured image set has reached the defined target count. If the robot control system110at step925determines that total number of the captured calibration images within the captured image set has not reached the defined target count, the robot control system110can return to step921to wait for a subsequent idle period, in order to capture more calibration images. If the robot control system110at step925determines that the total number of the captured calibration images within the captured image set has reached the defined target count, the process can continue to step915and the robot control system110determines whether a remaining time in the additional idle period is longer than or equal to the defined calibration time period.

At step915, if the robot control system110determines that the remaining time in the current idle period is longer than or equal to the defined calibration time period, the process can continue to step919. At step919, the robot control system110performs the additional camera calibration (e.g., second camera calibration).

After the robot control system110performs the additional calibration at step919, the robot control system110at step927may determine the amount of deviation between the camera calibration information of the first camera calibration and the updated camera calibration information of the additional calibration, and determine whether the amount of deviation exceeds a defined threshold. If the amount of deviation exceeds the defined threshold, the robot control system110outputs a notification at step929(e.g., to user interface device180). In some instances, the robot control system110may stop or pause the robot operation in response to such a determination. On the other hand, if the amount of deviation does not exceed the defined threshold, the robot control system110may reset the captured image set at step932and goes back to step903to detect an idle period to capture a new set of calibration images.

As stated above with respect toFIGS.6A and6B, some embodiments of the present disclosure may allow the calibration images to be captured without the use of an idle period, or regardless of whether there is an idle period.FIG.10depicts an example time line1000for such an embodiment. The time line1000may reflect an example scenario for the system600ofFIGS.6A and6B. The time line1000may begin with a calibration period1011, which may occur before a robot operation begins. In the calibration period1011, the robot control system110ofFIG.1A,1, or1C may perform the first camera calibration to determine camera calibration information for the camera670ofFIG.6A.

After the first camera calibration is complete, the robot operation begins with the task period1013. During the robot operation, the robot control system110controls the robot650ofFIGS.6A and6Bto perform the robot tasks during the task periods1013,1017,1021and1025, while the robot650is free from performing robot tasks during the idle periods1015,1019,1023.

In the example time line1000, the robot control system110does not need to wait for an idle period to capture the calibration images during the idle period. As discussed above, because the calibration pattern620ofFIGS.6A and6Bis placed on the robot650such that the calibration pattern620is regularly facing the camera670even while the robot650is performing a robot task, the robot control system110may capture calibration images any time during the robot operation. Thus, during the time period1031, the robot control system110captures calibration images and adds the captured calibration images to a captured image set. The time period1031may include only a robot task period(s), only an idle period(s), or any combination thereof. If, during the time period1031, the number of the captured calibration images in the captured image set reaches the defined target count, the robot control system110performs a second camera calibration to determine updated camera calibration information based on the captured calibration images. Further, the robot control system110may determine the amount of deviation between the camera calibration information from the first camera calibration and the updated camera calibration information from the second camera calibration. In this example, because the amount of deviation does not exceed the defined threshold, the robot control system110resets the captured image set by removing all calibration images in the captured image set, and continues to the time period1033. In another embodiment, the robot control system110does not perform the reset in the time period1031, such that the captured image set may act as a sliding window that uses a defined number of the most recently captured calibration images to perform a subsequent (e.g., third) camera calibration.

During the time period1033, the robot control system110captures another set of calibration images and adds the captured calibration images to the captured image set. Like with the time period1031, the time period1033may include only a robot task period(s), only an idle period(s), or any combination thereof. When the number of the captured calibration images in the captured image set reaches the defined target count, the robot control system110performs another (e.g., third) camera calibration to determine the updated camera calibration information based on the captured calibration images from the time period1033. The robot control system110may determine the amount of deviation between the camera calibration information from the first camera calibration and the updated camera calibration information from the third camera calibration of the time period1033. In this example, because the amount of deviation does not exceed the defined threshold, the robot control system110resets the captured image set by removing the calibration images in the captured image set, and continues to the time period1035.

During the time period1035, the robot control system110captures another set of calibration images and adds the captured calibration images to the captured image set. When the number of the captured calibration images in the captured image set reaches the defined target count, the robot control system110performs yet another (e.g., fourth) camera calibration to determine updated camera calibration information based on the captured calibration images. The robot control system110may determine the amount of deviation between the camera calibration information from the first camera calibration and the updated camera calibration information from the fourth camera calibration. In this example, because the amount of deviation exceeds the defined threshold, the robot control system110outputs the notification signal of the threshold being exceeded.

FIG.11depicts an example flow diagram1100that shows a camera calibration update process of the camera calibration relating to the processes discussed above forFIGS.6A,6B, and10. At step1101, the robot control system110ofFIG.1A,1, or1C performs a first camera calibration. At step1103, the robot control system110controls the robot650ofFIGS.6A and6Bto move the calibration pattern620ofFIGS.6A and6Band captures calibration images via the camera670ofFIG.6Aat respective locations within the camera field of view610ofFIGS.6A and6Bof the camera670. At step1105, the robot control system110determines whether the number of the captured calibration images has reached the defined target count. If the number of the captured calibration images has not reached the defined target count, the process returns to step1103and the robot control system110continues to capture more calibration images at respective locations.

Continuing with step1105, if the number of the captured calibration images has reached the target count, the process continues to step1107and the robot control system110performs an additional camera calibration (e.g., a second camera calibration) to determine updated camera calibration information. After the robot control system110performs the additional calibration at step1107, the robot control system110at step1109determines the amount of deviation between the camera calibration information of the first camera calibration and the updated camera calibration information of the additional camera calibration and determines whether the amount of deviation exceeds a defined threshold. If the amount of deviation exceeds the defined threshold, the robot control system110outputs the notification signal at step1111. In some cases, the robot control system110may stop or pause the robot operation in response to such a determination. On the other hand, if the amount of deviation does not exceed the defined threshold, the robot control system110may reset the captured calibration images at step1113and goes back to step1103to detect an idle period to capture new set of calibration images. In an embodiment, the reset step1113may be omitted.

Additional Discussion of Various Embodiments

Embodiment 1 relates to a robot control system comprising a communication interface configured to communicate with a robot having a base and a robot arm with a calibration pattern disposed thereon, and to communicate with a camera having a camera field of view. The robot control system further comprises a control circuit configured: a) to perform a first camera calibration to determine camera calibration information associated with the camera, b) to control, based on the camera calibration information, movement of the robot arm to perform a robot operation by outputting a first movement command that is based on the camera calibration information to the robot via the communication interface, c) to control the robot arm, after the first camera calibration, to move the calibration pattern to one or more locations within the camera field of view by outputting a second movement command to the robot via the communication interface, d) to receive one or more calibration images from the camera via the communication interface, wherein the one or more calibration images are respectively captured at the one or more locations, e) to add the one or more calibration images to a captured image set which includes calibration images, f) to perform a second camera calibration based on the calibration images in the captured image set, wherein the second camera calibration outputs updated camera calibration information, g) to determine a deviation between the camera calibration information and the updated camera calibration information, h) to determine whether the deviation exceeds a defined threshold, i) to output, in response to a determination that the deviation exceeds the defined threshold, a notification signal that indicates the deviation exceeds the defined threshold.

Embodiment 2 includes the robot control system of embodiment 1, wherein the second camera calibration is performed only when a total number of calibration images in the captured image set has reached a defined target count.

Embodiment 3 includes the robot control system of embodiment 1 or 2, wherein the control circuit is further configured to update the captured image set by adding a number of one or more calibration images that are most recently captured to the captured image set, and removing from the captured image set an equal number of one or more calibration images that were least recently captured, and wherein, when the second camera calibration is performed, a total number of calibration images in the captured image set is equal to a defined target count.

Embodiment 4 includes the robot control system of any one of embodiments 1-3, wherein the control circuit is further configured to detect an idle period during the robot operation, and to output the second movement command during the idle period such that the robot arm is controlled to move the calibration pattern to the one or more locations during the idle period, and wherein the one or more respective calibration images are captured at the one or more locations during the idle period.

Embodiment 5 includes the robot control system of embodiment 4, wherein the idle period is a first idle period, wherein the control circuit is further configured: a) to determine that a total number of calibration images in the captured image set has reached a defined target count, b) to detect, after the first idle period, a second idle period during the robot operation, c) to determine whether the second idle period is longer than or equal to a defined calibration time period, which indicates an amount of time necessary to complete the second camera calibration, d) in response to a determination that the second idle period is longer than or equal to the defined calibration time period, to perform the second camera calibration during the second idle period, e) in response to a determination that the second idle period is shorter than the defined calibration time period: a) to receive one or more additional calibration images from the camera via the communication interface during the second idle period, wherein the one or more additional calibration images are respectively captured at one or more additional locations within the camera field of view, b) to update the captured image set by adding the one or more additional images and by removing an equal number of one or more calibration images that were least recently captured so as to generate an updated captured image set, and c) to wait until a subsequent idle period that is longer than or equal to the defined calibration time period to complete the second camera calibration with the updated captured image set.

Embodiment 6 includes the robot control system of embodiment 4, wherein the idle period is a first idle period, wherein the control circuit is further configured: a) to determine, after the idle period, that a total number of calibration images in the captured image set has not reached a defined target count, b) to detect a second idle period that follows the first idle period, c) to receive one or more additional calibration images from the camera via the communication interface during the second idle period, wherein the one or more additional calibration images are respectively captured at one or more additional locations within the camera field of view, d) to add the one or more additional calibration images to the captured image set so as to generate an updated captured image set, e) to determine whether a total number of calibration images in the updated captured image set has reached the defined target count, f) in response to a determination that the total number of calibration images in the captured image set has reached the defined target count: to determine whether a remaining time in the second idle period is longer than or equal to a defined calibration time period which indicates an amount of time necessary to complete the second camera calibration, g) in response to a determination that the remaining time in the second idle period is longer than or equal to the defined calibration time period, to perform the second camera calibration during the remaining time in the second idle period.

Embodiment 7 includes the robot control system of embodiment 6, wherein the control circuit is further configured, in response to a determination that the remaining time in the second idle period is shorter than the defined calibration time period: a) to receive one or more subsequent additional calibration images from the camera via the communication interface during the remaining time in the second idle period, wherein the one or more subsequent additional calibration images are respectively captured at one or more subsequent additional locations within the camera field of view, b) to update the captured image set by adding the one or more subsequent additional calibration images and by removing an equal number of calibration images in the captured image set that were least recently captured, and c) to wait until a subsequent idle period that is longer than or equal to the defined calibration time period to complete the second camera calibration with the updated captured image set.

Embodiment 8 includes the robot control system of any one of embodiments 1-7, wherein the control circuit is further configured: a) to receive one or more additional calibration images from the camera via the communication interface and to add the one or more additional calibration images to the captured image set, wherein the one or more additional calibration images are respectively captured at one or more additional locations within the camera field of view, b) to determine, after each image of the one or more additional calibration images have been added to the captured image set, whether a total number of calibration images in the captured image set exceeds a defined target count, c) in response to a determination that the total number of calibration images in the captured image set exceeds the defined target count, to remove a least recently captured calibration image from the captured image set, wherein the second camera calibration is performed with all calibration images of the captured image set after the one or more additional images have been added to the captured image set.

Embodiment 9 includes the robot control system of any one of embodiments 1-8, wherein the camera calibration information and the updated camera calibration information include respective values for a calibration parameter.

While various embodiments have been described above, it should be understood that they have been presented only as illustrations and examples of the present invention, and not by way of limitation. It will be apparent to persons skilled in the relevant art that various changes in form and detail can be made therein without departing from the spirit and scope of the invention. Thus, the breadth and scope of the present invention should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the appended claims and their equivalents. It will also be understood that each feature of each embodiment discussed herein, and of each reference cited herein, can be used in combination with the features of any other embodiment. All patents and publications discussed herein are incorporated by reference herein in their entirety.