Patent Description:
The invention generally relates to machine vision calibration systems, and relates in particular to machine vision (e.g., camera) calibration systems that provide extrinsic calibration of cameras relative to the bases of articulated arms or programmable motion devices.

Extrinsic camera calibration refers to the determination of the coordinate frame of a camera relative to a system coordinate frame. By comparison, intrinsic camera calibration refers to the determination of the internal parameters of the camera's lens and imager, such as field of view, focal length, image center, and distortion parameters. In systems described herein it is assumed that the intrinsic parameters are known a priori.

In some calibration systems a planar calibration target is placed onto the robot's end-effector. The end-effector is then moved in the robot's environment. For example, with reference to <FIG>, a robotic system <NUM> that includes an articulated arm <NUM> and an end-effector <NUM>, as well as a perception system <NUM> for aiding in programming the articulated arm <NUM> in moving objects (e.g., within the robotic environment such as for example, into or out of a bin <NUM>). The robotic system <NUM> is shown grasping a calibration target <NUM> within perception range of the perception unit <NUM>. Using this system, the robotic system may calibrate the end-effector of the articulated arm with the perception system <NUM> and/or any other perception system <NUM> within the robotic environment. In some examples, the process involves viewing individual markers <NUM> on the calibration target <NUM> from a plurality of varied viewing positions and orientations. This may be time-consuming and may be labor intensive in certain applications. Further, in certain applications, some of the environmental features and possibly even robot attachments, need to be removed from the system prior to the calibration process in order to prevent the planar calibration target from colliding with any of these features or attachments.

There remains a need for an extrinsic camera calibration system that is efficient to perform and does not require significant changes to the robotic system and its environment. <CIT> appears to disclose a system and method for robustly calibrating a vision system and a robot is provided. The system and method enables a plurality of cameras to be calibrated into a robot base coordinate system to enable a machine vision/robot control system to accurately identify the location of objects of interest within robot base coordinates. <CIT> appears to disclose a method for calibrating a coordinate system of an image capture device and a coordinate system of a robot arm in a robot system that includes a display device, the image capture device, and the robot arm to which one of the display device and the image capture device is fixed, the robot arm having a drive shaft. The method includes: acquiring first captured image data based on first image data; acquiring second captured image data based on second image data different from the first image data; and calibrating the coordinate system of the image capture device and the coordinate system of the robot arm, using the first captured image data and the second captured image data. <CIT> appears to disclose a robot control device including a processor that creates a parameter of a camera including a coordinate transformation matrix between a hand coordinate system of an arm and a camera coordinate system of the camera. The processor calculates a relationship between an arm coordinate system and a pattern coordinate system at the time of capturing the pattern image of the calibration pattern, and estimates a coordinate transformation matrix between the hand coordinate system of the arm and the camera coordinate system of the camera with the relationship between the arm coordinate system and the pattern coordinate system, a position and attitude of the arm at the time of capturing a pattern image, and the pattern image.

In accordance with an aspect, the invention provides a system for providing extrinsic calibration of a camera to a relative working environment of a programmable motion device that includes an end-effector. The system includes a fiducial located at or near the end-effector, at least one camera system for viewing the fiducial as the programmable motion device moves in at least three degrees of freedom, and for capturing a plurality of images containing the fiducial, and a calibration system for analyzing the plurality of images to determine a fiducial location with respect to the camera to permit calibration of the camera with the programmable motion device.

In accordance with another aspect, the invention provides a method for providing extrinsic calibration of a camera to a relative working environment of a programmable motion device that includes an end-effector. The method includes viewing with a camera system a fiducial located at or near the end-effector as the programmable motion device moves in at least three degrees of freedom, capturing a plurality of images containing the fiducial, and analyzing the plurality of images to determine a fiducial location with respect to the camera to permit calibration of the camera with the programmable motion device.

In accordance with various aspects of the invention, a calibration system and method are provided that calibrate the extrinsic parameters of one or more cameras relative to the working environment of a robotic arm. The method estimates the extrinsic parameters (the coordinate frames) of one or more cameras relative to the robot's frame, and in certain aspects, benefits imaging cameras internally paired with depth sensors (structured light; stereo; or timeof-flight). The method also does not require significant changes to the robot, and does not require attachments to be removed. The system involves the use of a fiducial (marker) that may (or may not) be permanently mounted on the robot or end-effector.

<FIG> shows a system <NUM> employing calibration systems in accordance with certain aspects of the present invention. The system <NUM> includes articulated arm <NUM> with an end-effector <NUM>, as well as a perception system <NUM>. The system further includes a single fiducial <NUM> on the articulated arm <NUM> on or near the end-effector <NUM> in accordance with certain aspects of the invention. The use of the single fiducial (at a time) permits the calibration process of certain aspects of the invention to uniquely calibrate the extrinsic parameters of one or more cameras with respect to the articulated arm. The fiducial <NUM> may be in the form of any of an LED or other small but bright illumination source (e.g., brighter than ambient illumination), or a reflective (e.g., retroreflective) marker, such as an at least semi-spherical retroreflective sphere. In accordance with other aspects, a plurality of fiducials may be used, with one or more being used at a time.

A detection unit <NUM> includes a camera <NUM> for detecting illumination from the fiducial <NUM>. If using a retro-reflective ball as the fiducial <NUM>, the ball is illuminated from an illumination source <NUM> co-located with the camera <NUM>. The fiducial may be permanently attached to, or optionally removable from, the robot end-effector.

In accordance with certain aspects, the system provides calibration even when different positions of the articulated arm may result in a fiducial being located at the same location even when the articulated arm is in a very different position. <FIG>, for example, shows the articulated arm <NUM> in a very different position than that of <FIG>, yet the fiducial <NUM> (e.g., single fiducial) is in the same position relative the camera <NUM> of the detection unit <NUM>. The fiducial <NUM> (e.g., retro-reflective ball or LED or other point-like light source) means that the image position of the fiducial is unambiguous. In other words, if there were two or more fiducials, logic would be required to determine which image point - pixel coordinates - belonged to which fiducial. One fiducial means there can be one and only one assignment. An important benefit of the technique is that the position of the fiducial need not be precisely localized. Its position can be completely unknown.

Again, the fiducial <NUM> may be either a retro-reflective ball, or an LED that is brighter than ambient lighting. Additionally, if there is significant background lighting, the LED may be imaged in both its on and off state, and then the difference between images can be used to detect LED position. In either case the LED or the retro-reflective ball is designed such that it is visible from nearly a full half-sphere of directions, i.e., <NUM> degrees x <NUM> degrees illumination or reflective sphere as shown diagrammatically in <FIG>. In particular, with the fiducial <NUM> positioned as shown, illumination is emitted in <NUM> degrees as shown at <NUM>, and vertically <NUM> degrees as shown at <NUM>.

In accordance with an aspect of the invention, a portion of the articulated arm <NUM>, such as, for example the end-effector, is rotated through each of three degrees of freedom, while images are recorded of the fiducial <NUM> and the forward kinematics of the end-effector are also recorded (i.e., the position and orientation of the end-effector in space) <FIG> show the end-effector <NUM> with the associated fiducial <NUM> being moved from a first position (<FIG>) to a second position (<FIG>) such that the end-effector and fiducial move through at least three degrees of freedom of movement as shown diagrammatically in <FIG> (at <NUM> showing an initial position) and <FIG> (showing at <NUM> a later position having moved in x, y, and z dimensions). A plurality of images are captured during this movement.

The processes for automatically planning multiple joint configurations of the robot in order to present the fiducial to the one or more cameras, use the mechanical design (in which approximate extrinsic parameters may be known or given) to plan out the places where the LED is likely to be visible to the camera(s). These processes further compensate for known constraints for poses of the estimation algorithms. In particular, the estimation process requires that the robot not simply translate the fiducial between the views - they must be in general position. The robot must rotate the coordinate frame of the fiducial in order for the fiducial's position to be calculated. In addition, the rotation may not be solely around a single axis. For example, one instance may be: determine a cuboidal region of interest, choose a discretization, e.g., grid count (Nx,Ny,Nz) for each of the three axes, as well as a number of different orientations with which to present the LED, e.g. No = <NUM>. This collection of poses would be over-representative of poses putting the fiducial in "general position. " There is a balance between the number of poses (the more of which reduces the eventual estimation error) and the time it takes to the given number of poses.

With reference to <FIG> and <FIG> in accordance with an aspect of the invention, the calibration process begins (step <NUM>), with first rotating the end-effector through at least three degrees of freedom as shown diagrammatically in <FIG>, and capturing image data for a plurality of positions during this rotation of the end-effector (step <NUM>). The process next involves estimating the extrinsic parameters from the data by first estimating the position of the fiducial by thresholding each camera image, and then averaging the positions of the above-threshold (brightest) pixels (step <NUM>). Or as describe above, taking the difference of two images in which for example, the fiducial is a controlled LED that is turned on and off between images while the robot is stationary.

The process then obtains a linear estimate by constructing a coefficient matrix whose least singular vector encodes the extrinsics of the camera as well as the position of the fiducial (step <NUM>). The equation for positioning of the image point is as follows in accordance with an aspect: <MAT>.

where the equation is homogeneous and therefore equality is up to scale, and further, where:.

The location of the image point is then estimated (step <NUM>), and the process then involvers constructing a matrix for each image of the fiducial (step <NUM>). The coefficient matrix is obtained in accordance with an aspect, by constructing a <NUM> x <NUM> matrix for each imaged fiducial. If the measured fiducial has position q̃i, then the coefficient matrix are the <NUM> coefficients of the <NUM> polynomials: <MAT> of the expression: <MAT>.

The process then involves concatenating the coefficient matrices for each of N image points to form a 3N x <NUM> matrix (step <NUM>), whose least singular vector without noise would have the form of the <NUM>-dimensional vector above. Since noise will be present, and since it will have an arbitrary scaling, the first step is to determine the scale from the determinant of the coefficients corresponding to the rotation matrix. Having done so, the process then projects the estimated coefficients of the rotation matrix, projecting them to the manifold SO(<NUM>) of rotation matrices. Once an initial estimate of the rotation is made, a similar revised linear algorithm can be employed to estimate the unknown vectors t and p.

The system then uses the linear estimate, to employ nonlinear least squares optimization to refine the estimates of the extrinsics and the position of the fiducial (step <NUM>). The process uses equation (<NUM>) as a model, and with the initial estimates uses a nonlinear least squares optimization, such as Levenberg-Marquardt, to minimize the sum of the square norms of the projected points to the measurements (in non-homogenous image coordinates). In addition, outliers in the process can be removed using, for example, random sample consensus, or other approaches robust to outliers. The extrinsic parameters of the camera are thereby calibrated to the robotic arm via the fiducial (step <NUM>) and the process ends (step <NUM>).

The robotic system may also employ multiple cameras. In this instance, each camera can at first be treated individually as described above, but then combined in a final optimization step that would be expected to have reduced error. From the optimization procedure for the individual cameras, the estimates of the position of the fiducial relative to the end-effector should be close, but not exactly equal. First, the mean of these positions is taken to be the initial estimate for the combined multi-camera estimation; and the initial estimates for the individual cameras' poses are taken to be the estimates of poses from the individual camera estimates. The final optimization sums the square errors (or other robust penalty function) for all the re-projection errors. The output of this procedure is the final estimate encoding the poses of all cameras in the multi-camera system.

In accordance with various aspects, therefore, the system and process provide automatic planning and processes for verifying the calibration method. Such calibration may further be done as a robotic system is processing objects, provided that the articulated arm moves in at least three degrees of freedom while the fiducial is visible to camera undergoing the calibration in accordance with an aspect of the invention. In this way, not only fixed cameras, but also movable cameras may readily be calibrated with the articulated arm.

<FIG>, for example, shows an object processing system <NUM> that includes a programmable motion device (e.g., an articulated arm <NUM> mounted on a frame <NUM>). The articulated arm <NUM> is mounted above a portion of a discontinuous track system <NUM> on which carriers <NUM> are able to move in each of two mutually orthogonal directions by having the wheels on the carriers <NUM> pivot to either of two mutually perpendicular directions. The carriers <NUM> are adapted to receive any of supply bins <NUM> or destination bins <NUM>. The carriers <NUM> are able to move the supply bins <NUM> and destination bins <NUM> near the articulated arm such that the articulated arm may be employed to move objects from a supply bin to a destination bin under the articulated arm. The system may also include shelves <NUM> onto which any of supply bins or destination bins may be temporarily parked. The programmable motion device may also include a perception unit <NUM> attached to the articulated arm <NUM>, as well as one or more cameras <NUM> mounted to the frame <NUM>. Additionally, the carriers <NUM> may include a camera <NUM>. Further, the system may include one or more non-automated carriers (e.g., moving shelf unit <NUM>) that human worker may move near to the programmable motion device. The non-automated carriers may also include a camera <NUM> mounted thereon. The system may be controlled by one or more computer processing systems <NUM> that communicate with all devices via wired or wireless communication.

With reference to <FIG>, the articulated arm <NUM> includes an end-effector <NUM> with a fiducial <NUM> mounted on or near the end-effector <NUM>. In accordance with various aspects of the invention, any of the cameras <NUM>, <NUM>, <NUM>, and <NUM> may be calibrated by viewing the fiducial as the articulated arm moves. The automated carriers <NUM> may therefore be re-calibrated as they approach the processing station frame <NUM>. Additionally, and with reference to <FIG>, each supply bin <NUM> (or destination bin) may further include a camera <NUM> that may be calibrated to the articulated arm during processing. The supply bin <NUM> may contain objects <NUM>, and the automated carrier <NUM> may include one or two opposing rotatable paddles <NUM> for securing the bin onto the carrier <NUM>. The pivoting of the wheel assemblies <NUM> discussed above is shown diagrammatically at A in <FIG>. Additionally and with reference to <FIG>, the camera <NUM> on the non-automated movable carrier <NUM> may be calibrated to the articulated arm (as well as a bin <NUM> using the camera <NUM>).

The system may therefore use a fiducial to calibrate the articulated arm with the extrinsic parameters of any of a wide variety of cameras, some on moving and even non-automated carriers, permitting continuous calibration and re-calibration during the processing of objects by the object processing system. The automated carriers may therefore confirm calibration or re-calibrate with an articulated arm during processing. Additionally, a human worker may move a non-automated carrier (e.g., shelving unit <NUM>) into an area proximate the articulated arm <NUM>. Once any of the camera <NUM> and/or the camera <NUM> is calibrated with the articulated arm as discussed above, the articulated arm may be used to retrieve objects from the bin <NUM> on the unit <NUM>.

<FIG> shows at <NUM> a processing station that includes a programmable motion device <NUM> with an end-effector <NUM>. The end-effector <NUM> includes (in place of, for example a vacuum cup), a calibration unit <NUM>. The calibration unit <NUM> includes one or more features that may be detected by any or all of the detection units <NUM>. <FIG> shows a rear underside view of the calibration unit <NUM> as attached to the end-effector <NUM> of the programmable motion device <NUM>. The calibration unit <NUM> includes a coupling portion <NUM> for coupling to an end-effector coupling portion <NUM> of the end-effector <NUM> (shown in <FIG>), as well as a transverse portion <NUM> that extends transverse to the coupling portion <NUM>.

With reference to <FIG>, the transverse portion <NUM> includes on a top side thereof, a visible LED <NUM> as well as control circuitry <NUM>, <NUM> for controlling the visible LED and providing wireless communication. The transverse portion <NUM> may also include an underside (as shown in <FIG>) that also includes a visible LED <NUM>. The control circuitry <NUM>, <NUM> may, for example, provide timing control to cause each of the visible LEDs <NUM>, <NUM> to turn on and off in accordance with a timing signal (e.g., flash) at a rate that is synchronized with a processing system for processing information detected by the detection units <NUM>. This way, illumination from either or both visible LEDs <NUM>, <NUM> may be readily distinguished from any background illumination by the processing system (e.g., <NUM>, <NUM>). The control circuitry <NUM>, <NUM> may also provide wireless communication (e.g., infrared, Bluetooth, radio frequency etc.) with the processing system <NUM>, <NUM>.

The transverse portion <NUM> also includes on a top side thereof, an infrared LED <NUM> as well as control circuitry <NUM>, <NUM> for controlling the infrared LED, at for example, any of a plurality of infrared frequencies and providing wireless communication. The transverse portion <NUM> may also include on the underside (as shown in <FIG>) an infrared LED <NUM>. The control circuitry <NUM>, <NUM> may, for example, provide timing control to cause each of the infrared LEDs <NUM>, <NUM> to turn on and off in accordance with a timing signal (e.g., flash) at a rate that is synchronized with a processing system for processing information detected by the detection units <NUM>. This way, infrared illumination from either or both infrared LEDs <NUM>, <NUM> may be readily distinguished from any background infrared illumination by the processing system (e.g., <NUM>, <NUM>). The control circuitry <NUM>, <NUM> may also provide wireless communication (e.g., infrared, Bluetooth, radio frequency etc.) with the processing system. <FIG> shows a side view of the calibration unit <NUM>, also showing a connection portion <NUM> of the e end-effector <NUM> for coupling to the calibration unit <NUM>.

Claim 1:
A system for providing extrinsic calibration of a camera to a relative working environment of a programmable motion device (<NUM>) that includes an end-effector (<NUM>), said system comprising:
a calibration unit (<NUM>) attached to the end effector (<NUM>), the calibration unit including a plurality of fiducials (<NUM>, <NUM>, <NUM>, <NUM>);
at least one camera (<NUM>) for viewing the plurality of fiducials (<NUM>, <NUM>, <NUM>, <NUM>) of the calibration unit (<NUM>) as the programmable motion device (<NUM>) moves in at least three degrees of freedom, and for capturing a plurality of images containing one or more of the plurality of fiducials (<NUM>, <NUM>, <NUM>, <NUM>); and
a calibration processing system (<NUM>, <NUM>) for analyzing the plurality of images to determine a fiducial location with respect to the at least one camera (<NUM>) to permit calibration of the at least one camera (<NUM>) with the programmable motion device (<NUM>),
wherein the system is characterized by the calibration unit (<NUM>) including a coupling portion (<NUM>) for coupling to the end-effector (<NUM>) and a transverse portion (<NUM>) that is transverse to the coupling portion (<NUM>), the transverse portion (<NUM>) including the plurality of fiducials (<NUM>, <NUM>, <NUM>, <NUM>) that are positioned on mutually opposing sides of the transverse portion (<NUM>) and further positioned on mutually opposing sides of the end-effector (<NUM>).