Patent Description:
The invention generally relates to robotics, and relates in particular to robotic control systems that are designed to accommodate a wide variety of unexpected conditions and loads.

Most industrial robotic systems operate in a top-down manner, generally as follows: a controller samples a variety of sensors, and then logic on that same controller computes whether or not to take action. The benefit of this logic flow (usually referred to as "polling") is that all of the control logic is in the same place. The disadvantage is that in practical robotic systems, the signals are often sampled quite slowly. Also, all sensors must be wired to the control cabinet leading to long and error-prone cable runs.

A specific example of this traditional architecture would generally be implemented by a legacy robot supplier such as those sold by ABB Robotics, Inc. of Auburn Hills, Michigan, Kuka Roboter GmbH of Germany, Fanuc America Corporation of Rochester Hills, Michigan, or one of their top-tier integrators. All of these suppliers generally encourage the same architecture, and have similar form factors. For example: a welding cell used in an automotive facility might have an ABB IRC5 control cabinet, an ABB IRB2600 <NUM> reach <NUM> degree of freedom robot, a Miller GMAW welding unit wired over an industrial bus (Devicenet/CANbus) to the IRC5, and an endo-farm tooling package mounting a GMAW torch (e.g., a Tregaskiss Tough Gun). All programming is done on the IRC5, and the end effector has no knowledge of the world, and things like crashes can only be observed or prevented on the IRC5, which is itself quite limited.

Again, in such systems, however, the signals are often sampled relatively slowly and sensors must generally be wired to the control cabinet. There remains a need therefore, for a robotic control system that is able to efficiently and reliably provide dynamic control and responsiveness to conditions in the environment of the robot.

<CIT> discloses robotic end effector system and method having a plurality of end effectors which are selectively suitable for particular applications on a workpiece. The end effectors include a resident controller adapted to execute tasks specific to the end effector and are rapidly attachable and removable from the robot for easy change over to different workpieces.

In accordance with an embodiment, the invention provides an articulated arm system in accordance with the appended independent claim <NUM>. In accordance with another embodiment, the invention provides a method of providing a control signal to an end effector of an articulated arm in accordance with the appended independent claim <NUM>. Some further optional features are provided in the appended dependent claims.

In accordance with an embodiment, the invention provides an architecture for robotic end effectors that allows the end effector to alter the state of the robot. In accordance with certain embodiments, the end effector may observe the environment at a very high frequency and compare local sensor data and observations to a set of formulas or trigger events. This allows for robot-agnostic low latency motion primitive routines, such as for example move until suction and move until force without requiring the full response time of the robotic main controller. A robotic end effector is therefore provided that can alter the state of the robot, and further that may be modified during run time based on a variety of control policies. In accordance with further embodiments, the invention provides a multifaceted gripper design strategy has also been developed for multimodal gripping without tool changers.

A majority of industrial robotic systems execute their programming logic control in one place only - in the robot controller. The robot controller in these systems is often a large legacy controller with an obscure and (and sometimes poorly featured) programming language. In contrast, the majority of modern and emerging robotic systems contain logic distributed between a robot controller and several workstation computers running a modern operating system and software stack, such as the Ubuntu operating system as sold by Canonical Ltd. of Isle Of Man, the Linux operating system as provided by The Linux Foundation of San Francisco, California and the ROS robotic operating environment as provided by Open Source Robotics Foundation of San Francisco, California.

A positive aspect of these architectures is that they provide tremendous, even arbitrary, amounts of computing power that may be directed towards problems like motion planning, localization, computer vision, etc. The downsides of this architecture are primarily that going through high-level middleware such as ROS adds significant latency, and evaluating a control policy in a loop may see round trip times of well over <NUM>.

As a unifying solution for this problem, a gripper control system has been developed with onboard electronics, sensors, and actuators to which high level logic controlling the system uploads a set of 'triggers' at runtime. These are control policies, such as stop the robot when a force above X Newtons is observed, or when object is observed by depth sensor, slow down the trajectory. The end effector may then evaluate the policy natively at the kHz level, and trigger actions of situations where the gripper should take an action.

<FIG> shows a portion of an articulated arm assembly that includes a force sensor system <NUM>, on-board control electronics <NUM>, a vacuum end effector <NUM>, a three dimensional depth sensor system <NUM>, an input pressure sensor <NUM>, an output pressure sensor <NUM>, and another vacuum end effector <NUM>. The articulated arm therefore includes on-board control electronics <NUM> as well as multiple end effectors <NUM>, <NUM>. In certain embodiments, the articulated arm may include a further end effector similar to end effector <NUM> that is adjacent end effector <NUM> (and is therefore not shown in <FIG>).

<FIG> shows the on-board control electrics <NUM>, which includes connectors <NUM> for the force sensors, connectors <NUM> for the robot, connectors <NUM> for the pressure sensors, connectors <NUM> for LEDs such as RGB LEDs, and connector <NUM> for a microcontroller with serial and wireless connections.

In accordance with an embodiment, the invention provides an articulated arm control system that includes an articulated arm with an end effector, at least one sensor for sensing at least one of the position, movement or acceleration of the articulated arm, a main controller for providing computational control of the articulated arm, and an on-board controller for providing, responsive to the at least one sensor, a control signal to the main controller.

<FIG> shows a pre-programmed robot control routine that begins (step <NUM>), executes a first batch program (step <NUM>), polls sensors for inputs (step <NUM>), executes a second batch program (step <NUM>), polls the sensors again for inputs (step <NUM>), executes a third batch program (step <NUM>), and then ends (step <NUM>). If the system is relying on sensor inputs to cause a change in the program (e.g., stop due to readings of a force sensor), the system must wait for that sensor to be polled. In accordance with embodiments of the present invention, on the other hand, interrupt signals may be provided to the main robot controller to cause pre-defined specific responses. As diagrammatically shown in <FIG>, such interrupt signals may be received any time and immediately processed.

<FIG> shows a robotic system <NUM> in accordance with an embodiment of the present invention in which the articulated arm portion of <FIG> (including the force sensor system <NUM>, on-board control electronics <NUM>, the vacuum end effector <NUM>, the three dimensional depth sensor system <NUM>, the input pressure sensor <NUM>, the output pressure sensor <NUM>, and the other vacuum end effector <NUM>) is attached to further articulated arm sections <NUM>, <NUM>, <NUM>, <NUM> and <NUM>. The articulated arm section <NUM> is attached to a robot base <NUM>, which is coupled to a main robot controller <NUM> by connector cables <NUM>. An interrupt signal may be provided from the on-board control electronics <NUM> to the main robot controller <NUM> either by direct wire connection or wirelessly.

This solution conveys several tremendous advantages: First, one may add the advanced behaviors one generates to any robot, as long as the robot complies with a relatively simple API. Second, one may avoid long cable runs for delicate signals, from the end effector to the robot control box (which is often mounted some distance away from a work cell). Third, one may respond to changes in the environment at the speed of a native control loop, often thousands of times faster than going exclusively through high level logic and middleware. Fourth, one may alter these policies at runtime, switching from move until suction to stop on loss of suction, as well as chaining policies.

In accordance with a further embodiment, the invention provides a method of altering or overriding a control signal from a main controller to an end effector.

<FIG>, for example, shows an implementation of the on-board control electronics <NUM>. The electronics <NUM> receives at <NUM> control signals from the main robot controller <NUM> (shown in <FIG>), which causes motors M1, M2, M3 (shown at <NUM>, <NUM> and <NUM>) and the vacuum (shown at <NUM>) of the articulated arm to move. The motors may control, for example, elbow, wrist and gripper motors of the articulated arm. In the absence of any feedback signals from the environment, the control signals <NUM> are routed to the appropriate motors for control of the articulated arm in accordance with the program in the main controller.

The electronics <NUM> however, is also coupled to input sensors including pressure sensors <NUM>, <NUM> and <NUM>, a camera <NUM>, force / torque sensors <NUM>, <NUM> deflection / deformation sensor <NUM> and flow sensor <NUM>. These sensors are coupled to an on-board controller <NUM> that determines whether to send an interrupt signal to the main robotic controller, and determines whether to immediately take action by overriding any of the output signals to motors M1 - M3 and the vacuum. This is achieved by having the on-board controller <NUM> be coupled to control junctions <NUM>, <NUM>, <NUM> and <NUM> in the control paths of the signals <NUM>, <NUM>, <NUM> and <NUM>.

The robot, for example, may be working in very cluttered, dynamic environments. In order to manipulate objects in these conditions, one needs much more sensing than a typical, more structured, open-loop robotic system would need. The grippers are therefore instrumented with absolute pressure sensors, a 3D RGBD camera, force-torque sensor, and suction cup deflection sensing. By sensing and processing the sensor data directly at the wrist via a microcontroller hardware interrupts may be set (via digital inputs) immediately (hundreds/thousands of Hz). There is much more overhead in the other approach of communicating the sensor data back to the main robotic controller for analysis, which would be significantly slower. This allows one to modify robot motion/execution significantly faster, which in turn allows one to move the robot significantly faster, adapting at speeds not possible otherwise. In these dynamic and unpredictable environments, adapting and providing recovery quickly is vitally important.

The pressure sensors, for example, may provide binary gripping/not gripping, and threshold comparisons (> grip pressure, < required retract pressure, < drop pressure). The pressure sensors may also map material properties/selected grasps to expected pressure readings and in real-time modify trajectory execution (speeds, constraints) in order to ensure successful transportation. The pressure sensors may also provide real-time monitoring of upstream pressure (pressure from source) to ensure expected air pressure available, and modify expected suction measurements from downstream accordingly.

The camera may be an RGBD camera that provides data regarding environment registration, automated localization of expected environment components (conveyor, out shelves, out-bin stack) to remove hand tuning, and expected/unexpected objects/obstacles in the environment and modify trajectory execution accordingly.

The force-torque sensors may provide impulse interrupts. When an unusual or unexpected force or torque is encountered we can stop trajectory execution and recover, where the robot before would have continued its motion in collision with that object causing damage to the object or robot. The force-torque sensors may also provide mass/COM estimates, such as Model Free mass estimates that may inform trajectory execution to slow down as one may be dealing with higher mass and inertias at the endpoint, which are more likely to be dropped due to torqueing off. Model Based mass estimates may also be used to ensure quality of grasp above COM, make sure that the correct item is grasped, that the item is singulated, and that the item is not damaged (unexpected mass).

The deflection/deformation sensor may observes suction cup contact with the environment (typically when one wants to interrupt motion) as the bellows are deflected and have not modified pressure readings, and have not yet displayed a noticeable force impulse. The deflection sensor at its simplest will be used for interrupting motion to avoid robot Force Protective Stops by being that earliest measurement of contact. The deflection/deformation sensor may also measure the floppiness of the picks, which allows one in real-time to again modify trajectory execution, slowing down or constraining the motions to ensure successful transport, or putting it back in the bin if the floppiness is beyond a threshold at which the item may be safely transported.

The flow sensors may detect changes in the amount of airflow as compared to expected air flow values or changes. For example, upon grasping an object, it is expected that the airflow would decrease. Once an object is grasped and is being carried or just held, a sudden increase in air flow may indicate that the grasp has been compromised or that the object has been dropped. The monitoring of weight in combination with air flow may also be employed, particularly when using high flow vacuum systems.

With reference to <FIG>, the program begins (step <NUM>), by applying the end effector to an object at a selected grasp location (step <NUM>). A vacuum is applied to the end effector (step <NUM>), and the sensors are polled (step <NUM>). Responsive to the sensor inputs, the system determines whether it should try to pick up the object (step <NUM>). For example, if too much vacuum flow is detected, the system may determine that the grasp is insufficient for picking up the object. In this case, the system will determine (stop <NUM>) whether there have already been too many tries to pick up this particular object (possibly involving the main controller). If there have not already been too many retrys, the system may select another grasp location for the object (step <NUM>) and return to step <NUM> above. If the system determines that there have already been too many retries, the system will select a new object and a new associated grasp location (step <NUM>) and return to step <NUM> above.

If the system determines that the object should be picked up (step <NUM>), the system will then lift the object (step <NUM>) and then read the sensors (step <NUM>). If the orientation of the end effector needs to be adjusted, the system adjusts the orientation of the end effector (step <NUM>), for example to cause a heavy object to be held in tension (vertically) by the end effector as opposed to a combination of a vertical and horizontal grasp that would cause a sheer force to be applied. In another example, the system may choose the hold an lighter object with a combination of a vertical and horizontal grasp to accommodate a high speed rotation movement so that when the object is being moved, a centrifugal force will be applied in the direction aligned with the grasp of the object. Once the orientation of the end effector is chosen (step <NUM>), the system will choose a trajectory path (step <NUM>), and then begin execution of the trajectory, e.g., the batch program N (step <NUM>).

With reference to <FIG>, the execution of the batch program N may begin by polling the one or more sensors for inputs (step <NUM>). If none of the inputs exceeds a defined threshold for the main control command (step <NUM>), e.g., to move in a certain vector, then the system will continue to execute the batch program (step <NUM>) until done (whereupon the system returns to step <NUM>). If the batch program is not done, the system returns to step <NUM>, polling the sensor(s) for inputs. If any of the inputs from the sensor(s) do exceed a threshold (step <NUM>), then the system will determine whether the main control command should be altered (e.g., movement slowed or the path changed) (step <NUM>), and if so, the program will so alter the main control command (step <NUM>). If the main control command is not altered, the system will determine whether the main control command should be overridden (step <NUM>), e.g., movement of the end effector should be stopped or the object should be put down for a new grasp attempt, or the object has been dropped, in which case, the system will proceed to pick up a new object and signal for cleaning by a human that an object has been dropped. In any of the exemplary cases, the program will so override the main control command (step <NUM>). In either case, the system then returns to executing the batch program as either altered or overridden, returning to step <NUM> until done. If the main control signal for a batch program is changed (altered or overwritten), the main controller is also promptly notified.

In accordance with another embodiment, the invention provides an articulated arm control system includes an articulated arm with an end effector, at least one sensor for sensing at least one of the position, movement or acceleration of the articulated arm, and a main controller for providing computational control of the articulated arm, and an on-board controller for providing, responsive to the at least one sensor, a motion signal that directly controls at least a portion of the articulated arm.

<FIG>, for example shows the robotic system <NUM> of <FIG> except that the articulated arm portion of <FIG> is rotated with respect to the articulated arm section <NUM> such that the vacuum end effector <NUM> is now positioned to engage the work environment, while the vacuum end effector <NUM> is moved out of the way.

A unique contribution of the articulated arm is its multiple facets for multimodal gripping, e.g., having multiple grippers packaged on a single end effector in such a way that the robot can use different grippers by orienting the end effector of the robot differently. These facets can be combined in combinations as well as used individually. Other more common approaches are tool changers, which switch a single tool out with a different one on a rack. Multimodal gripping of the present invention reduces cycle time significantly compared to tool changers, as well as being able to combine multiple aspects of a single end effector to pick up unique objects.

The gripper designs in the above embodiments that involved the use of up to three vacuum cups, may be designed specifically for picking items of less than a certain weight, such as <NUM> lbs, out of a clutter of objects, and for grasping and manipulating the bins in which the objects were provided.

The same approach to instrumentation of a vacuum grasping end effector may be applied to any arbitrary configuration of vacuum cups as well. For example, if the robotic system needs to handle boxes such as might be used for shipping of things, then arbitrary NxM arrangements of the suction cells may be created to handle the weight ranges of such packages. <FIG> for example shows an end effector <NUM> that includes a <NUM> by <NUM> array of end effector sections <NUM>, each of which includes a vacuum cup <NUM>. Each end effector section <NUM> may include pressure sensors as discussed above, and each vacuum cup <NUM> may include a deformation sensor that is able to detect deformation along any of three dimensions. The end effector sections <NUM> are mounted to a common base <NUM> that includes a coupling <NUM> for attachment to an articulated arm.

<FIG> shows an end effector <NUM> that includes a <NUM> by <NUM> array of end effector sections <NUM>, each of which includes a vacuum cup <NUM>. Again, each end effector section <NUM> may include pressure sensors as discussed above, and each vacuum cup <NUM> may include a deformation sensor that is able to detect deformation along any of three dimensions. The end effector sections <NUM> are mounted to a common base <NUM> that includes a coupling <NUM> for attachment to an articulated arm.

The 3x3 array that may, for example, handle up to <NUM> pound packages, and the 6x6 array that may handle up to <NUM> pounds. Such scaling of end effector sections may be made arbitrarily large, and of arbitrary shapes (if, for example, the known objects to be handled are of a particular shape as opposed to generally square/rectangular).

It is significant that by extrapolating the standard vacuum cell to arbitrary sizes/shapes, such an instrumented end effector may be designed for any given object or class of objects that shares all the benefits of such instrumentation as the above embodiments.

Claim 1:
An articulated arm system, comprising:
an articulated arm (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) including an end effector (<NUM>, <NUM>); and
an articulated arm control system including:
at least one sensor (<NUM>, <NUM>, <NUM>, <NUM>) proximate the end effector for sensing at least one of a position, a movement, and an acceleration of the articulated arm;
a main controller (<NUM>) configured to automatically provide computational control of the articulated arm; and
characterized by an on-board controller (<NUM>) mounted on the articulated arm proximate the end effector and coupled to the at least one sensor, wherein the on-board controller is configured to poll the at least one sensor and automatically provide, responsive to output of the at least one sensor, an interrupt signal to the main controller to cause a predefined response to the output of the at least one sensor.