ROBOTIC GRIPPER ALIGNMENT MONITORING SYSTEM

A method and system are provided for real-time predictive failure monitoring of a robotic gripper alignment. A sensor board embedded in a robotic gripper includes a plurality of sensors to generate sensor data comprising motion data, inclination data and accelerometer data. Sensor board includes an alignment classification engine using machine learning based models to classify alignment and misalignment configuration of the robotic gripper based on received sensor data. Notification is sent to a user interface for both correct alignment and misalignment classifications in real-time.

TECHNICAL FIELD

This application relates to equipment monitoring. More particularly, this application relates to predictive failure monitoring for misalignment of a robotic gripper.

BACKGROUND

Robotic grippers are widely used in automated parcel distribution and material processing facilities. As an illustrative example,FIG.1shows an automated tote stacking system101designed to stack and unstack totes111(e.g., bins, trays, or the like) used as a transport container for moving materials on a system of conveyors. A key component in a stacking robot is the gripper201, as illustrated inFIG.2, which is a simple mechanical end effector with electromechanical actuation and a metal hook202to grab, manipulate and release the tote111.

Gripping functionality is achieved by the mechanical design and its precise alignment with respect to the tote stack. A failure point for robotic grippers is that proper alignment can be broken during installation or operation, preventing proper grasp of the tote, rendering the automated tote stacking system101inoperable. Gripper failure can arise from misalignment caused by various reasons, such as imprecise installation, or gradual slippage of fasteners over the course of repetitive task operations. The gripper alignment may not get any attention from maintenance staff until it fails to operate.

Hence, alignment calibration is required during installation and routine maintenance. This calibration, however, involves repeated manual testing and observation to make recommended alignments. Current procedures for calibration leave the validation and alignment to human installer/maintenance technician/operator to fully inspect all gripper alignment points after a functional failure or part of preventive maintenance. However, this procedure is incapable of predicting a functional failure, an early-stage misalignment, or a tote integrity-related issue related to the tote stacking system operation.

Currently, there are no automated means to monitor or diagnose gripper alignment deviations and repeated manual intervention is required to diagnose the alignment deviations. Nor is there any system to provide unambiguous information to maintenance technicians from various alignment points to determine which alignment point has the condition of interest and what procedure is to be followed. A need exists for a fully automated gripper alignment monitoring system that can easily notify installers and maintenance technicians of any alignment issue or other maintenance need of the gripper and provide unambiguous information regarding identified issues and failure predictions.

SUMMARY

Aspects of the present disclosure provide a method and system for real-time predictive failure monitoring of a robotic gripper alignment. A sensor board embedded in a robotic gripper includes a plurality of sensors to generate sensor data comprising motion data, inclination data and accelerometer data. Sensor board includes a processor and an alignment classification engine using machine learning based models to classify alignment and misalignment configuration of the robotic gripper based on received sensor data. Notification is sent to a user interface for both correct alignment and misalignment classifications in real-time.

Aspects of the present disclosure further provide engine incorporating proximity sensor data and event data detected by a main controller for classification of alignment and misalignment configuration of the robotic gripper.

Aspects of the present disclosure further provide classification and notification of a damaged tote based on audio sensor data.

DETAILED DESCRIPTION

Methods and systems are disclosed to solve the technical problem of continuous preventative failure monitoring a gripper alignment of a robotic device used in an automated material processing system. In contrast with conventional approaches for monitoring, the disclosed embodiments improve the reliability of robotic grippers through real time continuous monitoring of gripper alignment. Using the automated monitoring method of this disclosure eliminates slow and unreliable manual preventive maintenance and gripper alignment verification procedures. The disclosed system has a novel gripper design with an inbuilt sensing and processor core for accurate monitoring capability. The embedded sensing computer is configured as constrained edge device (i.e., limited single core processing functionality) with a machine learning core to execute machine learning (ML) processing for classification of detected misalignments. In contrast with the disclosed compact single multi-sensor node, conventional monitoring consists of binary failure notification without context, by a wide physical arrangement of multiple sensor installations. An Edge Gateway computer provides means for performing all edge processing from all constrained edge gripper devices. The disclosed system recommends the most probable alignments required, thereby reducing manual inspection time. The gripper alignment monitoring system provides automatic notification to the maintenance team of any gripper alignment deviations by triggering a warning to avoid functional failure and positive confirmation when the gripper is properly aligned. Moreover, the gripper alignment monitoring system provides a preventative measure against failure of the robotic gripper from executing grasping tasks by early detection of alignment deviations, unlike conventional maintenance which often finds misalignment only after system failure.

The embodiments of this disclosure are described with reference to a robotic gripper arrangement in which a pair of grippers, such as gripper201illustrated inFIG.2, operate in tandem to grasp a tote used for material transport in an automated material processing facility. Grasping may relate to stacking, unstacking, transporting, or other operations related to grasping. However, the functionality of the gripper alignment monitoring system is not limited to this particular arrangement of robotic grippers and is easily applicable to any number of robotic grippers to be monitored. Nor is the particular geometry or topology of the gripper201essential to the installation of the topology gripper alignment monitoring system; variations to the basic structure of gripper201can be accommodated by disclosed system as described.

FIG.3illustrates examples of various instances of detectable gripper misalignment locations for monitoring according to embodiments of this disclosure. In an embodiment, vertical gripper assembly screws311are used to fasten lateral adjuster plate312, which can slide as necessary for alignment of gripper201. In another embodiment, horizontal gripper adjustment screws313fasten the gripper to an actuator (e.g., a cam follower) and horizontal gripper adjustment screws314fasten gripper to a backstop support structure315, both of which are adjustment points for alignment of the gripper. In an embodiment, proximity sensor316, which is affixed to gripper201for guiding actuation of the gripper, may be misaligned or may malfunction, causing the actuator of gripper to overshoot or undershoot the target position for grasping the tote. Monitoring of such alignment fasteners and proximity sensors by the disclosed monitoring system can locate and prognosticate a potential misalignment before a failure occurs.

FIG.4illustrates an example of control system hardware for performing gripper alignment monitoring in accordance with embodiments of this disclosure. Control system400includes means to readily identify the alignment need of the robotic gripper or fleet of robotic grippers. Main controller411, which may be implemented as a programmable logic controller, is configured to control the robotic gripper along with the remainder of the automated process system in which the gripper operates. Control system400includes a sensor board415having multiple embedded sensors (motion, inclination, accelerometer, and microphone), memory for software/firmware to execute machine learning classification, and a processor, altogether arranged in a printed microcircuit board. One or more of the sensors are configured with 3-axis sensing. Firmware on the sensor board is programmed with a trained classification model that can determine type of misalignment and/or cause of misalignment. In an embodiment, gripper401is configured with a precisely milled pocket414for housing the sensor board415. The milled pocket414may be sealed with epoxy, polycarbonate cover, or the like for protection of the circuit board415and to fix the position of the sensor node flush with gripper body surface for optimized precision in calibration of gripper orientation. In an embodiment, the body of gripper401is configured with a physical channel to conceal wiring421which connects to sensor board415for power and data transmission. An advantage of a hard wired sensor board is for reliable power source rather than from a battery that requires recharging or periodic replacement. In an embodiment, wiring421conforms to a USB connection for both communication and powering the inbuilt gripper sensor node415.

The embedded sensors, firmware/software and processor enable local monitoring and real time classification, eliminate the need for transfer of high-frequency multiple sensor data to edge gateway412or cloud413for the classification analysis. Local classification of gripper misalignment is communicated to a user interface410via edge gateway412as a maintenance notification.

FIG.5illustrates an example of a software architecture for gripper alignment monitoring in accordance with embodiments of this disclosure. An automatic machine learning system511includes machine learning model generation module512and firmware building module513. A gripper sensor node system521includes a data collection module522, a firmware update module523, a real-time classification engine524, and an over-the-air (OTA) manager module525. Edge gateway node system531includes data storage535, OTA manager module532, cloud interface module533and programmable logic controller (PLC) interface module534. Cloud platform541includes a user interface module and cloud data storage. A local user interface551is available for communicating with the system components via edge gateway node system.

Automatic machine learning system511is used to generate misalignment classification models through machine learning model generation module512. In an embodiment, the machine learning model generation module512implements an automatic machine learning (AutoML) algorithm for model selection and to generate a dynamic model to achieve misalignment classification through establishing the relation between alignment points, output data concerning good alignment obtained from multiple sensors and other data (e.g., PLC event data), and different types of bad alignment. Models may include clustering algorithms, neural networks or statistical models. Sensor readings of the 3 axis sensors along with other sensor data are checked for true positives, false positives, true negatives and false negatives. Criteria for assessing the classification models include examination of precision and recall according to metrics. Once trained, the system can label this relationship to various alignment needs using a database of alignment and replacement recommendations using an automatic machine learning module. Firmware building module513creates a firmware package for the trained AutoML models, which is deployed to the firmware module of gripper sensor node system521using a wired connection or an OTA download.

Gripper sensor node system521receives sensor data at data collection module522and real-time classification engine524classifies the sensor data using the models stored in firmware523. Upon misalignment detection, the model determines which form of misalignment is most likely present. OTA manager525controls model downloads from automatic ML system511and transmissions of misalignment classification results to edge gateway node system531. Alternatively, the data is transmitted over wired links.

Edge gateway node system531receives real-time misalignment classifications from gripper sensor node system521via OTA manager or a wired connection and stores in data storage535. Misalignment classifications may be triggered by events detected by the PLC that controls the automated system in which the robotic gripper operates. PLC interface534receives the detected event information and stores in data storage535. Using the real-time classification data and the event data, maintenance manager536determines the required maintenance adjustment and generates a maintenance notification which is sent to local UI551or remote UI module of cloud server541. The maintenance notification may also be sent to a central condition and fleet monitoring system if used by the automation facility, either on premises or to cloud server541.

FIG.6is a flowchart illustration of an example method for continuous monitoring of the gripper alignment in accordance with embodiments of this disclosure. With trained models deployed in the gripper sensor node system511, the gripper alignment monitoring process starts at601by initializing the gripper alignment monitoring system at610. The system activates sources all data inputs which include one or more of the following: sensors providing inputs for motion data611, inclination data612, and accelerometer data613; PLC connection for event data614. Alignment classification engine621executes the classification models based on data inputs611,612,613and614. Results are examined at623and if no misalignment classifications are determined, than a proper alignment notification625is sent to the user interface. If one or more misalignment classifications at623are determined, then misalignment diagnosis is performed by alignment classification engine to identify the most likely cause(s) for misalignment. In this example, the causes may include one or more of the following: assembly screws631, lateral plate632, cam followers633, backstop634, proximity sensor635, or gripper deformation636. A misalignment notification is generated and an realignment recommendation notification is sent to the user interface indicating a particular fastener location as the most likely cause for misalignment.

In an embodiment, additional sensor monitoring may be incorporated into the gripper alignment monitoring system, including but not limited to an audio sensor such as a micro-electro-mechanical microphone system (MEMS) which generates audio sensor data615to detect anomalous sounds indicative of damage to tote integrity. Such an audio sensor is placed in a location to capture sounds during tote handling or manipulation in the automated processing system. A specialized classification engine is trained to learn from audio data sensed by the microphone at the robotic gripper location and is capable of classifying a tote that having flaws with integrity, which may be the cause for gripper failure instead of a misalignment cause. During monitoring, if tote integrity check at624indicates a detected anomaly, a tote integrity notification638is sent to indicate a potentially damaged tote, otherwise there is no notification.

The embodiments of the present disclosure may be implemented with any combination of hardware and software. In addition, the embodiments of the present disclosure may be included in an article of manufacture (e.g., one or more computer program products) having, for example, a non-transitory computer-readable storage medium. The computer readable storage medium has embodied therein, for instance, computer readable program instructions for providing and facilitating the mechanisms of the embodiments of the present disclosure. The article of manufacture can be included as part of a computer system or sold separately.

The program modules, applications, computer-executable instructions, code, or the like depicted inFIG.5are merely illustrative and not exhaustive and that processing described as being supported by any particular module may alternatively be distributed across multiple modules or performed by a different module. In addition, various program module(s), script(s), plug-in(s), Application Programming Interface(s) (API(s)), or any other suitable computer-executable code hosted locally, and/or hosted on other computing device(s) accessible via one or more of network, may be provided to support functionality provided by the program modules, applications, or computer-executable code and/or additional or alternate functionality. Further, functionality may be modularized differently such that processing described as being supported collectively by the collection of program modules may be performed by a fewer or greater number of modules, or functionality described as being supported by any particular module may be supported, at least in part, by another module. In addition, program modules that support the functionality described herein may form part of one or more applications executable across any number of systems or devices in accordance with any suitable computing model such as, for example, a client-server model, a peer-to-peer model, and so forth. In addition, any of the functionality described as being supported by any of the program modules depicted inFIG.5may be implemented, at least partially, in hardware and/or firmware across any number of devices.

Although specific embodiments of the disclosure have been described, one of ordinary skill in the art will recognize that numerous other modifications and alternative embodiments are within the scope of the disclosure. For example, any of the functionality and/or processing capabilities described with respect to a particular device or component may be performed by any other device or component. Further, while various illustrative implementations and architectures have been described in accordance with embodiments of the disclosure, one of ordinary skill in the art will appreciate that numerous other modifications to the illustrative implementations and architectures described herein are also within the scope of this disclosure. In addition, it should be appreciated that any operation, element, component, data, or the like described herein as being based on another operation, element, component, data, or the like can be additionally based on one or more other operations, elements, components, data, or the like. Accordingly, the phrase “based on,” or variants thereof, should be interpreted as “based at least in part on.”