IEEE Standards Study Group:

Standard Metrics and Test Methods for Assessing the 

Performance of Grasping-Type Robot End-Effectors

Berk Calli (WPI)

Joe Falco (NIST)

Kenny Kimble (NIST)

Adam Norton (NERVE)

October 29, 2020
Agenda

• Introductions

• IEEE Study Group – Step 1: Project Authorization Request (PAR).

• NIST– Performance Metrics and Test Methods for Robotic Hands (Joe) 

• Round Robin Testing and Test Resources (Adam)

• Discussion: Approach for developing this standard

• Future Grasping and Manipulation Standards Efforts (Berk, Kenny)
IEEE Standards Study Group: Standard Metrics and Test Methods for 

Assessing the Performance of Grasping-Type Robot End-Effectors

Goal: A unified set of metrics and performance tests

that the robotics community will use to help advance

robot grasp type end-effector technologies through

standardized benchmarking with the longer-term use

as a standard specification to define the characteristics

of commercially available end-effector technologies.

Scope: This standard contains a series of metrics and

supporting test methods to perform a characterization

of grasping type robot end-effectors. Developing these

characterizations will guide in the selection of

appropriate designs to support an application and will

help to direct research and development

advancements. Each test method describes the test

setup, artifacts, measurements, and guidance for

analysis of the measurements. The metrics provide a

common language for comparing different designs and

will strengthen the development, and deployment of

more-capable grasping-type robot end-effectors toward

increasing the flexibility and general-purpose

applicability of robots.

URL: https://www.nist.gov/el/intelligent-systems-division-73500/robotic-grasping-and-manipulation-assembly/standards-efforts
Grasping and Manipulation Benchmarks

• Why robotic hands?

• Problem

• What makes a ‘good’ hand/gripper?

• Informing consumers

• Goal

• Feedback for R&D, benchmarking

• Technical specifications

• Mechanism

• Metrics and test methods

• Artifacts and datasets

• Data analysis guidance
NIST Proposed Metrics and Test Methods

See: https://www.nist.gov/el/intelligent-systems-division-73500/robotic-grasping-and-manipulation-assembly/publications
Test Method Example Measurement 

Instrument

Finger Strength

Touch Sensitivity

Finger Force Tracking

Force Calibration

Finger Repeatability

Grasp Strength

Slip Resistance

Grasp Efficiency

Cycle Time 

In-Hand Manipulation

Object Pose Estimation

Elemental Performance Measures

http://www.nist.gov/el/isd/grasp.cfm
Finger Strength

• What: Maximum force a robotic finger can impose on its 

environment

• How: Measure peak quasi-static forces with load cell at full 

finger extension

• Why: Strength across fingers can vary, weakest at full 

extension, positively correlated with payload, guide control
Grasp Strength

• What: The maximum force a robotic 

hand can impose on an object

• How: Artifact with intrinsic force 

sensing

• Why: Estimate payload
Slip Resistance

• What: Maximum force hand can sustain in shear

• How: Measure peak force at the onset of gross slipping

• Why: Indicator of minimum payload, indicator of wholehand friction coefficient, safer to test
Cycle Time

• What: A measure of the minimum time required for a 

robotic hand to cycle from pre-grasp, to grasp, to 

pre-grasp

• How: Requires two consecutive cycles with a force 

sensing object artifact

• Why: estimating productivity, guide controllers and 

planners
Finger Repeatability

• What: A measure of the difference in pose results 

when a finger is commanded to a position multiple 

times from the same direction

• How: Requires two consecutive cycles with a force 

sensing object artifact

• Why: Ability to re-establish pose
Touch Sensitivity

• What: A measure of the smallest, self-registered 

contact force exerted by a robotic finger on an object

• How: Measure maximum impact force at full finger 

extension at various joint speeds

• Why: Force dependent on speed, force maximized at 

full extension, minimize disturbance during object 

acquisition
Grasp Efficiency

• What: Ability to force-modulate 

grasp with increasing object 

disturbance

• How: Computed at each data point 

collected from initial grasp force 

until maximum pull force (gross 

slipping)

• Why: Automated grasping efficiency, 

minimize power consumption, 

minimize wear-and-tear
Finger Force Tracking

• What: Finger ability to impose desired contact forces 

on its environment

• How: Measure disparity between hand-reported forces 

and reference force-torque transducer

• Why: Prerequisite for force-modulated grasping and 

manipulation
Force Calibration

• What: Quantifying force calibration accuracy of 

onboard tactile sensing

• How: Coupled with finger force tracking test 

• Why: Non-removable force sensors, force accuracy 

often not reported/available, impacts force control 

strategies
In-Hand Manipulation

• What: Measure of a robotic hand’s ability to control the 

pose of an object

• How: MoCap system and objects with optical targets

• Why: Quantifies range-of-motion, frequency response, 

controller accuracy and repeatability, useful for 

functional-level tasks
Object Pose Estimation

• What: Measure a robotic hand’s ability to estimate the 

pose of an object (contact or non-contact)

• How: MoCap system and objects with optical targets

• Why: Object pose information can inform task-level 

operations, objects can move within grasps, visual 

occlusion
Publications

• J. Falco, D. Hemphill, K. Kimble, E. Messina, A. Norton, R. Ropelato, H. Yanko, 

“Benchmarking Protocols for Evaluating Grasp Strength, Grasp Cycle Time, 

Finger Strength, and Finger Repeatability of Robot End-effectors”, IEEE RA-L 

Special Issue on Benchmarking Protocols in Robotic Manipulation (2020).

• J. Falco, K. Van Wyk, E. Messina, Performance Metrics and Test Methods for 

Robotic Hands, NIST Special Publication 1227, Oct 2018

• J. Falco, K. Van Wyk, E. Messina, Terminology for Robotic Hands and 

Associated Performance Metrics, NIST Special Publication 1229, Oct 2018 

• J. Falco, K. Van Wyk, S. Liu, S. Carpin, “Robotic Grasping: Facilitating 

Replicable Performance Measures via Benchmarking and Standardized 

Methodologies”, IEEE Robotics and Automation Magazine, December 2015.

• Stefano Carpin, Shu Liu, Joe Falco, Karl Van Wyk, “Multi-Fingered Robotic 

Grasping: A Primer,” arXiv 1607.06620, online.
Discussion

• Suggestions for additional performance measures / 

test methods - ongoing

• Start the effort by choosing three existing performance 

measures

• For each, review suggested revisions to date (rhgm

feedback) including implementation by papers that 

reference NIST work.