Grasping and Manipulation 

Performance Measures

and Benchmarking

Joe Falco, Karl Van Wyk, Elena Messina, Kenny Kimble

Intelligent Systems Division

National Institute of Standards and Technology
Agenda

• Overview of NIST Benchmarking work

– Grasping and Manipulation (elemental level)

– Robotic Assembly (functional/task level)

• Participant discussion of related work

• Formulation of a unified effort
NIST Goals

• Develop metrics, test methods and artifacts with example datasets 

to characterize the performance of grasping and manipulation with 

emphasis on deployment for manufacturing tasks.

• Provide the robotics community with unbiased measurement 

methods for both elemental characteristics and function-level 

performance capabilities.

• Short Term: Provide researchers and developers insight for 

improving their hardware and software designs 

• Long Term: Used to develop specifications that will help match 

capabilities to end-user manufacturing needs
NIST Testbed

• Hands/Grippers

- SCHUNK Dexterous Hand II

- ROBOTIQ 3-Finger Gripper

- Wonik Robotics Allegro Hand

- Empire Robotics VERSABALL Gripper

- Soft Robotics Inc. 

- Conventional parallel grippers

• Tactile Sensors

- Syntouch BioTac, BioTac SP, & Numatac

- OptoForce 3D Force sensors

- ATI Industrial Automation Nano17 F/T 

transducers

- Weiss Robotics Tactile sensors

• Arms 

- KUKA LWR 4+

- Universal Robots UR5, UR10

- ABB YuMi

- Rethink Robotics Baxter
Test Method Measurement Instrument

Finger Strength

Touch Sensitivity

Finger Force Tracking

Force Calibration

Grasp Strength

Slip Resistance

Grasp Efficiency

Cycle Time 

In-Hand Manipulation

Object Pose Estimation

Elemental Test Methods

http://www.nist.gov/el/isd/grasp.cfm
Touch Sensitivity

• What: A measure of the smallest, self-registered 

contact force exerted by a robotic finger on an object

• How: Measure maximum impact force at full finger 

extension at various joint speeds

• Why: Force dependent on speed, force maximized at 

full extension, minimize disturbance during object 

acquisition
Grasp Strength

• What: The maximum force a robotic 

hand can impose on an object

• How: Artifact with intrinsic force 

sensing

• Why: Estimate payload
In-Hand Manipulation

• What: Measure of a robotic hand’s ability to control the 

pose of an object

• How: MoCap system and objects with optical targets

• Why: Quantifies range-of-motion, frequency response, 

controller accuracy and repeatability, useful for 

functional-level tasks
Functional Performance Testing

• Quantify performance of a robotic system completing a 

task

• Tests target assembly operations: pick-place, insertion, 

fastening, meshing, wire harnessing, pulley belt routing

• Whole system-system testing

• Component testing

x

F(x)

https://www.nist.gov/programs-projects/performance-metrics-and-benchmarks-advance-state-robotic-assembly
Test Design

• Assembly Operations

• Design for Assembly (DFA)

• Human performance factor analysis

• Parameterizes objects

• Handling times

• Insertion times

• Guide design space

• Direct human comparison

Geoffrey Boothroyd, Peter Dewhurst, and Winston 

Knight. Product Design for Manufacture and 

Assembly. CRC press, 1994.
Performance Metrics

• Modes

• Disassembly

• Assembly

• Primary metrics

• Speed  completion time

• Reliability  probability of success

• Granularity

• Per-part/operation

• Whole board
Data Analytics

• Ordinal or Attribute Data

• Detecting statistical difference in datasets – Kolmogorov-Conover

• Check for differences as a whole or on a per rank basis

• Primary performance measure: probability of success (PS)

• Continuous Data

• Detecting statistical difference in datasets – Kolmogorov-Smirnov

• Check for differences between sample means and variances

• If no detectable differences, difference exists somewhere else 

(skewness, kurtosis)

• Matlab, R

http://www.nist.gov/el/isd/software.cfm
Example Peg-in-Hole

• Functional test method to measure the performance of 

robot systems at basic insertions

• Triangular design facilitates cyclical testing

• Peg-hole parameters, spacing based on human data

System 1 System 2

K. Van Wyk, M. Culleton, J. Falco, K. Kelley, “Comparative Peg-in-Hole Testing of a Force-based Manipulation 

Controlled Robotic Hand”, IEEE Transactions on Robotics, 2018, DOI: 10.1109/TRO.2018.2791591.
Example Peg-in-Hole

Robotic System Correlation KS μ

(s)

σ2(s) PS (%)

System 1 0.01 18.31 107.3 87.6

System 2 Spiral 0.07 * 37.13* 399.6* 95.2

System 2 Random -0.01 * 15.62 417.72 95.2

System 2 QuasiRandom-0.11 * 8.2* 50.25* 95.2

System 1 System 2

*Indicates statistical significance in comparison with System 1.
Task Boards

• Series of themed boards

• Each instance focuses on particular assembly facets

• Design with reference to DFA

• Low-cost, internationally replicable

• Real components

Task Board #1 Concepts
Task Board #1

• Focuses simple insertions, nut threading, gear 

meshing, plug connections

• Design intersection

• Spans DFA tables

• Real components

• Low-cost

• Internationally replicable

• IROS 2017 competition

• Distribution
Documentation

• Grasping and Manipulation

https://www.nist.gov/programs-projects/performance-metrics-andbenchmarks-advance-state-robotic-grasping

• Robotic Assembly

https://www.nist.gov/programs-projects/performance-metrics-andbenchmarks-advance-state-robotic-assembly

• Working Publications – (NIST Special Publication Format):

Terminology document – Proposed Standard Terminology for 

Robotic Hands and Associated Performance Metrics

Test method document – Performance Metrics and Test Methods 

for Robotic Hands
Publications

• K. Van Wyk, M. Culleton, J. Falco, K. Kelley, “Comparative Peg-in-Hole Testing 

of a Force-based Manipulation Controlled Robotic Hand”, IEEE Transactions on 

Robotics, 2018, DOI: 10.1109/TRO.2018.2791591.

• J. Falco, K. Van Wyk, S. Liu, S. Carpin, “Robotic Grasping: Facilitating 

Replicable Performance Measures via Benchmarking and Standardized 

Methodologies”, IEEE Robotics and Automation Magazine, December 2015.

• Stefano Carpin, Shu Liu, Joe Falco, Karl Van Wyk, “Multi-Fingered Robotic 

Grasping: A Primer,” arXiv 1607.06620, online.

• Shneier, Michael, et al., “Measuring and Representing the Performance of 

Manufacturing Assembly Robots”, NIST Interagency/Internal Report (NISTIR)-

8090, 2015.

• J. Falco, J. Marvel, R. Norcross, K. Van Wyk, “Benchmarking Robot Force 

Control Capabilities: Experimental Results”, NIST IR 8097, January, 2016.

• Falco, J., Marvel, J., Messina, E., A Roadmap to Progress Measurement 

Science in Robot Dexterity and Manipulation, NISTIR 7993, May 2014.

• Falco, Marvel, J., Messina, E., Dexterous Manipulation for Manufacturing 

Applications Workshop, NISTIR 7940, June 2013.
Competitions

• Mechanism for introducing benchmarking concepts to 

the research community in a competitive environment

• IROS 2016 household tasks with some manufacturing 

tasks introduced

• IROS 2017 dedicated manufacturing track

• World Robot Summit (WRS) Industrial Robotics 2018

• IROS 2019 - TBD

• WRS Industrial Robotics 2020 

• European Robotics League
IROS 2017

• Robotic Grasping and Manipulation 

Competition: Manufacturing Track 

https://www.nist.gov/el/intelligent-systemsdivision-73500/robotic-grasping-andmanipulation-competition-manufacturing
WRS 2018

• WRS 2018 World Robot Challenge (WRC) 

Industrial Robotics Category 

http://worldrobotsummit.org/en/wrc2018/industrial/
Related Work

• Yale-CMU-Berkley (YCB) Object Benchmarks for 

Robotic Manipulation

• Advanced Robotics for Manufacturing (ARM) Institute

• Berkeley led open discussion of robot grasping 

benchmarks, protocols and metrics

• UMass Lowell – NERVE Center

• Other? 
Formulation of Unified Effort

• Propose unifying efforts with regular meetings under 

IEEE RHGM TC

• Periodic NIST hosted online 

– What is an appropriate frequency?

– Sub-focus areas/sub-working groups could meet independently

• Yearly face-to-face at an IEEE robotics conference

• Consensus on tests, metrics, analyses will facilitate 

benchmarking

• Working publications as precursors to standards efforts

• Competitions