http://nerve.uml.edu

Adam Norton, Associate Director

University Of Massachusetts Lowell

New England Robotics Validation And Experimentation (NERVE) Center

adam_norton@uml.edu 

Development and Validation of Robotic 

Grasping and Manipulation Test Methods
http://nerve.uml.edu

• The NERVE Center is an interdisciplinary robotics testing, research, and training facility 

that evaluates robot capabilities, human performance, and human-robot interaction. 

• Our mission is to improve the development of robot systems by enabling evaluation 

across many domains including industrial automation, exoskeletons and wearable 

robots, and disaster response.

• Test and evaluation services are available on-site, in the field, and remotely; we can 

provide test apparatuses, sensors, data collection, personnel, logistics, and analysis.

• We participate in the development of consensus standards through ASTM: E54.09 

response robots, F45 industrial mobile vehicles, F48 exoskeletons and exosuits.

• Director: Holly Yanco

• Associate Director: Adam Norton

• Affiliated Faculty: 

– Biomedical Engineering: Bryan Buchholz

– Computer Science: Reza Ahmadzadeh, Holly Yanco

– Electrical and Computer Engineering: Yan Luo, Paul Robinette, Jay Weitzen, 

Thanuka Wickramarathne

– Mechanical Engineering: Yan Gu, Murat Inalpolat, Kshitij Jerath, Kelilah Wolkowicz

– Physical Therapy and Kinesiology: Pei-Chun Kao, Alexandre Lopes, Yi-Ning Wu

– Plastics Engineering: Ramaswamy Nagarajan

University of Massachusetts Lowell

New England Robotics Validation and Experimentation (NERVE) Center
http://nerve.uml.edu

ARMada Industrial Manipulator Testbed

Rethink Robotics 

Vacuum Gripper

Robotiq Epick Robotiq 2-Finger 

Adaptive Gripper 

Robotiq 3-

Finger Adaptive 

Gripper 

Wonik

Robotics 

Allegro Hand

RightHand

Robotics ReFlex

TakkTile 2

OnRobot RG2-FT Vaccon

Vacuum

Grippers

Schunk 2-Finger 

Grippers

SAKE EZGripper

Gen2

Soft Robotics

Gripper

Schunk 3-

Finger

Grippers

Universal Robots Omron TM700

UR5e

Omron LD90 ABB YuMi Rethink Robotics 

Sawyer

Agility Robotics 

Digit

Fetch Mobile KUKA LBR iiwa

Manipulator

Kinova Gen3 Kinova JACO2 Yaskawa Motoman

GP7

Cognex Camera Asus Xtion Pro OptiTrack V120:Trio ATI 6-axis 

F/T Sensor

Robotiq F/T 

Sensor

OptoForce OMD20-SE-40NIntel RealSense OptiTrack Prime 17w Fusion3 F410 Computer Stations
http://nerve.uml.edu

Manipulation Test Methods, Benchmarks, and Metrics

Grasp and finger performance test methods

SAKE Robotics EZ Gripper

Grasping

Manufacturing assembly task boards

Rethink Robotics Sawyer with Robotiq 2-finger

Assembly Collaboration

Example HRC scenario: artifact assembly, 

handoff between agents, transfer, organization

YCB Object 

and Model Set

Common objects used for 

manipulation research

Remote Experimentation of Manipulation for 

Online Test and Evaluation (REMOTE)
http://nerve.uml.edu

Manipulation Test Methods, Benchmarks, and Metrics

Complexity

End Effector Capabilities Object Grasping Object Manipulation Fine Object Manipulation

NIST Split Cylinder Artifact OSU Grasp Reset Mechanism YCB Object Set NIST Assembly Task Board
http://nerve.uml.edu

Grasp Strength and Grasp Cycle Time

Related publication: Joe Falco, Daniel Hemphill, Kenneth Kimble, Elena Messina, Adam Norton, Rafael Ropelato, and Holly 

Yanco. Benchmarking Protocols for Evaluating Grasp Strength, Grasp Cycle Time, Finger Strength, and Finger 

Repeatability of Robot End-effectors. IEEE Robotics and Automation Letters, Special Issue on Benchmarking Protocols for 

Robotic Manipulation, Volume 5, Issue 2, pp. 644-651, April 2020.
http://nerve.uml.edu

• Split cylinder or split block artifacts

– Encompassing/wrap grasps (cylinder artifact)

– Pinch grasps (block artifact)

• Orientations of grasping forces: 0° or 90°

Minimum, median, and maximum grasp sizes

0° 90°

Grasp Strength and Grasp Cycle Time

Related publication: Joe Falco, Daniel Hemphill, Kenneth Kimble, Elena Messina, Adam Norton, Rafael Ropelato, and Holly 

Yanco. Benchmarking Protocols for Evaluating Grasp Strength, Grasp Cycle Time, Finger Strength, and Finger 

Repeatability of Robot End-effectors. IEEE Robotics and Automation Letters, Special Issue on Benchmarking Protocols for 

Robotic Manipulation, Volume 5, Issue 2, pp. 644-651, April 2020.

4x
http://nerve.uml.edu

• Resources:

– 3D models for artifact 

cores, versions for 

dimensions every 5mm

– Software tools for 

extracting quasi-static force 

readings for grasp strength

– Software tools for 

extracting grasp cycle time 

in progress

Related publication: Joe Falco, Daniel Hemphill, Kenneth Kimble, Elena Messina, Adam Norton, Rafael Ropelato, and Holly 

Yanco. Benchmarking Protocols for Evaluating Grasp Strength, Grasp Cycle Time, Finger Strength, and Finger 

Repeatability of Robot End-effectors. IEEE Robotics and Automation Letters, Special Issue on Benchmarking Protocols for 

Robotic Manipulation, Volume 5, Issue 2, pp. 644-651, April 2020.

Grasp Strength and Grasp Cycle Time

https://tinyurl.com/UML-NIST-Manip-Bench
http://nerve.uml.edu

YCB Object and Model Set 

Common objects used for grasping and manipulation research; ~200 kits distributed internationally to date

UPDATES

More to come!

(unfortunately)
http://nerve.uml.edu

NIST Assembly Task Boards

NIST-ATB #1 IROS 2020 ATB
http://nerve.uml.edu

NIST Assembly Task Boards

• Robotics: Science and Systems (RSS) 2020 Workshop: Benchmarking Tools 

for Evaluating Robotic Assembly of Small Parts

– https://www.uml.edu/research/nerve/assembly-workshop-rss-2020.aspx

– Authors of accepted submissions received a NIST-ATB #1

• IROS 2020 Manipulation Competition: Manufacturing Track

– https://www.nist.gov/el/intelligent-systems-division-73500/iros-2020-roboticgrasping-and-manipulation-competition

– Competitors received two assembly task boards: one for practice, one for 

competition, hidden until competition time

– Variable task placement between practice and competition requires perception
http://nerve.uml.edu

NIST Assembly Task Boards

29 NIST-ATB #1 kits distributed to the United States, United Kingdom, Germany, Italy, New Zealand, Switzerland
http://nerve.uml.edu

NIST Assembly Task Boards

6 pairs of task boards distributed to the United States, China, Denmark, Germany, Japan, New Zealand

Practice Competition

Hole for

Large Round

Peg

Green Mark Red Mark

Practice Competition

Hole for

Large Round

Peg

Green Mark Red Mark

IROS 2020 Task Board Configurations
http://nerve.uml.edu

Remote Experimentation of Manipulation for 

Online Test and Evaluation (REMOTE) Testbed

• Infrastructure to support remote experimentation of 

robotic manipulation and benchmarking

• Set of arms and grippers available and a set of test 

stations for performing repeatable, measurable tasks

• All test stations sensorized for ground truth measurements 

and high-fidelity playback of activities
http://nerve.uml.edu

GRASPING TASKS

Remote Experimentation of Manipulation for 

Online Test and Evaluation (REMOTE) Testbed
http://nerve.uml.edu

Grasp Reset Mechanism (Oregon State University)

REMOTE: Repeatable Grasp Testing

8x
http://nerve.uml.edu

• NIST Assembly Task Board #1

• Must be positioned and reset manually

• Developing automated evaluation method 

utilizing computer vision to compare before 

and after states

Disassembled

Assembled

AN

REMOTE: Automatic Evaluation of NIST-ATB #1
http://nerve.uml.edu

• Automated evaluation

– Stage 1: Position invariant task board detection

– Stage 2: Data collection

– Stage 3: Component evaluation

• Future Work

– Adding more cameras for multiple views

– Measuring agreement between cameras

– Working towards empty and full conditions; how 

to treat in between states?

– Experiment with error states; e.g., misaligned 

gear placement

Stage 1

Stage 2

Empty Reference Evaluation Image

Component Regions

Full Reference

Grayscale, Bilateral Filter, Normalize

Stage 3

AN

REMOTE: Automatic Evaluation of NIST-ATB #1
http://nerve.uml.edu

Empty Eval Full Score

+1135

+2598

-9779

Bars Only Rods Only Gears Only Nuts Only Connectors Only

AN

REMOTE: Automatic Evaluation of NIST-ATB #1
http://nerve.uml.edu

Test Method Validation

• Round robin testing to exercise test methods

– Repeatability of fabricating the test artifacts and apparatuses

– Repeatability of results for the same robot in different facilities

– Evaluating robots available in different facilities

– Environmental effects of different facilities

• Gathering test data to demonstrate the effectiveness and robustness of the 

test method, modifying as needed based on these results

• Common practice for other robot test method efforts: response robots, 

mobile robots, etc.
http://nerve.uml.edu

Benchmarking Tools Available

YCB Object and Model Set NIST Assembly Task Board #1

Purchasing information available online
http://nerve.uml.edu

Thank you!

adam_norton@uml.edu

UMass Lowell NERVE Center

University Of Massachusetts Lowell

New England Robotics Validation And Experimentation (NERVE) Center