Benchmarking 

Protocols for 

Evaluating Small 

Parts Robotic 

Assembly Systems

Kenneth Kimble
About me

Mechanical Engineer from 

National Institute of Standards 

and Technology (NIST)

6+ years of experience testing 

robots

2 years working with assembly 

robotics and NIST assembly 

task boards (ATBs)
Introduction

• Automation in robotics has not seen enough adoption due to the inability of 

robotics to support component variation, tight tolerances, and cost 

effectiveness. 

• Benchmarking performance of assembly based robotics will push design to 

solve the problems associated with low-volume, high-mixture 

manufacturing. 

• These solutions are needed by small and medium sized manufactures which 

make up most of the worlds manufacturing base.

• Task boards are designed to address the most prevalent operations in 

assembly, i.e. insertions, threaded fasteners, and fittings. 

• They also address more complex operations such as handling and routing 

flexible parts.
Assembly Task

Boards (ATB)

• NIST ATB 1 (Fig 1): Peg insertions, gear 

meshing, electrical connectors, and nut 

threading

• NIST ATB 2 (Fig 2): Alignment and insertions, 

handling flexible parts, actuating tensioners, 

and threading bolts

• NIST ATB 3 (Fig 3): Manipulating flexible 

cables, routing cables through obstacles, and 

inserting electrical connectors

Fig. 1 Insertions and 

Connectors

Fig. 2 Belt Drives

Fig. 3 Cable Routing
Protocol

• Disassembly

 Boards begin fully assembled with a kit tray.

 Parts are removed from the board and placed in the tray.

• Assembly

 Boards begin disassembled with a kit layout locating the parts in 

predetermined orientations.

 Parts are taken from the kit tray and assembled on the task board.

• Points are awarded based on the level of completion of individual 

tasks

 Time is recorded as a secondary metric for benchmarking performance.

Assembled

Disassembled
NIST ATB 1

Insertions and Connectors

Insertion of pegs: 4 cylindrical pegs and 4 square 

pegs (4mm, 8mm, 12mm, 16mm)

 Points for partial insertion and fully seated

Threading nuts: 4 sizes of nuts (4mm, 8mm, 

12mm, 16mm)

 Points for threading and fully seated

Meshing gears: 3 sizes of gears with interlocking 

teeth

 Points for insertion and seated with meshed teeth

Attaching connectors: 4 orientationally specific 

electrical connectors (BNC, USB, RJ45, and 

DSUB)

 Points for partial insertion and full insertion
NIST ATB 2

Belt Drives

Alignment and insertion: 6 metal collars of various 

sizes

 Points for insertion

Insertion of Pulleys: 2 round pulleys, 2 sprockets, 2 

Timing pulleys

 Points for partial insertion and fully seated

Threading bolts: 6 bolts w/ washers (6mm)

 Points for threading and fully seated

Installation of belts: 1 round belt, 1 timing belt, 

and 1 chain

 Points for each correctly routed pulley

Actuating tensioners: 1 sliding tensioner, 1 elastic 

tensioner, and 1 screw tensioner

 Points for tension
NIST ATB 3

Cable Routing

• Acquiring/Handling loose parts: 1 thin flexible 

cable, 1 stiff thick cable, 1 flat cable (audio jack, 

ethernet, and a MIL connector flat cable.)

 Points for grasping and control

• Routing cables: 2 tall pegs with washer heads for 

each cable type (sized for each cable thickness)

 Points for correct routing direction and maintained 

rout through other tasks

• Weaving/Placing loose parts: 3 printed plastic 

tubes in different orientations for each cable

 Points for partial weave and fully weaved through 

each tube

• Connecting cable: 1 female connector for each 

cable

 Points for partial insertion and fully seated
Design Reasoning

• Test common assembly tasks and especially difficult assembly tasks that 

need solutions

• G. Boothroyd, P. Dewhurst and W. Knight in “Design for Manufacture and 

Assembly”

 Manual operation times are recorded for each subtask

 Compare human performance and robot system performance when handling parts
Competitions

• IROS 2020 virtual

 Repeat of 2019 competition with some Taskboard design changes. 

(Teams were given minimal time to program a task board that 

combined the competencies of NIST ATB 1-3.)

• IROS 2019

 Teams were given minimal time to program a task board that 

combined the competencies of NIST ATB 1-3.

• WRS 2018

 Teams competed to complete as much of a belt drive unit as possible. 

Belt drives of increasing difficulty were available. This utilized the 

elements presented in the belt drive task board. 

• IROS 2017

 Teams constructed a gear assembly used at future competitions. 

Teams also competed in the assembly and disassembly of NIST ATB 1. 

IROS competition board

WRS Belt Drive assembly 
Solutions

• Previous competitions were heavily weighted with manual programming 

solutions. This is not what these competitions are trying to promote.

• Use of expedient lead-through programming and application based user 

interfaces is a step in the right direction, but fully autonomous assembly is 

the future.

• Teams are starting to incorporate the use of force control, CAD data, AI and 

perception systems into their solutions. These systems with easy to use 

interfaces (no robot expertise necessary) is the future of adopting robotics 

into small and medium sized manufacturers.

• Teams have started to prove competency on competition taskboards with 

time to spare.
Feedback for Improvements

Limitations of the benchmark

• Threaded holes in the task boards are subject to wear over time that affects performance; when used in 

competitions or other testing, are new boards used?

• Cable routing board design may get augmented with channels to accommodate robotics,

• Looking to improve tolerances in the boards; possibly through using an aluminum board instead of 

plexiglass

Ideas for future benchmarks

• Develop assemblies that utilize a subset of the components of each task board and can be powered once 

it is finished being built, so if it is built correctly then it will operate

• Not just vertical insertions; insertions that require more complicated movements and/or orientations to 

be inserted

• Objects that can be deformed during insertion (e.g., blister); tasks that are more sensitive to force 

application

• Tying knots, manipulating flexible materials

• Two-hand manipulation where an object gets stabilized by one and assembled by the other
Closing

• By developing these benchmarks for robot system performance on assembly task boards NIST 

hopes to encourage progress in the field as well as to provide tools that assess a robot system’s 

assembly capabilities for choosing the best system for an intended application.

• Feedback on board designs, future task boards and/or better applications are welcome.

• Up and coming task board designs...

Wire harness

Vision 

challenge:

Object 

identification

Transparent objects

Negative spaces (not just holes)

Reflective surfaces

Similar shapes and sizes

Assembly that is removed 

when finished

Built from cable routing
Resources

Benchmarking Protocols for Evaluating Small Parts Robotic Assembly Systems Paper:

 https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8957300

NIST assembly performance metrics and test methods website:

 https://www.nist.gov/el/intelligent-systems-division-73500/robotic-grasping-and-manipulationassembly/assembly

NERVE center website for ordering boards:

 https://www.uml.edu/research/nerve/

To apply for NIST ATB #1 click the link at the bottom of the page

 https://www.uml.edu/Research/NERVE/NIST-Assembly-Task-Board-Form.aspx

Thank you