Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Measuring and Comparing Response Robot

Capabilities and Remote Operator Proficiency

Test Director:

Adam Jacoff

Intelligent Systems Division

National Institute of Standards and Technology

U.S. Department of Commerce

Sponsor:

Science and Technology Directorate 

U.S. Department of Homeland Security

Internet

RobotTestMethods.nist.gov

Adam	Jacoff

Intelligent	Systems	Division

NATIONAL	INSTITUTE	OF	STANDARDS	AND	TECHNOLOGY

U.S.	Department	of	Commerce

Email

RobotTestMethods@nist.gov

1

Version 2020B
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Collaborators

Tom Haus, Los Angeles Fire Dept. & CA-TF1, CA

Parry Boogard, Valley Regional Fire Authority & WA-TF1, WA

Clint Arnett, TEEX/Disaster City & TX-TF1, TX

George Hough, Fire Dept. of New York City & NY-TF1, NY

Jim Ingledue, Virginia Beach Fire Dept. & VA-TF2, VA

Mark Hundley, Virginia Beach Fire Dept. & VA-TF2, WA

Michael O’Shea, FAA UAS Integration Office (formerly U.S. DOJ)

Martin Hutchings, Sacramento Sheriff & IAB, CA

John Delaney, Arlington County Fire, Dept., & IAB, VA

Mike Marino, Prince George’s County Fire Dept. & IAB, MD

Coitt Kessler, Austin Fire Dept., TX

Chris Sadler, York County Fire Dept., VA

Andy Moore, Southwest Research Institute, San Antonio, TX

Al Frazier, Grand Forks County Sheriff’s Dept., ND

Ben Miller, CDPS COE for Aerial Technology Fire Fighting, CO

Mark Blanks, Virginia Tech University, VA

Daniele Nardi, Sapienza Universita di Roma, Italy

Max Delo, ESF-13, U.S. Marshals Service, DOJ

Bryan Gillespy, ESF-13, U.S. Marshals Service, DOJ

Gabriele Ferri, NATO CMRE, Italy

Howie Stockhowe, Virginia Beach Fire Dept, Virginia Beach, VA

Tony Galladora, Montgomery County Police, MD

Acknowledgements

Acknowledgements

This work was sponsored by Philip Mattson and Kai-Dee Chu from the 

Department of Homeland Security, Science and Technology Directorate, 

through an interagency agreement with the National Institute of 

Standards and Technology (NIST). 

The NIST Team includes: 

Adam Jacoff, Ann Virts, Raymond Sheh, Kamel Saidi, 

Kenny Kimble.

Dozens more people have contributed to the development and 

validation of these test methods. They include FEMA urban search and 

rescue task force teams, firefighters, law enforcement, collaborating 

test facilities, other civilian and military organizations, and commercial 

manufacturers. There are far too many to mention, but some of the 

ongoing (non-commercial) collaborators are listed below, roughly in 

order of their involvement:

Disclaimer

Commercial equipment shown in this document are for illustrative 

purposes only. This does not imply recommendation or endorsement by 

the National Institute of Standards and Technology, nor does it imply 

that the products identified are necessarily the best available for the 

purpose. 

Measurement Units

The International System of Units (a.k.a. SI Units) and U.S. Customary 

Units (a.k.a. Imperial Units) are used throughout this document. 

Approximate equivalents in each system of units enable use of readily 

available materials in different countries. This avoids excessive 

purchasing and fabrication costs. The differences between the stated 

unit dimensions are insignificant for comparison of test method results, 

so each set of units are considered standard for the purposes of these 

test methods. 

Satoshi Tadokoro, Tohoku University, Sendai, Japan

Tetsuya Kimura, Nagoaka Univ. of Technology, Nagoaka, Japan

Bob Gann, CDPS COE for Aerial Technology Fire Fighting, CO

Andy Olesen, Canadian Explosives Technicians Assoc., Canada

Tom Prentice, Reveille Peak Ranch, Burnet, TX

Michael Leo, Fire Department of New York City, NY

Luke Bergan, New South Whales Police Dept., Sydney, Australia

Katie Thielmeyer, Woodlawn Fire Dept. OH

Oliver Huke, RACE Test Facility, UKAEA, Oxfordshire, United Kingdom

2
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

POST

T-TOP

LEGS

POST

4x4x24in (10x10x60cm)

LEGS

2x4x20in (5x10x50cm)

with 45° opposing ends

T-TOP

2x4x12in (5x10x30cm)

BUCKETS

2 or 5 gal (7 or 19 liter)

Launch/Land

12in (30cm) Radius Center

(A,B,C,D)

10 ft (3 m) Sp

FLIGHT LINE

SIGHT LINE

Launch/Land

12in (30cm) Radius Center

FLIGHT LINE

SIGHT LINE

Bucket Stand #1

(A,B,C,D)

Bucket Stand #2 

(E,F,G,H)

Bucket Stand #3

(I,J,K,L)

10ft (3m) Spacings

10 FT (3 M)

ALTITUDE LAUNCH LAND

Test Director 

Adam Jacoff

Intelligent Systems Division

Nat’l Institute of Standards and Technology 

U.S. Dept. of Commerce 

RobotTestMethods@nist.gov | 301-975-4235 

Sponsor 

Philip Mattson

Science and Technology Directorate 

U.S. Dept. of Homeland Security

These test methods are primarily intended for 

vertical takeoff and landing systems with an 

onboard camera and remote pilot display. Some 

are also applicable to fixed wing systems when the 

lane dimensions are extended to accommodate 

the orbit radius of forward flying systems. 

Bucket stands on a level surfaces ensure the top bucket is 

vertical and the angled buckets are 45 degrees. 

Fl

kn

te

re

no 

de

T

re

p

p

gr

ai

in

TRAVERSE TEST

Right and Left Ovals Around All Three Bucket Stands

ALIGNED

WHEN RING IS 

COMPLETELY 

VISIBLE

Objectives:

Develop the measurements and standards infrastructure necessary to 

quantitatively evaluate robotic capabilities and remote operator proficiency.

Outcomes: 

Test methods, performance metrics, and data collection tools to facilitate 

integration of emerging technologies for hazardous and essential missions. 

Impacts: 

• Objective test methods help researchers and manufacturers push the state 

of the science by measuring progress and highlighting breakthroughs.

• Resulting quantitative performance data helps compare systems, specify 

purchases, and train with measures of remote operator/pilot proficiency.

GROUND

AQUATIC

AERIAL

3

Project Overview

Safety | Capabilities | Proficiency
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

POST

T-TOP

LEGS

POST

4x4x24in (10x10x60cm)

LEGS

2x4x20in (5x10x50cm)

with 45° opposing ends

T-TOP

2x4x12in (5x10x30cm)

BUCKETS

2 or 5 gal (7 or 19 liter)

Launch/Land

12in (30cm) Radius Center

(A,B,C,D)

10 ft (3 m) Sp

FLIGHT LINE

SIGHT LINE

Launch/Land

12in (30cm) Radius Center

FLIGHT LINE

SIGHT LINE

Bucket Stand #1

(A,B,C,D)

Bucket Stand #2 

(E,F,G,H)

Bucket Stand #3

(I,J,K,L)

10ft (3m) Spacings

10 FT (3 M)

ALTITUDE LAUNCH LAND

Test Director 

Adam Jacoff

Intelligent Systems Division

Nat’l Institute of Standards and Technology 

U.S. Dept. of Commerce 

RobotTestMethods@nist.gov | 301-975-4235 

Sponsor 

Philip Mattson

Science and Technology Directorate 

U.S. Dept. of Homeland Security

These test methods are primarily intended for 

vertical takeoff and landing systems with an 

onboard camera and remote pilot display. Some 

are also applicable to fixed wing systems when the 

lane dimensions are extended to accommodate 

the orbit radius of forward flying systems. 

Bucket stands on a level surfaces ensure the top bucket is 

vertical and the angled buckets are 45 degrees. 

Fl

kn

te

re

no 

de

T

re

p

p

gr

ai

in

TRAVERSE TEST

Right and Left Ovals Around All Three Bucket Stands

ALIGNED

WHEN RING IS 

COMPLETELY 

VISIBLE

• Develop reproducible test methods that are cheap and easy to conduct.

• Measure combinations of existing capabilities and emerging technologies.

• Inspire innovation using tests to communicate operational needs and gaps.

• Guide purchasing and deployment decisions with objective data.

• Focus training with repeatable tasks to measure and compare proficiency.

• Identify readiness issues with equipment and/or training through 

comparisons with local, regional, or national averages.

REPEAT

GROUND

AQUATIC

AERIAL

4

Project Approach

Safety | Capabilities | Proficiency
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Same Tests Help Different Users

Safety | Capabilities | Proficiency

Robot Developers

§ Understand missions through tangible test apparatuses

§ Practice and refine robot designs, make trade-off decisions

§ Highlight “Best-In-Class” capabilities

Responders, Soldiers, and Other Users

§ Compare robots with objective data, not marketing

§ Specify purchases based on existing combinations of capabilities

§ Align expectations with deployment decisions

Program Managers 

§ Describe objectives with a collection of tangible tasks

§ Challenge conventional approaches and inspire innovation

§ Measure baseline capabilities and document progress
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Comprehensive Suites of 50 Test Methods 

Ground, Aerial, and Aquatic Systems

Mobility

Dexterity

Endurance

Sensors

Radio Comms

Durability

Logistics

Safety

Autonomy

Proficiency

Aerial (netted) Underwater (tank)

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Comprehensive Suites of 50 Test Methods 

Ground, Aerial, and Aquatic Systems

ity

ity

ce

ors

ms

ity

cs

ety

for Small Unmanned Aircraft Systems

 Committee on Homeland Security Applications; 

9) | Website: RobotTestMethods.nist.gov

uites of 50 Test Methods 

l, and Aquatic Systems

Standard Test Lanes Scorable Scenarios

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Comprehensive Suites of 50 Test Methods 

Ground, Aerial, and Aquatic Systems

ity

ity

ce

rs

ms

ity

cs

ety

my

cy

120cm (48in) Scale 60cm (24in) Scale 30cm (12in) Scale Confined Access

Standard Test Methods for 

ASTM International Standards Com

Response Robots (E54.09) | 

Comprehensive Suit

Ground, Aerial, 

Mobility

Dexterity

Endurance

Sensors

Radio Comms

Durability

Logistics

Safety

Autonomy

Proficiency

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Comprehensive Suites of 50 Test Methods 

Ground, Aerial, and Aquatic Systems

Mobility

Dexterity

Endurance

Sensors

Radio Comms

Durability

Logistics

Safety

Autonomy

Proficiency

Repeatable Tasks Competitions

6

Comprehensive Suites of Standard Test Methods

Safety | Capabilities | Proficiency

Aerial (hangar)

ALTITUDE

SPACING

Aerial (outdoor)
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Aquatic

Underwater ROV Test Methods

Safety | Capabilities | Proficiency
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Aquatic

Underwater ROV Test Methods

Safety | Capabilities | Proficiency
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Underwater ROV Test Methods

Safety | Capabilities | Proficiency
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Test Methods for sUAS

Safety | Capabilities | Proficiency

Initial focus is VTOL, but some tests apply to forward flying aircraft when scaled up to the appropriate orbit radius.

Small Unmanned Aircraft Systems (sUAS)

10
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Repeatable Maneuvering and Payload Functionality Tests 

Safety | Capabilities | Proficiency

vely evaluate 

ncy to align 

e safety and 

ey save lives 

ots is to get 

at’s where 

ese test 

nd Payload 

cross the 

with 

ency. They are 

n measure 

eir proficiency 

to enable 

te 

M 

land Security 

). They are also 

s in 

dard for Small 

Safety 

ard Guide for 

manned 

F3266-18). 

e common 

ults. They can 

mbedded into 

and 

alitative 

set their own 

rformance to 

ission 

o make and 

formance data 

INSIDE EACH BUCKET

1A

EVALUATE SENSORS

SMALL SYSTEMS LARGE SYSTEMS

INTERFACES PROCEDURES

SENSORS MANEUVERING

MEASURE & COMPARE SCALABLE TEST LANES (ALTITUDE = SPACING) ALTITUDE

SPACING

OMNI BUCKETS

11
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

BLACK BUCKETS & COLOR RINGS 

EMBEDDED INTO SCENARIOS

1

TOP

FRONT 1A

LEFT 1B 1C BACK

1D RIGHT

ALL “A” BUCKETS

POINT TOWARD THE 

LAUNCH/LAND

HANDLE

CENTERLINE

ONLY THE NUMBERS 

CHANGE FOR EACH 

OMNI BUCKET STAND

Omni Bucket Stands

Open Test Lane

1A

BROKEN INNER BLACK EDGE

IS THE JUDGEMENT POINT

12

POST

LEGS 

FLUSH WITH TOP

SHOWN NOT ALIGNED SHOWN ALIGNED

WHITE BUCKETS & GREEN RINGS 

IN STANDARD TEST LANES
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

13

Scalable for Indoor/Outdoor Venues

Open Test Lane

MEASURING TAPE CENTERLINE 

UP TO 90 M (300 FT)

1

2

3

FLIGHT LINE

4

5(S)

S

S

S

z S

ALTITUDE S

ALTITUDE 2(S)

S

ALTITUDE 5(S)

LAUNCH/LAND 

PLATFORM

OMNI BUCKET STANDS

(A-B-C-D)

SPACING = S ALTITUDES = S, 2(S), 5(S) LENGTH = 10(S) 

S = 3 m (10 ft) 3, 6, 15 m (10, 20, 50 ft) 30 m (100 ft) 

S = 6 m (20 ft) 6, 12, 30 m (20, 40, 100 ft) 60 m (200 ft) 

S = 9 m (30 ft) 9, 18, 45 m (30, 60, 150 ft) 90 m (300 ft)

Spacing (S) = Altitude

DATE/TIME

PILOT CODE

48IN SQUARE LANDING WITH

12IN CENTER RADIUS TARGET

100-300 FT MEASURING TAPE

(4) OMNI STANDS

24 INCH 0R 48 INCH 

LAUNCH / LAND

PLATFORM

ACUITY TARGET

FOR ALIGNMENT

DIGITAL CLOCK CAPTURES

DATE | TIME (SYNCED)

WHITEBOARD CAPTURES

MAKE | MODEL | CONFIG

(4) OMNI BUCKET STANDS

12 INCH RADIUS

CIRCLE FOR ACCURACY

CENTERLINE 

MEASURING TAPE

DATE/TIME

PILOT CODE

48IN SQUARE LANDING WITH

12IN CENTER RADIUS TARGET

100-300 FT MEASURING TAPE

(4) OMNI STANDS

1–4

TOP

FRONT 1A

LEFT 1B 1C BACK

1D RIGHT “A” BUCKET

ALWAYS POINTING 

TOWARD THE 

LAUNCH/LAND

PLATFORM

HANDLE

CENTERLINE

LAUNCH / LAND PLATFORM

DIGITAL CLOCK

DATE & TIME

WHITEBOARD INFO

PILOT & AIRCRAFT

(4) OMNI 

BUCKET STANDS

CENTERLINE 

MEASURING TAPE
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Circuit Training and Credentialing

Open Test Lane

Position

MAN/PAY 1

• Hover stably

• Basic maneuvers

• Land accurately

• 20 Buckets in 1 lap

Standard Test Methods for Small Unmanned Aircraft Systems ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

14

Position 

(MAN 1 | PAY 1)

FLIGHT LINE

4

z

z

z

ALTITUDE S

START | END

3

2

1

5(S)

ALTITUDE 2(S)

S

S S S S

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

18

1

2

3

FLIGHT LINE

4

ALTITUDE S

START | END

5(S)

S

S S S S

Traverse 

(MAN 2 | PAY 2)

Traverse

MAN /PAY 2

• Fly sideways along a line

• Left and right directions

• Land accurately

• 20 Buckets in 2 laps

14
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

1

2

3

FLIGHT LINE

4

10FT 10FT 10FT 10FT

50FT

ALTITUDE 2(S)

START | END

ALTITUDE S

ALTITUDE 7(S)

ALTITUDE 5(S)

ALTITUDE 6(S)

7(S)

6(S)

5(S)

2(S)

Evaluate Various Flight Paths and Sensors

Open Test Lane

• 7(S) altitude

• 6(S) altitude 

• 5(S) altitude 

• 2(S) altitude 

• S altitude 

5 Different Orbits in Every Lane Spacing

(S) = 10ft, 20ft, 30ft, or other

15

12

1

2

3

4

z

z

FREE FLIGHT

z z

z

CW

CCW

CW

CCW

LAND

S

S S S S

FLIGHT LINE

1

2

3

FLIGHT LINE

4

START | END

S

S S S S

5(S)

ALTITUDE S

Standard Test Methods for Small Unmanned Aircraft Systems ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

18

1

2

3

FLIGHT LINE

4

ALTITUDE S

START | END

5(S)

S

S S S S

Traverse 

(MAN 2 | PAY 2)

1

2

3

FLIGHT LINE

4

z

z

ALTITUDE X

FLY STRAIGHT AND LEVEL OVER THE LANE CENTERLINE

START | END

ALIGN

NON-STOP

NON-STOP NON-STOP

NON-STOP

NON-STOP

NON-STOP

z

S

S S S S

5(S)

S

ALIGN CW

CCW

Position 

(MAN 1 | PAY 1)

FLIGHT LINE

4

z

z

z

ALTITUDE S

START | END 3

2

1

5(S)

ALTITUDE 2(S)

S

S S S S

1

2

3

4

z

z

FREE FLIGHT

z z

z

CW

CCW

CW

CCW

LAND

S

S S S S

FLIGHT LINE

Recon

Spiral 

Orbit 

Traverse

Position

DATE/TIME

PILOT CODE

48IN SQUARE LANDING WITH

12IN CENTER RADIUS TARGET

100-300 FT MEASURING TAPE

(4) OMNI STANDS

1–4

TOP

FRONT 1A

LEFT 1B 1C BACK

1D RIGHT “A” BUCKET

ALWAYS POINTING 

TOWARD THE 

LAUNCH/LAND

PLATFORM

HANDLE

CENTERLINE

LAUNCH / LAND PLATFORM

DIGITAL CLOCK

DATE & TIME

WHITEBOARD INFO

PILOT & AIRCRAFT

(4) OMNI 

BUCKET STANDS

CENTERLINE 

MEASURING TAPE
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Embedded Into Scenarios as Repeatable Scoring

Training and Evaluation

16

VEHICLE INSPECTION

(EXTERIOR AND INTERIOR)

DHS Responder Familiarization Exercise

ROOTOP 

FLUSH

OBSTACLE / SUNNY

RECESSED

OPEN / SUNNY

OPEN / SHADY

CONFINED

WIDE AREA SEARCH

(DOWNWARD OBJECTS)

BUILDING EXTERIOR SEARCH

(DOWNWARD OBJECTS)
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Embedded Into Scenarios as Repeatable Scoring

Training and Evaluation

17

Standard Disaster Response Robot Challenge and Plant Disaster Prevention Challenge, World Robot Summit, Japan

METAL BUCKETS 

BLEND INTO THE ENVIRONMENT
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Test Methods Scale to Measure All Robot Sizes

Safety | Capabilities | Proficiency

Cobham Telemax

80kg (175lbs)

ICOR Caliber MK3

84kg (185lbs)

Remotec HD-SEL

111kg (245lb)

iRobot 710 Kobra

166.5kg (367lbs)

ICOR Caliber T5

64kg (140lbs)

iRobot 110 FirstLook

2.4kg (5.2lbs)

Qinetiq Dragon Runner 10

4.5kg (10lbs)

ICOR Caliber Mini

27kg (65lbs)

iRobot 310 SUGV

13.2kg (29lbs)

Remotec Titus

61kg (135lbs)

Remotec Wolverine

367kg (810lbs)

Remotec Mark 5-A1

358kg (790lbs)

Remotec F6B

220kg (485lb)

WM Robotics Knight

249kg (550lbs)

Howe & Howe Thermite RS1 & RS3

550kg (1200lbs) 1200 Gallons per Minute
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

“Start Remote, Stay Remote?”

Safety | Capabilities | Proficiency
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Remotely controlled R/C helicopters 

(10 years ago). An expensive hobby!

Remote Control of all Degrees of Freedom

-- Hard to Remotely Operate --

Onboard Sensors and Coordinated Control 

-- Easy to Remotely Operate --

Multi-rotor systems that self level, hold 

station, and move in intuitive ways are 

much easier to fly. 

Intelligent Control Systems

Aerial Example

Aerial Example
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Rugged robot 

manipulators 

with 6 DOF 

coordinated 

control move in 

ways that are 

easy for remote 

operators to gain 

proficiency.

Industrial pedestal 

robotics (circa 

1980’s) meets 

mobility in rugged 

environments.

Intelligent Control Systems

Dexterity Example

Measuring Assistive Capabilities
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Centering Between Obstacles, Avoiding 

Obstacles, Wall Following, Exploration, 

Automatic Appendages to Improve Efficiency 

and Reliability in Terrains and Obstacles

Intelligent Control Systems

Mobility Example

Measuring Assistive Capabilities
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

We use embedded mapping fiducials, half round shapes, on BOTH sides of walls to measure Coverage, Consistency, Local 

Accuracy, and Global Accuracy. This enables evaluation of ladar/lidar scanned maps of interiors with complex flooring 

(could be a killer app for aerials).

Useless Map Broken Map

Consistent/Accurate Map

Low/High Fiducials

Localization and Mapping

Measuring Assistive Features
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

§ Apparatus: A repeatable, reproducible, and inexpensive 

representation of a task you expect the robot to perform.

§ Procedure: A script for the robot operator to follow.

§ Metric: A quantitative way to measure the performance.

Visual Acuity Targets Manipulator Dexterity Tasks Mobility Terrains and Obstacles

Standard Test Methods NOT Standard Robots

Safety | Capabilities | Proficiency
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Gravel

Sand

Crossing 

Ramps

Symmetric

Stepfields

Rubble Pile, Disaster City, College Station, TX

Scale: 120 cm (48 in) 

Scale: 240 cm (96 in)

Apparatuses Scale to Intended Environments

Safety | Capabilities | Proficiency
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Ground Robot Validation Exercises

30 Test Methods: 10 Basic Skills, 15 IED/PBIED/VBIED, 5 Building Access

120cm (4ft) Lateral Clearance Apparatuses

Safety | Capabilities | Proficiency
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

“Standard Test Methods in a Box”

Idea for ISO Container Facility

• ISOs have easy lights out testing, radio attenuated, all weather.

• Between containers can have higher arched canvas tarp, good for stairs, doors, taller dexterity test methods.

Parking Lot ISO Container Facilities (Rent or Buy)

Safety | Capabilities | Proficiency
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

“Standard Test Methods in a Box”

Idea for ISO Container Facility

Parking Lot ISO Container Facilities (Rent or Buy)

Safety | Capabilities | Proficiency
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Apparatuses Scale to Intended Environments

Increasing Complexity with Same Procedure and Metric

Scale: 60 cm (24 in) Lateral Clearance

Trains, Busses, Planes, Dwellings, Parked Cars, etc.

1.2 m (4 ft) Lateral Clearance

2.4 m (8 ft) Lateral Clearance

Four nested lanes contain 20 

test methods and fit into one 

ISO container.

60cm (24in) Lateral Clearance for Confined Environments

Safety | Capabilities | Proficiency
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

1.2 m (4 ft) Lateral Clearance

2.4 m (8 ft) Lateral Clearance

Confined Access, Throwable Robots, 3D Printed Robots (Disposable), STEM Kits

30cm (12in) Lateral Clearance for Confined Environments

Safety | Capabilities | Proficiency
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Rubble Pile, Disaster City, College Station, TX

1.2 m (4 ft) Lateral Clearance

2.4 m (8 ft) Lateral Clearance

Robots Scale By Size, Weight, or Access Hole

Safety | Capabilities | Proficiency
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Example of Quantitative Performance Comparisons

Robots Capabilities, Operator Proficiency, and/or Autonomous Behaviors

MOBILITY TEST METHODS

MOBILITY: Confined Area Terrains (figure-8 path, minimum 150 meters) 

 Average rate of advance comparison of robots on increasingly complex terrains: 

0

10

20

30

40

50

60

70

80

ReconScout FirstLook Armad-Base SandFlea DR10-Base Armad-Manip DR10-SCK Pointman

0 3 0 0 8 4

41

11 9 6 7

25

13 11

38

14 17 19 19

30

21

15

64

19 17 19

8 12 14 11

75

38

30

38

32 30 30 30

38 38

25

38 38

30

38

0

(meters per minute)

Sand Gravel Flat Line Following Continuous Ramps Crossing Ramps Stepfields 

ROBOT A ROBOT B ROBOT C ROBOT D ROBOT E ROBOT F ROBOT G ROBOT H

Robot Capabilities Data to Support Procurement/Deployment

Average Rate of Advance on Terrain for At Least 100m 

(meters/minute)

Compare Robot Capabilities or Proficiency

Safety | Capabilities | Proficiency
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Example of Quantitative Performance Comparisons

Robots Capabilities, Operator Proficiency, and/or Autonomous Behaviors

Compare Robot Capabilities or Proficiency

Safety | Capabilities | Proficiency

Sampling of Bomb Tech Readiness Nationwide

Sequence of 10 Basic Maneuvering and Dexterity Test Methods in 30 Minutes

Score

100+ Bomb Techs Using Their Operational Robots (USA & Canada)

Civilian Average

Best In Class Robot with 

Expert Operator (Orange)

Conventional Robot with 

Expert Operator (Orange

Military Average
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Simple Rules for Comparing Scores 

Focus Training and Evaluate Proficiency for Credentialing

• Compare scores in the same size lanes or embedded scenarios

Use ALL tests and scenarios that apply to the robot or to the intended missions.

• Compare scores with the same trial times

Trial time limits help normalize fatigue across several tests, so novices don’t get worn out unnecessarily. They 

provide enough time for an “expert” to perform a complete trial with 20 tasks. Longer times can be used for 

scenarios with embedded test apparatuses.

• Compare scores to “average” or “best-in-class” operators across organizations or regions

• ”Expert” operators designated by the manufacturer are used to capture the system’s best possible 

performance in each test. Those scores are considered the 100th percentile of remote operator proficiency 

for that system. Your score is some percentage of that “expert” score (1-100%).

• For training, use the entire time limit and track your scores. When they become repeatable, your learning 

phase is over. The best indicator of your proficiency in each test is the average of your last 5 trials. 

• We will post “average” and ”expert” scores and rates for each aircraft in each test as we collect them.

34
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Conducting Standard Test Methods

Safety | Capabilities | Proficiency

Conducting Standard Test Methods

Safety | Capabilities | Proficiency
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Set Your Minimum Thresholds for Pass/Fail

Focus Training and Evaluate Proficiency for Credentialing

• Organizations can set their own threshold for pass/fail in these tests based on their tolerance for reliability 

and/or efficiency. Complete trials are assumed.

• Measure everybody repeatedly over time and graph the results to help people understand their strengths and 

weaknesses. Then set minimum thresholds relative to the average or “expert” scores. Or adopt other 

organization’s thresholds as a central credentialing reference.

• At deployment time, each organization needs to consider their environmental variables, and mission 

complexity (night ops, BVLOS, etc.) to select a robot and operator that is likely to succeed.

Average Military 66%

Average Civilian 55%

Example proficiency data shown from bomb squads in ground robot tests 36
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Related Awards

Project Recognition

37

2020 Presidential Gears of Government Award

Recognizing people across the Federal workforce whose dedication supports exceptional delivery of key outcomes for the 

American people, specifically around mission results, customer service, and accountable stewardship. 

For developing the first ever comprehensive suite of emergency response robot test methods and data collection tools 

to evaluate and improve bomb-disposal robots and operators. These efforts led to enhanced testing and use of 

advanced robot capabilities that enable emergency responders to perform extremely hazardous missions from safer 

standoff distances. 

2019 Secretary Ron Brown Excellence in Innovation Award, U.S. Department of Commerce

The most prestigious singular honor given by the Department, chosen from among the Gold Medal Awards each year. 

2019 Gold Medal Award, U.S. Department of Commerce 

The highest award given by the Department for extraordinary contributions that impact key mission objectives.

Our same process originally applied to 

ground robots will work similarly for 

sUAS and remote pilots.
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Test Director:

Adam Jacoff

Intelligent Systems Division

National Institute of Standards and Technology

U.S. Department of Commerce

Sponsor:

Phil Mattson

Science and Technology Directorate 

U.S. Department of Homeland Security

Internet

RobotTestMethods.nist.gov

Adam	Jacoff

Intelligent	Systems	Division

NATIONAL	INSTITUTE	OF	STANDARDS	AND	TECHNOLOGY

U.S.	Department	of	Commerce

Email

RobotTestMethods@nist.gov

38