Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

ASTM E54.09 Homeland Security Applications; Response Robots

Committee Update

Sub Committee Chair:

Adam Jacoff

Intelligent Systems Division

National Institute of Standards and Technology

U.S. Department of Commerce

Committee Chair:

Phil Mattson

Science and Technology Directorate 

U.S. Department of Homeland Security

1

Version 2022A

STARTS AT 10:00 AM EST 

WASHINGTON, DC TIME

Internet

RobotTestMethods.nist.gov

Adam	Jacoff

Intelligent	Systems	Division

NATIONAL	INSTITUTE	OF	STANDARDS	AND	TECHNOLOGY

U.S.	Department	of	Commerce

Email

RobotTestMethods@nist.gov
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

2

• Reminder that electronic recording of ASTM meetings is prohibited.

• Meeting will run in accordance with the ASTM Antitrust Statement.

Antitrust Statement (also in meeting minutes)

ASTM International is a not-for-profit organization and developer of voluntary consensus standards. ASTM's leadership in international 

standards development is driven by the contributions of its members: more than 30,000 technical experts and business professionals 

representing 135 countries.

The purpose of antitrust laws is to preserve economic competition in the marketplace by prohibiting, among other things, unreasonable 

restraints of trade. In ASTM activities, it is important to recognize that participants often represent competitive interests. Antitrust laws 

require that all competition be open and unrestricted.

It is ASTM's policy, and the policy of each of its committees and subcommittees, to conduct all business and activity in full compliance with 

international, federal and state antitrust and competition laws. The ASTM Board of Directors has adopted an antitrust policy which is found 

in Section 19 of ASTM Regulations Governing Technical Committees. All members need to be aware of and compliant with this policy. The 

Regulations are accessible on the ASTM website (http://www.astm.org/COMMIT/Regs.pdf) and copies of the antitrust policy are available 

at the registration desk. For a complete list of standards, see: http://www.astm.org/COMMIT/SUBCOMMIT/E5409.htm

Call To Order

Committee Update
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

3

Ground Test Day Agenda

Committee Update

.

10:00 am EST Introduction, Committee Overview, Events

11:00 am EST Maneuvering and Mobility

12:00 pm EST Dexterity and C-IED/EOD Tasks

1:00 pm EST Mapping, Sensing and Radio Comms

2:00 pm EST Open Discussion 

POST

T-TOP

LEGS

POST

4x4x24in (10x10x60cm)

LEGS

2x4x20in (5x10x50cm)

with 45° opposing ends

T-TOP

2x4x12in (5x10x30cm)

BUCKETS

2 or 5 gal (7 or 19 liter)

Launch/Land

12in (30cm) Radius Center

Bucket Stand #1

(A,B,C,D)

10 ft (3 m) Spacings

FLIGHT LINE

SIGHT LINE

Launch/Land

12in (30cm) Radius Center

FLIGHT LINE

SIGHT LINE

Bucket Stand #1

(A,B,C,D)

Bucket Stand #2 

(E,F,G,H)

Bucket Stand #3

(I,J,K,L)

10ft (3m) Spacings

10 FT (3 M)

ALTITUDE LAUNCH LAND

Test Director 

Adam Jacoff

Intelligent Systems Division

Nat’l Institute of Standards and Technology 

U.S. Dept. of Commerce 

RobotTestMethods@nist.gov | 301-975-4235 

Sponsor 

Philip Mattson

Science and Technology Directorate 

U.S. Dept. of Homeland Security

These test methods are primarily intended for 

vertical takeoff and landing systems with an 

onboard camera and remote pilot display. Some 

are also applicable to fixed wing systems when the 

lane dimensions are extended to accommodate 

the orbit radius of forward flying systems. 

Bucket stands on a level surfaces ensure the top bucket is 

vertical and the angled buckets are 45 degrees. 

TRAVERSE TEST

Right and Left Ovals Around All Three Bucket Stands

ALIGNED

WHEN RING IS 

COMPLETELY 

VISIBLE

GROUND

AQUATIC

AERIALALTITUDE

SPACING

Page 4 11/9/21

The Open Test Lanes evaluates 5 different flight paths to identify objects from safe altitudes in open 

environments. These tests are scalable for all sizes of aircraft to demonstrate positive control at all times 

with accurate perches. They can be performed outdoors or indoors to control lighting and weather. The 

smallest size lane fits on an indoor basketball or tennis court for small drones and/or novice pilots to 

practice safely without flying in the national airspace.

Figure 4) The standard tests use well defined altitudes, positions, orientations, and perches to evaluate both 

Maneuvering and/or Payload Functionality.

Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Scalable Test Lanes for Various Sizes of Aircraft

Project Overview

procurement professionals, and users objectively evaluate 

system capabilities and remote pilot proficiency to align 

with mission requirements. Thisimproves the safety and 

effectiveness of emergency responders as they save lives 

and protect property in our communities

The first step toward credentialing remote pilots is to get 

everybody onto the same measuring stick. That’s where 

standard test methods can play a key role. These test 

methods for Basic Maneuvering (MAN 1-5) and Payload 

Functionality (PAY 1-5) are being replicated across the 

country and internationally to focus training with 

quantitative measures of remote pilot proficiency. They are 

low cost and easy to replicate so everyone can measure 

their own progress over time and compare their proficiency 

to regional or national averages on similar 

systems. Concurrent test lanes can be set up to enable 

multiple systems and pilots to train or evaluate 

simultaneously. 

They are being standardized through the ASTM 

International Standards Committee on Homeland Security 

Applications; Response Robots (ASTM E54.09). They are also 

referenced as Job Performance Requirements in 

the National Fire Protection Association Standard for Small 

Unmanned Aircraft Systems Used For Public Safety 

Operations (NFPA 2400) and the ASTM Standard Guide for 

Training for Remote Pilot in Command of Unmanned 

Aircraft Systems Endorsement (ASTM F38.03 F3266-18). 

These suites of standard test methods provide common 

measures of capabilities with quantitative results. They can 

be conducted individually, in sequences, or embedded into 

operational training scenarios as repeatable and 

reproducible tasks with scores to augment qualitative 

assessments. Organizations using these tests set their own 

thresholds of acceptable system and pilot performance to 

align with their airspace, environment, and mission 

complexities. But those decisions are easier to make and 

trust when they’re based on quantitative performance data 

captured within standard test methods.

INSIDE EACH BUCKET

EVALUATE SENSORS

SMALL SYSTEMS LARGE SYSTEMS

INTERFACES PROCEDURES

SENSORS MANEUVERING

MEASURE & COMPARE

1A

SCALABLE TEST LANES (ALTITUDE = SPACING) ALTITUDE

SPACING

OMNI BUCKETS

Page 4 

ASTM International Standards Committee on Homeland Security Appli

Response Robots (E54.09) | Website: RobotTestMethods.nist.go

• Wide area search of a plane crash site

• Vehicle identifications

• Hazardous tanker truck accident and fire

• Suspicious vehicle takedown

• Bus hostage situation

• Collapsed structure search

• Subway terrorist attack

• Flooded house rescue with delivery of personal floatation devices

• Flash flood bridge collapse with school bus and vehicle rescue

OPEN TEST LANES

The Open Test Lanes evaluates 5 different flight paths to identify objects from sa

environments. These tests are scalable for all sizes of aircraft to demonstrate pos

with accurate perches. They can be performed outdoors or indoors to control lig

smallest size lane fits on an indoor basketball or tennis court for small drones an

practice safely without flying in the national airspace.

Figure 4) The standard tests use well defined altitudes, positions, orientations, a

Maneuvering and/or Payload Functionality.

Standard Test Methods for Response Robots ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Scalable Test Lanes for Various Sizes o

Project Overview

Standard Test Methods for Small Unmanned Aircraft Systems ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

3

Introduction

Remotely operated aerial systems enable emergency 

responders to perform extremely hazardous tasks from 

safer stand-off distances. The U.S. National Institute of 

Standards and Technology is leading an international effort 

to develop standard test methods to help manufacturers, 

procurement professionals, and users objectively evaluate 

system capabilities and remote pilot proficiency to align 

with mission requirements. Thisimproves the safety and 

effectiveness of emergency responders as they save lives 

and protect property in our communities

The first step toward credentialing remote pilots is to get 

everybody onto the same measuring stick. That’s where 

standard test methods can play a key role. These test 

methods for Basic Maneuvering (MAN 1-5) and Payload 

Functionality (PAY 1-5) are being replicated across the 

country and internationally to focus training with 

quantitative measures of remote pilot proficiency. They are 

low cost and easy to replicate so everyone can measure 

their own progress over time and compare their proficiency 

to regional or national averages on similar 

systems. Concurrent test lanes can be set up to enable 

multiple systems and pilots to train or evaluate 

simultaneously. 

They are being standardized through the ASTM 

International Standards Committee on Homeland Security 

Applications; Response Robots (ASTM E54.09). They are also 

referenced as Job Performance Requirements in 

the National Fire Protection Association Standard for Small 

Unmanned Aircraft Systems Used For Public Safety 

Operations (NFPA 2400) and the ASTM Standard Guide for 

Training for Remote Pilot in Command of Unmanned 

Aircraft Systems Endorsement (ASTM F38.03 F3266-18). 

These suites of standard test methods provide common 

measures of capabilities with quantitative results. They can 

be conducted individually, in sequences, or embedded into 

operational training scenarios as repeatable and 

reproducible tasks with scores to augment qualitative 

assessments. Organizations using these tests set their own 

thresholds of acceptable system and pilot performance to 

align with their airspace, environment, and mission 

complexities. But those decisions are easier to make and 

trust when they’re based on quantitative performance data 

captured within standard test methods.

Basic Maneuvering (MAN 1-5) and Payload Functionality (PAY 1-5)

Safety | Capabilities | Proficiency

INSIDE EACH BUCKET

EVALUATE SENSORS

SMALL SYSTEMS LARGE SYSTEMS

INTERFACES PROCEDURES

SENSORS MANEUVERING

MEASURE & COMPARE

1A

SCALABLE TEST LANES (A

SPAC

OMNI BUCKETS

PROCEDURES

SENSORS MANEUVERING

INTERFACES

SMALL DRONES

MEASURE & COMPARE SCALABLE (ALTITUDE = SPACING)

Page 4 11/9/21

OPEN TEST LANES

The Open Test Lanes evaluates 5 different flight paths to identify objects from safe altitudes in open 

environments. These tests are scalable for all sizes of aircraft to demonstrate positive control at all times 

with accurate perches. They can be performed outdoors or indoors to control lighting and weather. The 

smallest size lane fits on an indoor basketball or tennis court for small drones and/or novice pilots to 

practice safely without flying in the national airspace.

Figure 4) The standard tests use well defined altitudes, positions, orientations, and perches to evaluate both 

Maneuvering and/or Payload Functionality.

Standard Test Methods for Response Robots ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Scalable Test Lanes for Various Sizes of Aircraft

Project Overview

Introduction

Remotely operated aerial systems enable emergency 

responders to perform extremely hazardous tasks from 

safer stand-off distances. The U.S. National Institute of 

Standards and Technology is leading an international effort 

to develop standard test methods to help manufacturers, 

procurement professionals, and users objectively evaluate 

system capabilities and remote pilot proficiency to align 

with mission requirements. Thisimproves the safety and 

effectiveness of emergency responders as they save lives 

and protect property in our communities

The first step toward credentialing remote pilots is to get 

everybody onto the same measuring stick. That’s where 

standard test methods can play a key role. These test 

methods for Basic Maneuvering (MAN 1-5) and Payload 

Functionality (PAY 1-5) are being replicated across the 

country and internationally to focus training with 

quantitative measures of remote pilot proficiency. They are 

low cost and easy to replicate so everyone can measure 

their own progress over time and compare their proficiency 

to regional or national averages on similar 

systems. Concurrent test lanes can be set up to enable 

multiple systems and pilots to train or evaluate 

simultaneously. 

They are being standardized through the ASTM 

International Standards Committee on Homeland Security 

Applications; Response Robots (ASTM E54.09). They are also 

referenced as Job Performance Requirements in 

the National Fire Protection Association Standard for Small 

Unmanned Aircraft Systems Used For Public Safety 

Operations (NFPA 2400) and the ASTM Standard Guide for 

Training for Remote Pilot in Command of Unmanned 

Aircraft Systems Endorsement (ASTM F38.03 F3266-18). 

These suites of standard test methods provide common 

measures of capabilities with quantitative results. They can 

be conducted individually, in sequences, or embedded into 

operational training scenarios as repeatable and 

reproducible tasks with scores to augment qualitative 

assessments. Organizations using these tests set their own 

thresholds of acceptable system and pilot performance to 

align with their airspace, environment, and mission 

complexities. But those decisions are easier to make and 

trust when they’re based on quantitative performance data 

captured within standard test methods.

INSIDE EACH BUCKET

EVALUATE SENSORS

SMALL SYSTEMS LARGE SYSTEMS

INTERFACES PROCEDURES

SENSORS MANEUVERING

MEASURE & COMPARE

1A

SCALABLE TEST LANES (ALTITUDE = SPACING) ALTITUDE

SPACING

OMNI BUCKETS

LARGE DRONES

5 F

OPEN TEST LANE AND RELATED OPERATIONAL SCENARIOS

FLIGHT LINE

z

z

ALTITUDE S 

S 

S

RECON

MAN/PAY 5

FLIGHT LINE

S 

ALTITUDE 

INSPECT

MAN/PAY 4

FLIGHT LINE

ALTITUDE S

S 

TRAVERSE

MAN/PAY 2

FLIGHT LINE

z

ALTITUDE S

AL

S 

POSITION

MAN/PAY 1

FLIGHT LINE

S 

ORBIT

MAN/PAY 3

AERIAL
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

4

Previous Meeting Minutes

Committee Update

ASTM E54.09 Aerial Test Methods Introduction (2021A)

ASTM E54.09 Aerial Test Methods Sensing and Radio Comms (2021A)

ASTM E54.09 Aerial Test Methods Use Case Examples (2021A)

ASTM E54.09 Aquatic Tests Introduction (v2021A)

ASTM E54.09 Ground Test Methods Introduction (2021A)

ASTM E54.09 Ground Test Methods Dexterity and Strength (2021A)

ASTM E54.09 Ground Test Methods Maneuvering and Mobility (2021A)

ASTM E54.09 Ground Test Methods Sensors and Radio Comms (2021A)

POST

T-TOP

LEGS

POST

4x4x24in (10x10x60cm)

LEGS

2x4x20in (5x10x50cm)

with 45° opposing ends

T-TOP

2x4x12in (5x10x30cm)

BUCKETS

2 or 5 gal (7 or 19 liter)

Launch/Land

12in (30cm) Radius Center

Bucket Stand #1

(A,B,C,D)

10 ft (3 m) Spacings

FLIGHT LINE

SIGHT LINE

Launch/Land

12in (30cm) Radius Center

FLIGHT LINE

SIGHT LINE

Bucket Stand #1

(A,B,C,D)

Bucket Stand #2 

(E,F,G,H)

Bucket Stand #3

(I,J,K,L)

10ft (3m) Spacings

10 FT (3 M)

ALTITUDE LAUNCH LAND

Test Director 

Adam Jacoff

Intelligent Systems Division

Nat’l Institute of Standards and Technology 

U.S. Dept. of Commerce 

RobotTestMethods@nist.gov | 301-975-4235 

Sponsor 

Philip Mattson

Science and Technology Directorate 

U.S. Dept. of Homeland Security

These test methods are primarily intended for 

vertical takeoff and landing systems with an 

onboard camera and remote pilot display. Some 

are also applicable to fixed wing systems when the 

lane dimensions are extended to accommodate 

the orbit radius of forward flying systems. 

Bucket stands on a level surfaces ensure the top bucket is 

vertical and the angled buckets are 45 degrees. 

TRAVERSE TEST

Right and Left Ovals Around All Three Bucket Stands

ALIGNED

WHEN RING IS 

COMPLETELY 

VISIBLE

GROUND

AQUATIC

AERIALALTITUDE

SPACING

Page 4 11/9/21

The Open Test Lanes evaluates 5 different flight paths to identify objects from safe altitudes in open 

environments. These tests are scalable for all sizes of aircraft to demonstrate positive control at all times 

with accurate perches. They can be performed outdoors or indoors to control lighting and weather. The 

smallest size lane fits on an indoor basketball or tennis court for small drones and/or novice pilots to 

practice safely without flying in the national airspace.

Figure 4) The standard tests use well defined altitudes, positions, orientations, and perches to evaluate both 

Maneuvering and/or Payload Functionality.

Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Scalable Test Lanes for Various Sizes of Aircraft

Project Overview

procurement professionals, and users objectively evaluate 

system capabilities and remote pilot proficiency to align 

with mission requirements. Thisimproves the safety and 

effectiveness of emergency responders as they save lives 

and protect property in our communities

The first step toward credentialing remote pilots is to get 

everybody onto the same measuring stick. That’s where 

standard test methods can play a key role. These test 

methods for Basic Maneuvering (MAN 1-5) and Payload 

Functionality (PAY 1-5) are being replicated across the 

country and internationally to focus training with 

quantitative measures of remote pilot proficiency. They are 

low cost and easy to replicate so everyone can measure 

their own progress over time and compare their proficiency 

to regional or national averages on similar 

systems. Concurrent test lanes can be set up to enable 

multiple systems and pilots to train or evaluate 

simultaneously. 

They are being standardized through the ASTM 

International Standards Committee on Homeland Security 

Applications; Response Robots (ASTM E54.09). They are also 

referenced as Job Performance Requirements in 

the National Fire Protection Association Standard for Small 

Unmanned Aircraft Systems Used For Public Safety 

Operations (NFPA 2400) and the ASTM Standard Guide for 

Training for Remote Pilot in Command of Unmanned 

Aircraft Systems Endorsement (ASTM F38.03 F3266-18). 

These suites of standard test methods provide common 

measures of capabilities with quantitative results. They can 

be conducted individually, in sequences, or embedded into 

operational training scenarios as repeatable and 

reproducible tasks with scores to augment qualitative 

assessments. Organizations using these tests set their own 

thresholds of acceptable system and pilot performance to 

align with their airspace, environment, and mission 

complexities. But those decisions are easier to make and 

trust when they’re based on quantitative performance data 

captured within standard test methods.

INSIDE EACH BUCKET

EVALUATE SENSORS

SMALL SYSTEMS LARGE SYSTEMS

INTERFACES PROCEDURES

SENSORS MANEUVERING

MEASURE & COMPARE

1A

SCALABLE TEST LANES (ALTITUDE = SPACING) ALTITUDE

SPACING

OMNI BUCKETS

Page 4 

ASTM International Standards Committee on Homeland Security Appli

Response Robots (E54.09) | Website: RobotTestMethods.nist.go

• Wide area search of a plane crash site

• Vehicle identifications

• Hazardous tanker truck accident and fire

• Suspicious vehicle takedown

• Bus hostage situation

• Collapsed structure search

• Subway terrorist attack

• Flooded house rescue with delivery of personal floatation devices

• Flash flood bridge collapse with school bus and vehicle rescue

OPEN TEST LANES

The Open Test Lanes evaluates 5 different flight paths to identify objects from sa

environments. These tests are scalable for all sizes of aircraft to demonstrate pos

with accurate perches. They can be performed outdoors or indoors to control lig

smallest size lane fits on an indoor basketball or tennis court for small drones an

practice safely without flying in the national airspace.

Figure 4) The standard tests use well defined altitudes, positions, orientations, a

Maneuvering and/or Payload Functionality.

Standard Test Methods for Response Robots ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Scalable Test Lanes for Various Sizes o

Project Overview

Standard Test Methods for Small Unmanned Aircraft Systems ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

3

Introduction

Remotely operated aerial systems enable emergency 

responders to perform extremely hazardous tasks from 

safer stand-off distances. The U.S. National Institute of 

Standards and Technology is leading an international effort 

to develop standard test methods to help manufacturers, 

procurement professionals, and users objectively evaluate 

system capabilities and remote pilot proficiency to align 

with mission requirements. Thisimproves the safety and 

effectiveness of emergency responders as they save lives 

and protect property in our communities

The first step toward credentialing remote pilots is to get 

everybody onto the same measuring stick. That’s where 

standard test methods can play a key role. These test 

methods for Basic Maneuvering (MAN 1-5) and Payload 

Functionality (PAY 1-5) are being replicated across the 

country and internationally to focus training with 

quantitative measures of remote pilot proficiency. They are 

low cost and easy to replicate so everyone can measure 

their own progress over time and compare their proficiency 

to regional or national averages on similar 

systems. Concurrent test lanes can be set up to enable 

multiple systems and pilots to train or evaluate 

simultaneously. 

They are being standardized through the ASTM 

International Standards Committee on Homeland Security 

Applications; Response Robots (ASTM E54.09). They are also 

referenced as Job Performance Requirements in 

the National Fire Protection Association Standard for Small 

Unmanned Aircraft Systems Used For Public Safety 

Operations (NFPA 2400) and the ASTM Standard Guide for 

Training for Remote Pilot in Command of Unmanned 

Aircraft Systems Endorsement (ASTM F38.03 F3266-18). 

These suites of standard test methods provide common 

measures of capabilities with quantitative results. They can 

be conducted individually, in sequences, or embedded into 

operational training scenarios as repeatable and 

reproducible tasks with scores to augment qualitative 

assessments. Organizations using these tests set their own 

thresholds of acceptable system and pilot performance to 

align with their airspace, environment, and mission 

complexities. But those decisions are easier to make and 

trust when they’re based on quantitative performance data 

captured within standard test methods.

Basic Maneuvering (MAN 1-5) and Payload Functionality (PAY 1-5)

Safety | Capabilities | Proficiency

INSIDE EACH BUCKET

EVALUATE SENSORS

SMALL SYSTEMS LARGE SYSTEMS

INTERFACES PROCEDURES

SENSORS MANEUVERING

MEASURE & COMPARE

1A

SCALABLE TEST LANES (A

SPAC

OMNI BUCKETS

PROCEDURES

SENSORS MANEUVERING

INTERFACES

SMALL DRONES

MEASURE & COMPARE SCALABLE (ALTITUDE = SPACING)

Page 4 11/9/21

OPEN TEST LANES

The Open Test Lanes evaluates 5 different flight paths to identify objects from safe altitudes in open 

environments. These tests are scalable for all sizes of aircraft to demonstrate positive control at all times 

with accurate perches. They can be performed outdoors or indoors to control lighting and weather. The 

smallest size lane fits on an indoor basketball or tennis court for small drones and/or novice pilots to 

practice safely without flying in the national airspace.

Figure 4) The standard tests use well defined altitudes, positions, orientations, and perches to evaluate both 

Maneuvering and/or Payload Functionality.

Standard Test Methods for Response Robots ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Scalable Test Lanes for Various Sizes of Aircraft

Project Overview

Introduction

Remotely operated aerial systems enable emergency 

responders to perform extremely hazardous tasks from 

safer stand-off distances. The U.S. National Institute of 

Standards and Technology is leading an international effort 

to develop standard test methods to help manufacturers, 

procurement professionals, and users objectively evaluate 

system capabilities and remote pilot proficiency to align 

with mission requirements. Thisimproves the safety and 

effectiveness of emergency responders as they save lives 

and protect property in our communities

The first step toward credentialing remote pilots is to get 

everybody onto the same measuring stick. That’s where 

standard test methods can play a key role. These test 

methods for Basic Maneuvering (MAN 1-5) and Payload 

Functionality (PAY 1-5) are being replicated across the 

country and internationally to focus training with 

quantitative measures of remote pilot proficiency. They are 

low cost and easy to replicate so everyone can measure 

their own progress over time and compare their proficiency 

to regional or national averages on similar 

systems. Concurrent test lanes can be set up to enable 

multiple systems and pilots to train or evaluate 

simultaneously. 

They are being standardized through the ASTM 

International Standards Committee on Homeland Security 

Applications; Response Robots (ASTM E54.09). They are also 

referenced as Job Performance Requirements in 

the National Fire Protection Association Standard for Small 

Unmanned Aircraft Systems Used For Public Safety 

Operations (NFPA 2400) and the ASTM Standard Guide for 

Training for Remote Pilot in Command of Unmanned 

Aircraft Systems Endorsement (ASTM F38.03 F3266-18). 

These suites of standard test methods provide common 

measures of capabilities with quantitative results. They can 

be conducted individually, in sequences, or embedded into 

operational training scenarios as repeatable and 

reproducible tasks with scores to augment qualitative 

assessments. Organizations using these tests set their own 

thresholds of acceptable system and pilot performance to 

align with their airspace, environment, and mission 

complexities. But those decisions are easier to make and 

trust when they’re based on quantitative performance data 

captured within standard test methods.

INSIDE EACH BUCKET

EVALUATE SENSORS

SMALL SYSTEMS LARGE SYSTEMS

INTERFACES PROCEDURES

SENSORS MANEUVERING

MEASURE & COMPARE

1A

SCALABLE TEST LANES (ALTITUDE = SPACING) ALTITUDE

SPACING

OMNI BUCKETS

LARGE DRONES

5 F

OPEN TEST LANE AND RELATED OPERATIONAL SCENARIOS

FLIGHT LINE

z

z

ALTITUDE S 

S 

S

RECON

MAN/PAY 5

FLIGHT LINE

S 

ALTITUDE 

INSPECT

MAN/PAY 4

FLIGHT LINE

ALTITUDE S

S 

TRAVERSE

MAN/PAY 2

FLIGHT LINE

z

ALTITUDE S

AL

S 

POSITION

MAN/PAY 1

FLIGHT LINE

S 

ORBIT

MAN/PAY 3

AERIAL

Internet

RobotTestMethods.nist.gov

Adam	Jacoff

Intelligent	Systems	Division

NATIONAL	INSTITUTE	OF	STANDARDS	AND	TECHNOLOGY

U.S.	Department	of	Commerce

Email

RobotTestMethods@nist.gov
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

5

Membership

Committee Update

POST

T-TOP

LEGS

POST

4x4x24in (10x10x60cm)

LEGS

2x4x20in (5x10x50cm)

with 45° opposing ends

T-TOP

2x4x12in (5x10x30cm)

BUCKETS

2 or 5 gal (7 or 19 liter)

Launch/Land

12in (30cm) Radius Center

Bucket Stand #1

(A,B,C,D)

10 ft (3 m) Spacings

FLIGHT LINE

SIGHT LINE

Launch/Land

12in (30cm) Radius Center

FLIGHT LINE

SIGHT LINE

Bucket Stand #1

(A,B,C,D)

Bucket Stand #2 

(E,F,G,H)

Bucket Stand #3

(I,J,K,L)

10ft (3m) Spacings

10 FT (3 M)

ALTITUDE LAUNCH LAND

Test Director 

Adam Jacoff

Intelligent Systems Division

Nat’l Institute of Standards and Technology 

U.S. Dept. of Commerce 

RobotTestMethods@nist.gov | 301-975-4235 

Sponsor 

Philip Mattson

Science and Technology Directorate 

U.S. Dept. of Homeland Security

These test methods are primarily intended for 

vertical takeoff and landing systems with an 

onboard camera and remote pilot display. Some 

are also applicable to fixed wing systems when the 

lane dimensions are extended to accommodate 

the orbit radius of forward flying systems. 

Bucket stands on a level surfaces ensure the top bucket is 

vertical and the angled buckets are 45 degrees. 

TRAVERSE TEST

Right and Left Ovals Around All Three Bucket Stands

ALIGNED

WHEN RING IS 

COMPLETELY 

VISIBLE

GROUND

AQUATIC

AERIALALTITUDE

SPACING

Page 4 11/9/21

The Open Test Lanes evaluates 5 different flight paths to identify objects from safe altitudes in open 

environments. These tests are scalable for all sizes of aircraft to demonstrate positive control at all times 

with accurate perches. They can be performed outdoors or indoors to control lighting and weather. The 

smallest size lane fits on an indoor basketball or tennis court for small drones and/or novice pilots to 

practice safely without flying in the national airspace.

Figure 4) The standard tests use well defined altitudes, positions, orientations, and perches to evaluate both 

Maneuvering and/or Payload Functionality.

Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Scalable Test Lanes for Various Sizes of Aircraft

Project Overview

procurement professionals, and users objectively evaluate 

system capabilities and remote pilot proficiency to align 

with mission requirements. Thisimproves the safety and 

effectiveness of emergency responders as they save lives 

and protect property in our communities

The first step toward credentialing remote pilots is to get 

everybody onto the same measuring stick. That’s where 

standard test methods can play a key role. These test 

methods for Basic Maneuvering (MAN 1-5) and Payload 

Functionality (PAY 1-5) are being replicated across the 

country and internationally to focus training with 

quantitative measures of remote pilot proficiency. They are 

low cost and easy to replicate so everyone can measure 

their own progress over time and compare their proficiency 

to regional or national averages on similar 

systems. Concurrent test lanes can be set up to enable 

multiple systems and pilots to train or evaluate 

simultaneously. 

They are being standardized through the ASTM 

International Standards Committee on Homeland Security 

Applications; Response Robots (ASTM E54.09). They are also 

referenced as Job Performance Requirements in 

the National Fire Protection Association Standard for Small 

Unmanned Aircraft Systems Used For Public Safety 

Operations (NFPA 2400) and the ASTM Standard Guide for 

Training for Remote Pilot in Command of Unmanned 

Aircraft Systems Endorsement (ASTM F38.03 F3266-18). 

These suites of standard test methods provide common 

measures of capabilities with quantitative results. They can 

be conducted individually, in sequences, or embedded into 

operational training scenarios as repeatable and 

reproducible tasks with scores to augment qualitative 

assessments. Organizations using these tests set their own 

thresholds of acceptable system and pilot performance to 

align with their airspace, environment, and mission 

complexities. But those decisions are easier to make and 

trust when they’re based on quantitative performance data 

captured within standard test methods.

INSIDE EACH BUCKET

EVALUATE SENSORS

SMALL SYSTEMS LARGE SYSTEMS

INTERFACES PROCEDURES

SENSORS MANEUVERING

MEASURE & COMPARE

1A

SCALABLE TEST LANES (ALTITUDE = SPACING) ALTITUDE

SPACING

OMNI BUCKETS

Page 4 

ASTM International Standards Committee on Homeland Security Appli

Response Robots (E54.09) | Website: RobotTestMethods.nist.go

• Wide area search of a plane crash site

• Vehicle identifications

• Hazardous tanker truck accident and fire

• Suspicious vehicle takedown

• Bus hostage situation

• Collapsed structure search

• Subway terrorist attack

• Flooded house rescue with delivery of personal floatation devices

• Flash flood bridge collapse with school bus and vehicle rescue

OPEN TEST LANES

The Open Test Lanes evaluates 5 different flight paths to identify objects from sa

environments. These tests are scalable for all sizes of aircraft to demonstrate pos

with accurate perches. They can be performed outdoors or indoors to control lig

smallest size lane fits on an indoor basketball or tennis court for small drones an

practice safely without flying in the national airspace.

Figure 4) The standard tests use well defined altitudes, positions, orientations, a

Maneuvering and/or Payload Functionality.

Standard Test Methods for Response Robots ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Scalable Test Lanes for Various Sizes o

Project Overview

Standard Test Methods for Small Unmanned Aircraft Systems ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

3

Introduction

Remotely operated aerial systems enable emergency 

responders to perform extremely hazardous tasks from 

safer stand-off distances. The U.S. National Institute of 

Standards and Technology is leading an international effort 

to develop standard test methods to help manufacturers, 

procurement professionals, and users objectively evaluate 

system capabilities and remote pilot proficiency to align 

with mission requirements. Thisimproves the safety and 

effectiveness of emergency responders as they save lives 

and protect property in our communities

The first step toward credentialing remote pilots is to get 

everybody onto the same measuring stick. That’s where 

standard test methods can play a key role. These test 

methods for Basic Maneuvering (MAN 1-5) and Payload 

Functionality (PAY 1-5) are being replicated across the 

country and internationally to focus training with 

quantitative measures of remote pilot proficiency. They are 

low cost and easy to replicate so everyone can measure 

their own progress over time and compare their proficiency 

to regional or national averages on similar 

systems. Concurrent test lanes can be set up to enable 

multiple systems and pilots to train or evaluate 

simultaneously. 

They are being standardized through the ASTM 

International Standards Committee on Homeland Security 

Applications; Response Robots (ASTM E54.09). They are also 

referenced as Job Performance Requirements in 

the National Fire Protection Association Standard for Small 

Unmanned Aircraft Systems Used For Public Safety 

Operations (NFPA 2400) and the ASTM Standard Guide for 

Training for Remote Pilot in Command of Unmanned 

Aircraft Systems Endorsement (ASTM F38.03 F3266-18). 

These suites of standard test methods provide common 

measures of capabilities with quantitative results. They can 

be conducted individually, in sequences, or embedded into 

operational training scenarios as repeatable and 

reproducible tasks with scores to augment qualitative 

assessments. Organizations using these tests set their own 

thresholds of acceptable system and pilot performance to 

align with their airspace, environment, and mission 

complexities. But those decisions are easier to make and 

trust when they’re based on quantitative performance data 

captured within standard test methods.

Basic Maneuvering (MAN 1-5) and Payload Functionality (PAY 1-5)

Safety | Capabilities | Proficiency

INSIDE EACH BUCKET

EVALUATE SENSORS

SMALL SYSTEMS LARGE SYSTEMS

INTERFACES PROCEDURES

SENSORS MANEUVERING

MEASURE & COMPARE

1A

SCALABLE TEST LANES (A

SPAC

OMNI BUCKETS

PROCEDURES

SENSORS MANEUVERING

INTERFACES

SMALL DRONES

MEASURE & COMPARE SCALABLE (ALTITUDE = SPACING)

Page 4 11/9/21

OPEN TEST LANES

The Open Test Lanes evaluates 5 different flight paths to identify objects from safe altitudes in open 

environments. These tests are scalable for all sizes of aircraft to demonstrate positive control at all times 

with accurate perches. They can be performed outdoors or indoors to control lighting and weather. The 

smallest size lane fits on an indoor basketball or tennis court for small drones and/or novice pilots to 

practice safely without flying in the national airspace.

Figure 4) The standard tests use well defined altitudes, positions, orientations, and perches to evaluate both 

Maneuvering and/or Payload Functionality.

Standard Test Methods for Response Robots ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Scalable Test Lanes for Various Sizes of Aircraft

Project Overview

Introduction

Remotely operated aerial systems enable emergency 

responders to perform extremely hazardous tasks from 

safer stand-off distances. The U.S. National Institute of 

Standards and Technology is leading an international effort 

to develop standard test methods to help manufacturers, 

procurement professionals, and users objectively evaluate 

system capabilities and remote pilot proficiency to align 

with mission requirements. Thisimproves the safety and 

effectiveness of emergency responders as they save lives 

and protect property in our communities

The first step toward credentialing remote pilots is to get 

everybody onto the same measuring stick. That’s where 

standard test methods can play a key role. These test 

methods for Basic Maneuvering (MAN 1-5) and Payload 

Functionality (PAY 1-5) are being replicated across the 

country and internationally to focus training with 

quantitative measures of remote pilot proficiency. They are 

low cost and easy to replicate so everyone can measure 

their own progress over time and compare their proficiency 

to regional or national averages on similar 

systems. Concurrent test lanes can be set up to enable 

multiple systems and pilots to train or evaluate 

simultaneously. 

They are being standardized through the ASTM 

International Standards Committee on Homeland Security 

Applications; Response Robots (ASTM E54.09). They are also 

referenced as Job Performance Requirements in 

the National Fire Protection Association Standard for Small 

Unmanned Aircraft Systems Used For Public Safety 

Operations (NFPA 2400) and the ASTM Standard Guide for 

Training for Remote Pilot in Command of Unmanned 

Aircraft Systems Endorsement (ASTM F38.03 F3266-18). 

These suites of standard test methods provide common 

measures of capabilities with quantitative results. They can 

be conducted individually, in sequences, or embedded into 

operational training scenarios as repeatable and 

reproducible tasks with scores to augment qualitative 

assessments. Organizations using these tests set their own 

thresholds of acceptable system and pilot performance to 

align with their airspace, environment, and mission 

complexities. But those decisions are easier to make and 

trust when they’re based on quantitative performance data 

captured within standard test methods.

INSIDE EACH BUCKET

EVALUATE SENSORS

SMALL SYSTEMS LARGE SYSTEMS

INTERFACES PROCEDURES

SENSORS MANEUVERING

MEASURE & COMPARE

1A

SCALABLE TEST LANES (ALTITUDE = SPACING) ALTITUDE

SPACING

OMNI BUCKETS

LARGE DRONES

5 F

OPEN TEST LANE AND RELATED OPERATIONAL SCENARIOS

FLIGHT LINE

z

z

ALTITUDE S 

S 

S

RECON

MAN/PAY 5

FLIGHT LINE

S 

ALTITUDE 

INSPECT

MAN/PAY 4

FLIGHT LINE

ALTITUDE S

S 

TRAVERSE

MAN/PAY 2

FLIGHT LINE

z

ALTITUDE S

AL

S 

POSITION

MAN/PAY 1

FLIGHT LINE

S 

ORBIT

MAN/PAY 3

AERIAL

46 MEMBERS
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

6

Recent Ballot Activity

Committee Update

Still Being Balloted:

E2853-2021 Human-System Interaction: Search Tasks: Random Mazes 

with Complex Terrain

Successfully Balloted:

E3310-2021 Mobility: Parallel Rail Obstacles

E2802 2021 Mobility: Variable Hurdle Obstacles

E3311-2021 Mobility: Variable Diagonal Rail Obstacles

E2854-2021 Radio Comms: Line-of-Sight Range 

E2855-2021 Radio Comms: Non-Line-of-Sight Range 

POST

T-TOP

LEGS

POST

4x4x24in (10x10x60cm)

LEGS

2x4x20in (5x10x50cm)

with 45° opposing ends

T-TOP

2x4x12in (5x10x30cm)

BUCKETS

2 or 5 gal (7 or 19 liter)

Launch/Land

12in (30cm) Radius Center

Bucket Stand #1

(A,B,C,D)

10 ft (3 m) Spacings

FLIGHT LINE

SIGHT LINE

Launch/Land

12in (30cm) Radius Center

FLIGHT LINE

SIGHT LINE

Bucket Stand #1

(A,B,C,D)

Bucket Stand #2 

(E,F,G,H)

Bucket Stand #3

(I,J,K,L)

10ft (3m) Spacings

10 FT (3 M)

ALTITUDE LAUNCH LAND

Test Director 

Adam Jacoff

Intelligent Systems Division

Nat’l Institute of Standards and Technology 

U.S. Dept. of Commerce 

RobotTestMethods@nist.gov | 301-975-4235 

Sponsor 

Philip Mattson

Science and Technology Directorate 

U.S. Dept. of Homeland Security

These test methods are primarily intended for 

vertical takeoff and landing systems with an 

onboard camera and remote pilot display. Some 

are also applicable to fixed wing systems when the 

lane dimensions are extended to accommodate 

the orbit radius of forward flying systems. 

Bucket stands on a level surfaces ensure the top bucket is 

vertical and the angled buckets are 45 degrees. 

TRAVERSE TEST

Right and Left Ovals Around All Three Bucket Stands

ALIGNED

WHEN RING IS 

COMPLETELY 

VISIBLE

GROUND

AQUATIC

AERIALALTITUDE

SPACING

Page 4 11/9/21

The Open Test Lanes evaluates 5 different flight paths to identify objects from safe altitudes in open 

environments. These tests are scalable for all sizes of aircraft to demonstrate positive control at all times 

with accurate perches. They can be performed outdoors or indoors to control lighting and weather. The 

smallest size lane fits on an indoor basketball or tennis court for small drones and/or novice pilots to 

practice safely without flying in the national airspace.

Figure 4) The standard tests use well defined altitudes, positions, orientations, and perches to evaluate both 

Maneuvering and/or Payload Functionality.

Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Scalable Test Lanes for Various Sizes of Aircraft

Project Overview

procurement professionals, and users objectively evaluate 

system capabilities and remote pilot proficiency to align 

with mission requirements. Thisimproves the safety and 

effectiveness of emergency responders as they save lives 

and protect property in our communities

The first step toward credentialing remote pilots is to get 

everybody onto the same measuring stick. That’s where 

standard test methods can play a key role. These test 

methods for Basic Maneuvering (MAN 1-5) and Payload 

Functionality (PAY 1-5) are being replicated across the 

country and internationally to focus training with 

quantitative measures of remote pilot proficiency. They are 

low cost and easy to replicate so everyone can measure 

their own progress over time and compare their proficiency 

to regional or national averages on similar 

systems. Concurrent test lanes can be set up to enable 

multiple systems and pilots to train or evaluate 

simultaneously. 

They are being standardized through the ASTM 

International Standards Committee on Homeland Security 

Applications; Response Robots (ASTM E54.09). They are also 

referenced as Job Performance Requirements in 

the National Fire Protection Association Standard for Small 

Unmanned Aircraft Systems Used For Public Safety 

Operations (NFPA 2400) and the ASTM Standard Guide for 

Training for Remote Pilot in Command of Unmanned 

Aircraft Systems Endorsement (ASTM F38.03 F3266-18). 

These suites of standard test methods provide common 

measures of capabilities with quantitative results. They can 

be conducted individually, in sequences, or embedded into 

operational training scenarios as repeatable and 

reproducible tasks with scores to augment qualitative 

assessments. Organizations using these tests set their own 

thresholds of acceptable system and pilot performance to 

align with their airspace, environment, and mission 

complexities. But those decisions are easier to make and 

trust when they’re based on quantitative performance data 

captured within standard test methods.

INSIDE EACH BUCKET

EVALUATE SENSORS

SMALL SYSTEMS LARGE SYSTEMS

INTERFACES PROCEDURES

SENSORS MANEUVERING

MEASURE & COMPARE

1A

SCALABLE TEST LANES (ALTITUDE = SPACING) ALTITUDE

SPACING

OMNI BUCKETS

Page 4 

ASTM International Standards Committee on Homeland Security Appli

Response Robots (E54.09) | Website: RobotTestMethods.nist.go

• Wide area search of a plane crash site

• Vehicle identifications

• Hazardous tanker truck accident and fire

• Suspicious vehicle takedown

• Bus hostage situation

• Collapsed structure search

• Subway terrorist attack

• Flooded house rescue with delivery of personal floatation devices

• Flash flood bridge collapse with school bus and vehicle rescue

OPEN TEST LANES

The Open Test Lanes evaluates 5 different flight paths to identify objects from sa

environments. These tests are scalable for all sizes of aircraft to demonstrate pos

with accurate perches. They can be performed outdoors or indoors to control lig

smallest size lane fits on an indoor basketball or tennis court for small drones an

practice safely without flying in the national airspace.

Figure 4) The standard tests use well defined altitudes, positions, orientations, a

Maneuvering and/or Payload Functionality.

Standard Test Methods for Response Robots ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Scalable Test Lanes for Various Sizes o

Project Overview

Standard Test Methods for Small Unmanned Aircraft Systems ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

3

Introduction

Remotely operated aerial systems enable emergency 

responders to perform extremely hazardous tasks from 

safer stand-off distances. The U.S. National Institute of 

Standards and Technology is leading an international effort 

to develop standard test methods to help manufacturers, 

procurement professionals, and users objectively evaluate 

system capabilities and remote pilot proficiency to align 

with mission requirements. Thisimproves the safety and 

effectiveness of emergency responders as they save lives 

and protect property in our communities

The first step toward credentialing remote pilots is to get 

everybody onto the same measuring stick. That’s where 

standard test methods can play a key role. These test 

methods for Basic Maneuvering (MAN 1-5) and Payload 

Functionality (PAY 1-5) are being replicated across the 

country and internationally to focus training with 

quantitative measures of remote pilot proficiency. They are 

low cost and easy to replicate so everyone can measure 

their own progress over time and compare their proficiency 

to regional or national averages on similar 

systems. Concurrent test lanes can be set up to enable 

multiple systems and pilots to train or evaluate 

simultaneously. 

They are being standardized through the ASTM 

International Standards Committee on Homeland Security 

Applications; Response Robots (ASTM E54.09). They are also 

referenced as Job Performance Requirements in 

the National Fire Protection Association Standard for Small 

Unmanned Aircraft Systems Used For Public Safety 

Operations (NFPA 2400) and the ASTM Standard Guide for 

Training for Remote Pilot in Command of Unmanned 

Aircraft Systems Endorsement (ASTM F38.03 F3266-18). 

These suites of standard test methods provide common 

measures of capabilities with quantitative results. They can 

be conducted individually, in sequences, or embedded into 

operational training scenarios as repeatable and 

reproducible tasks with scores to augment qualitative 

assessments. Organizations using these tests set their own 

thresholds of acceptable system and pilot performance to 

align with their airspace, environment, and mission 

complexities. But those decisions are easier to make and 

trust when they’re based on quantitative performance data 

captured within standard test methods.

Basic Maneuvering (MAN 1-5) and Payload Functionality (PAY 1-5)

Safety | Capabilities | Proficiency

INSIDE EACH BUCKET

EVALUATE SENSORS

SMALL SYSTEMS LARGE SYSTEMS

INTERFACES PROCEDURES

SENSORS MANEUVERING

MEASURE & COMPARE

1A

SCALABLE TEST LANES (A

SPAC

OMNI BUCKETS

PROCEDURES

SENSORS MANEUVERING

INTERFACES

SMALL DRONES

MEASURE & COMPARE SCALABLE (ALTITUDE = SPACING)

Page 4 11/9/21

OPEN TEST LANES

The Open Test Lanes evaluates 5 different flight paths to identify objects from safe altitudes in open 

environments. These tests are scalable for all sizes of aircraft to demonstrate positive control at all times 

with accurate perches. They can be performed outdoors or indoors to control lighting and weather. The 

smallest size lane fits on an indoor basketball or tennis court for small drones and/or novice pilots to 

practice safely without flying in the national airspace.

Figure 4) The standard tests use well defined altitudes, positions, orientations, and perches to evaluate both 

Maneuvering and/or Payload Functionality.

Standard Test Methods for Response Robots ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Scalable Test Lanes for Various Sizes of Aircraft

Project Overview

Introduction

Remotely operated aerial systems enable emergency 

responders to perform extremely hazardous tasks from 

safer stand-off distances. The U.S. National Institute of 

Standards and Technology is leading an international effort 

to develop standard test methods to help manufacturers, 

procurement professionals, and users objectively evaluate 

system capabilities and remote pilot proficiency to align 

with mission requirements. Thisimproves the safety and 

effectiveness of emergency responders as they save lives 

and protect property in our communities

The first step toward credentialing remote pilots is to get 

everybody onto the same measuring stick. That’s where 

standard test methods can play a key role. These test 

methods for Basic Maneuvering (MAN 1-5) and Payload 

Functionality (PAY 1-5) are being replicated across the 

country and internationally to focus training with 

quantitative measures of remote pilot proficiency. They are 

low cost and easy to replicate so everyone can measure 

their own progress over time and compare their proficiency 

to regional or national averages on similar 

systems. Concurrent test lanes can be set up to enable 

multiple systems and pilots to train or evaluate 

simultaneously. 

They are being standardized through the ASTM 

International Standards Committee on Homeland Security 

Applications; Response Robots (ASTM E54.09). They are also 

referenced as Job Performance Requirements in 

the National Fire Protection Association Standard for Small 

Unmanned Aircraft Systems Used For Public Safety 

Operations (NFPA 2400) and the ASTM Standard Guide for 

Training for Remote Pilot in Command of Unmanned 

Aircraft Systems Endorsement (ASTM F38.03 F3266-18). 

These suites of standard test methods provide common 

measures of capabilities with quantitative results. They can 

be conducted individually, in sequences, or embedded into 

operational training scenarios as repeatable and 

reproducible tasks with scores to augment qualitative 

assessments. Organizations using these tests set their own 

thresholds of acceptable system and pilot performance to 

align with their airspace, environment, and mission 

complexities. But those decisions are easier to make and 

trust when they’re based on quantitative performance data 

captured within standard test methods.

INSIDE EACH BUCKET

EVALUATE SENSORS

SMALL SYSTEMS LARGE SYSTEMS

INTERFACES PROCEDURES

SENSORS MANEUVERING

MEASURE & COMPARE

1A

SCALABLE TEST LANES (ALTITUDE = SPACING) ALTITUDE

SPACING

OMNI BUCKETS

LARGE DRONES

5 F

OPEN TEST LANE AND RELATED OPERATIONAL SCENARIOS

FLIGHT LINE

z

z

ALTITUDE S 

S 

S

RECON

MAN/PAY 5

FLIGHT LINE

S 

ALTITUDE 

INSPECT

MAN/PAY 4

FLIGHT LINE

ALTITUDE S

S 

TRAVERSE

MAN/PAY 2

FLIGHT LINE

z

ALTITUDE S

AL

S 

POSITION

MAN/PAY 1

FLIGHT LINE

S 

ORBIT

MAN/PAY 3

AERIAL
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

7

Overdue Standards (5-8 Years Old)

Committee Update

Need to be Updated and Re-Balloted:

E2991-17 Mobility: Traverse Gravel Terrain

E2992-17 Mobility: Traverse Sand Terrain

E2566-17 Sensing: Visual Acuity

E2592-16 Logistics: Packaging for US&R Task Force Equipment Caches

E3132-17 Logistics: System Configuration Identification

E2521-16 Terminology for Evaluating Response Robot Capabilities

POST

T-TOP

LEGS

POST

4x4x24in (10x10x60cm)

LEGS

2x4x20in (5x10x50cm)

with 45° opposing ends

T-TOP

2x4x12in (5x10x30cm)

BUCKETS

2 or 5 gal (7 or 19 liter)

Launch/Land

12in (30cm) Radius Center

Bucket Stand #1

(A,B,C,D)

10 ft (3 m) Spacings

FLIGHT LINE

SIGHT LINE

Launch/Land

12in (30cm) Radius Center

FLIGHT LINE

SIGHT LINE

Bucket Stand #1

(A,B,C,D)

Bucket Stand #2 

(E,F,G,H)

Bucket Stand #3

(I,J,K,L)

10ft (3m) Spacings

10 FT (3 M)

ALTITUDE LAUNCH LAND

Test Director 

Adam Jacoff

Intelligent Systems Division

Nat’l Institute of Standards and Technology 

U.S. Dept. of Commerce 

RobotTestMethods@nist.gov | 301-975-4235 

Sponsor 

Philip Mattson

Science and Technology Directorate 

U.S. Dept. of Homeland Security

These test methods are primarily intended for 

vertical takeoff and landing systems with an 

onboard camera and remote pilot display. Some 

are also applicable to fixed wing systems when the 

lane dimensions are extended to accommodate 

the orbit radius of forward flying systems. 

Bucket stands on a level surfaces ensure the top bucket is 

vertical and the angled buckets are 45 degrees. 

TRAVERSE TEST

Right and Left Ovals Around All Three Bucket Stands

ALIGNED

WHEN RING IS 

COMPLETELY 

VISIBLE

GROUND

AQUATIC

AERIALALTITUDE

SPACING

Page 4 11/9/21

The Open Test Lanes evaluates 5 different flight paths to identify objects from safe altitudes in open 

environments. These tests are scalable for all sizes of aircraft to demonstrate positive control at all times 

with accurate perches. They can be performed outdoors or indoors to control lighting and weather. The 

smallest size lane fits on an indoor basketball or tennis court for small drones and/or novice pilots to 

practice safely without flying in the national airspace.

Figure 4) The standard tests use well defined altitudes, positions, orientations, and perches to evaluate both 

Maneuvering and/or Payload Functionality.

Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Scalable Test Lanes for Various Sizes of Aircraft

Project Overview

procurement professionals, and users objectively evaluate 

system capabilities and remote pilot proficiency to align 

with mission requirements. Thisimproves the safety and 

effectiveness of emergency responders as they save lives 

and protect property in our communities

The first step toward credentialing remote pilots is to get 

everybody onto the same measuring stick. That’s where 

standard test methods can play a key role. These test 

methods for Basic Maneuvering (MAN 1-5) and Payload 

Functionality (PAY 1-5) are being replicated across the 

country and internationally to focus training with 

quantitative measures of remote pilot proficiency. They are 

low cost and easy to replicate so everyone can measure 

their own progress over time and compare their proficiency 

to regional or national averages on similar 

systems. Concurrent test lanes can be set up to enable 

multiple systems and pilots to train or evaluate 

simultaneously. 

They are being standardized through the ASTM 

International Standards Committee on Homeland Security 

Applications; Response Robots (ASTM E54.09). They are also 

referenced as Job Performance Requirements in 

the National Fire Protection Association Standard for Small 

Unmanned Aircraft Systems Used For Public Safety 

Operations (NFPA 2400) and the ASTM Standard Guide for 

Training for Remote Pilot in Command of Unmanned 

Aircraft Systems Endorsement (ASTM F38.03 F3266-18). 

These suites of standard test methods provide common 

measures of capabilities with quantitative results. They can 

be conducted individually, in sequences, or embedded into 

operational training scenarios as repeatable and 

reproducible tasks with scores to augment qualitative 

assessments. Organizations using these tests set their own 

thresholds of acceptable system and pilot performance to 

align with their airspace, environment, and mission 

complexities. But those decisions are easier to make and 

trust when they’re based on quantitative performance data 

captured within standard test methods.

INSIDE EACH BUCKET

EVALUATE SENSORS

SMALL SYSTEMS LARGE SYSTEMS

INTERFACES PROCEDURES

SENSORS MANEUVERING

MEASURE & COMPARE

1A

SCALABLE TEST LANES (ALTITUDE = SPACING) ALTITUDE

SPACING

OMNI BUCKETS

Page 4 

ASTM International Standards Committee on Homeland Security Appli

Response Robots (E54.09) | Website: RobotTestMethods.nist.go

• Wide area search of a plane crash site

• Vehicle identifications

• Hazardous tanker truck accident and fire

• Suspicious vehicle takedown

• Bus hostage situation

• Collapsed structure search

• Subway terrorist attack

• Flooded house rescue with delivery of personal floatation devices

• Flash flood bridge collapse with school bus and vehicle rescue

OPEN TEST LANES

The Open Test Lanes evaluates 5 different flight paths to identify objects from sa

environments. These tests are scalable for all sizes of aircraft to demonstrate pos

with accurate perches. They can be performed outdoors or indoors to control lig

smallest size lane fits on an indoor basketball or tennis court for small drones an

practice safely without flying in the national airspace.

Figure 4) The standard tests use well defined altitudes, positions, orientations, a

Maneuvering and/or Payload Functionality.

Standard Test Methods for Response Robots ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Scalable Test Lanes for Various Sizes o

Project Overview

Standard Test Methods for Small Unmanned Aircraft Systems ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

3

Introduction

Remotely operated aerial systems enable emergency 

responders to perform extremely hazardous tasks from 

safer stand-off distances. The U.S. National Institute of 

Standards and Technology is leading an international effort 

to develop standard test methods to help manufacturers, 

procurement professionals, and users objectively evaluate 

system capabilities and remote pilot proficiency to align 

with mission requirements. Thisimproves the safety and 

effectiveness of emergency responders as they save lives 

and protect property in our communities

The first step toward credentialing remote pilots is to get 

everybody onto the same measuring stick. That’s where 

standard test methods can play a key role. These test 

methods for Basic Maneuvering (MAN 1-5) and Payload 

Functionality (PAY 1-5) are being replicated across the 

country and internationally to focus training with 

quantitative measures of remote pilot proficiency. They are 

low cost and easy to replicate so everyone can measure 

their own progress over time and compare their proficiency 

to regional or national averages on similar 

systems. Concurrent test lanes can be set up to enable 

multiple systems and pilots to train or evaluate 

simultaneously. 

They are being standardized through the ASTM 

International Standards Committee on Homeland Security 

Applications; Response Robots (ASTM E54.09). They are also 

referenced as Job Performance Requirements in 

the National Fire Protection Association Standard for Small 

Unmanned Aircraft Systems Used For Public Safety 

Operations (NFPA 2400) and the ASTM Standard Guide for 

Training for Remote Pilot in Command of Unmanned 

Aircraft Systems Endorsement (ASTM F38.03 F3266-18). 

These suites of standard test methods provide common 

measures of capabilities with quantitative results. They can 

be conducted individually, in sequences, or embedded into 

operational training scenarios as repeatable and 

reproducible tasks with scores to augment qualitative 

assessments. Organizations using these tests set their own 

thresholds of acceptable system and pilot performance to 

align with their airspace, environment, and mission 

complexities. But those decisions are easier to make and 

trust when they’re based on quantitative performance data 

captured within standard test methods.

Basic Maneuvering (MAN 1-5) and Payload Functionality (PAY 1-5)

Safety | Capabilities | Proficiency

INSIDE EACH BUCKET

EVALUATE SENSORS

SMALL SYSTEMS LARGE SYSTEMS

INTERFACES PROCEDURES

SENSORS MANEUVERING

MEASURE & COMPARE

1A

SCALABLE TEST LANES (A

SPAC

OMNI BUCKETS

PROCEDURES

SENSORS MANEUVERING

INTERFACES

SMALL DRONES

MEASURE & COMPARE SCALABLE (ALTITUDE = SPACING)

Page 4 11/9/21

OPEN TEST LANES

The Open Test Lanes evaluates 5 different flight paths to identify objects from safe altitudes in open 

environments. These tests are scalable for all sizes of aircraft to demonstrate positive control at all times 

with accurate perches. They can be performed outdoors or indoors to control lighting and weather. The 

smallest size lane fits on an indoor basketball or tennis court for small drones and/or novice pilots to 

practice safely without flying in the national airspace.

Figure 4) The standard tests use well defined altitudes, positions, orientations, and perches to evaluate both 

Maneuvering and/or Payload Functionality.

Standard Test Methods for Response Robots ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Scalable Test Lanes for Various Sizes of Aircraft

Project Overview

Introduction

Remotely operated aerial systems enable emergency 

responders to perform extremely hazardous tasks from 

safer stand-off distances. The U.S. National Institute of 

Standards and Technology is leading an international effort 

to develop standard test methods to help manufacturers, 

procurement professionals, and users objectively evaluate 

system capabilities and remote pilot proficiency to align 

with mission requirements. Thisimproves the safety and 

effectiveness of emergency responders as they save lives 

and protect property in our communities

The first step toward credentialing remote pilots is to get 

everybody onto the same measuring stick. That’s where 

standard test methods can play a key role. These test 

methods for Basic Maneuvering (MAN 1-5) and Payload 

Functionality (PAY 1-5) are being replicated across the 

country and internationally to focus training with 

quantitative measures of remote pilot proficiency. They are 

low cost and easy to replicate so everyone can measure 

their own progress over time and compare their proficiency 

to regional or national averages on similar 

systems. Concurrent test lanes can be set up to enable 

multiple systems and pilots to train or evaluate 

simultaneously. 

They are being standardized through the ASTM 

International Standards Committee on Homeland Security 

Applications; Response Robots (ASTM E54.09). They are also 

referenced as Job Performance Requirements in 

the National Fire Protection Association Standard for Small 

Unmanned Aircraft Systems Used For Public Safety 

Operations (NFPA 2400) and the ASTM Standard Guide for 

Training for Remote Pilot in Command of Unmanned 

Aircraft Systems Endorsement (ASTM F38.03 F3266-18). 

These suites of standard test methods provide common 

measures of capabilities with quantitative results. They can 

be conducted individually, in sequences, or embedded into 

operational training scenarios as repeatable and 

reproducible tasks with scores to augment qualitative 

assessments. Organizations using these tests set their own 

thresholds of acceptable system and pilot performance to 

align with their airspace, environment, and mission 

complexities. But those decisions are easier to make and 

trust when they’re based on quantitative performance data 

captured within standard test methods.

INSIDE EACH BUCKET

EVALUATE SENSORS

SMALL SYSTEMS LARGE SYSTEMS

INTERFACES PROCEDURES

SENSORS MANEUVERING

MEASURE & COMPARE

1A

SCALABLE TEST LANES (ALTITUDE = SPACING) ALTITUDE

SPACING

OMNI BUCKETS

LARGE DRONES

5 F

OPEN TEST LANE AND RELATED OPERATIONAL SCENARIOS

FLIGHT LINE

z

z

ALTITUDE S 

S 

S

RECON

MAN/PAY 5

FLIGHT LINE

S 

ALTITUDE 

INSPECT

MAN/PAY 4

FLIGHT LINE

ALTITUDE S

S 

TRAVERSE

MAN/PAY 2

FLIGHT LINE

z

ALTITUDE S

AL

S 

POSITION

MAN/PAY 1

FLIGHT LINE

S 

ORBIT

MAN/PAY 3

AERIAL
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

8

Validation Exercises

Committee Update

• 2021.10 Unmanned Tactical Applications Conference, Guardian Center, Georgia (3 days)

• 2021.10 Law Enforcement Drone Association Conference, Bend, Oregon (3 days)

• 2020.09 DHS sUAS Assessement, Ft. Meyers, FL (5 days)

• 2021.08 FAA Safety Team Online Course (1 hour, quiz, certificate)

• 2021.08 AUVSI Xponential with DroneResponders Fly-In at Merceded Benz Stadium (3 days)

• 2021.08 Civil Air Patrol Advanced Training, Ft. Atterbury, Indiana (3 days)

• 2021.08 Eastern Regional Robot Rodeo, NAVEODTECHDIV, Indianhead, MD (3 days)

• 2021.06 RoboCupRescue Robot Competition – Remote Video Trials (months)

• 2021.06 ASTM E54.09 Response Robots Meeting, Online Only

Color Key: Ground Aerial Aquatic Multiple Standards
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

9

Validation Exercises

Committee Update

Disaster-Resilient Buildings, Infrastructure, 

and Communities Goal

Technical Expertise: Standards Committees

Slide37

Assess the portfolio of scientific expertise

Chair/Fellow, ASTM International Standards Committee on Homeland 

Security Applications: Response Robots (ASTM E54.09) 

This committee is standardizing 50 test methods for ground, aerial, and 

aquatic systems. It hosts two meetings annually to facilitate development 

and balloting of standards. Typically we conduct an associated test 

method validation exercise.

Member, NFPA Standards Committee on Small Unmanned Aircraft 

Systems Used for Public Safety Applications (NFPA 2400)

This committee has adopted our test methods to support Job 

Performance Requirements for remote pilot proficiency and will likely set 

thresholds of performance for firefighting and other public safety 

applications. 

Member, NATO SCI-ET-052 Research Task Group on EOD Robot TeleManipulation This committee is developing NATO’s roadmap for 

dexterous robot applied to C-IED/EOD applications.

2019 Host: Houston Fire Dept

2017 Host: Canadian CETA

2018 Host: San Diego Fire Dept

• 2020.08 DHS/DOJ sUAS Procurement Testing ($35M), Montgomery County Police Facility, MD (1 days)

• 2020.10 Air Force Large Ground Robot Procurement ($70M), Tyndall AFB, FL (Weeks)

• 2020.08 DHS/DOJ sUAS Procurement Testing ($35M), Montgomery County Police Facility, MD (5 days)

• 2020.09 Canadian Fire Training Facility Opening Exercise, Toronto Airport, Ontario, Canada (4 days)

• 2020.08 World Robot Summit Disaster Response Championship, Fukushima, Japan (4 days)

• 2020.06 RoboCupRescue International Championship, Bordeaux, France (5 days)

• 2020.05 AUVSI Exponential Conference (netted aviary), Boston, MA (3 days)

• 2020.04 Fire Dept. International Conference (FDIC) Hands-On Training, Indianapolis, IN (3 days)

• 2020.03 UTAC UAS Conference, Guardian Center, Perry, GA (4 days)

• 2020.03 Public Safety UAS Conference Validation Exercise, Crozet, VA (5 days)

• 2020.02 ASTM E54.09 Response Robots Meeting, Atlanta, GA (3 days)

Color Key: Ground Aerial Aquatic Multiple Standards
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

10

Validation Exercises

Committee Update

Disaster-Resilient Buildings, Infrastructure, 

and Communities Goal

Technical Expertise: Standards Committees

Slide37

Assess the portfolio of scientific expertise

Chair/Fellow, ASTM International Standards Committee on Homeland 

Security Applications: Response Robots (ASTM E54.09) 

This committee is standardizing 50 test methods for ground, aerial, and 

aquatic systems. It hosts two meetings annually to facilitate development 

and balloting of standards. Typically we conduct an associated test 

method validation exercise.

Member, NFPA Standards Committee on Small Unmanned Aircraft 

Systems Used for Public Safety Applications (NFPA 2400)

This committee has adopted our test methods to support Job 

Performance Requirements for remote pilot proficiency and will likely set 

thresholds of performance for firefighting and other public safety 

applications. 

Member, NATO SCI-ET-052 Research Task Group on EOD Robot TeleManipulation This committee is developing NATO’s roadmap for 

dexterous robot applied to C-IED/EOD applications.

2019 Host: Houston Fire Dept

2017 Host: Canadian CETA

2018 Host: San Diego Fire Dept

Color Key: Ground Aerial Aquatic Multiple Standards

• 2020.01 Ohio Fire Training Facility Opening, Ohio (2 days)

• 2020.01 FDIC Fire/Rescue East, Daytona, FL (2 days)

• 2020.01 Los Angeles Fire Dept. Training, Los Angeles, CA (3 days)

• 2019.12 FAA Requirements Workshop for Fire Depts and Emergency Services, NIST (1 day)

• 2019.11 Atlantic Future Forum, UK HMS Queen Elizabeth, Annapolis, MD (2 days)

• 2019.11 DHS Familiarization Exercise, Army Camp Shelby, MS (5 days)

• 2019.10 World Robot Summit, Fukushima, Japan (5 days)

• 2019.09 NATO Aerial and Ground Exercise, Base Borden, Ontario, Canada (3 days)

• 2019.07 Aerial Validation Exercise at NIST (3 days)

• 2019.06 RoboCupRescue International Championship, Sydney, Australia (5 days)

• 2019.06 ASTM E54.09 Response Robots Meeting and Exercise, Denver, CO (5 days)
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

11

Validation Exercises

Committee Update

Disaster-Resilient Buildings, Infrastructure, 

and Communities Goal

Technical Expertise: Standards Committees

Slide37

Assess the portfolio of scientific expertise

Chair/Fellow, ASTM International Standards Committee on Homeland 

Security Applications: Response Robots (ASTM E54.09) 

This committee is standardizing 50 test methods for ground, aerial, and 

aquatic systems. It hosts two meetings annually to facilitate development 

and balloting of standards. Typically we conduct an associated test 

method validation exercise.

Member, NFPA Standards Committee on Small Unmanned Aircraft 

Systems Used for Public Safety Applications (NFPA 2400)

This committee has adopted our test methods to support Job 

Performance Requirements for remote pilot proficiency and will likely set 

thresholds of performance for firefighting and other public safety 

applications. 

Member, NATO SCI-ET-052 Research Task Group on EOD Robot TeleManipulation This committee is developing NATO’s roadmap for 

dexterous robot applied to C-IED/EOD applications.

2019 Host: Houston Fire Dept

2017 Host: Canadian CETA

2018 Host: San Diego Fire Dept

Color Key: Ground Aerial Aquatic Multiple Standards

• 2019.05 Western Regional Robot Rodeo, Sandia/Kirtland, Albuquerque, NM (5 days)

• 2019.05 Canadian Police College Training Exercise, London, ON Canada (7 days)

• 2019.04 Thermite RS2 firefighting robot capabilities evaluation (1 day)

• 2019.04 Army Tank Automotive Research and Development facility fabrication (remote)

• 2019.04 Fire Dept Training Conference (FDIC), Indianapolis, IN (3 days)

• 2019.04 Guardian Center Training, Perry, GA (2 days remote)

• 2019.04 Reveille Ranch Calibration, Texas Dept of Public Safety, Burnet, TX (2 days)

• 2019.04 InstantEye UAS capabilities evaluation, NIST (3 days)

• 2019.03 ASTM F38 standard balloted referencing 6 of our aerial test methods

• 2019.03 Navy Explosive Ordinance Disposal Tech Division facility fabrication (remote)

• 2019.03 Virginia UAS Summit on Public Safety, Crozet, VA (3 days)

• 2020.02 ASTM E54.09 Response Robots Meeting and Exercise, Atlanta, CO (3 days)
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

12

Eastern National Robot Rodeo

Example Use Cases

Lead Agency

Navy NAVEODTECHDIV 

(2021.08)

Location:

Indianhead, MD

Outcomes:

• Implemented Maneuvering, Mobility, 

Dexterity tests to host a training rodeo.

• Set up new indoor facility plus three 

shipping container sequential scenario.

• No pictures/video allowed!

Eastern National Robot Rodeo (ENRR

August 2-6, 2021; Indian Head, Maryland

Description

The use of advanced robotics in public safety and military 

bomb squad communities has grown significantly over 

the past few decades. As technology increases, bomb 

technicians need to better understand capabilities and 

limitations of these systems in order to align industry 

opportunity for participants to network with oth

technicians and exchange information on 

techniques, and procedures.

Scenarios

Participants will have an opportunity to utilize 
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

13

Air Force Large Robot Purchase

Example Use Cases

Lead Agency

Air Force Civil Engineer Center (AFCEC/CXAE)

2021.01

Primary Location:

Tyndall Air Force Base

Panama Beach, FL USA

Outcomes:

• Implemented Maneuvering, Dexterity, 

Sensing, and Radio Comms tests to capture 

comparable data across a roster of large CIED/EOD robots.

• $70M total funding

L3 HARRIS T4

(NEEDS TESTING)

L3 HARRIS T7 

(PURCHASED)

322 KG (710 LBS)

103 KG (226 LBS)
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

14

NATO SCI-342 Research Task Group

Example Use Cases

• Adopting and extending our tests to measure robot 

dexterity and remote operator proficiency necessary 

to perform explosive ordinance disposal (EOD) 

missions from remote standoff distances. 

• Will replicate the modular dexterity tests in several 

collaborating NATO countries, including the NATO 

Center of Excellence for C-EOD Operations in 

Slovakia.

• Adam Jacoff leads the Subgroup 4 Evaluation (Testing 

& Metrics)

• The charter will be active 2020-2023 

Subgroup 4 Evaluation (Testing & Metrics)

• Adam Jacoff

• Aurélie Lepoil

• André Volk

• Johannes Pellenz

• Eric den Breejen
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

15

Canadian CETA and CERRA Training/Credentialing

Example Use Cases

Lead Agencies;

CETA- Canadian Explosives Technicians Association

CERRA- Canadian Emergency Responders Robotics Association

Primary Locations:

Pearson International Airport ( Toronto Canada)

Grimsby Regional Training Centre ( Grimsby, Ontario , Canada)

CETA

CETA is the national association for police/military/government agencies tasked with response to 

explosives , chemical, biological, and radiological incidents in Canada. Current projects include 

EOD Standard training methods for both robots and bomb techs deployed in bomb suits.

CERRA

Spring 2020 established with focus on the public safety deployment of ground, air, water based 

robotics. Membership is open to any current or former public safety member or agency or any 

supporting government agency with an interest in response robots.
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

16

RoboCupRescue Robot League (2000-present)

Example Use Cases

Dissemination: Robotics Research Competitions

Assess the effectiveness of disseminating project outputs

• We conduct annual international robotics research competitions, sometimes two a year.

• The RoboCupRescue Championships (shown below) use 20 ground robot tests set up in a large maze so 

they can be conducted individually as preliminaries then a comprehensive search mission for finals. 

• These competition focus on autonomous behaviors vs. remote teleoperation using all three sizes of

robot test lanes. Typically more than 30 teams participate.

• Most teams fabricate the test methods at their facilities to refine designs and practice.

p

ASTM International Standards Committee for Homeland Security Applications;

Response Robots (E54.09)

This annual competition, chaired by Adam Jacoff since its inception in 2001, has evolved over the years into 

a rigorous validation exercise for the suite of standard test methods being used. It also provides multiple 

layers of outreach to different communities. The test methods inspire innovation in teams of robotics 

researchers needing reproducible measurements to evaluate and compare a wide variety of 

implementations. International test administrators learn how to conduct standardized testing in their own 

facilities. Program managers and companies use the quantitative results to identify best-in-class 

performers. Emergency responders and others learn how to use standard test methods to focus training

and measure operator proficiency prior to deploying into operational scenarios. In general, these 

competitions have become a very effective way to disseminate standard test methods internationally. As a 

result of this competition, facilities based on our test methods are being set up in Japan, Korea, and 

Thailand bringing the total to 15 worldwide.

Figure 1: The venue was an indoor sports arenaAll the test apparatuses were purchased and fabricated by 

Standard Test Methods for Response Robots

ASTM International Standards Committee for Homeland Security Applications;

Response Robots (E54.09)

Figure 2: More than 35 ground and aerial test methods were arranged as individual testsrepeatable 

NAGOYA, JAPAN LARGE MAZE OF 20 GROUND ROBOT 

TEST LANES 120cm (48in) SCALE

Disaster-Resilient Buildings, Infrastructure, 

and Communities Goal

Dissemination: Robotics Research Competitions

Slide52

Assess the effectiveness of disseminating project outputs

• Typically more than 30 teams participate from all over the world, 

roughly 300 people each year.

• The annual Championships move internationally with teams typically 

qualified from several regional competitions.

• These competitions are recruiting tools for manufacturers and test 

facilities around the world. 

Standard Test Methods for Response Robots

ASTM International Standards Committee for Homeland Security Applications;

Response Robots (E54.09)

RoboCupRescue

Championships

2020 Bordeux, France

2019 Sydney, Australia

2018 Montreal, Canada

2017 Nagoya, Japan

2016 Leipzig, Germany

2015 Hefei, China

2014 Joao Pessoa, Brazil

2013 Eindhoven, Netherlands

2012 Mexico City, Mexico

2011 Istanbul, Turkey

2010 Singapore, Singapore

2009 Graz, Austria

2008 Suzhou, China

2007 Atlanta, USA

2006 Bremen, Germany

2005 Osaka, Japan

2004 Lisbon, Portugal

2003 Padua, Italy

2002 Fukuoka, Japan

2001 Seattle, USA

2000 AAAI Conf, Austin, TX
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

17

RoboCupRescue Robot League (Remote 2021)

Example Use Cases

100’S OF SELF SCORED VIDEO SUBMISSIONS IN TESTS

Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Video Trials For Remote Scoring

Dexterity and Strength Classification Tests

OVERVIEW OF ROBOT & OPERATOR ALL OPERATOR ACTIONS

SHOW EASILY 

READABLE 

TIMESTAMP, PRINTED

TRIAL INFO, AND ALL 

OPERATOR ACTIONS

SAVE SCREEN TO

VIDEO OR ZOOM IN

WITH CAMERA AS

SHOWN HERE

SHOW THE 

OPERATOR IN THE 

BACKGROUND

WITH BACK TOWARD

THE APPARATUS

SHOW AS MUCH 

DETAILOF THE 

TASK AS 

POSSIBLE

ADD PLENTY OF LIGHTING

TASK DETAIL ROBOT VIEW

GET INVOLVED THROUGH THE

NATO SCI-342 RTG 

BEST IN CLASS

RESULTS
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

18

New Test Facility in Germany

Example Use Cases

Dortmond, Germany Opened 2021.10

Host of the 2021

RoboCupRescue

German Open 

Robot Competition
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

19

Test Facility in Brisbane, Australia

Example Use Cases

Modular, fork-liftable, and 

stowable apparatuses set into a 

lane containment perimeter.
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

20

World Robot Summit (2018-2020)

Example Use Cases

Standard Disaster Robotics Category, Fukushima Robot Test Field, Fukushima, Japan
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Sub Committee Chair:

Adam Jacoff

Intelligent Systems Division

National Institute of Standards and Technology

U.S. Department of Commerce

Committee Chair:

Phil Mattson

Science and Technology Directorate 

U.S. Department of Homeland Security

Internet

RobotTestMethods.nist.gov

Adam	Jacoff

Intelligent	Systems	Division

NATIONAL	INSTITUTE	OF	STANDARDS	AND	TECHNOLOGY

U.S.	Department	of	Commerce

Email

RobotTestMethods@nist.gov

21