Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

ASTM E54.09 Homeland Security Applications; Response Robots

Search Tasks in Labyrinths, Mazes, and Scenarios

Sub Committee Chair:

Adam Jacoff

Intelligent Systems Division

National Institute of Standards and Technology

U.S. Department of Commerce

Committee Chair:

Phil Mattson

Science and Technology Directorate 

U.S. Department of Homeland Security

1

Version 2022A

STARTS AT 10:00 AM EST 

WASHINGTON, DC TIME

Internet

RobotTestMethods.nist.gov

Adam	Jacoff

Intelligent	Systems	Division

NATIONAL	INSTITUTE	OF	STANDARDS	AND	TECHNOLOGY

U.S.	Department	of	Commerce

Email

RobotTestMethods@nist.gov
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

2

Linear Search Tasks with Measures of Acuity

Search Tasks

Response Robots (E54.09) | 

 

WITH EMBEDDED LINEAR INSPECT TASKS
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

3

Linear Search Tasks with Measures of Acuity

Search Tasks

CONCENTRIC 

C

GAP

SIZE

C1 LARGEST

10.3 mm 

(0.4 in)

C2

4.1 mm 

(0.16 in)

C3

1.6 mm 

(0.06 in)

C4

0.7 mm 

(0.03 in)

C5 SMALLEST

0.3 mm 

(0.01 in)

ALIGNED ALIGNED NOT ALIGNED
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

4

Always Placed in 10 Different Elevations and Orientations

Search Tasks Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

93

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

4

FAR SIDE OF OBJECT INTERIOR BOTTOM

ORIENTED TO READ LETTER UPRIGHT WHEN MOUNTED

Compiled SCENARIO 2020-06-26 10:37 (On Objects)

1C

1C

1C

1C

1C

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

3

LEFT SIDE OF OBJECT INTERIOR BOTTOM

ORIENTED TO READ LETTER UPRIGHT WHEN MOUNTED

Compiled SCENARIO 2020-06-26 10:37 (On Objects)

1B

1B

1B

1B

1B

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

3

LEFT SIDE OF OBJECT INTERIOR BOTTOM

ORIENTED TO READ LETTER UPRIGHT WHEN MOUNTED

Compiled SCENARIO 2020-06-26 10:37 (On Objects)

1B

1B

1B

1B

1B

OPEN

BUCKET

BUCKET 

LID HOLE

TARGET

ON LID

10 Search Elevations and Orientations

Search and Inspect Tests

FLOOR 0 cm (0 ft)

TABLE 60 cm (2 ft)

COUNTER 120 cm (4 ft)

SHELF 180 cm (6 ft)

CEILING 240 cm (8 ft) “FORWARD” “OVER” ”UNDER”

The open buckets and 

buckets with lid holes 

require different 

elevations, orientations, 

and proximities to align 

with and identify 

all targets.

Example point of view 

from a small robot:

ALIGNED

OPEN

BUCKET

BUCKET 

LID HOLE

OPEN

BUCKET

BUCKET 

LID HOLE

OPEN

BUCKET

BUCKET 

LID HOLE

OPEN

BUCKET

BUCKET 

LID HOLE

OPEN

BUCKET

BUCKET 

LID HOLE

OPEN

BUCKET

BUCKET 

LID HOLE

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

4

FAR SIDE OF OBJECT INTERIOR BOTTOM

ORIENTED TO READ LETTER UPRIGHT WHEN MOUNTED

Compiled SCENARIO 2020-06-26 10:37 (On Objects)

1C

1C

1C

1C

1C

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

3

LEFT SIDE OF OBJECT INTERIOR BOTTOM

ORIENTED TO READ LETTER UPRIGHT WHEN MOUNTED

Compiled SCENARIO 2020-06-26 10:37 (On Objects)

1B

1B

1B

1B

1B

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

3

LEFT SIDE OF OBJECT INTERIOR BOTTOM

ORIENTED TO READ LETTER UPRIGHT WHEN MOUNTED

Compiled SCENARIO 2020-06-26 10:37 (On Objects)

1B

1B

1B

1B

1B

OPEN

BUCKET

BUCKET 

LID HOLE

TARGET

ON LID

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

4

FAR SIDE OF OBJECT INTERIOR BOTTOM

ORIENTED TO READ LETTER UPRIGHT WHEN MOUNTED

Compiled SCENARIO 2020-06-26 10:37 (On Objects)

1C

1C

1C

1C

1C

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

3

LEFT SIDE OF OBJECT INTERIOR BOTTOM

ORIENTED TO READ LETTER UPRIGHT WHEN MOUNTED

Compiled SCENARIO 2020-06-26 10:37 (On Objects)

1B

1B

1B

1B

1B

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

3

LEFT SIDE OF OBJECT INTERIOR BOTTOM

ORIENTED TO READ LETTER UPRIGHT WHEN MOUNTED

Compiled SCENARIO 2020-06-26 10:37 (On Objects)

1B

1B

1B

1B

1B

OPEN

BUCKET

BUCKET 

LID HOLE

TARGET

ON LID

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

4

FAR SIDE OF OBJECT INTERIOR BOTTOM

ORIENTED TO READ LETTER UPRIGHT WHEN MOUNTED

Compiled SCENARIO 2020-06-26 10:37 (On Objects)

1C

1C

1C

1C

1C

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

3

LEFT SIDE OF OBJECT INTERIOR BOTTOM

ORIENTED TO READ LETTER UPRIGHT WHEN MOUNTED

Compiled SCENARIO 2020-06-26 10:37 (On Objects)

1B

1B

1B

1B

1B

Standard Test Methods for Small Unmanned Aircraft Systems

ASTM International Standards Committee on Homeland Security Applications; Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

3

LEFT SIDE OF OBJECT INTERIOR BOTTOM

ORIENTED TO READ LETTER UPRIGHT WHEN MOUNTED

Compiled SCENARIO 2020-06-26 10:37 (On Objects)

1B

1B

1B

1B

1B

OPEN

BUCKET

BUCKET 

LID HOLE

TARGET

ON LID

ALIGNED NOT

ALIGNED

NOT

ALIGNED
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

5

Rectangular Labyrinth Top View with Search Task Locations

Search Tasks

RIGHT Hand

Wall Following

LEFT Hand

Wall Following
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

6

Freeform Maze Top View with Search Task Locations

Search Tasks
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

7

Embedded Scenario with Search Task Locations

Search Tasks
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Labyrinth/Maze Navigation

Exploration and Mapping

8

)

OM MAZE SEARCH

Security Applications;

.01)

y Response 

(HSI): 

rrain 

Mapping Fiducials (Half Barrels) Mapping Fiducials (Half Barrels)
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

9

Complex Terrains Challenge Mapping Systems

Search Tasks
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

10

Changes in Elevation and Movable Clutter Do Too

Search Tasks
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

11

Three Different Scales For Different Environments

Search Tasks
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

ASTM E54.09 Homeland Security Applications; Response Robots

Exploration and Mapping

Sub Committee Chair:

Adam Jacoff

Intelligent Systems Division

National Institute of Standards and Technology

U.S. Department of Commerce

Committee Chair:

Phil Mattson

Science and Technology Directorate 

U.S. Department of Homeland Security

12

Version 2022A

STARTS AT 10:00 AM EST 

WASHINGTON, DC TIME

Internet

RobotTestMethods.nist.gov

Adam	Jacoff

Intelligent	Systems	Division

NATIONAL	INSTITUTE	OF	STANDARDS	AND	TECHNOLOGY

U.S.	Department	of	Commerce

Email

RobotTestMethods@nist.gov
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Labyrinth/Maze Navigation

Exploration and Mapping

13

)

OM MAZE SEARCH

Security Applications;

.01)

y Response 

(HSI): 

rrain 

Mapping Fiducials (Half Barrels) Mapping Fiducials (Half Barrels)
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

14

Exploration and Mapping Tests

Ground Robots

Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

100

Measure Robot Capabilities and/or Operator Proficiency

Exploration and Mapping Tests

• Validate easy to replicate 

exploration and mapping tests 

in YOUR scenarios.

• Focus on reconfigurable task 

apparatuses that are easy to lay 

out temporarily and store 

between trials.

• Compare your 2D and 3D map 

results over time in 

variable/repeatable layouts 

within the same scenario.

• Try different scenarios using all 

the same layout rules, in 

houses, workplaces, industrial 

facilities, outdoor settings.

Distribute QR codes (all the same or all unique) and/or 

half round shape fiducials in pairs on both sides of walls to 

measure map consistency and accuracy.
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

15

Exploration and Mapping Tests

Ground Robots

Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

101

Measure Robot Capabilities and/or Operator Proficiency

Exploration and Mapping Tests

• Validate easy to replicate 

exploration and mapping tests 

in YOUR scenarios.

• Focus on reconfigurable task 

apparatuses that are easy to lay 

out temporarily and store 

between trials.

• Compare your 2D and 3D map 

results over time in 

variable/repeatable layouts 

within the same scenario.

• Try different scenarios using all 

the same layout rules, in 

houses, workplaces, industrial 

facilities, outdoor settings.

Distribute QR codes (all the same or all unique) and/or 

half round shape fiducials in pairs on both sides of walls to 

measure map consistency and accuracy.
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

16

Exploration and Mapping Tests

Ground Robots

Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

102

Distribute Fiducials Throughout Your Scenarios

Exploration and Mapping Tests

• Validate easy to replicate 

exploration and mapping tests 

in YOUR scenarios.

• Focus on reconfigurable task 

apparatuses that are easy to lay 

out temporarily and store 

between trials.

• Compare your 2D and 3D map 

results over time in 

variable/repeatable layouts 

within the same scenario.

• Try different scenarios using all 

the same layout rules, in 

houses, workplaces, industrial 

facilities, outdoor settings.

UPPER MAPPING

FIDUCIALS CENTERED 

AT 2 M (6 FT)

LOWER MAPPING

FIDUCIALS CENTERED 

AT 1 M (3 FT)

PAIRS OF HALF-ROUND FIDUCIALS ARE ALWAYS ON BOTH SIDES OF WALLS

60 cm (24 in) diameter 

cylindrical shapes provide 

shape fiducials in maps.
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

17

Exploration and Mapping Tests

Ground Robots

Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

108

Purchase Mapping Fiducials

Exploration and Mapping Tests

Purchase enough sign panels to make 20 half-cylinders roughly 

60 cm (24 in) diameter x 120 cm (48 in) tall.

UPPER MAPPING

FIDUCIALS CENTERED 

AT 2 M (6 FT)

LOWER MAPPING

60 x 90 cm

(24 x 36 in)

60 x 90 cm

(24 x 36 in)

Wood beam on top, 

middle, and bottom 

maintains the

diameter and 

enables hanging. 

The middle beam is 

be shared by both 

the upper and lower 

panels.

Bend the panels as 

smoothly possible

Need 40 panels this size, or 5 packs as shown = $180 US plus wood beams

OPTION A
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

18

Exploration and Mapping Tests

Ground Robots

Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

109

Purchase Mapping Fiducials

Exploration and Mapping Tests

40 x 120 cm (16 x 48 in) = $16 US 45 x 120 cm (18 x 48 in) = $31 US

Purchase 10 concrete form tubes (cylindrical)

then CUT THEM IN HALF to make half-round pairs.

60 cm (24 in) diameter fiducials are preferred, although they can be expensive.

These could work for finer resolution maps.

$160 US TOTAL $320 US TOTAL

OPTION B
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

19

Exploration and Mapping Tests

Ground Robots

Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

103

Distribute Fiducials Throughout Your Scenarios

Exploration and Mapping Tests

(5) Upper fiducials

(5) Lower fiducials

• Validate easy to replicate 

exploration and mapping tests 

in YOUR scenarios.

• Focus on reconfigurable task 

apparatuses that are easy to lay 

out temporarily and store 

between trials.

• Compare your 2D and 3D map 

results over time in 

variable/repeatable layouts 

within the same scenario.

• Try different scenarios using all 

the same layout rules, in 

houses, workplaces, industrial 

facilities, outdoor settings.

UPPER MAPPING

FIDUCIALS CENTERED 

AT 2 M (6 FT)

LOWER MAPPING

FIDUCIALS CENTERED 

AT 1 M (3 FT)

FIDUCIALS ALWAYS ON BOTH SIDES OF WALLS

Example Workplace Scenario
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

20

Exploration and Mapping Tests

Ground Robots

Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

104

Distribute Lower Fiducials Throughout Your Scenarios

Exploration and Mapping Tests

2021 Embedded Remote Trials

UPPER MAPPING

FIDUCIALS CENTERED 

AT 2 M (6 FT)

LOWER MAPPING

FIDUCIALS CENTERED 

AT 1 M (3 FT)

FIDUCIALS ALWAYS ON BOTH SIDES OF WALLS

Example Workplace Scenario

(5) Upper fiducials

(5) Lower fiducials

Lower Fiducial Map

5 pairs of fiducials shown in a 3D map 

layer at 1 m (3 ft) elevation. The fiducial 

pairs should form circles in 2D maps.

Map Metrics:

• Coverage (how many of 10 are visible?)

• Consistency (how close are the pairs?)

• Local Accuracy (by room)

• Global accuracy (average overall)
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

21

Exploration and Mapping Tests

Ground Robots

Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

105

Evaluative 2D/3D Maps with Lower Fiducial Layer

Exploration and Mapping Tests

UPPER MAPPING

FIDUCIALS CENTERED 

AT 2 M (6 FT)

LOWER MAPPING

FIDUCIALS CENTERED 

AT 1 M (3 FT)

FIDUCIALS ALWAYS ON BOTH SIDES OF WALLS

Example Workplace Map

Lower Fiducial Map

5 pairs of fiducials shown in a 3D map 

layer at 1 m (3 ft) elevation. The fiducial 

pairs should form circles in 2D maps.

Map Metrics:

• Coverage (how many of 10 are visible?)

• Consistency (how close are the pairs?)

• Local Accuracy (by room)

• Global accuracy (average overall)
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

22

Exploration and Mapping Tests

Ground Robots

Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

106

Distribute Upper Fiducials Throughout Your Scenarios

Exploration and Mapping Tests

UPPER MAPPING

FIDUCIALS CENTERED 

AT 2 M (6 FT)

LOWER MAPPING

FIDUCIALS CENTERED 

AT 1 M (3 FT)

FIDUCIALS ALWAYS ON BOTH SIDES OF WALLS

Example Workplace Scenario

(5) Upper fiducials

(5) Lower fiducials

Upper Fiducial Map

5 pairs of fiducials shown in a 3D map 

layer at 2 m (6 ft) elevation. The fiducial 

pairs should form circles in 2D maps.

Map Metrics:

• Coverage (how many of 10 are visible?)

• Consistency (how close are the pairs?)

• Local Accuracy (by room)

• Global accuracy (average overall)
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

23

Exploration and Mapping Tests

Ground Robots

Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

107

Evaluative 2D/3D Maps with Upper Fiducial Layer

Exploration and Mapping Tests

UPPER MAPPING

FIDUCIALS CENTERED 

AT 2 M (6 FT)

LOWER MAPPING

FIDUCIALS CENTERED 

AT 1 M (3 FT)

FIDUCIALS ALWAYS ON BOTH SIDES OF WALLS

Example Workplace Map

Upper Fiducial Map

5 pairs of fiducials shown in a 3D map 

layer at 2 m (6 ft) elevation. The fiducial 

pairs should form circles in 2D maps.

Map Metrics:

• Coverage (how many of 10 are visible?)

• Consistency (how close are the pairs?)

• Local Accuracy (by room)

• Global accuracy (average overall)
Standard Test Methods for Response Robots

ASTM International Standards Committee on Homeland Security Applications; 

Response Robots (E54.09) | Website: RobotTestMethods.nist.gov

Sub Committee Chair:

Adam Jacoff

Intelligent Systems Division

National Institute of Standards and Technology

U.S. Department of Commerce

Committee Chair:

Phil Mattson

Science and Technology Directorate 

U.S. Department of Homeland Security

Internet

RobotTestMethods.nist.gov

Adam	Jacoff

Intelligent	Systems	Division

NATIONAL	INSTITUTE	OF	STANDARDS	AND	TECHNOLOGY

U.S.	Department	of	Commerce

Email

RobotTestMethods@nist.gov

24