Standard Test Methods for Response Robots

ASTM International Standards Committee for Homeland Security Applications;

Response Robots (E54.09)

DHS-NIST-ASTM	E54.09	Response	Robots	Meeting	and	Exercise,	Toronto,	Canada	(v2017.2).docx Page	1

Project Report

Date: 2017-06-12

Event:	 ASTM	E54.09	Response	Robots	Committee	Meeting	and	Exercise	Summary

Location: Hamilton	Port	Authority,	Toronto,	Canada

Emphasis: Validated 50	test	methods	for	Ground,	Aerial,	and	Aquatic	Response	Robots

Prepared	By: Adam	Jacoff

Intelligent	Systems	Division

National	Institute	of	Standards	and	Technology

U.S.	Department	of	Commerce

Gaithersburg,	MD	20899j

Adam.Jacoff@nist.gov		|	RobotTestMethods@nist.gov

http://RobotTestMethods.nist.gov		|		301-975-4235

General	ASTM	E54.09	Standards	Committee Overview:

Over the past few years our suite of 30 ground robot test methods have been adopted

widely across the U.S. and internationally to support purchasing and training. We are

now expanding our efforts to develop test methods for small unmanned aircraft systems

(sUAS) defined by the FAA as under 25kg (55lbs) and for underwater remotely operated

vehicles for hazardous environments.

Remotely operated robots enable emergency responders to perform extremely hazardous

tasks from safer stand-off distances. Standard test methods help robot manufacturers and

users objectively evaluate system capabilities to align with mission requirements. This

improves the safety and effectiveness of emergency responders as they attempt to save

lives and protect property in our communities.

The ASTM International Standards Committee on Homeland Security Applications:

Response Robots (E54.09) is developing the standards infrastructure necessary to inspire

innovation, inform purchasing decisions, and focus training with measures of operator

proficiency. Adam Jacoff from NIST’s Intelligent Systems Division chairs the subcommittee with his team of robotics researchers and collaborators from all over the

world. The standards committee is made up of robot manufacturers, emergency

responders, test administrators, program managers, and others interested in developing

the underlying measurement science necessary to quantitatively evaluate and compare

robotic system capabilities and operator proficiencies.

This effort has been sponsored by the Department of Homeland Security, Science &

Technology Directorate, Capability Development Support Group, Office of Standards.

They along with other sponsors have facilitated these standards committee meetings and
Standard Test Methods for Response Robots

ASTM International Standards Committee for Homeland Security Applications;

Response Robots (E54.09)

DHS-NIST-ASTM	E54.09	Response	Robots	Meeting	and	Exercise,	Toronto,	Canada	(v2017.2).docx Page	2

associated test method validation exercises for nearly fifty test methods for ground,

aerial, and aquatic systems.

Summary of Standards Committee Meeting and Validation Exercise hosted in

Hamilton, Ontario:

This meeting and exercise was hosted at the Hamilton Port Authority, Ontario, Canada.

More than twenty robots provided by manufacturers, emergency responders, and military

organizations evaluated their robotic capabilities and operator proficiencies within the

test methods. Test administrators from collaborating facilities around the world

conducted trials to learn about and validate the test methods. They also helped devise

new tests to address emerging requirements articulated by responders at the meeting.

Figure 1: A) Front view of the Hamilton Port Authority boat storage facility used to host

the event. It was mostly empty for the summer. B) Back view of the same facility. They

also provided dock space in the Lake Ontario to embed our underwater test methods to

conduct trials in the opaque water more typically associated with response operations.

Ground Robot Test Methods Evaluating Bomb Squad Proficiency

Ann Virts from the Engineering Laboratory, Adam Norton from the University of

Massachusetts - Lowell, and Donny Boyd from the Montgomery County MD Fire

Department administered training trials with Canadian bomb squad technicians. The

bomb techs used their own service robots to validate a suite of basic skills test methods

used to focus training, measuring operator proficiency for countering improvised

explosive devices (C-IED). The suite of 120cm (48in) wide ground robot test methods

have been used across the U.S. at several such events. They are also hosted in dozens of

responder facilities enabling them to measure and compare their own proficiency against

“expert” operators provided by the manufacturers, or regional and national averages.

These same test methods also help identify deficiencies in equipment. This was the first

time we have conducted such an exercise in Canada. See the media pointers below for the
Standard Test Methods for Response Robots

ASTM International Standards Committee for Homeland Security Applications;

Response Robots (E54.09)

DHS-NIST-ASTM	E54.09	Response	Robots	Meeting	and	Exercise,	Toronto,	Canada	(v2017.2).docx Page	3

reaction from the local bomb squad commanders.

 Figure 2: A) The apparatuses set up to the left are the Basic Maneuvering and Dexterity

suite of test methods. They are performed sequentially in time limited trials. B) The

Access Buildings suite of test methods includes Terrains, Stoops, Stairs, Doors, and

Hallway Labyrinths all with embedded Dexterity and Mapping tasks.

 Figure 3: The main service robot used by these regional bomb squads is manufactured in

Canada. A) It is shown performing the Dexterity: Rotate task using the parallel

apparatus at door knob elevation. B) Here it is performing theDexterity: Inspect task

using the omni-directional apparatus.
Standard Test Methods for Response Robots

ASTM International Standards Committee for Homeland Security Applications;

Response Robots (E54.09)

DHS-NIST-ASTM	E54.09	Response	Robots	Meeting	and	Exercise,	Toronto,	Canada	(v2017.2).docx Page	4

 Figure 4: A variety of other robots were also run through the test methods.

 

Aquatic Test Methods in the Clear Water Tank and Harbor Scenarios

Jesse Pentzer from Pennsylvania State University, Capt. Tom Haus from the Los Angeles

Fire Department, CA, and Battalion Chief Parry Boogard from the Valley Regional Fire

Department, WA administered underwater system testing in the clear water tank and off

the harbor dock. These fire fighters are members of the new FEMA US&R Robotics

Standards Committee. They are tasked with selecting suites of standard test methods to

evaluate ground, aerial, and aquatic systems to purchase for US&R Task Force

equipment caches nationwide. Since the facility had docks in Lake Ontario, we took the

opportunity to embed our various underwater test methods as operational training

scenarios in the brackish water. Embedding the test methods into operational scenarios is

always the next step. The embedded standard test methods enable quantitative evaluation

and comparison of capabilities from clear water tank tests with turbulence from

submerged pumps to more operationally significant and uncontrolled environments with

brackish water in the harbor (submerged pumps can be added as well to add turbulence).
Standard Test Methods for Response Robots

ASTM International Standards Committee for Homeland Security Applications;

Response Robots (E54.09)

DHS-NIST-ASTM	E54.09	Response	Robots	Meeting	and	Exercise,	Toronto,	Canada	(v2017.2).docx Page	5

Figure 5: Local responders collaborated with manufacturers demonstrating the newest

available remotely operated vehicle technologies in an 18,000 gallon water tank with

more than a dozen test methods. The repeatable tasks and clear water videos provided

key insights into the capabilities of the systems and helped measure important upgrades

implemented by manufacturers since the previous exercise. Submerged pumps

surrounding the test apparatuses provide constant turbulence while performing the tasks,

encouraging automatic holding of position and orientation to improve performance. A

blackout tarp over the top of the tank can provide darkened conditions. Beyond the

obvious training benefits, these test methods provide useful de-bugging tools and

readiness assessments, especially after repairs.

Figure 6: A) The C-frame apparatuses provide five different orientations for each task

(0°-45°-90°-135°-180°). B) The walkways on top of the tank appear helpful, but they

actually hindered inserting the large apparatus frames into the tank. So open top tanks

are preferred. C) The same test methods were replicated off the dock in the brackish

water with natural currents and sediment in the harbor. We embed the apparatuses into

these operationally relevant scenarios to measure the degradation in performance due to

the uncontrolled environmental variables.
Standard Test Methods for Response Robots

ASTM International Standards Committee for Homeland Security Applications;

Response Robots (E54.09)

DHS-NIST-ASTM	E54.09	Response	Robots	Meeting	and	Exercise,	Toronto,	Canada	(v2017.2).docx Page	6

Figure 7: A variety of underwater systems participated. A) The smallest system is

typically intended to perform reconnaissance tasks. B) The largest system is much more

capable and includes a $100K sonar.

Aerial System Test Methods

Kam Saidi from the Engineering Laboratory and Al Frazier from the North Dakota

Airborne Law Enforcement Association administered two indoor netted aviaries

containing more than a dozen test methods to evaluate small unmanned aircraft systems

(sUAS). These systems were provided and operated by manufacturers and responder

organizations leading the effort to safely and effectively incorporate these emerging

technologies into their public safety operations. The test methods quantify essential safety

features and airworthiness while providing useful practice tasks that measure proficiency

for responders or hobbyists prior to flying in the national airspace.

 Figure 8: A) The indoor netted aviaries provided safe flight testing while validating the

test methods and collecting essential data for the standardization process. B) A wall of

fans provided turbulence for the aerial systems, encouraging automatic holding of

position and orientation to improve performance of tasks.
Standard Test Methods for Response Robots

ASTM International Standards Committee for Homeland Security Applications;

Response Robots (E54.09)

DHS-NIST-ASTM	E54.09	Response	Robots	Meeting	and	Exercise,	Toronto,	Canada	(v2017.2).docx Page	7

 Figure 9: One aviary focused on Maneuvering test methods.

 

Figure 10: The other aviary focused on Situational Awareness and Sensor test methods.

Figure 11: A) The outdoor testing took place in a more remote section of Ontario where

the local responders are cleared to fly their systems outdoors. B) We set up our

Situational Awareness test methods to practice and validate them using an aerial system

manufactured in Canada and used by all the responders. C) Because aerial systems need

to handle weather conditions, we adapted a sprinkler system to the Spiral Inspect test

apparatus to enable testing resilience of the system to rain.
Standard Test Methods for Response Robots

ASTM International Standards Committee for Homeland Security Applications;

Response Robots (E54.09)

DHS-NIST-ASTM	E54.09	Response	Robots	Meeting	and	Exercise,	Toronto,	Canada	(v2017.2).docx Page	8

Radio Communications Attenuation Tests

Rick Candell, Kam Saidi, and Kenny Kimble fabricated a new radio communications

attenuation test apparatus at the event. The intent of the apparatus is to attenuate the radio

signal from the operator interface as if the remote robot has entered a structure. We have

attempted a few different apparatus concepts in the past but this one seemed to work

while accommodating even the smallest handheld operator interfaces and detachable

antennas.

Figure 12: Kam is shown operating an aerial system in the netted aviary from an

operator control unit within the new Radio Attenuation test method apparatus. It is

essentially a glove box coated with radio attenuation tiles. The signal from the robot is

shown on a remote display outside the box. The next revision will contain a camera

inside the box so the operator can see his/her own hands as well.

Media	Pointers

News	interview	with	President	of	the	Canadian	Explosives	Technicians	Association

https://www.thespec.com/videopopup/7369896?popUp=true

Local	news

http://www.chch.com/bomb-disposal-robots/

Other	media

https://motherboard.vice.com/en_us/article/these-robots-will-explore-tunnels-dug-by-lootersin-egypt
Standard Test Methods for Response Robots

ASTM International Standards Committee for Homeland Security Applications;

Response Robots (E54.09)

DHS-NIST-ASTM	E54.09	Response	Robots	Meeting	and	Exercise,	Toronto,	Canada	(v2017.2).docx Page	9

					

More	information	is	available	on	our	website: RobotTestMethods.nist.gov see	

Meetings/Exercises.

-Adam