GCR 15-995

A Measure of Domestic Security: 

Economic Benefits of NIST’s Support of Public Safety 

and Security

David P. Leech

Dakota Consulting, Inc.

This publication is available free of charge from:

http://dx.doi.org/10.6028/NIST.GCR.15-995
NIST GCR 15-995

A Measure of Domestic Security: 

Economic Benefits of NIST’s 

Support of Public Safety and 

Security

Prepared for

U.S. Department of Commerce

Special Programs Office

Standards Coordination Office 

National Institute of Standards and Technology

Gaithersburg, MD 20899-2100

By

David P. Leech

Dakota Consulting, Inc.

This publication is available free of charge from:

http://dx.doi.org/10.6028/NIST.GCR.15-995

June 2015

U.S. Department of Commerce 

Penny Pritzker, Secretary

National Institute of Standards and Technology 

Willie May, Under Secretary of Commerce for Standards and Technology and Director 
i

Acknowledgements

William Billotte, and Erik Puskar, of NIST’s Special Programs Office, and Standards 

Coordination Office respectively, selected the six case study areas examined herein. The 

following former and current NIST technical staff described the nature of programs 

involved, provided points of contact among industry and end user beneficiaries, and 

entertained numerous questions of fact and interpretation: Nelson Bryner, Bert Coursey, 

Greg Gillen, Lawrence Hudson, Nancy Lin, William MacCrehan, Elena Messina, Jayne 

Morrow, Marcela Najarro, Walter Rossiter, Michael Verkouteren, and Christopher White.

Numerous collaborators and beneficiaries from industry, other government agencies, and 

the first-responder community discussed the benefits of working with NIST scientists and 

the relevant standards development organizations. They did so, for the most part, on the 

condition of anonymity. Nevertheless, their contributions are much appreciated. 
ii

Table of Contents

Introduction .................................................................................................................................................... 1

Bomb-Resistant Trash Receptacles ................................................................................................................ 3

Background & Problem ........................................................................................................................... 3

Solution...................................................................................................................................................... 4

NIST Impacts............................................................................................................................................ 5

Personal Protective Equipment (PPE)............................................................................................................ 7

Background & Problem ........................................................................................................................... 7

Solution...................................................................................................................................................... 9 

NIST Impacts...........................................................................................................................................10

Explosives Trace Detection ...........................................................................................................................11

Setting & Problem ...................................................................................................................................11

Solution.....................................................................................................................................................13

NIST Impacts...........................................................................................................................................15

X-Ray Standards for Bulk Explosives Detection ..........................................................................................16

Background & Problem ..........................................................................................................................16

Solution.....................................................................................................................................................17

NIST Impacts...........................................................................................................................................19

White Powder................................................................................................................................................21

Setting & Problem ...................................................................................................................................21

Solution.....................................................................................................................................................24

NIST Impacts...........................................................................................................................................25

Emergency Response Robots ........................................................................................................................27

Background & Problem ..........................................................................................................................27

Solution.....................................................................................................................................................29

NIST Impacts...........................................................................................................................................30

Conclusion.....................................................................................................................................................33

Appendix A: Approach to the Selection and Development of Cases............................................................35
1

Introduction

The events of September 11, 2001 (“9/11”) punctuated, consolidated, and accelerated 

many of NIST’s long-standing concerns about technologies that help to ensure public 

safety.

1 The scope of these concerns about the performance and interoperability of public 

safety technology is wide and deep. There are many types of first-responder equipment;

sophisticated system-of-systems technologies that pervade the contemporary public 

safety environment and engage NIST’s measurement scientists and technologists in some 

of their most challenging and beneficial work. NIST’s participation in Standards 

Development Organizations (SDOs), most often early, and behind the scenes, is the 

subject of this report. Six priority areas are explored where NISTs collaboration with 

industry, universities, international partners, trade associations, responders, standard 

development organizations and Federal, State and local agencies has led to important 

enhancements of public safety:

• Blast Resistant Trash Receptacles;

• Personal Protective Equipment;

• Explosives Trace Detection;

• X-Ray Standards for Bulk Explosives Detection;

• Biothreat Agents (“White Powder”); and

• Emergency Response Robots.

These cases focus on six areas, among hundreds, where NIST’s contributions to public 

safety and security, funded substantially by the Department of Homeland Security’s 

Science and Technology Directorate (hereafter, DHS S&T), have been remarkable.2

There are many more examples, but these illustrate the mission range and impact of a 

cadre of scientists and technologists dedicated to the question, “By what measure?”3 As 

these cases show, NIST typically leverages vast public and private investments by 

coordinating the technical efforts of Federal, State, and local agencies; industries; and, 

often, nations. In doing so, NIST personnel cultivate a depth and breadth of measurement 

science expertise that are un-matched. NIST’s capabilities help to ensure that public 

security capabilities are the most effective they can be, given the state of applied 

 1 For example, not long after the establishment of the NIST’s precursor organization, the National Bureau of 

Standards (NBS), the Baltimore Fire of 1904 burned for 30 hours, destroying more than 1,500 buildings in 

the city’s business district, principally because of incompatibility of fire hydrant and fire hose threading. 

NIST was instrumental in the subsequent development of the national standard for fire hydrant and hose 

coupling diameters and threads. See, Rexmond Cochrane, Measures for Progress: A History of the National 

Bureau of Standard, U.S. Department of Commerce, U.S. Government Printing Office, 1974, pp. 84-86. 2 Appendix A contains a methodological note on the selection and development of the six cases. 

3

 The purpose of the collaboration between NIST and DHS is to provide the latter with direct access to 

expertise in measurement science and standards development, See, “Memorandum of Understanding 

between the Science and Technology Directorate, Department of Homeland Security and the National 

Institute for Standards and Technology, U.S. Department of Commerce,” 2008.
2

knowledge, the complexities of first responders’ missions, and the spatial vastness of the 

public safety mission. 

These cases will demonstrate, NIST’s unique expertise and facilities across a broad 

spectrum of security-related technologies generate economic benefits for public and 

private sector collaborators by:

• Helping other Federal agencies and industries resolve complex 

measurement-related issues;

• Bridging “first responder” and “emerging technology developer” 

communities;

• Reducing the time, often dramatically, involved in coming to a national, 

even international, consensus within the context of standards development 

organizations;

• Transferring technology to private sector developers and facilitating 

technology diffusion; and

• Reducing risks, improving quality, enhancing competitiveness, and saving 

lives.

The technologies NIST and its collaborators advance are, though often hidden, employed 

pervasively and are integral with first responder systems. The testimonies of first 

responders and the product developers that support them provide insight into the 

considerable impact that NIST is making to the effectiveness public safety personnel and 

the state of public safety.

When security matters to the nation, NIST is always there to answer the vexing question, 

“By what measure?” Having the highest confidence in the answer to that question is an 

important public investment. 
3

Bomb-Resistant Trash Receptacles

ASTM 2639-12 Standard Test Method for Blast Resistant Trash Receptacles (2012)

ASTM 2740-12 Standard Specification for Trash Receptacles Subjected to Blast Resistant Testing (2012)

ASTM E2831M-11 Standard Guide for the Deployment of Trash Receptacles in Crowded Places (2011)

Background & Problem4

Like “hidden pictures,” the technology of public security is often embedded in the 

surroundings. Blast Resistant Trash Receptacles (BRTR) is a case in point. Their 

sophistication is not detectable to the untrained eye. Wherever people routinely gather —

on subway platforms or at public events — ever-present trash receptacles have the 

potential to become instruments of terror when explosive devices are placed inside them. 

They represent what the planners at the Department of Homeland Security (DHS) call a 

“specific vulnerability within the transit environment.” 

In 2012, rather than remove these potential targets (as New York City was reported to 

have routinely done in anticipation of public events like the annual Times Square New 

Year’s Eve celebration), the city of Chicago purchased and installed BRTR that could 

limit the damage done to people and facilities if the receptacle contained explosive 

devices. BRTR were made possible when DHS S&T enlisted NIST in 2007 to quickly 

convene a set of industry participants to develop an ASTM International (formerly the 

American Society for Testing and Materials, ASTM) performance standard.

For many Federal, State, local and industry procurement officials, documentary standards 

that have been developed under a process recognized by the American National 

Standards Institute (ANSI) — like the ASTM International BRTR standards cited above 

— provide the confidence that often technically complex products will perform, “as 

advertised,” and interoperate with complementary products and services.

It will come as a surprise to many that trash receptacles can be among the technically 

complex products that public and private agents procure. Although many companies 

manufacture trash receptacles, fewer design and manufacture trash receptacles with the 

intent of keeping the public safe from terrorist bombing attacks. Among these are the 

manufactures of BRTR.

 4 Trash and Recycling Receptacles for Transit Facilities (APTA SS-SIS-WP-014-13), American Public 

Transportation Association, March 26, 2013; and Homeland Security, Science and Technology, Tech Note, 

“Blast Resistant Trash Receptacles (BRTRs), System Assessment and Validation for Emergency 

Responders (SAVER), July 2012.
4

Blasts within enclosed spaces, such as underground or enclosed subway platforms, are 

typically more damaging than explosions in open environments due to the limited paths a 

blast wave can travel. BRTRs, unlike ordinary trash receptacles should be used in such 

environments. They reduce the potential fragmentation and overpressure associated with 

the detonation of explosive devices placed in a trash receptacles. BRTRs direct the force 

of the blast upward and eliminate metal components that, if made airborne by a blast, 

would pose as a threat to bystanders. Tests conducted in 2005 by the Naval Surface 

Warfare Center, Indian Head Explosive Ordnance Disposal Technology Division (NSWC 

IHEODTD) found varying performance across vendors and even across the product lines 

of a single vendor.5

DHS S&T’s goal in calling on NIST to lead an industry consensus effort was to develop

BRTR designs that directed the potential blasts upward, thereby reducing the risks to 

people and infrastructure. Since every facility and venue has unique locational and 

functional characteristics, the associated risk levels vary widely. Times Square or a 

Pentagon subway station would be higher risk locations than, say, an ordinary suburban 

subway stop. The BRTR standard solution needed to be scalable and testable; providing 

useful information on receptacle performance and use at any threat level. 

Solution 

As a result of a NIST-led collaboration, in just three years (2008-2011) the ASTM 

International published three standards that help BRTR buyers verify manufacturers’ 

claims of the performance of their products; stipulate the tests manufacturers’ products 

must pass; and deploy BRTR in crowded places. With NIST in a technical leadership 

role, the ASTM technical committee was able to organize the wide spectrum of 

collaborators identified below.

Representative ASTM International Technical Committee Members

• Industry: Aigis Blast Protection, Inc.; Applied Engineering; American Innovations;

Centerpoint Manufacturing; Concentric Security; Mistral

• Trade Associations: American Public Transportation Association; American

Society of Mechanical Engineers;, Energetic Materials Research and Testing

Center

• Transit Agencies: Bay Area Rapid Transit; Niagara Frontier Transportation Authority; 

New York Transit authority; National Railroad Passenger Corporation (AMTRAK)

• Federal Agencies: Department of Homeland Security; Department of Defense; Federal 

Aviation Administration; U.S. Navy; National Institute of Standards and Technology

ASTM E2639-09a “Standard Test Method for Blast Resistance of Trash Receptacles,” 

provides a standard procedure for characterizing the blast resistance of trash receptacles 

when an explosive is detonated within the receptacle. ASTM E2740-10 “Standard 

Specification for Trash Receptacles Subjected to Blast Resistance Testing,” provides 

 5 DHS funded the U.S. Navy to test BRTRs as part of a DHS program (System Assessment and Validation 

for Emergency Responders (SAVER)). NSWC adapted a British test procedure to do the testing. The final 

test method that NSWC used for testing the BRTRs was the basis for developing the ASTM standard.
5

performance requirements for trash receptacles when subjected to the explosive tests 

described in E2639-09a. Thirdly, ASTM E2831/E 2831M-11 “Standard Guide for 

Deployment of Blast Resistant Trash Receptacles in Crowded Places,” provides basic 

recommendations for a variety of operational and explosive effect situations. For 

example, operational considerations account for the impact of BRTR use during the 

collection and removal of trash from a venue. Explosive considerations look at mitigation 

of explosive effects resulting from the detonation of an explosive device placed in a 

receptacle.

NIST Impacts

In the absence of a consumer guide, buyers must rely on their own expertise, or 

manufacturers’ assertions that products will meet the uses for which they are sold. When 

products don’t meet buyers’ expectations buyers are faces with unplanned expenditures 

of time and money. 

Where safety is concerned, it is worth the time it takes to investigate the suitability and 

veracity of product specifications and claims. These “transaction costs” can be 

significantly reduced if a buyer can turn to a comparative guide or the product carries a 

trusted “seal of approval.”

In the 2007-2008 time frame, in response to Department of Homeland Security (DHS) 

concerns about the safety of the nation’s transit system platforms, NIST scientists and 

engineers led an effort that, in relatively short order, led to the establishment of a

cohesive set of BRTR consensus standards that: 

• Responded to DHS S&T’s security concerns;

• Could be performance-tested to high standards of measurement; and

• Provided a much-needed aid in cost-effectively determining the optimal number and

 placement of secure receptacles for crowded places.6

The speed with which the suite of standards was developed (three standards in three 

years) no doubt saved the public time and money by reducing the time spent conducting 

product information searches, verifications, and evaluations and money spent buying too 

much or too little ineffective equipment. Also significantly reduced was the time spent by 

manufacturers and users in the often time-consuming process of standards consensusmaking. According to industry sources familiar with trash receptacle technology and the 

documentary standards development process, “without NIST’s involvement we would 

still be without a standard.” NIST’s “honest broker” role allows the freer flow of 

communication among manufacturers, and between them and their product users. NIST

possesses the well-grounded knowledge required to challenge vested interests in the 

consensus process on the basis of sound measurement science. According to one ASTM 

participant, “Passing three standards in three years from scratch is a miracle.”

 6 Trash and Recycling Receptacles for Transit Facilities (APTA SS-SIS-WP-014-13), American Public 

Transportation Association, March 26, 2013.
6

Manufacturers also gained valuable knowledge about how existing receptacle designs 

performed and what design changes would enable their products to meet the test criteria 

for improved safety. BRTR are now ubiquitous in settings where people gather, such as 

subway station platforms and large public events.
7

Personal Protective Equipment (PPE)

NFPA 1981, Standard on Open-Circuit Self-Contained 

Breathing Apparatus (SCBA) for Emergency Service

Background & Problem

It almost goes without saying that firefighting is a hazardous occupation and that the 

success of firefighting operations brings with it significant economic benefits. In 2012, 

U.S. fire departments responded to an estimated 1,375,000 fires. These fires caused 2,855 

civilian deaths and 16,500 civilian injuries. Firefighters suffered 69,400 injuries in 2012 

and 64 firefighters were fatally injured while on duty.7 Adjusted for the deaths of 340 

firefighters at the World Trade Center in 2001, the annual average of firefighter deaths 

over the past decade is 87, representing a decline over the last few decades.8 The cost of 

direct property damage associated with all fires in 2013 is estimated at $11.5 billion.9

Despite a long-term decline in average annual firefighters deaths — due in large part to 

better training and equipment — the hazards associated with firefighting appear to be 

increasing because the nature of fires confronted (the “fire ground”) has taken a turn for 

the worse. According to a veteran firefighter and Personal Protective Equipment (PPE) 

specialist, “This is not the same world that we used to fight fires in.” Firefighters across 

the nation know this, with objective certainty, because of research conducted by NIST 

and the expertise they bring to standards development organizations like the National Fire 

Protection Association (NFPA), a long-term NIST collaborator.

The most vulnerable component of a firefighter's protective gear in high-heat conditions 

is the facepiece lens of self-contained breathing apparatus (SCBA). Once a lens is 

seriously damaged or compromised, firefighters are exposed to toxic gases, respiratory 

tract injuries, and asphyxiation. 

In 2008, a National Institute for Occupational Safety and Health (NIOSH) firefighter 

fatality investigation team contacted the chairman of NFPA Technical Committee on 

Respiratory Protective Equipment to report that there appeared to have been firefighter 

fatalities in which the SCBA facepiece lens had melted. That was an unexpected 

development. The SCBA standard is reviewed every 5 years, and the latest revision had 

been published the year before. The specifications for the lens had not changed in more 

than 15 years. What had changed, however, was the intensity of the fires confronting

firefighters. As a result, the SCBA facepiece lens was the piece of equipment in which 

 7 An Overview of the U.S. Fire Problem, NFPA’s Fire Analysis and Research Division, May 2014. 8 Rita Fahy, et. al., Firefighter Fatalities in the United States-2013, National Fire Protection Association, 

June 2014.

9 Michael Karter, Fire Loss in the United States During 2013, National Fire Protection Association, 

September 2014.
8

those changes in exposure conditions showed up first, and with deadly results.10 Recent 

testing performed by Underwriters Laboratories, comparing fires in legacy homes versus 

modern homes have confirmed that today’s firefighters are dealing with fuels, heat, 

flashover, and chemicals that they have not experienced historically.

In light of these developments, NIOSH, NIST, and the NFPA Fire Protection Research 

Foundation (FPRF) jointly hosted a research needs workshop to identify thermal 

degradation issues and prioritize research needed to address the performance of SCBA 

facepiece lenses.11 As a result, NIST set out to determine how temperature and heat-flow 

affects standard firefighter breathing equipment and possibly contribute to first-responder 

fatalities and injuries. They also conducted furnished-house fire experiments in 

abandoned townhouses in an effort to improve the match between the actual fire ground 

conditions and requirements to be incorporated in standards.12

In order to sell their products to fire departments, SCBA manufacturers must obtain 

NIOSH certification to sell their products to firefighters. In addition, independent 

laboratories test SCBAs to verify their compliance with relevant NFPA standards. When 

tested to realistic fire conditions SCBA facepiece lenses manufactured by five different 

companies were all damaged. Yet all the lenses tested were NFPA compliant and met the 

requirements of heat and flame tests specified in the existing standards.13 At this time, the 

existing standard specified that SCBAs be exposed to 95o C (203o F) oven for 15 minutes 

followed by exposure to direct flame for 10 seconds.14 The research concluded that the 

relatively low temperature of the oven test and short duration of the direct flame exposure 

did not appear to adequately reproduce temperature and flame exposure experienced by 

today’s fire fighters.

DHS S&T funded the “live burn” studies and laboratory tests conducted by NIST. NIST 

modeled the fire and its effect on the SCBA facepiece lenses and conducted a series of 

well controlled experiments, both small- and full-scale, in NIST laboratories. The 

experimental results demonstrated a range of exposure conditions under which a radiant 

thermal flux would cause SCBA facepiece lenses to craze, bubble, soften, and deform. 

Using temperature and thermal flux data collect from lab- and field-scale burns, the NIST 

team developed new radiant exposure tests and performance criteria that are more 

representative of the high temperature and radiant exposure that fire fighters actually 

experience. On the basis of this research, SCBA test methods and performance criteria 

were presented to the NFPA Technical Committee on Respiratory Protection Equipment.

 10 “A Preview of the New Edition of NFPA 1981 SCBA Standard” (Interview with Dan Rossos, NFPA), 

FireRescue, November 2012. http://www.firefighternation.com/article/firefighter-safety/preview-newedition-nfpa-1981-scba-standard 11 See, Amy Mensch and Nelson Bryner, Emergency First Responder Respirator Thermal Characteristics: 

Workshop Proceedings, NIST Special Publication 1123, NIST, June 2011. 12 “Study Finds Failure Points in Firefighter Protective Equipment,” NIST Tech Beat, December 6, 2011. 13 NFPA 1981, Standard on Open-Circuit Self-Contained Breathing Apparatus (SCBA) for Emergency 

Services, 2007 Edition. 14 Ibid.
9

Solution

NFPA incorporated the new test methods and performance criteria for facepiece lenses 

into the 2013 edition of NFPA 1981, Standard on Open-Circuit Self-Contained Breathing 

Apparatus (SCBA) for Emergency Services. In addition, the NFPA issued a Safety 

Awareness Notice to over 32,000 fire departments in the United States warning that 

SCBA facepiece lenses may deform or degrade in severe firefighting conditions. The new 

tests were designed to enable firefighters to gain an increased ability to survive in 

extreme conditions, such as when escaping from a flashover.

In order to comply with the new standard, all SCBA facepiece lenses for the nation’s fire 

fighters will be replaced with improved lenses that perform at a level required by NFPA’s 

newly developed standard test method. New and better performing lens are now available 

and will be phased into all fire departments in the United States at an estimated cost of 

more than $750 million to enhance the safety of over one million fire fighters. NFPA 

1981-compliant SCBA are included on the DHS “Authorized Equipment List.”

Numerous DHS grant programs help firefighters and other first responders obtain 

critically needed and authorized equipment, protective gear, emergency vehicles, 

training, and other resources to protect the public and emergency personnel from fire and 

related hazards.15

Representative NFPA Technical Committee Members

• Industry: BPI Performance Products; Intertek Testing Services; Globe Manufacturing 

Company; Goodyear; International Personal Protection, Inc.; HIP Consulting, LLC.; Michael 

McKenna & Assoc.; Safety Equipment Institute; Tyco/Scott Safety Fire & Industrial (P.P.E.) 

Ltd.

• Trade Associations: Fire & Emergency Manufacturers and Services Assoc.; International

Association of Fie Fighters; International Safety Equipment Association; National Fire 

Protection Association; International Fire Service Training Assoc.

• Firefighters: Northborough Fire, MA; Columbus (OH) Firefighters Union; Addison Fire 

Department (TX)

• Federal Agencies: Department of Agriculture; Department of Homeland Security; Fire 

Administration; Lawrence Livermore National Laboratory; National Institute of Standards 

and Technology; National Institute for Occupational Safety and Health; U.S. Navy 

The 2013 edition of NFPA 1981 establishes the minimum respiratory protection and 

functional requirements for SCBAs used by emergency services workers. It applies to 

SCBAs used during firefighting, rescue operations, hazardous materials and terrorist 

incidents, and similar operations where responders may encounter confined spaces, 

unknown atmospheres, atmospheres that could become oxygen deficient, or that present 

imminent danger to the life and health of the firefighter.

 15 http://www.fema.gov/media-library-data/20130726-1825-25045-7138/fema_preparedness_grants_authorized_equipment_list.pdf
10

NIST Impacts

An experienced first-responder advocate observes that NIST research on the causes of 

SCBA facepiece lens failures has quite literally changed the face of critical firefighter 

equipment as well as what firefighters are taught (and trained) concerning the true 

capability of their protective equipment. As a result, injuries and deaths among 

firefighters due to these malfunctions have declined. A senior executive of the 

International Association of Fire Fighters believes that safety and effectiveness of the 

nation’s fire fighters has been radically improved because of NIST ‘s research on the 

thermal performance of SCBA facepiece lenses. Informed senior scientists believe that 

the search for the cause of the degraded and deformed SCBA facepiece lens would still 

be ongoing if it had not been for NIST’s unique fire research capabilities. The 

improvements in safety and effectiveness that have resulted from the new technology 

were the direct result of the NIOSH—NIST—NFPA collaboration.

Experts think there is much more to come. An unintended consequence of the NIST 

research on SCBA facepiece lenses was that new dimensions of the fire-fighting 

environment were discovered. According to one expert, “NIST’s involvement was a 

watershed event and a huge leap forward to better understanding of the investigative 

findings we were seeing in the field.” 

Getting the results of their research transferred and diffused is another dimension of 

NIST’s mission. According to seasoned first-responder advocates, for decades NIST’s 

fire research reports have been changing how decision makers set policy at the local level 

and how industry standards are written. It is widely believed that NIST’s neutrality and 

credibility alone drive change post-research; that other organizations producing similar 

work would not have enjoyed the same level of credibility for implementation. Any such 

delay would have resulted in more firefighter injuries and deaths, civilian injuries and 

deaths, and property loss. 

The benefits to the manufacturers that support first responders are also significant. 

Sounding a familiar refrain, NFPA technical committee members believe that their 

standards would have taken many more years to complete without NIST’s assistance. 

One industry representative estimated that his company had collectively dedicated 

thousands of hours to this and closely related standards efforts. Without NIST’s expertise 

in the process, that number would have been considerably higher. NIST efforts to test 

various materials, designs, and models of equipment have contributed to new products 

and to the industry standards that regulate firefighter safety in the field. Without NIST 

involvement, technical committee members believe that alternative standards would have 

included ineffective, and therefore more costly, tests. Finally, SCBA manufacturers 

believe that collaborating with NIST assures them that equipment tests are robust, 

repeatable, and technology neutral. Industry representatives estimate that product 

development cycles are shortened as a result of collaboration with NIST.
11

Explosives Trace Detection

ASTM E2520-07 Standard Practice for Verifying 

Minimum Acceptable Performance of Trace Explosives Detectors

NIST’s precision inkjet-produced test materials

Standardized Methods for Swiping

Setting & Problem

A natural gas leak in our home is detectable because gas suppliers have added the 

noxious scent to their products as an olfactory warning that harmful vapors are in the air. 

These additives help secure our homes and families from potentially deadly explosions. 

Securing airlines, airports, and other public places from terrorist bombers is not so easy. 

NIST’s Explosives Program provides the Department of Homeland Security (DHS) with 

trace explosives detection metrology resources and expertise not available anywhere else 

in the world. Explosives are the weapons of choice for terrorists in both civilian and 

military settings. Fortunately for the law enforcement community, the manufacture and 

transport of explosive devices leave detectable microscopic traces that can help identify 

concealed devices, their carriers, and their makers. 

In the wake of the September 11, 2001 terrorist attacks, Congress passed the Aviation and 

Transportation Security Act (ATSA) that established the Transportation Security 

Administration (TSA) within the DHS. TSA is responsible for identifying the 

requirements for new checked-baggage screening technologies; operationally testing and 

evaluating new screening equipment in airports; and procuring, deploying, and 

maintaining equipment that embodies new technologies. TSA shares responsibilities for 

the research and development of checked-baggage-screening technologies with DHS 

S&T and its Transportation Security Laboratory (TSL). TSL tests and evaluates 

technologies under development against TSA-established detection requirements.16

Following the 2004 report of the National Commission on Terrorist Attacks Upon the 

United States (the “9/11 Commission”), Congress, and the TSA, gave priority attention to 

screening airline passengers for explosives.17 Provisions to improve checkpoint 

technologies to detect explosives were included in the Intelligence Reform and Terrorism 

Prevention Act of 2004.18 In close collaboration with TSL, NIST has been involved since 

2003 in DHS-funded research to support standards and measurement needs for trace 

explosives screening and to support the development of next generation detection 

 16 TSA Has Enhanced Its Explosives Detection Requirements for Checked Baggage, but Additional 

Screening Actions Are Needed, USGAO, July 2011, (GAO-11-740). 17 Dana Shea and Daniel Morgan, Detection of Explosives on Airline Passengers: Recommendation of the 

9/11 Commission and Related Issues, Congressional Research Service, August 9, 2006. 18 Bart Elias, Airport Passenger Screening: Background and Issues for Congress, Congressional Research 

Service, April 23, 2009.
12

technologies. NIST has acquired an extensive collection of deployed explosives trace 

detection (ETD) systems and has developed unique measurement capabilities and 

standard test materials.19

By 2004, more than 10,000 ion mobility spectrometry (IMS)-based explosive detectors 

were already deployed at airports worldwide and by the military. Today, it is estimated 

that there are more than 15,000 ETDs that are in service.20 Standards for their 

performance, for evaluation and certification, regulations for use, and screening 

procedures for trace detection on luggage all were developed by the TSL. 

In approximately 2005, NIST was tasked by DHS with developing a supporting 

metrology infrastructure to evaluate the reliability and effectiveness of these ETDs, 

including fundamental research and the production of trace explosives standards to 

evaluate detector response. The primary means of calibrating IMS detectors is with 

vendor-supplied materials. Generally, the vendor-supplied materials were not sensitive to 

the amount of explosive available in the sample, only to its presence or absence.

An important facet of NIST’s efforts to support the TSL was the development of 

metrology tools for verifying ETD performance, including Standard Reference Materials 

(SRMs). Sponsored by DHS S&T, and working through the ASTM International’s 

Subcommittee on Chemical, Biological, Radiological, Nuclear, Explosives (CBRNE) 

Sensors and Detectors (E54.01), NIST had a technical leadership role in developing the 

measurement technology needed to evaluate commonly-deployed swipe detection 

systems: ASTM E2520-07 Standard Practice for Verifying Minimum Acceptable 

Performance of Trace Explosive Detectors.

 

ASTM E2520-07 stipulates a simple test, designed to be used by first responders that 

only tests the response of the detector to pure analyte deposited from solution.21 The 

residue samples swiped in the field differ considerably from the bulk materials used in 

most vendor-supplied calibrants. To test the effectiveness and reliability of ETDs, 

therefore, test materials that simulate the small amounts of explosive residue that are 

likely to be available in field samples is required. 

To support field calibrations, NIST also adopted and developed a capability to produce 

ETD test materials using inkjet printing technology whereby drop-on-demand inkjets 

 19 Bert Coursey, “Passenger Screening R&D: Responding to President Obama’s Call to Develop and 

Deploy the Next Generation of Screening Technologies” (Testimony before the U.S. House of 

Representatives, Committee on Science and Technology, Subcommittee on Technology and Innovation), 

February 3, 2010.

20 The number of ETD in service is estimated by G. W. Cook, et. al., “Using Gas Chromatography with Ion 

Mobility Spectrometry to Resolve Explosive Compounds in the Presence of Interferents, J. Forensic Sci., 

2010, 55, 1582–1591. That estimate is cited by J. Verkouteren, et. al., “Performance Metrics Based on 

Signal Intensity for Ion Mobility Spectrometry-based Explosive Trace Detectors Using Inkjet Printed 

Materials,” Analyst, 2014. 21 William A. MacCrehan, “A NIST Standard Reference Material (SRM) to Support the Detection of Trace 

Explosives,” Analytical Chemistry, July 28, 2009.
13

deposit precisely-measured and precisely-placed explosives solutions on test materials.22

To be useful to “front line” users, such test material must be safe to handle and transport, 

store well, and function uniformly. NIST SRMs, as well as field deployed “inkjet printed 

test materials,” were designed to meet these criteria.23 NIST reference and test materials 

complement those provided by the vendor by giving more information regarding the 

performance of the instruments. 

The efficiency and standardization of trace explosives sampling in the field has also been 

an important area of NIST’s support to DHS S&T and TSA.24 How hard to swipe and the 

best surfaces to swipe are not trivial technical issues. The physical forces underlying the 

effectiveness of passenger screening for explosives are the same as those affecting 

particle adhesion and removal for semiconductor manufacturing. But there are many 

complexities in field swiping that render those better-studied models inadequate. In 

contrast to the smooth, flat surfaces of semiconductor wafers, for example, the surfaces of 

the exteriors and interiors of carry-on luggage, laptop computers, and cosmetics bags, 

etc., are far from uniform. The swipe material “traps” used in collection devices are 

typically made of various cloths, papers, and fiberglass. The particles collected are likely 

to be non-spherical and contain body oils. All these change the contact area of the traps in 

use and their adhesion forces acting on the explosive traces in the sample.25

Solution

NIST personnel played a pivotal role in the ASTM International technical committee that 

created ASTM E2520-07. NIST continues to develop the basis for even broader protocols 

covering a wider range of explosives. Future standards will allow a ‘scoring’ protocol for 

ETDs that accounts for their detection sensitivity and the sensitivity range of explosives 

they can detect. 

ASTM International CBRNE Sensors and Detectors Subcommittee Members

• Industry/University: Airsense Analytical; Bahia Corp.; GE-Ion Track; Smiths Detection; 

CCDEX Inc.; Penn State University; Svagen Inc.

• First Responders: Maryland Fire & Rescue Institute-Terrorism Response; Montgomery 

County, Maryland, Bomb Squad; Virginia State Police; Sacramento County (CA) Sheriff’s 

Dept. Supervisor, Bomb Squad; Huston Police Bomb Squad; U. S. Capitol Police

• Federal Agencies: Bureau of Alcohol, Tobacco, Firearms and Explosives; Federal Drug 

Administration /Department of Health and Human Services-Physical Security; Sandia 

National Laboratories; Transportation Security Administration; U.S. Air Force; National 

Institute of Standards and Technology, Department of the Interior, Law Enforcement 

Department; U.S. Postal Service-Forensic Lab NIH-Physical Security; Coast Guard; Secret 

Service

• International: Defence Science & Technology Lab, Detection Department,

 22 Verkouteren, 2014, op. cit. 23 Ibid. 24 Greg Gillen, et. al., “Measurement Infrastructure for Calibration, Standardization and Optimization of 

Trace Contraband Detection Systems,” DHS Standards Division Annual Review, August 20, 2014.

25 Verkouteren, 2008, op. cit. 
14

 U.K.; Explosives Detection, Home Office Police Scientific Development

 Branch, U.K

ASTM E2520-07 is primarily intended to assist first responders and security screeners in 

verifying the minimum acceptable performance of IMS detectors used to identify traces 

of high explosives such as cyclotrimethylene trinitramine (RDX), pentaerythritol 

tetranitrate (PETN), and trinitrotoluene (TNT). The standard entails procedures intended 

to evaluate the response of ETDs to evaporated residues of low-concentration solutions of 

explosive compounds placed on test swipes. These solutions are available in the form of 

NIST SRM 2906.

SRM 2905 Trace Particulate Explosives materials simulate the residues that remain on 

the hands after fabricating an explosive device. One material is a common plastic 

explosive (composition C-4). The other material is a common military explosive (TNT). 

The size of the particles used is consistent with the size of the residue particles left on the 

hands and detected by swipe-type ETD. 

SRM 2906 Trace Explosives Calibration Solutions contains three solutions of trace 

explosives for TNT, PETN, and RDX (the active ingredients in military explosives like 

C-4 and Semtex) and container of solvent without explosive (a blank). They are used to 

test ETDs. ASTM International E2520-07 specifies the concentrations of explosives 

provided in SRM 2906. 

SRM 2907 Trace Terrorist Explosives Simulants, like 2905, provides residue simulants, 

but those in 2907 are intended to detect the explosive, triacetone triperoxide (TATP), 

used as the detonator in many terrorist improvised explosives devices, and the widely 

available plastic explosive Semtex 1A (certified for pentaerythritol tetranitrate, PETN). 

These materials provide accurately known amounts of the explosives to optimize swipe 

collection and trace explosives detector responses in a form that is safe to handle. 

In addition to its SRM-related activities, other elements of NIST’s Explosives Program 

are working with DHS and TSA to further develop and distribute NIST’s precision 

inkjet-produced test materials to U.S. agencies. Plans have been developed for

transitioning the production of inkjet printed materials to agencies or companies outside 

of NIST to meet future demands for wide-scale quality control of ETD.26

NIST researchers have also developed methods to determine optimized swiping 

procedures; developed standardized methods for swiping; and led the development of a 

draft ASTM swiping standard.27

 26 Ibid. 27 Gillen, op. cit.
15

NIST Impacts

Thousands of ETDs have been purchased and placed in service across the nation. 

Standard Reference Materials (SRMs), and the widely deployed inkjet test materials 

developed by NIST, improve the effectiveness of deployed ETD by assuring optimal 

sensitivity and lower false alarm rates. To the extent that this improved effectiveness 

reduces screening delays, the value of the queuing time saved by millions — at airport 

security checkpoints, marine terminals, loading docks, and building entrances — is surely 

considerable. The same detection technology and sampling is also being used for illicit 

drug detection in prisons and will have additional applications, and efficiencies, for 

border security and customs.28

In support of DHS S&T and the TSA, NIST scientists played a leadership role in the 

creation of standard practices for verifying the performance of ETDs and they continue to 

develop the technical basis for future revisions. By facilitating the ASTM documentary 

standards development process, and developing related SRMs, NIST supported the 

optimization of instrument design and validation and probably reduced the ASTM’s 

consensus formation time. Going forward, these efforts will promote the evaluation of 

ETD performance prior to procurement as well as throughout the lifetime of these 

devices. The associated efficiencies are likely to be considerable.

Collaborating with NIST’s explosives program has also spurred innovation among ETD 

manufacturers, leading to the creation of products (test “swipes”) that are primarily 

targeted at calibrating ETDs. One small company estimates that it might have taken them 

close to $1 million to develop the ETD consumables that NIST routinely makes available 

to other Federal agencies. In the unlikely event that this small company was able to make 

that investment, they say that the rigorously effective approach to the problem 

implemented by NIST would probably never have occurred to them. Another 

manufacturer claims that collaborating with NIST researchers significantly reduced time 

to market for its products. And NIST’s rigor reduced the number of costly failures that 

would otherwise certainly have occurred on the product development path. A large ETD 

manufacturer has used SRMs to quantitatively evaluate their swiping materials. Some 

ETD manufacturers certify their detectors against ASTM E2520-07 and SRMs.

 28 Source: Jenifer Verkouteren, et. al., “A Method to Determine Collection Efficiency of Particles by Swipe 

Sampling,” Measurement Science and Technology, Vol. 19 (2008).
16

X-Ray Standards for Bulk Explosives Detection

IEEE/ANSI N42.44 – 2008: American National Standard for the Performance of

Checkpoint Cabinet X-Ray Imaging Security Systems

IEEE/ANSI N42.45-2011: American National Standard for Evaluating the Image Quality of

X-Ray Computed Tomography (CT) Security-Screening Systems

IEEE/ANSI N42.46-2008: American National Standard for Measuring the Imaging Performance of

X-Ray and Gamma-Ray Systems for Cargo and Vehicle Security Screening

IEEE/ANSI N42.47-2010: American National Standard for Measuring the Imaging Performance of

X-Ray and Gamma-Ray Systems for Security Screening of Humans

Background & Problem

Following the terrorist attacks of September 11, 2001, the U.S. government, and its 

international allies, launched a series of efforts to up-grade their most vulnerable 

infrastructural systems. Not the least of these was the air transportation system. 

The nation’s air, land, and marine transportation systems are designed for accessibility 

and efficiency, two characteristics that make them highly vulnerable to terrorist attack. 

Since the system of air travel is an international system whose nodes extend beyond our 

borders into the geographical space of other nations, security concerns and actions by one 

nation can result in significant costs and benefits for other nations. 

In 2001 there were no comprehensive standards for the technical performance of x-ray or 

gamma-ray security-screening equipment, and, with the increasing focus on using such 

technologies for homeland-security applications, there was obvious need for criteria 

against which the performance of these technologies could be evaluated.

Requirements for new checked-baggage-screening technologies; operationally testing and 

evaluating technologies in airports; and procuring, deploying, and maintaining 

technologies are all TSA responsibilities under the Aviation and Transportation Security 

Act of 2001. TSA and TSL share responsibilities for the research and development of 

baggage- and cargo-screening technologies.

29

Under its mandate TSA certifies the x-ray screening equipment it deploys to commercial 

airports for screening passengers, their carry-on luggage and checked baggage. These 

equipment certifications are based on tests performed by the TSL. TSA certifies that 

screening equipment can detect the amounts, configurations, and types of explosive 

material that would be likely to be used to cause catastrophic damage to an aircraft. 

Periodically, the requirements for screening systems are reviewed in consultation with 

experts from outside TSA.30 TSA requires that screening equipment undergo three types 

of testing: certification testing, integration testing, and operational testing.

The U.S. x-ray screening infrastructure has two facets: air travel screening and air cargo 

screening. Regarding air travel screening, TSA deploys explosives detection systems 

 29 TSA Has Enhanced Its Explosives Detection Requirements for Checked Baggage, but Additional 

Screening Actions Are Needed, USGAO, July 2011, (GAO-11-740). 30 Ibid.
17

(EDS) and explosives trace detection (ETD) machines to screen all checked baggage 

transported by U.S. and foreign air carriers departing from U.S. commercial airports. 

There are over 18,000 airports in the U.S. Passenger and carry-on luggage screening is 

conducted at more than 450 airports (designated as “primary” airports), and checked 

baggage is screened at approximately 460 commercial airports throughout the United 

States. According to one projection, TSA will likely be screening over one billion people 

annually by 2024.31

Regarding air cargo screening, TSA’s Certified Cargo Screening Program (CCSP) strives 

to minimize logistical complexities by allowing screening to occur at factories, 

warehouses, third party logistics providers, and off-airport cargo consolidation facilities. 

TSA approves the screening procedures of applicants as well as supply chain security 

measures to prevent tampering with shipments once they have been screened. TSA audits 

participants’ performance. TSA has regulatory oversight with regard to air cargo security 

matters of about 4,400 freight forwarders, about 300 air carriers, and more than 1,000 

facilities that are participating in the CCSP. TSA has hundreds of transportation security 

inspectors overseeing the air cargo sector, including international cargo transportation 

security inspectors deployed to field offices in Los Angeles, Dallas-Fort Worth, Miami, 

and Frankfurt, Germany to examine cargo operations at the last points of departure to the 

United States. TSA also has international industrial representatives who work with 

hundreds of foreign passenger and all-cargo air carriers that operate flights to the United 

States. 

Solution

In 2005, NIST and DHS launched an effort to develop a suite of national voluntary 

consensus standards working through the Institute of Electrical and Electronics Engineers 

(IEEE). Collaborators from industry, government, and academia are identified below. 

IEEE/ANSI Aviation Bulk Explosives Detection Working Group Participants

• Industry: American Science & Engineering, Inc.; Analogic; Astrophysics Corp.; Battelle 

Memorial Institute; Control Screening; ENSCO Inc.; GE Security; HighCom Security; L3Comm; Morpho Detection Inc.; Rapisca Systems Inc.; SAIC; Secure Path; Smiths 

Detection; SRA International; SureScan Corp.; Tek84 Inc.; Underwriter’s Laboratory; Valley 

Forge Imaging; Varian

• Government Agencies: Canadian Nuclear Safety Commission; Center for Devices and 

Radiological Health; Domestic Nuclear Detection Office; Federal Bureau of Prisons;

Hazardous Devices School (National Bomb Squad Commanders Advisory Board); National 

Institute of Justice; National Institute of Standards and Technology; Savannah River National 

Laboratory; Florida State; New York State; Thunder Mountain Evaluation Center 

(Department of Defense); Transport Canada; Transportation Security Laboratory;

Transportation Systems Integration Facility; Customs and Border Protection; Secret Service

• Standards Development Organizations: American National Standards Institute; ASTM 

International; Institute of Electrical and Electronics Engineers; International Electrotechnical 

 31 Bart Elias, Airport Passenger Screening: Background and Issues for Congress, Congressional Research 

Service, April 23, 2009.
18

Commission

NIST and its collaborators developed a suite of standards that span the use of x-rays and 

gamma rays in the screening of carried items and human subjects at airline checkpoints, 

airline checked baggage, air cargo, and other venues, as well as the associated radiation 

safety concerns.32,33

These standards have evolved as TSA requirements have evolved. In the case of x-ray 

imaging applications, the process has produced x-ray image standard artifacts that gauge 

resolution, useful penetration, and materials discrimination among other attributes. Welldefined test methods and well-specified test objects, test & evaluation procedures allow 

the comparison of x-ray screening system models and allow “front lines” x-ray screening 

system users to assess the relative strength and weaknesses of various manufacturers’ 

equipment and to assess that equipment, in use, over time as systems age or are upgraded. 

To support the screening of carry-on baggage in airports, DHS and NIST worked with 

industry to develop IEEE IEEE/ANSI N42.44, American National Standard for the 

Performance of Checkpoint Cabinet X-Ray Imaging Security Systems. This standard uses 

the method of ASTM F792-08: Standard Practice for Evaluating the Imaging 

Performance of Security X-Ray Systems and establishes minimum performance 

requirements for resolution, useful penetration, and materials discrimination. 

Each piece of luggage undergoes inspection using the multi-view Computed Tomography

(CT) technique that sends data to automated explosives-detection algorithms that produce 

and analyze a three-dimensional image of the luggage contents. IEEE/ANSI N42.45-

2011: American National Standard for Evaluating the Image Quality of X-Ray Computed 

Tomography (CT) Security-Screening Systems is limited to test artifacts and test methods 

developed by NIST. It does not establish acceptable test results that are considered 

sensitive for reasons of national security. 

IEEE/ANSI N42.46-2008: American National Standard for Measuring the Imaging 

Performance of X-Ray and Gamma-Ray Systems for Cargo and Vehicle Security 

Screening, provides a common test method to consistently compare the performance of 

inspection systems.

To address the safety and effectiveness of Advanced Imaging Technology (AIT) systems, 

the development of two related standards was facilitated by NIST. IEEE/ANSI N42.47-

2010: American National Standard for Measuring the Imaging Performance of X-Ray 

 32 Larry Hudson, Steve Seltzer, Paul Bergstrom, and Frank Cerra, “In God We Trust, X-Ray Everything 

Else! Standards for X-Ray and Gamma-Ray Security Screening Systems,” DSP JOURNAL, 

July/December, 2007.

33 In the 2002-2006 period, NIST was also intensively engaged with DHS in the “fast-track” development 

of radiation instrumentation standards through IEEE/ANSI’s N42 National Committee on Radiation 

Instrumentation See, Erik Puskar and David Leech, “Bottom-Line Impact: the Economic Value of 

Documentary Standards,” ISO Focus+, Vol. 1, No. 6, June 2010, ISO Central Secretariat; and Erik Puskar 

Selected Impacts of Documentary Standards Supported by NIST 2008 Edition (NISTIR 7548), National 

Institute for Standards and Technology, January 2009, pp. 23-28.
19

and Gamma-Ray Systems for Security Screening of Humans, provides standard methods 

for measuring and reporting imaging quality characteristics and establishes minimally 

acceptable performance requirements for security-screening systems used to inspect 

people who are not inside vehicles, containers, or enclosures. 

The proliferation of security products that use ionizing radiation has exposed gaps in the 

national and international standards where security screening is being applied. Current 

trends that are informing the development of second- and third-generation standards 

include efforts to harmonize standards internationally, the testing of automated-target 

recognition algorithms, designing test objects that may be scored objectively, using the 

digital imagery that is produced by today’s (post-film) technologies, and designing 

standards that reward only technical improvements that contribute to task performance.

NIST Impacts

There are three broad categories of potential beneficiaries of x-ray standards for bulkexplosives detection in air transportation venues:

• End users of air transportation services

• Public and private sector buyers of sophisticated x-ray screening equipment 

• Manufacturers of x-ray screening equipment.

The economic benefits that accrue to these beneficiaries are estimated to be in the 

hundreds of millions of dollars.34

Enhanced security has value to airline passengers. Given greater security, we would 

expect them to place a higher value on airline services. Passengers get value above and 

beyond the fare actually paid. A knowledgeable airline association representative 

estimates that consumers receive approximately 5 percent more value than they actually 

pay for. The economic benefits of air transportation security also accrue to indirect 

beneficiaries whose lives and businesses are more secure because air transportation is 

more secure (for example, occupants of other potential terrorist targets, such as high rise 

buildings, nuclear power plants, and government buildings, and their families).

Public and private sector buyers of sophisticated x-ray screening equipment also benefit 

from the development and promulgation of consensus security standards. There are at 

least two kinds of economic impacts for procurement officials. When equipment 

manufacturers comply with standards, the “search costs” required to identify reliable 

suppliers, and the “transaction costs” of specifying and assessing contract performance, 

are reduced. From a TSA procurement perspective, a reduction of the number of design 

versions as a result of common requirements, common measurement language, and 

common configuration controls leads to significant procurement efficiencies. Moreover, 

the image quality standard, alone, has enabled TSA to significantly reduce the number 

 34 David Leech, The Economic Benefits of NIST’s Role in Security Standards Development: X-Ray 

Standards for Bulk- Explosives Detection (GCR 12-967), National Institute of Standards and Technology, 

March 2012.
20

and duration of laboratory compliance tests involving live explosives. This, in turn, has 

enabled the agency to keep pace with industry’s explosives detection systems 

improvements even as maintain pace with technology refreshment of deployed 

Explosives Detection Systems. The end result has result has been significant savings in 

cost, labor, and reduced risk to personnel.35

There are also likely to be downward pressure on prices paid by procurement agencies if 

suppliers of x-ray screening equipment make the investment to comply with standards 

and subsequently bid on contracts (“enter the market”).36 A few of the x-ray screening 

equipment manufacturers estimate that competitive pressures reduced the prices buyers 

would otherwise pay by 20 percent. If x-ray security standards contributed a small 

fraction to public and private procurement agents’ ability to assess the comparative value 

of competing x-ray screening equipment vendors on an “apples to apples” basis, and 

resulted in the entry of competing firms, the economic value of that contribution would 

be substantial. 

Finally, x-ray security standards reduce the development, testing, and compliance cost of 

manufacturing sophisticated x-ray screening equipment. A few representatives of 

screening equipment manufacturers estimated that, on average, their machines would be 

40 percent more costly in the absence of consensus standards. With more than 6,000 such 

machines in public and private inventories, the “front-end” cost savings resulting from xray equipment standards has been significant. 

 35 “New X-Ray Standards Help the World Screen for Explosives,” Office of Standards, DHS Science and 

Technology Directorate, April 3, 2013.

36 Past studies have shown, as a general matter, across industries, that the market entry of a third supplier of 

sufficient size (> 16 % market share) has the effect of reducing industry profit margins by 13-14%. See, 

John Kwoka, “The effects of Market Share Distribution on Industry Performance,” Review of Economics 

and Statistics, Volume 61, February 1979, pp. 101-109;Willard Mueller and Douglas Greer, “The Effect of 

Market Share on Industry Performance Reexamined,” Review of Economics and Statistics, Volume 66, 

May 1984, pp. 353-358; John Kwoka, “The effects of Market Share Distribution on Industry Performance: 

Reply,” Review of Economics and Statistics, Volume 66, May 1984, pp. 358-361. For more general 

theoretical and empirical evidence that “entry” reduces prices, see F.M. Scherer and David Ross, Industrial 

Market Structure and Economic Performance, Houghton Mifflin Company, 1990.
21

White Powder

ASTM E2770-10 

Standard Guide for Operational Guidelines for Initial Response to a Suspected Biothreat Agent

ASTM 2458-10 

Standard Practices for Bulk Sample Collection of Visible Powders Suspected of Being Biothreat Agents

Setting & Problem

In the lingo of CBRNE first-responders, “white powder” is the euphemism for a 

suspected biological agent of disease and death. Federal, State, and local first responders 

in the United States routinely respond to reports of suspicious substances. Since the 2001 

anthrax attacks there have been more than 30,000 such events reported.37 The efficiency 

and effectiveness of law enforcements’ responses since 2001 have improved, thanks in 

part to NIST’s role in the development of national standards. 

In September and October of 2001, letters laced with anthrax spores (Bacillus anthracis) 

were sent to two U.S. senators and to members of the media. The distribution vehicle: the 

U.S. postal system. The letters led to the first U.S. cases of anthrax disease related to 

bioterrorism. Twenty-two people, in four states and Washington, D.C., contracted 

anthrax. Five died.38

Following the 2001 anthrax attacks the surge in biothreat samples was too great for 

emergency responders and the laboratories to handle. At the time there were no validated 

methods or a coordinated concept of operations to aid in sample collection.39 Each 

biothreat event required a law enforcement investigation, sample collection by a qualified 

hazardous materials (HAZMAT) team, and laboratory analysis to clear the suspected 

materials. 

NIST had been working with the Department of Defense (DOD) and others to develop 

characterization tools to improve the sensitivity and reliability of bioagent and toxin

detection. Previous collaborations with NIST and the U.S. Army Dugway (Utah) Proving 

Ground had developed reliable methods based on DNA typing techniques to assess the 

concentration and viability of anthrax spores. The techniques and data developed through 

this collaboration have been important steps in developing a reliable reference standard 

 37 “New Guidance Issued for First Responders Collecting Suspected Biothreat Agents” NIST, November 

24, 2010. http://www.nist.gov/mml/bbd/powders_112310.cfm

38 Agencies Need to Validate Sampling Activities in Order to Increase Confidence in Negative Results, U.S. 

General Accountability Office (GAO), Apr 5, 2005.

39 Bryon Marsh, Implementing the National Framework for a Biothreat Field Response Mission Capability, 

Naval Postgraduate School, September 2013.
22

for anthrax detection and decontamination.40

NIST began interacting with DHS about these matters early in 2003 and has been actively 

supported by DHS S&T since 2004. In 2005, culminating numerous related inquiries, a 

Government Accountability Office (GAO) investigation of the 2001 anthrax attacks 

concluded that federal agencies were not well-prepared to deal with anthrax released 

through the mail; that the response activities lacked validation and that their sampling 

strategies were limited, leading to unreliable results.41

The federal agencies involved in sampling the postal facilities in 2001 to detect anthrax 

— the Centers for Disease Control (CDC), the Environmental Protection Agency (EPA), 

and the United States Postal Service (USPS) — had to develop a sampling strategy, 

collect samples, transport the samples, extract particles from samples, and analyze the 

samples. There was considerable variation within and between the agencies and within 

the overall process.42 The GAO concluded that:

Validating the overall process is important because operational and health related 

decisions are made on the basis of testing results generated by that process. In 

addition, validation would offer assurance that the results of using a particular 

method, which is part of that process, are robust enough to be reproduced, 

regardless of which agency, contractor, or laboratory is involved. Thus, agencies 

and the public could be reasonably confident that any test results generated by a 

process that includes that method would be reliable and, in particular, that any 

negative results would mean that a sample was free from contamination (within 

the method’s limits of detection).43

While these results called for a more effective response system overall, they did not 

specifically address the primary concerns of first responders: the on-site biological 

assessment and threat evaluation process. 

In 2007, with DHS S&T funding, the Association of Analytical Communities (AOAC)

 40 Ibid. 41 GAO, op. cit. The 2005 reports cited other GAO investigations, including: GAO, U.S. Postal Service: 

Issues Associated with Anthrax Testing at the Wallingford Facility, GAO-03-787T (Washington D.C., May 

19, 2003). See also GAO, U.S. Postal Service: Better Guidance Is Needed to Improve Communication 

Should Anthrax Contamination Occur in the Future, GAO-03-316 (Washington D.C., Apr. 7, 2003), and 

U.S. Postal Service: Better Guidance Is Needed to Ensure an Appropriate Response to Anthrax 

Contamination, GAO-04-239 (Washington D.C., Sept. 9, 2004). 42 The first activity involved agencies’ developing a sampling strategy (which included deciding how many 

samples to collect, where to collect them from, and what collection methods to use.) Probability sampling 

would have allowed agencies to determine whether the building was contaminated with some defined level 

of confidence but probability sampling was not deemed practical. Agencies and their contractors used 

various methods to collect samples during sampling events. (USPS used dry swabs to collect samples; CDC 

and EPA used multiple methods—dry swabs, pre-moistened swabs, wet wipes, and a high-efficiency 

particulate air (HEPA) vacuum). In transporting samples to laboratories for analysis, the agencies employed 

various approaches without knowing how transportation conditions might have affected sample viability. 

Finally, the GAO found that the analysis results were problematic because different laboratories used their 

own procedures for extracting samples in the absence of extraction efficiency data. See, GAO, op. cit., 

2005.

43 Ibid.
23

International established the Stakeholder Panel on Agent Detection Assays (SPADA).44

SPADA establishes method performance requirements and panels of reference materials 

through an extensive consensus process.45 SPADA set out to develop a method to screen 

aerosol collection samples for Francisella tularensis, Yersinia pestis, or anthrax, using 

polymerase chain reaction (PCR) technology widely used by Federal law enforcement 

agencies in forensic DNA typing. In 2010, SPADA began work on two additional efforts: 

detectors that use PCR techniques to screen aerosol collection of Burkholderia 

psuedomallei and Burkholderia mallei samples; and the development of portable devices 

to screen suspicious powders for anthrax using PCR rather than immunoassays.46

By 2008, on-site biological assessment and threat evaluation were the target of 

widespread concern, as reflected in another GAO report entitled: First Responders’ 

Ability to Detect and Model Hazardous Releases in Urban Areas is Significantly 

Limited.

47 It concluded that handheld detection devices for biological agents were not 

reliable or effective and that DHS has the primary mission to develop, test, and certify 

emergency responder detection equipment. 

The report served as a call to action among federal agencies according to close observers. 

In 2010, ASTM issued revised biothreat standards. By 2011, DHS published a national 

framework for biothreat response: The Framework for a Biothreat and Response Mission 

Capability. It addresses the challenges of technology integration for detection technology 

and the need to standardize.48

Meanwhile, since the anthrax attacks in 2001, the development, manufacture, sale, and 

deployment of biothreat detectors (including biothreat detection systems, technologies, 

tools, and assays) has been extensive. First responders increasingly employ biothreat 

detectors to protect the public.49

Formulating consensus guidance on biothreat response and collection was challenging. 

Many technical issues were subject to debate including the question of what reference 

materials biothreat detectors would be tested against. There was also variation in thirdparty testing protocols. Developing consensus standards was complicated by the diversity 

of the stakeholder community and the technical complexity of the biothreat itself. With 

respect to stakeholder community diversity, the perspectives of first responders and those 

who purchase their equipment; State and local public health officials; the Centers for 

 44 SPADA is a voluntary consensus standards body of more than 100 volunteer from across the broad 

stakeholder community, including stakeholders from federal, state, and local governments; the firstresponse and public health communities; academia; and industry. 45 Matthew Davenport, “Standards for Biothreat Detection,” ANSI Homeland Security Standards Panel, 

Tenth Annual Plenary Meeting: Achievements from the Past Decade and Charting the Path Forward,”

November 9, 2010

46 Matthew Davenport, “Consensus Standards That Define the Performance of Biological Threat 

Detectors,” DSP JOURNAL, Vol. 16, April/September 2012. 47 First Responders’ Ability to Detect and Model Hazardous Releases in Urban Areas is Significantly 

Limited (GAO-08-180), U.S. General Accountability Office, June 2008. 48 Marsh, op, cit.

49 Davenport, 2012, op. cit. 
24

Disease Control (CDC); the Laboratory Response Network (LRN); the National Institute 

for Occupational Safety and Health (NIOSH); the Federal Bureau of Investigation (FBI); 

biothreat detector developers and leading-edge biothreat researchers are all essential. 

Each community needed to be part of the consensus process to ensure its mission needs 

was represented in any resulting standard, test and evaluation criteria, and certification 

process.

With respect to technical complexity, first, a biothreat is often not one threat but a 

“family” of pathogens, any one of which could be used in an act of biological terrorism. 

Bacillus anthracis, for example, is not a single biothreat but a species of organisms that 

include more than 200 individual human pathogenic strains. In addition, a Bacillus 

anthracis detector has to be sensitive to genetic differences between the strains to reduce 

the likelihood that an important strain will go undetected. Biothreats also typically have 

biologically near neighbors that are nonthreatening. Additionally, biothreat detectors 

need to be sensitive to these differences in order to prevent assessment errors. Finally, 

with respect to technical complexity, the state of scientific understanding is limited but 

rapidly evolving.50

Solution

NIST led the initial effort to develop the ASTM E2458 standard for the bulk sample and 

swab sample collection for suspected biological threat agents, and ASTM E2770 for 

operational guidelines for the initial response to a suspected biothreat agent.51 Based on 

the recommendations of focus groups representing the first responder community, NIST 

and a broad team of federal, state and local agencies and other organizations worked 

together to update the existing standards (published in 2006) for sample collection and 

develop overall guidance on when to collect a sample and how to coordinate with other 

agencies and organizations. The updated standards were created through a consensus 

process involving the following organizations.

ASTM International CBRNE Sensors and Detectors Subcommittee Members

• Industry/Universities: Bioscience Inc.; Tauri Group/Johns Hopkins University

• Trade Associations: Association of Analytical Communities/Stakeholder Panel on Agent 

Detection Assays; Association of Public Health Laboratories

• First Responders: Fire Department of the City of New York; Georgia National Guard; 

Massachusetts Department of Fire Services; National Guard Bureau

• Federal Agencies: Biomedical Advanced Research and Development Authority; Centers 

for Disease Control and Prevention; Critical Reagents Program; Defense Intelligence 

Agency; Environmental Protection Agency; Federal Bureau of Investigation; Federal Drug 

Administration; Joint Program Executive Office for Chemical and Biological Defense;

National Institute of Standards and Technology; Office of Science and Technology Policy 

 50 Ibid. 51 Testimony of Dr. Richard R. Cavanagh Director, Office of Special Programs, National Institute of 

Standards and Technology Department of Commerce Before the United States House of Representatives 

Committee on Science, Space, and Technology, July 19, 2012.
25

The publication of ASTM E2770-10 established a biothreat response concept of 

operations with four critical elements. First it defines initial responsibilities for hazard 

assessment and threat evaluation. The most effective use of technology occurs here, 

potentially changing the way a suspect event is resolved. The second and third elements 

concern certified training and field delivered proficiency testing. The fourth critical 

element, sample-handling standards, was achieved through publication of ASTM E2458-

10, a sample collection standard. ASTM E2458-10 has two parts. “Method A” is for the 

collection of bulk powder and any accompanying letters. It applies to state public health 

laboratories. “Method B” is for the on-site collection of remaining trace residual powder. 

By recognizing the need for on-site testing, E2458-10 marks a significant change in the 

national response doctrine for suspected biological threats.52

The ASTM and AOAC standards have improved communication, cooperation, and 

interoperability within the stakeholder community and have given purchasers more 

assurance about the performance of available products for field screening/detection. 

ASTM E2770-10 and ASTM E2458-10 have been adopted by numerous states, the 

Association of Public Health Laboratories, and other stakeholder organizations. Some 

first-responder agencies, such as the National Guard Civil Support Teams (CST), are 

working to integrate the powder-collection standards into practice through real-world 

scenario training events like Operation Vigilant Sample in February 2011 and April 2012.

The updated standards have been widely implemented due to the extensive involvement 

of customers and stakeholders in the standards development process. 

NIST Impacts

In 2001 Federal, State, and local first responders dealing with suspicious white powders 

had little idea how to do so in a coordinated and effective manner. Since then, firstresponder standard practices have been developed and implemented nationwide. 

Important “front line” organizations, such as the CST, are integrating these standards into 

their training events.

Nationally, for example, the CST program has utilized the ASTM publications E2770-10

and 2458-10 to establish best practices with regard to sampling operations and on-site 

analytics. ASTM guidance documents have supported improved interoperability, aligning 

CST’s concept of operations with that of the Laboratory Response Network (LRN) and 

Federal Bureau of Investigation (FBI).53

According to state and local fire department officials, the first responder community 

relies on national standards to frame a response and then train and drill in order to perfect 

 52 Marsh, op. cit. 53 Established in 1999, the Laboratory Response Network (LRN) is a collaborative effort within the US 

federal government involving the Centers for Disease Control and Prevention (CDC), the Federal Bureau of 

Investigation (FBI) and the Association of Public Health Laboratories (APHL) to coordinate clinical 

diagnostic testing for bioterrorism. The primary technical focus was to detect the presence of biothreat 

agents in field samples in a manner that is consistent with the needs for law enforcement.
26

it. What seasoned professionals regard as “staggering” variations in the responses to the 

2001 anthrax attacks, nationally, were evidence that training gaps needed to be closed. 

At the local level, interactions with NIST personnel have had a systemic impact on first 

responders. According to one major county fire department official “The relationship 

with NIST has really shaped our department’s entire white powder response.” That said, 

if forced to prioritize the impacts, “Deployment was affected the most. Our precise and 

methodical response [to biothreat events] has been honed by the work of NIST.” 

Interactions with NIST’s cutting edge technology applications led this same organization 

to investigate the newest biothreat detection methods (DNA typing technologies). These 

investigations have, in turn, affected his organizations R&D and procurement priorities. 

Collaborating with NIST has led some organizations to develop relationships with DHS 

and the FBI. These, in turn, helped local first responders to frame the big picture of 

biothreat incidents so that, going forward, the organization’s response to biothreat events 

will be more effective overall. 

According to observers, an ever increasing number of people and departments are not just 

showing interest in proficiency but large numbers of people have become proficient and 

actually perform very well in these responses, repeatedly, and are now working to hone in 

their skill. 

Participants have noted NIST’s role as a catalyst in the national standard consensusmaking process. Working through the ASTM International, NIST brought the validity 

and respect needed to forge a national consensus despite disparate interests in multiple 

dimensions. Having NIST standing behind the data presented to the technical committee 

was invaluable in bringing the group together to agree on a valid process and plan. 
27

Emergency Response Robots

ASTM Series for Robots 

(ASTM E2552-07a, E2566-08, E2592-07, E2801-11, E2802-11, E2803-11, E2804-11, 

E2826-11, E2827-11, E2828-11, E2829-11, E2830-11, E2853-12, E2854-12, E2855-12)

Background & Problem

When disaster strikes today, Urban Search and Rescue (USAR) Task Forces spring into 

action. These are local government entities, certified, trained, and accredited by the 

federal Government. Each USAR Task Force is comprised of at least 70 persons whose 

skills include engineering, emergency medicine, canine handling, firefighting, hazardous 

material handling, communications, logistics, and other areas. They work to stabilize 

damaged structures, locate and extricate victims, identify risks of additional collapses, 

and meet other needs at disaster sites. Among USAR Task Force capabilities are the 

deployment of “response robots.” The robots deployed by first responders must have 

previously demonstrated functionality, adequacy, and appropriateness for the task, as 

well as interoperability, efficiency, and sustainability.54

The federal effort to stimulate the development and deployment of robots in urban search 

and rescue missions goes all the way back to Earthquake Hazards Reduction Act of 1977.

NIST is the lead agency for the programs established under that legislation. After the 

creation of Federal Emergency Management Agency (FEMA) in 1979 it was designated 

the lead agency for the USAR Task Force program. USAR Task Forces responded to the 

bombing of the Murrah federal building in Oklahoma City in 1995, the terrorist attacks of 

2001, Hurricane Katrina (2005), the earthquakes that devastated Haiti in 2010, to name a 

few.55

In 2004, noting a lack of standards or performance metrics, DHS S&T tapped NIST to 

coordinate the creation of performance standards for response robots. NIST leveraged the 

prior work on performance requirements and test methods for bomb disposal robots 

supported by the National Institute of Justice and began working through the ASTM 

International Standards Committee on Homeland Security Applications; Operational 

Equipment; Robots (E54.08.01).56 NIST personnel have chaired the subcommittee since 

its formation. At the time a lack of consensus national requirements and standards was 

slowing the development of robot systems and leading to confusion among Federal, state 

and local emergency planners equipment manufacturers, software developers, and 

shippers. NIST continues to work closely with the National Bomb Squad Commanders 

 54 Statement of Requirements for Urban Search and Rescue Robot Performance Standards (Preliminary 

Version), Department of Homeland Security, Science and Technology Directorate and National Institute of 

Standards and Technology, MAY 13, 2005. 

55 Keith Bea, Urban Search and Rescue Task Forces: Facts and Issues, Congressional Research Service, 

January 29, 2010. “More recently, the 108th Congress transferred that authority [“lead agency for the 

program”] to the National Institute of Standards and Technology of the Department of Commerce. Cites 

Sec. 103, P.L. 108-360, 42 U.S.C. 7704(b)(1). 56 http://www.nist.gov/el/isd/ms/robottestmethods.cfm
28

Advisory Board to define suites of tests for performance of robots in their main 

deployment scenarios, especially requirements for vehicle-borne improvised explosive 

devices (IEDs) and for measuring the proficiency of operators of these robots to guide 

training. 

DHS began supporting the bomb squad test method development that fit comfortably 

under the existing ASTM E54 task group. Most of the robots that could be considered for 

USAR missions were repurposed bomb squad robots. The DHS-sponsored project began 

through a series of workshops hosted by NIST at which FEMA USAR team members 

defined the performance requirements for the robots and began itemizing the types of 

deployment scenarios to which the robots may be applied.57

As part of its standards development efforts, NIST established a Robot Test Facility 

(RTF).58 The RTF enables engineers to collect performance data from test methods 

designed to mimic rescue scenarios for bomb and urban search and rescue robots. A wide 

variety of robots are run through a series of demanding tasks in experiments designed to 

understand how to best capture data and to refine the physical artifacts and methodology. 

The data are collected and submitted to the standards balloting process of the ASTM’s 

E54.08.01 committee.59

Robot competitions are also organized by NIST. These aid the development and 

dissemination of standard test methods. The competitions push the boundaries of current 

capabilities and provide critical insight into how the test methods will respond to the 

robots of the future.60

NIST also works directly with emergency response teams, conducting robot evaluation 

exercises in the first responders’ own training facilities. Emergency rescue teams and 

bomb squads engage in these exercises to help validate emerging standard robot test 

methods, to familiarize themselves with available robot capabilities, and to advise 

developers about burgeoning operational requirements. The exercises also introduce 

emerging robotic capabilities to emergency responders within the context of their home 

training facilities. These activities aid in the drafting of standard test methods tuned to the 

deployment tasks and scenarios.61

 57 Messina, 2007, op. cit. Most of the robots that could be considered for USAR missions were repurposed 

bomb squad robots. 58 The RTF was funded as part of the American Recovery and Reinvestment Act, 

http://www.recovery.gov/arra/Transparency/RecoveryData/reporting/agency_reporting5program.aspx?agen

cy_code=13&progplanid=7720) 

59 http://www.nist.gov/el/isd/ms/roboticsbldg.cfm 60 http://www.nist.gov/manuscript-publication-search.cfm?pub_id=910662 61 Adam Jacoff, et. al., “Emergency Response Robot Evaluation Exercise,” Association for Computing 

Machinery, 2012. 
29

Solution

Today, urban search and rescue robots capable of locating victims at disaster sites are 

entering the marketplace. To accelerate development and deployment of this potentially 

life-saving technology, NIST and DHS host a public forum on robot standards to explore 

ideas and timelines for standards related to urban rescue robot development, testing, and 

certification.62

The DHS-NIST-ASTM collaboration, and the network of collaborations it enables, has 

led to a suite of standard test methods for response robots. These standards allow DHS to 

provide guidance to local, state, and federal homeland security organizations regarding 

the purchase, deployment, and use of robotic systems for USAR applications.63 The 

ASTM E54.08.01 committee includes equal representation of robot developers, 

emergency responders, and civilian/military test administrators.64

ASTM International Homeland Security Applications, 

Operational Equipment —Robots Task Group Members65

• Industry: AirRobot; Applied Research Associates, Inc.; Boston Dynamics; Cobham;

ICOR Technology; iRobot; MacroUSA; Mesa Robotics, Inc.; Qinetiq North America; Recon 

Robotics; Remotec; Northrop Grumman; RoboTe; Roboteam; SuperDroid Robots; WM 

Robot, LLC

• Federal or National Organizations: Federal Emergency Management Agency; National 

Bomb Squad Commanders Advisory Committee; National Capitol Region Bomb Squad 

Working Group; National Institute of Standards and Technology; Urban Search and Rescue 

Technical Search Working Group

• National and Federal Facilities with Standard Test Methods: Applied Research 

Associates, Inc.; Camp Dawson Training Center; National Institute of Standards and 

Technology; Penn State Applied Research Laoratory; Robolit LLC; Space and Naval Warfare 

Systems Command System Center Pacific; Texas A&M Engineering Extension Service; 

Southwest Research Institute; Tyndall Air Force Base; University of Massachusetts Lowell, 

NERVE Center; 

• International Facilities with Standard Test Methods: Bundeswehr Technical Center for 

Automotive and Armored Vehicles – Germany; Curtin University of Technology – Western 

Australia; Jacobs University – Germany; Mahidol University – Thailand; Nagaoka University 

of Technology – Japan; Sapienza University of Rome – Italy; Tohoku University – Japan

The objective of this suite of standards is to facilitate quantitative comparisons of 

different robot configurations based on statistically significant robot capabilities data 

captured within standard test methods to understand deployment capabilities, guide 

 62 http://www.nist.gov/public_affairs/techbeat/tb2005_0412.htm#robot 63 Messina, 2007, op. cit. 64 http://www.nist.gov/el/isd/ms/robottestmethods.cfm 65 The actual collaboration of robot developers with NIST’s response robots program has been much 

broader — practically universal — than that depicted by ASTM collaboration alone. See, 

http://www.nist.gov/el/isd/ms/upload/2010_RobotGuide-2.pdf
30

purchasing decisions, and support operator training with measures of proficiency.66

This suite of standards measures robot maneuvering, mobility, manipulation, sensing, 

endurance, radio communication, durability, reliability, logistics, and safety for remotely 

operated ground vehicles, aquatic vehicles, and small, unmanned aerial systems.

These standard test methods measure baseline robot/operator capabilities necessary to 

perform operational tasks defined by emergency responders, soldiers, and their respective 

organizations. 

The overall suite of standards is expanding to answer new mission requirements every 

year, and some test methods have already been updated to widen their scope for testing 

autonomous systems.67

NIST Impacts

Emergency responders risk life and limb interacting with known hazards to protect the 

public. Some of those hazards are well suited to being addressed by remotely operated 

robots. Examples include: disabling or dismantling improvised explosive devices (pipes, 

packages, vehicle); searching for survivors in collapsed or compromised structures; 

investigating illicit border tunnels; establishing situational awareness during police 

actions; monitoring large scale industrial or transportation accidents; or assessing 

potential terrorist attacks using chemical, biological, or radiological sources. 

Many responder organizations already own robots but have had difficulty deploying them 

effectively. New robots are promising advanced capabilities and easier operator 

interfaces, but it is hard for responders to sift through companies’ product marketing 

without rigorous measures of what functions a robot is capable of performing. First 

responder organizations also need ways to train operators and reliably measure their 

skills.68 

The standard test methods for response robots are intended to do just that. The economic 

benefits of doing so are many: operational cost savings; government budget leverage 

(increased robot capability per government budget dollar); procurement process cost 

savings; proliferation of ASTM test methods; response robot market expansion; 

acceleration of technology development, transfer, and diffusion; and significant savings 

in robot test & evaluation costs.

Standard test methods generated from first responder robot requirements have enabled 

the development of objective performance metrics and repeatable performance testing. 

These, in turn, are accelerating the development and deployment of mobile robotic tools 

 66 Ibid. 67 Ibid. 68 http://www.nist.gov/el/isd/ms/robottestmethods.cfm
31

for USAR responders.69

The impacts NIST’s response robotics program are widely recognized. A Congressional 

Representative who visited the NIST facility where the test methods are developed 

declared it to be, “an excellent example of government and private sector cooperation.” A 

senior executive of the National Bomb Squad Advisory Board (NBSCAB), which acts as 

the final decision making authority on guidelines and standards for the profession, stated, 

“The work [NIST is] doing to develop standard test methods for vehicle-borne 

improvised explosive device ... extremely important, as it will help us better define such 

robots and clarify exactly what technology requirements we need to pursue on the 

research and development front.” NBSCAB plans to use the data produced by NIST’s 

response robots program in its decision-making in support of bomb squad professionals. 

As a result of the NIST-led standards development activities of the ASTM International, 

important procurement decisions are now based on quantitative, statistically significant 

data. The Interagency Board (IAB) (whose equipment list guides procurements for 

federal, state, and local emergency response) has adopted the standard test methods 

developed under NIST’s program.70 Purchases are no longer based on the vendor’s 

brochure, but rather on consensus standard test methods that characterize the capabilities 

of robots to help find the right match for user needs. 

Other government agencies are also leveraging the results of this program. For example:

The DOD Joint Improvised Explosive Device Defeat Office (JIEDDO) based a $30M 

procurement of 500 small, “throwable” robots on results from the test method suite. And 

the Defense Advanced Projects Research Agency (DARPA) is sponsoring NIST’s 

participation in their latest Robotics Challenge, which addresses robotic solutions for 

disaster responses inspired by the Fukushima Daiichi nuclear disaster of 2011. A 

Japanese robotics researcher who deployed a response robot to the fifth floor of the 

Fukushima Daiichi reactor plant attributed the advances that enabled that feat to NISTdeveloped test methods. NIST’s leadership in an ongoing international response robots 

competition (under the RoboCup umbrella) enabled advances in the capabilities of the 

Japanese robot used.

Other robot manufacturers are improving their commercial robots thanks to the lessonslearned when their robots run through test methods. The test methods contained the 

ASTM International suite of robot standards also provide the means to verify any claims 

of “new improved” robots. 

The test methods are being propagated within the U.S. and internationally. Several 

facilities have been populated with replicas of the NIST’s Robot Test Facility, including a 

 69 Elena Messina, “Performance Standards for Urban Search and Rescue Robots Enabling Deployment of 

New Tools for Responders,” DSP Journal, July/December 2007. 70 IAB members advocate for and assist the development and implementation of performance criteria, stand

ards, test protocols, and technical, operating, and training requirements for all-hazards incident response eq

uipment with a special emphasis on Chemical, Biological, Radiological, Nuclear, and Explosive issues. See

: https://iab.gov/SEL.aspx
32

facility at Southwest Research Institute (SwRI), in San Antonio, Texas, an independent, 

nonprofit, applied R&D organization; and the New England Robot Validation and 

Experimentation (NERVE) Center at the University of Massachusetts-Lowell. Japan and 

Germany have full replicas. New locations coming on line include SPAWAR in San 

Diego and Curtin University in Australia. Several other locations have facilities that 

implement subsets of the test methods. 

One of these applied research organizations reports that with the growing acceptance of 

the ASTM E54.08.01 standards, the robot evaluation processes have been streamlined 

enabling a 40-50% reduction of test event operational expenses.

Ultimately, thanks to the ASTM suite of robot test standards, there are more market 

opportunities for robot vendors and first response organizations have more choices and

fewer risks when adopting this new technology. 
33

Conclusion

NIST scientists and engineers have been answering the question, “by what measure?,” for 

over a century. The technical complexity, scale, and scope of many public safety issues 

have grown and changed over the decades. For all those years, realizing public safety 

goals and objectives has required the best answer to the question available and NIST has 

been there to provide it.

In the last few decades NIST has attempted to measure the economic impact — the return 

on investment (ROI) — of applying its unique measurement expertise to issues of 

national, local, or industry-wide concern. Looking at 34 economic impact case studies 

conducted from 1981 to 2011, a dollar of NIST resources returned an average of $183 in 

societal benefits of all kinds (ranging from $32 to over $1000 returned to society for 

every NIST dollar invested); and an average benefit-to-cost ratio of 29-to-1 (ranging from 

3:1 to 113:1) for 24 of the 34 cases.71 It is in that spirit that the preceding six cases have 

distilled, qualitatively, the kinds of benefits that flow from NIST’s collaboration with 

DHS concerning critical security issues. 

The testimonies of first responders, the product developers that support them, and other 

SDO collaborators demonstrate that the DHS-NIST partnership, and the network of 

partnerships it enables, is making a unique and considerable impact on the effectiveness 

of law enforcement and the state of public safety. These impacts are summarized in the 

table .

 71 Calculated from data originally compiled by Gregory Tassey, 2010, updated by the author. See “Outputs 

and Outcomes of NIST Laboratory Research,” http://www.nist.gov/director/planning/studies.cfm. The BCR 

and SRR of NIST’s contributions to the development of the FPDM Standard (assessed in 2011), 4:1 and 

48% respectively, were added to Tassey’s data to compute the averages shown here.
34

All of the cases describe changes in “front lines” operational practices that positively 

affect millions, from subway passengers across the nation, to airline passengers in 

baggage check queues, to an untold numbers of victims of random or intentional 

disasters. 

The scope of the operations to which NIST’s measurement technologies are applied and 

the frequency of use vary among the six cases. While all the technologies involved have 

wide application, the scale and pace of the detection activities associated with airport and 

cargo screening, and biothreat detection operations, appear relatively greater.

All the cases described implemented standards that had a significant impact on the 

resources procurement agents expend to sort out the previously bewildering claims about 

how various types of equipment will actually perform. 

All of the cases described NIST’s contribution to the improved effectiveness with which 

“front lines” operators and first responders do their jobs; in one way or another carrying 

out their objectives with improved quality for every dollar invested in getting the first 

responder on the scene. 

In all cases industry collaborators report that NIST’s measurement technology 

applications have spurred innovation and accelerated the product development-to-market 

cycle. This, in turn, enabled industry to provide more effective equipment more rapidly. 

In all cases the standards development processes was accelerated due to NIST’s 

participation in typically wide-ranging collaborations involving representatives of 

government procurement activities, first responders, industry associations, individual 

companies and experts, and government agencies and laboratories with policy interests 

and expertise. In such complex collaborative settings, NIST’s widely respected “honest 

broker” role typically reduces the costs, to public and private participants, of “consensus 

formation.” By the same token, this streamlining contributes to the faster delivery of 

more effective equipment to first responders, no doubt preventing injuries, saving lives, 

and reducing operations costs; all leveraging Federal, State, local, and industry 

investments in the public safety mission.

Because of NIST’s role in making CBRNE-related equipment more effective, in all six 

cases it is likely that lives have been, or will be, saved. In the case of SCBA facepiece 

lenses, since fatalities motivated the NIST research that improved the lenses, the absence 

of fatalities related to lens degradation strongly suggests that firefighters’ lives have been 

saved and their safety has been substantially improved.

The DHS has turned, and continues to turn, to NIST when the question, “By what 

measure?,” matters. It so often does when domestic security goals and objectives are 

being implemented. NIST, in turn, exercises its relatively unique capability to leverage a 

vast network of collaborators in industry, government, academia, and non-governmental 

organizations, both domestic and international. 
35

Appendix A:

Approach to the Selection and Development of Cases

NIST’s Program Manager for the Special Programs Office selected groups of standards to 

be investigated for their economic benefits. Questions about NIST’s participation in the 

subject documentary standards were posed to, and answered by, lead researchers via 

email exchanges. Lead researchers provided points of contact among Standards 

Development Organization (SDO) collaborators and end users. Email and telephone 

exchanges with SDO collaborators, equipment manufacturers, and first responders 

solicited qualitative information about how collaboration with NIST and/or the 

availability of documentary standards made a difference in their operations. 

While the benefits reported here are qualitative in nature, the questions posed to NIST 

researchers, documentary standards developers, manufacturers, and “front lines” users 

were such that, with more time, the qualitative results presented here could be quantified.