AN ASSESSMENT OF THE 

NATIONAL INSTITUTE OF STANDARDS 

AND TECHNOLOGY 

INFORMATION TECHNOLOGY 

LABORATORY 

FISCAL YEAR 2011 

Panel on Information Technology 

Laboratory Assessments Board 

Division on Engineering and Physical Sciences 

THE NATIONAL ACADEMIES PRESS 

Washington, D.C. 

www.nap.edu
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, N.W. Washington, DC 20001 

NOTICE: The project that is the subject of this report was approved by the Governing Board 

of the National Research Council, whose members are drawn from the councils of the 

National Academy of Sciences, the National Academy of Engineering, and the Institute of 

Medicine. The members of the panel responsible for the report were chosen for their special 

competences and with regard for appropriate balance. 

This study was supported by Contract No. SB134106Z0011, TO #10, between the National 

Academy of Sciences and the National Institute of Standards and Technology, an agency of 

the U.S. Department of Commerce. Any opinions, findings, conclusions, or 

recommendations expressed in this publication are those of the authors and do not 

necessarily reflect the views of the agency that provided support for the project. 

International Standard Book Number-13: 978-0-309-22002-6 

International Standard Book Number-10: 0-309-22002-5 

Copies of this report are available from 

Laboratory Assessments Board 

Division on Engineering and Physical Sciences 

National Research Council 

500 Fifth Street, N.W. 

Washington, DC 20001 

Additional copies of this report are available from the National Academies Press, 500 Fifth 

Street, N.W., Lockbox 285, Washington, DC 20055; (800) 624-6242 or (202) 334-3313 (in 

the Washington metropolitan area); Internet, http://www.nap.edu. 

Copyright 2011 by the National Academy of Sciences. All rights reserved. 

Printed in the United States of America 
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished 

scholars engaged in scientific and engineering research, dedicated to the furtherance of science and 

technology and to their use for the general welfare. Upon the authority of the charter granted to it by the 

Congress in 1863, the Academy has a mandate that requires it to advise the federal government on 

scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences. 

The National Academy of Engineering was established in 1964, under the charter of the National 

Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its 

administration and in the selection of its members, sharing with the National Academy of Sciences the 

responsibility for advising the federal government. The National Academy of Engineering also sponsors 

engineering programs aimed at meeting national needs, encourages education and research, and recognizes 

the superior achievements of engineers. Dr. Charles M. Vest is president of the National Academy of 

Engineering. 

The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the 

services of eminent members of appropriate professions in the examination of policy matters pertaining to 

the health of the public. The Institute acts under the responsibility given to the National Academy of 

Sciences by its congressional charter to be an adviser to the federal government and, upon its own 

initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president 

of the Institute of Medicine. 

The National Research Council was organized by the National Academy of Sciences in 1916 to associate 

the broad community of science and technology with the Academy’s purposes of furthering knowledge and 

advising the federal government. Functioning in accordance with general policies determined by the 

Academy, the Council has become the principal operating agency of both the National Academy of 

Sciences and the National Academy of Engineering in providing services to the government, the public, 

and the scientific and engineering communities. The Council is administered jointly by both Academies 

and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Charles M. Vest are chair and vice chair, 

respectively, of the National Research Council. 

www.national-academies.org 
iv 

PANEL ON INFORMATION TECHNOLOGY 

DAVID R. FERGUSON, Applied Mathematical Analysis, LLC, Chair 

NAOMI S. ALTMAN, Pennsylvania State University 

PHILLIP COLELLA, E.O. Lawrence Berkeley National Laboratory 

L. PAMELA COOK-IOANNIDIS, University of Delaware 

GEORGE V. CYBENKO, Dartmouth College 

LUIS A. ESCOBAR, Louisiana State University 

ERIC H. GROSSE, Google, Inc. 

ANIL K. JAIN, Michigan State University 

BIING-HWANG (FRED) JUANG, Georgia Institute of Technology 

JAMES F. KUROSE, University of Massachusetts 

PATRICK D. LINCOLN, SRI International Corporation 

STEVEN B. LIPNER, Microsoft Corporation 

ALEXA T. McCRAY, Harvard Medical School 

GREGORY H. MILLER, University of California, Davis 

DEBASIS MITRA, Bell Labs, Alcatel-Lucent 

J. MARC OVERHAGE, Indiana University School of Medicine and Regenstrief Institute 

W. TIMOTHY STRAYER, Raytheon BBN Technologies 

STEPHEN B. VARDEMAN, Iowa State University 

Staff 

JAMES P. McGEE, Director 

CY BUTNER, Senior Program Officer 

LIZA HAMILTON, Administrative Coordinator 

EVA LABRE, Program Associate 
v 

Acknowledgment of Reviewers 

This report has been reviewed in draft form by individuals chosen for their 

diverse perspectives and technical expertise, in accordance with procedures approved by 

the National Research Council’s Report Review Committee. The purpose of this 

independent review is to provide candid and critical comments that will assist the 

institution in making its published report as sound as possible and to ensure that the 

report meets institutional standards for objectivity, evidence, and responsiveness to the 

study charge. The review comments and draft manuscript remain confidential to protect 

the integrity of the deliberative process. We wish to thank the following individuals for 

their review of this report: 

Suzanne Bakken, Columbia University, 

Steven M. Bellovin, Columbia University, 

Carl de Boor, University of Wisconsin-Madison (Emeritus), 

Stuart I. Feldman, Google, Inc., 

James M. Landwehr, Avaya Laboratories, 

Richard W. Pew, Raytheon BBN Technologies, 

Jeffrey D. Ullman, Stanford University, and 

Stephen A. Vavasis, University of Waterloo. 

Although the reviewers listed above have provided many constructive comments 

and suggestions, they were not asked to endorse the conclusions or recommendations, nor 

did they see the final draft of the report before its release. The review of this report was 

overseen by Alton D. Slay, Warrenton, Virginia. Appointed by the National Research 

Council, he was responsible for making certain that an independent examination of this 

report was carried out in accordance with institutional procedures and that all review 

comments were carefully considered. Responsibility for the final content of this report 

rests entirely with the authoring panel and the institution. 

vii 

Contents 

SUMMARY 1 

1 THE CHARGE TO THE PANEL AND THE ASSESSMENT PROCESS 5 

2 GENERAL ASSESSMENT OF THE INFORMATION TECHNOLOGY 

 LABORATORY 7 

Laboratory Mission and Programs, 7 

Observations and Recommendations, 10 

Laboratory Responses to the 2009 Panel Report, 14 

3 ASSESSMENT OF THE LABORATORY DIVISIONS 16 

Applied and Computational Mathematics Division, 16 

Advanced Network Technologies Division, 18 

Computer Security Division, 21 

Information Access Division, 24 

Software and Systems Division, 27 

Statistical Engineering Division, 30 

4 SUMMARY OF CONCLUSIONS 34 

1 

Summary 

The Information Technology Laboratory (ITL) of the National Institute of 

Standards and Technology (NIST) has been assessed by the Panel on Information 

Technology, appointed by the National Research Council (NRC). The panel of experts 

visited the six divisions of the laboratory and reviewed their activities. The scope of the 

assessment included the following criteria, provided by the Director of NIST in his 

charge to the NRC: (1) the degree to which laboratory programs in measurement science, 

standards, and services achieve their stated objectives and fulfill the mission of the 

operating unit (laboratory); (2) the technical merits and scientific caliber of the current 

laboratory programs relative to comparable programs worldwide; and (3) the alignment 

between laboratory research and development (R&D) efforts and those services and other 

mission-critical deliverables for which the laboratory is responsible. On the basis of its 

assessment using these criteria, the panel formed the observations and recommendations 

presented below, among others discussed in this report. 

OBSERVATIONS 

 Observations 1 through 3 below pertain directly to how the ITL is performing 

with respect to the three assessment criteria from the Director of NIST. Observations 4 

through 6 address changes that have taken place since the 2009 assessment performed by 

the NRC panel appointed for that review.1 Observations 7 through 10 focus on areas of 

concern. 

1. The programs of the Information Technology Laboratory are focused on 

research and development that advance measurement science, standards, and 

technology. As an example, the Virtual Measurement Systems Program has 

identified the importance of understanding virtual measurements and 

uncertainties in advancing industry’s increasing reliance on software 

modeling and simulation in, for example, the design of new, advanced 

products. Similarly, the Cloud Computing Program is working on a U.S. 

government cloud computing technology roadmap that is focused on the 

highest-priority national cloud computing interoperability, portability, and 

security requirements. The Health Information Technology Program is 

working to improve standards for health technologies. These programs and 

the others reviewed are all making substantial progress toward meeting their 

objectives and are well aligned with the ITL mission and responsibilities. 

2. The technical merits and scientific caliber of the current ITL programs are 

very high relative to comparable programs worldwide as measured by 

publications and especially by outstanding products such as the Digital 

Library of Mathematical Functions (DLMF) and the NIST Special 

Publication 800* series. The DLMF is without peer in the broader 

 1

 National Research Council, An Assessment of the National Institute of Standards and Technology 

Information Technology Laboratory: Fiscal Year 2009. Washington, D.C.: The National Academies Press, 

2009. 
2 

community, and the NIST Special Publication 800* series is renowned for 

providing technically sound, unbiased, relevant guidelines that are frequently 

adopted voluntarily in private-sector procurements and practices and often 

mandated by the Office of Management and Budget for use by the federal 

government. 

3. The ITL R&D efforts appear to be carefully aligned with the mission-critical 

deliverables for which the ITL is responsible. Programs in cloud computing, 

health information technology, identity management, cybersecurity 

education, trusted identities, and voting standards are all addressing national 

priorities in information technology. National priorities with critical 

information technology aspects are being addressed by projects in 

biosciences and bioimaging, cyber physical systems, forensics, greenhouse 

gas measurement, optical medical imaging, public safety communications, 

quantum information, smart grid, and trusted networking (Internet Protocol 

Version 6 [IPv6], Domain Name System Security Extensions [DNSSEC]). 

4. The Software and Systems Division (SSD) has made great strides since the 

previous assessment panel registered concerns in its 2009 report.2

 The most 

prominent concern was “the lack of strong scientific and administrative 

leadership within the SSD and also, in some cases, at the programmatic 

level.”3 Today those concerns are being aggressively addressed, and the SSD 

has become more focused and better able to respond to its current challenges. 

5. The ITL leadership has done an excellent job in filling two critical 

management positions: division chief for the Computer Security Division 

(CSD) and division chief for the Software and Systems Division. The ITL 

management is still faced with finding a permanent chief for the Advanced 

Network Technologies Division (ANTD). 

6. The ITL has struggled with how crosscutting programs—those that involve 

work in a collaborative fashion across divisions—would be managed, since 

they do not fit neatly into the divisional structure. The ITL answer has been 

to use a matrix management structure (a structure in which an individual 

reports to two supervisors, one functional and one operational). In 2007,4

less so in 2009, the panel was aware of considerable angst on the part of 

management and staff as to how that would work. This year there were no 

signs of that distress. It appears that the ITL has done an excellent job of 

working out the kinks and implementing matrix management. 

7. The Statistical Engineering Division (SED) is continuing on an even keel 

with strong leadership and technical expertise. However, as observed in the 

2009 assessment report, the division workload is growing but the division is 

not. The SED is seriously understaffed, and this problem needs to be 

addressed with some urgency.5

 2

 Ibid. 

3

 Ibid., p. 15. 

4

 National Research Council, An Assessment of the National Institute of Standards and Technology 

Information Technology Laboratory: Fiscal Year 2007. Washington, D.C.: The National Academies Press, 

2007. 5

 National Research Council, An Assessment of the National Institute of Standards and Technology 

Information Technology Laboratory: Fiscal Year 2009. Washington, D.C.: The National Academies Press, 

2009, pp. 2, 9, 14. 
3 

8. The Computer Security Division is also understaffed, although neither 

performance nor morale has as yet been affected. 

9. The work of the Applied and Computational Mathematics Division (ACMD) 

continues to be excellent. However, the scientific culture of the division may 

not be sufficiently focused on collaboration to address the problems of 

multiscale and multiphysics involving complex geometries that are emerging 

as national priorities. 

10. The Advanced Network Technologies Division is doing an excellent job in 

responding to several national priorities in both the short and long term, 

including its continued outstanding activities in Internet infrastructure 

protection and its newer efforts in smart grids and public safety 

communications. The division has also improved the quality of its internal 

and external collaborations, as well as the quality of its publications. The 

ANTD is understaffed for the portfolio of activities that it is undertaking. The 

various teams handling projects with short deadlines do not have as much 

time to dig into the subjects as they would like or would be useful. Another 

consequence of the understaffing is that basic research activities are perhaps 

below levels that are healthy. ANTD management has not yet articulated a 

long-term, strategic view of networking. 

RECOMMENDATIONS 

 

1. At least two ITL divisions, the Statistical Engineering Division and the 

Computer Security Division, are feeling the constraints of increasing 

workloads and insufficient staffing (the SED more so than the CSD). If the 

ITL is to maintain its prominence in these areas, it should consider plans to 

address the growth that will be needed to support the expanding workload of 

each of these divisions. 

2. Because the trend toward simulations of increasing model fidelity and 

numerical accuracy is expected to continue, the Applied and Computational 

Mathematics Division will be called on to play an increasing role in 

addressing problems that are multidisciplinary. To ensure that the ITL is 

ready to support this work, the ACMD should devise a strategy to change the 

scientific culture of the division to meet those increased challenges. 

3. The ITL should fill the position of chief of the Advanced Network 

Technologies Division with a permanent chief. ANTD management should 

address the understaffing issue in the division, and in particular it should 

ensure that there are adequate resources to handle both the short- and longterm needs of the division. ANTD management should create a strategic 

roadmap for the technical work of the division. The roadmap should be 

useful in managing the division’s resources and portfolio of activities. 

4. The ITL should devote attention now to strategic, long-term technical needs 

in cloud computing that NIST may be called on to address in the future, 

including questions surrounding the scale of cloud computing and how such 

a scale could be accommodated in a laboratory or simulation environment. 

5. The ITL should consider creating a collaborative effort between the 

Computer Security Division and the Software and Systems Division that 
4 

would be responsible for the creation of standards and guidelines on secure 

software development for application by government, industry, and 

academia. 

6. The ITL and the Software and Systems Division should reconsider the SSD 

mission statement, given the fresh focus of the new leadership, and after the 

SSD strategic planning process is complete. 

7. The ITL and the Software and Systems Division should hire additional 

formally trained individuals in the SSD’s core foundational areas. 

8. The Information Access Division (IAD) supports the development of 

technologies and their transition into the commercial marketplace as well as 

government applications. The division currently relies on substantial and 

sustained amounts of other agency (OA) funding (approximately 60 percent 

of IAD funding). Most of the OA funding is security-related (from the 

Department of Homeland Security, the Department of Defense, the Federal 

Bureau of Investigation, and the Intelligence Advanced Research Projects 

Activity). The reports, standards, and evaluation studies of the IAD are 

closely followed by academia and industry. In light of increasing foreign 

dominance of the biometric industry, IAD’s reliance on OA funding, and 

IAD’s work in support of biometrics technology development, it is important 

that the IAD and the ITL remain mindful of the NIST mission to promote 

U.S. innovation and industrial competitiveness, and so IAD efforts should 

continue to place highest priority on the needs of the nation’s commerce even 

while pursuing activities involving international sponsors. 

9. The ITL should review the approval process of the Institutional Review 

Board6

 to maximize the efficiency of the process and minimize unnecessary 

latency. 

 6

 See http://www.hhs.gov/ohrp/humansubjects/commonrule/. Accessed July 11, 2011. The Office 

for Human Research Protections at the Department of Health and Human Services provides oversight for 

the protection of human subjects in research through the regulations that are spelled out for Institutional 

Review Boards in the so-called Common Rule (45 C.F.R. 46). 
5 

1 

The Charge to the Panel and the Assessment Process 

 At the request of the National Institute of Standards and Technology (NIST), the 

National Research Council (NRC) has, since 1959, annually assembled panels of experts 

from academia, industry, medicine, and other scientific and engineering environments to 

assess the quality and effectiveness of the NIST measurements and standards 

laboratories, of which there are now six,7 as well as the alignment of the laboratories’ 

activities with their missions. NIST requested that three of its laboratories be assessed in 

2011: the Center for Nanoscale Science and Technology, the NIST Center for Neutron 

Research, and the Information Technology Laboratory (ITL). Each of these was assessed 

by a separate panel of experts; the findings of the respective panels are summarized in 

separate reports. This report summarizes the findings of the Panel on Information 

Technology. 

 For the fiscal year (FY) 2011 assessment, NIST requested that the panel focus on 

the following criteria as part of its assessment: 

1. Assess the degree to which laboratory programs in measurement science, 

standards, and services achieve their stated objectives and fulfill the mission 

of the operating unit (laboratory); 

2. Assess the technical merits and scientific caliber of the current laboratory 

programs relative to comparable programs worldwide; and 

3. Assess the alignment between laboratory research and development (R&D) 

efforts and those services and other mission-critical deliverables for which the 

laboratory is responsible. 

The context of this technical assessment is the mission of NIST, which is to 

promote U.S. innovation and industrial competitiveness by advancing measurement 

science, standards, and technology in ways that enhance economic security and improve 

the quality of life. The NIST laboratories conduct research to anticipate future metrology 

and standards needs, to enable new scientific and technological advances, and to improve 

and refine existing measurement methods and services. 

 In order to accomplish the assessment, the NRC assembled a panel of 18 

volunteers whose expertise matches that of the work performed by the ITL staff.8

 The 

panel members were also assigned to six subgroups (division review teams), whose 

members’ expertise matched that of the work performed by staff in the six divisions in 

the ITL: Applied and Computational Mathematics Division (ACMD), Advanced Network 

Technologies Division (ANTD), Computer Security Division (CSD), Information Access 

Division (IAD), Software and Systems Division (SSD), and Statistical Engineering 

Division (SED). 

 7

 The six NIST laboratories are the Material Measurement Laboratory, the Physical Measurement 

Laboratory, the Engineering Laboratory, the Information Technology Laboratory, the Center for Nanoscale 

Science and Technology, and the NIST Center for Neutron Research. 8

 See http://www.nist.gov/itl/ for more information on ITL programs. Accessed April 18, 2011. 
6 

The panel met at the NIST facilities in Gaithersburg, Maryland, on March 21-23, 

2011. After the full panel had met for a session of welcoming comments from the NIST 

Acting Associate Director for Laboratory Programs and overview presentations on the 

ITL and six ITL programs by the laboratory’s management and staff, the panel divided 

into its six review teams, and each team (led by a team leader chosen from within the 

panel) then visited its respective ITL division for about a day. During these visits, the 

review team members attended presentations, tours, demonstrations, and interactive 

sessions with the ITL staff. Subsequently, the entire panel assembled for about a day and 

a half, during which it interacted with ITL and NIST management and also met in closed 

session to deliberate on its findings and to define the contents of this assessment report. 

 The approach of the panel to the assessment relied on the experience, technical 

knowledge, and expertise of its members, whose backgrounds were carefully matched to 

the technical areas of ITL activities. The panel reviewed selected examples of the 

technological research covered by the ITL; because of time constraints, it was not 

possible to review the ITL programs and projects exhaustively. The examples reviewed 

by the panel were selected by the ITL in consultation with the panel chair and NRC staff. 

The panel’s goal was to identify and report salient examples of accomplishments and 

opportunities for further improvement with respect to the following: the degree to which 

the ITL programs achieve their stated objectives and fulfill the ITL mission, the technical 

merit and scientific caliber of the ITL work, and the alignment between ITL R&D efforts 

and ITL services and other mission-critical deliverables. These examples are intended 

collectively to portray an overall impression of the laboratory, while preserving useful 

suggestions specific to projects and programs that the panel examined. The panel applied 

a largely qualitative rather than a quantitative approach to the assessment, although it is 

possible that future assessments will be informed by further consideration of various 

analytical methods that can be applied. 

 For its assessment, the panel relied primarily on presentations made by NIST and 

ITL managers and staff and by other researchers associated with NIST projects and 

programs, and on informational notes prepared by NIST and ITL staff for use by the 

panel. This report does not contain extensive citations of technical articles and reports. 

Other documents and resources used by the panel are cited in the report, as appropriate. 

The comments in this report are not intended to address each program within the 

ITL exhaustively. Instead, this report identifies key issues. Given the necessarily nonexhaustive nature of the review process, the omission of any particular ITL program or 

project should not be interpreted as a negative reflection on the omitted program or 

project. 

 Chapter 2 of this report presents a more detailed summary of the panel’s 

assessment of the Information Technology Laboratory. Chapter 3 presents the panel’s 

assessment of the divisions within the laboratory. Chapter 4 summarizes the overall 

conclusions of the panel. 
7 

2 

General Assessment of the Information Technology 

Laboratory 

LABORATORY MISSION AND PROGRAMS 

The Information Technology Laboratory supports the NIST mission through its 

own mission “to promote U.S. innovation and industrial competitiveness by advancing 

measurement science, standards, and technology through research and development in 

information technology, mathematics, and statistics.”9

 In support of this mission, the ITL 

has posed two strategic goals: 

 “Accelerate, through standards, tests, and metrics, the development, 

deployment and use of secure, usable, interoperable and reliable 

information systems that make American businesses more innovative 

and more competitive. 

 “Enable world-class measurement and testing through research 

innovations in the areas of computer science and systems engineering, 

mathematics and statistics.”10

In support of its mission and strategic goals, the ITL has formed a very strong 

scientific and technical team with core competencies in technology development in 

information technology (IT) measurement and testing, mathematical and statistical 

analyses for measurement science, modeling and simulation for measurement science, 

and information technology standards development and deployment. Further, the ITL 

has in recent years focused its R&D agenda on eight broad program areas: complex 

systems; cyber and network security; the enabling of scientific discovery; identitymanagement systems; information discovery, use, and sharing; pervasive information 

technologies; trustworthy information systems; and virtual measurement systems. 

The ITL now has a number of programs in these broad areas.11 The ITL program 

portfolio contains the following: 

 Complex Systems Program: This program examines systems that are 

composed of large, interrelated, and interacting entities that, taken together, 

show a macroscopic behavior that is not predictable through an examination 

of the individual entities. This program pursues an understanding of the 

fundamental science of complex systems and the development of rigorous 

 9

 Cita M. Furlani, ITL Director, “The Information Technology Laboratory,” presentation to the 

panel, Gaithersburg, Maryland, March 21, 2011, p. 3. 10 Ibid., p. 9. 

11 The program descriptions that follow were drawn from the descriptions provided to the panel by 

the ITL staff. 
8 

descriptions (analytic, statistical, or semantic) that enable the prediction and 

control of the behavior of such systems. The program is initially focused on 

the Internet and grid computing, and it will facilitate predictability and 

reliability in these areas and in other complex systems (e.g., biotechnology, 

nanotechnology, semiconductors, and complex engineering). 

 Cloud Computing Program: The purpose of this program is to accelerate the 

federal government’s secure adoption of cloud computing by building a U.S. 

government cloud computing standards roadmap12 that focuses on the highestpriority U.S. government cloud computing security, interoperability, and 

portability requirements; and by leading efforts to develop standards and 

guidelines in close consultation and collaboration with standards bodies, the 

private sector, and other stakeholders. 

 National Initiative for Cybersecurity Education: This initiative was 

established to “build a comprehensive framework that promulgates the 

availability of education, training, and awareness resources, designed to 

improve the cybersecurity knowledge, skills, and behavior of every segment 

of the population.”13

 Quantum Information Program: In order to develop a measurement and 

standards infrastructure for information systems based on the principles of 

quantum physics, this program pursues the following objectives: to understand 

the potential for quantum information to revolutionize information science; to 

develop theory, methods, architectures, and algorithms to enable the 

engineering and testing of quantum computing components and systems; and 

to demonstrate and to test secure, commercial-grade communication 

components, systems, and protocols for the quantum era. 

 Identity Management Systems Program: The purpose of this program is to 

advance the development and adoption of fingerprint, face, and iris 

identification and surveillance technologies through the designing of 

appropriate performance metrics, evaluation methodologies, test suites and 

test data, prototypes and testbeds, workshops, and standards and guidelines. 

 Health Information Technology Program (Health IT Program): This program 

was established to support the accelerated development and harmonization of 

standards for health technologies, to create a health IT testing infrastructure, 

to consult on certification processes, to expand R&D and the deployment of 

security protocols, to support the usability of health technologies, and to 

address health care development beyond traditional physical locations, such as 

telemedicine and pervasive health care. 

 Pervasive Information Technology Program: This program studies the trend 

toward increasingly ubiquitous connected computing sensors, devices, and 

networks that monitor and respond transparently to human needs. The 

program promotes the development of standards and measurement methods 

 12 The first edition of the NIST Cloud Computing Standards Roadmap, NIST SP 500-291, July 

2011, is available at http://collaborate.nist.gov/twiki-cloudcomputing/pub/CloudComputing/StandardsRoadmap/NIST_SP_500-291_Jul5A.pdf. Accessed August 12, 

2011. 13 Contained in the program descriptions provided to the panel by the ITL staff. 
9 

for reliable, interoperable, and ubiquitous communication and networking of 

personal and medical devices by facilitating the creation of standards for 

sensor communication, networking interoperability, and sensor information 

security enabling the use of pervasive information technologies to enhance 

personal and professional productivity and quality of life. 

 National Strategy for Trusted Identities in Cyberspace (NSTIC): This is a 

White House initiative for improving the privacy, security, and convenience 

of sensitive online transactions. This work is to be done collaboratively with 

the private sector, advocacy groups, public-sector agencies, and other 

organizations. The goals of the NSTIC are “to protect individuals, businesses, 

and public agencies from the high costs of cyber crimes such as identity theft 

and fraud, while simultaneously helping to ensure that the Internet continues 

to support innovation and a thriving marketplace of products and ideas.”14

 Virtual Measurement Systems Program: This program was established to 

investigate uncertainties produced primarily by computer simulations or by 

computer-assisted measurements. The program introduces metrological 

constructs (i.e., standard references, uncertainty characterization and 

propagation, and traceability) into scientific computation and computerassisted measurement technologies. A “virtual measurement” is information 

related to a physical model or system, but gleaned from analysis and 

measurement of a computer model or a computer simulation together with 

uncertainties in the computed quantities. Examples might include 

computational models of physical systems and visualizations of the results. As 

with physical measurement systems, the development of a virtual metrology 

infrastructure will yield predictive computing with quantified reliability, 

resulting in better-informed decision making when the results of computer 

simulations are used. 

Five of the above programs (Cloud Computing, Health IT, Pervasive IT, NSTIC, 

and Virtual Measurement Systems) are led from the ITL Headquarters Office. 

In addition to these programs in the focused R&D areas, the ITL conducts the 

following program: 

 Voting Standards Program: This program responds to the mandates in the 

Help America Vote Act of 2002 (HAVA; Public Law 107-252) and the 

Military and Overseas Voter Empowerment (MOVE) Act of 2009 (Public 

Law 111-84) by developing new standards and test methods; this program 

also conducts research that supports innovative technologies. 

 The ITL works on programs supporting national priorities and on other programs 

deemed to be strategic to the ITL. The Quantum Information, Health IT, and Voting 

Standards Programs are examples of programs addressing national priorities. Strategic 

programs include the Complex Systems, Pervasive IT, and Virtual Measurement Systems 

Programs. 

 14National Institute of Standards and Technology, Information Technology Laboratory, June 2011, 

p. 5. See http://www.nist.gov/itl/upload/ITLbrochure2011.pdf. Accessed June 11, 2011. 
10 

 The ITL’s approach to program management is to work either within a division 

or to work in a collaborative, crosscutting fashion across divisions. Examples of 

crosscutting programs addressing national priorities are the Quantum Information, Health 

IT, and Voting Standards Programs. 

 The ITL continues to produce products of national and international import. 

Some examples include the following: 

 The Digital Library of Mathematical Functions (DLMF): This work provides 

carefully selected, edited, and validated mathematical reference information 

covering a broad area of applicable mathematics; it is a unique and enduring 

accomplishment without peer in the broader community. Ongoing work on 

the DLMF includes maintenance, graphics, infrastructure for Math-on-theWeb, tables on demand, and the Painleve Project (addressing Painleve 

transcendents, a new class of functions represented in the DLMF). 

 Performance metrics, evaluation methodologies, test suites and test data for 

fingerprint, face, and iris identification and multibiometrics. 

 Standards (American National Standards Institute [ANSI]-NIST/ITL): These 

standards are for biometric data-exchange formats, biometric sample quality, 

biometric acquisition and processing protocols, and conformance testing 

methodologies. 

 NIST Special Publication 800* series: These publications are renowned for 

providing technically sound, unbiased, relevant guidelines that are frequently 

adopted voluntarily in private-sector procurements and practices and often 

mandated by the Office of Management and Budget (OMB) for use by the 

federal government. 

 Cryptographic standards and guidelines: These include the Advanced 

Encryption Standard (FIPS-197), Recommendation for Random Number 

Generation Using Deterministic Random Bit Generators (SP800-90), and 

Recommendation for Block Cipher Modes of Operation (SP800-38 series). 

OBSERVATIONS AND RECOMMENDATIONS 

 Following are observations and recommendations of the panel resulting from its 

2011 assessment of the Information Technology Laboratory. Observations 1 through 3 

pertain directly to how the ITL is performing with respect to the three assessment criteria 

from the Director of NIST. Observations 4 through 6 address changes that have taken 

place since the 2009 assessment by the NRC panel appointed for that assessment.15 

Observations 7 through 10 focus on areas of concern. 

Observations 

1. The programs of the Information Technology Laboratory are focused on 

research and development that advance measurement science, standards, and 

 15 National Research Council, An Assessment of the National Institute of Standards and 

Technology Information Technology Laboratory: Fiscal Year 2009. Washington, D.C.: The National 

Academies Press, 2009. 
11 

technology. As an example, the Virtual Measurement Systems Program has 

identified the importance of understanding virtual measurements and 

uncertainties in advancing industry’s increasing reliance on software 

modeling and simulation in, for example, the design of new, advanced 

products. Similarly, the Cloud Computing Program is working on a U.S. 

government cloud computing technology roadmap that is focused on the 

highest-priority national cloud computing interoperability, portability, and 

security requirements. The Health Information Technology Program is 

working to improve standards for health technologies. These programs and 

the others reviewed are all making substantial progress toward meeting their 

objectives and are well aligned with the ITL mission and responsibilities. 

2. The technical merits and scientific caliber of the current ITL programs are 

very high relative to comparable programs worldwide as measured by 

publications and especially by outstanding products such as the Digital 

Library of Mathematical Functions (DLMF) and the NIST Special 

Publication 800* series. The DLMF is without peer in the broader 

community, and the NIST Special Publication 800* series is renowned for 

providing technically sound, unbiased, relevant guidelines that are frequently 

adopted voluntarily in private-sector procurements and practices and often 

mandated by the Office of Management and Budget for use by the federal 

government. 

3. The ITL R&D efforts appear to be carefully aligned with the mission-critical 

deliverables for which the ITL is responsible. Programs in cloud computing, 

health information technology, identity management, cybersecurity 

education, trusted identities, and voting standards are all addressing national 

priorities in information technology. National priorities with critical 

information technology aspects are being addressed by projects in 

biosciences and bioimaging, cyber physical systems, forensics, greenhouse 

gas measurement, optical medical imaging, public safety communications, 

quantum information, smart grid, and trusted networking (Internet Protocol 

Version 6 [IPv6], Domain Name System Security Extensions [DNSSEC]). 

4. The Software and Systems Division (SSD) has made great strides since the 

previous assessment panel registered concerns in its 2009 report.16 The most 

prominent concern was “the lack of strong scientific and administrative 

leadership within the SSD and also, in some cases, at the programmatic 

level.”17 Today those concerns are being aggressively addressed, and the 

SSD has become more focused and better able to respond to its current 

challenges. 

5. The ITL leadership has done an excellent job in filling two critical 

management positions: division chief for the Computer Security Division 

(CSD) and division chief for the Software and Systems Division. The ITL 

management is still faced with finding a permanent chief for the Advanced 

Network Technologies Division (ANTD). 

6. The ITL has struggled with how crosscutting programs—those that involve 

work in a collaborative fashion across divisions—would be managed, since 

 16 Ibid. 

17 Ibid., p. 15. 
12 

they do not fit neatly into the divisional structure. The ITL answer has been 

to use a matrix management structure. In 2007,18 less so in 2009, the panel 

was aware of considerable angst on the part of management and staff as to 

how that would work. This year there were no signs of that distress. It 

appears that the ITL has done an excellent job of working out the kinks and 

implementing matrix management. 

7. The Statistical Engineering Division (SED) is continuing on an even keel 

with strong leadership and technical expertise. However, as observed in the 

2009 assessment report, the division workload is growing but the division is 

not. The SED is seriously understaffed, and this problem needs to be 

addressed with some urgency.19

8. The Computer Security Division is also understaffed, although neither 

performance nor morale has as yet been affected. 

9. The work of the Applied and Computational Mathematics Division (ACMD) 

continues to be excellent. However, the scientific culture of the division may 

not be sufficiently focused on collaboration to address the problems of 

multiscale and multiphysics involving complex geometries that are emerging 

as national priorities. 

10. The Advanced Network Technologies Division is doing an excellent job in 

responding to several national priorities in both the short and long term, 

including its continued outstanding activities in Internet infrastructure 

protection and its newer efforts in smart grids and public safety 

communications. The division has also improved the quality of its internal 

and external collaborations, as well as the quality of its publications. The 

ANTD is understaffed for the portfolio of activities that it is undertaking. The 

various teams handling projects with short deadlines do not have as much 

time to dig into the subjects as they would like or would be useful. Another 

consequence of the understaffing is that basic research activities are perhaps 

below levels that are healthy. ANTD management has not yet articulated a 

long-term, strategic view of networking. 

Recommendations 

1. At least two ITL divisions, the Statistical Engineering Division and the 

Computer Security Division, are feeling the constraints of increasing 

workloads and insufficient staffing (the SED more so than the CSD). If the 

ITL is to maintain its prominence in these areas, it should consider plans to 

address the growth that will be needed to support the expanding workload of 

each of these divisions. 

2. Because the trend toward simulations of increasing model fidelity and 

numerical accuracy is expected to continue, the Applied and Computational 

 18 National Research Council, An Assessment of the National Institute of Standards and 

Technology Information Technology Laboratory: Fiscal Year 2007. Washington, D.C.: The National 

Academies Press, 2007. 19 National Research Council, An Assessment of the National Institute of Standards and 

Technology Information Technology Laboratory: Fiscal Year 2009. Washington, D.C.: The National 

Academies Press, 2009, pp. 2, 9, 14. 
13 

Mathematics Division will be called on to play an increasing role in 

addressing problems that are multidisciplinary. To ensure that the ITL is 

ready to support this work, the ACMD should devise a strategy to change the 

scientific culture of the division to meet those increased challenges. 

3. The ITL should fill the position of chief of the Advanced Network 

Technologies Division with a permanent chief. ANTD management should 

address the understaffing issue in the division, and in particular it should 

ensure that there are adequate resources to handle both the short- and longterm needs of the division. ANTD management should create a strategic 

roadmap for the technical work of the division. The roadmap should be 

useful in managing the division’s resources and portfolio of activities. 

4. The ITL should devote attention now to strategic, long-term technical needs 

in cloud computing that NIST may be called on to address in the future, 

including questions surrounding the scale of cloud computing and how such 

a scale could be accommodated in a laboratory or simulation environment. 

5. The ITL should consider creating a collaborative effort between the 

Computer Security Division and the Software and Systems Division that 

would be responsible for the creation of standards and guidelines on secure 

software development for application by government, industry, and 

academia. 

6. The ITL and the Software and Systems Division should reconsider the SSD 

mission statement, given the fresh focus of the new leadership, and after the 

SSD strategic planning process is complete. 

7. The ITL and the Software and Systems Division should hire additional 

formally trained individuals in the SSD’s core foundational areas. 

8. The Information Access Division (IAD) supports the development of 

technologies and their transition into the commercial marketplace as well as 

government applications. The division currently relies on substantial and 

sustained amounts of other agency (OA) funding (approximately 60 percent 

of IAD funding). Most of the OA funding is security-related (from the 

Department of Homeland Security, the Department of Defense, the Federal 

Bureau of Investigation, and the Intelligence Advanced Research Projects 

Activity). The reports, standards, and evaluation studies of the IAD are 

closely followed by academia and industry. In light of increasing foreign 

dominance of the biometric industry, IAD’s reliance on OA funding, and 

IAD’s work in support of biometrics technology development, it is important 

that the IAD and the ITL remain mindful of the NIST mission to promote 

U.S. innovation and industrial competitiveness, and so IAD efforts should 

continue to place highest priority on the needs of the nation’s commerce even 

while pursuing activities involving international sponsors. 

9. The ITL should review the approval process of the Institutional Review 

Board20 to maximize the efficiency of the process and minimize unnecessary 

latency. 

 20 See http://www.hhs.gov/ohrp/humansubjects/commonrule/. Accessed July 11, 2011. The Office 

for Human Research Protections at the Department of Health and Human Services provides oversight for 

the protection of human subjects in research through the regulations that are spelled out for Institutional 

Review Boards in the so-called Common Rule (45 C.F.R. 46). 
14 

LABORATORY RESPONSES TO THE 2009 PANEL REPORT 

In the 2009 report of the NRC review panel,21 seven recommendations were made 

to the ITL. The panel recommended as follows in the 2009 assessments: 

1. ITL staff, perhaps led by the program managers, should look for linkages with 

external organizations such as research universities and laboratories. The 

recent addition of temporary funding associated with the economic recovery 

can help build these connections. 

2. The ITL should make efforts to raise its profile through outreach (connections 

with major research universities and laboratories, hosting faculty, postdoctoral 

researchers, and other short-term visitors; and staff participation in professional 

service) and publication (in highly respected journals and conferences). 

3. Program managers who are capable of providing technical leadership and also 

devote effort to promoting the interests of their programs should be regarded 

by the staff as positive contributors, even if they are no longer writing code or 

doing other technical tasks associated with individual projects. 

4. There is a need for additional senior technical leadership. 

 The Software and Systems Division (SSD) needs to hire a strong health 

informatics leader. 

 NIST should appoint a full-time chief for the SSD, which currently has an 

acting chief who divides time between leading the division and working in 

the Office of the ITL Director. 

 The panel found multiple cases of the SSD’s suffering from a lack of 

sufficient focused leadership at a time when the SSD is being asked to be 

the lead in several important efforts, such as health care. 

5. SSD leadership should encourage its staff toward greater innovation and 

redirection in keeping with developments in the broader research and 

scientific community. 

6. Apart from the current chief, there has been no perceptible growth in the 

permanent staff of the Statistical Engineering Division for years. The division 

is short-staffed, and such growth should be pursued with urgency before the 

next review. 

7. The ITL needs a process for sunsetting programs and encouraging the bottomup development of new programs. 

Overall, the ITL provided to the current panel adequate responses to the seven 

recommendations in the 2009 report. Several observations need to be made, however, 

regarding the ITL responses: 

1. The combined responses to the first two recommendations in the list above 

were appropriate. Prestigious publications and professional activities should 

certainly raise awareness of NIST in the scientific community. However, 

missing from the ITL response was any discussion about how effective these 

 21 National Research Council, An Assessment of the National Institute of Standards and 

Technology Information Technology Laboratory: Fiscal Year 2009. Washington, D.C.: The National 

Academies Press, 2009, pp. 1-2. 
15 

activities have been in the R&D area. Such a discussion would have been 

enlightening and should be included in all future responses to panel 

recommendations. 

2. The sixth recommendation was not completely satisfied. Two new employees 

were hired, but their impact was lessened by the loss of two staff members. 

The ITL’s action maintained the status quo, and once again it is recommended 

that there be increased staffing for the SED. 

3. The ITL’s response to the seventh recommendation was a description of how 

programs were evaluated, changed, or moved during 2010, but there was no 

mention of sunsetting or of a process for sunsetting, so the response was not 

adequate. The ITL still needs to address the issue of a sunsetting process. 
16 

3 

Assessment of the Laboratory Divisions 

The Information Technology Laboratory is a very strong scientific and technical 

resource for the nation with a long history of making lasting contributions. The ITL is 

composed of six divisions: Applied and Computational Mathematics, Advanced Network 

Technologies, Computer Security, Information Access, Software and Systems, and 

Statistical Engineering. The ITL’s staff resources total 562, comprising the following: 

322 technical federal staff members (57 percent of the total), 176 technical guest 

researchers and contractors (31 percent), 46 administrative staff (8 percent), 15 faculty (3 

percent), and 3 postdoctoral researchers (1 percent). Of the ITL staff, 38 percent are 

computer scientists, 20 percent are mathematicians or statisticians, 17 percent are 

information technology specialists, 12 percent are engineers or physicists, 12 percent are 

administrative and support personnel, and 1 percent are social scientists. The following 

sections present summary assessments for each of the ITL divisions. 

APPLIED AND COMPUTATIONAL MATHEMATICS DIVISION 

 The Applied and Computational Mathematics Division provides leadership within 

NIST in the use of applied and computational mathematics to solve scientific and 

engineering problems that arise in measurement science and related applications. The 

division’s staff does research and development in mathematics and computation and 

participates with NIST colleagues in peer-to-peer collaborations on NIST problems. 

Further, the ACMD supports the national and worldwide science and engineering 

communities by developing and disseminating mathematical reference data, software, 

and related tools. The division works with internal and external groups to develop 

standards, test procedures, reference implementations, and other measurement 

technologies for scientific computation. 

 The ACMD is composed of four groups: Mathematical Analysis and Modeling, 

Mathematical Software, Computing and Communications Theory, and High Performance 

Computing and Visualization. The division has 57 federal employees; 51 are at the 

Gaithersburg, Maryland, NIST campus, and 6 are at the Boulder, Colorado, NIST 

campus. Three of the federal employees are postdoctoral researchers. The ACMD also 

has 29 associates, of whom 23 are guest researchers, 1 is a postdoctoral researcher, and 5 

are students. 

 In regard to the charge of the Director of NIST to assess the degree to which 

laboratory programs in measurement science, standards, and services achieve their stated 

objectives and fulfill the mission of the ITL: the division has an impressive array of 

technical expertise, and it is attacking a wealth of research areas. The quality of the work 

meets the computational accuracy needs of consumer laboratories, which vary widely 

among applications (e.g., from accuracy sufficient for visualization to highly accurate 

computations for atomic structure). In most areas, the work of the division is very good 

and comparable to what one would find in top 10 to top 20 university 

applied/computational mathematics programs. In some cases—for example, the Digital 
17 

Library of Mathematical Functions—the work is unique and without peer in the broader 

community. 

 The work of the division is accomplished primarily through a culture of small 

“atomic” collaborations common in the work of applied mathematics. Impressively, 

members of the division have established effective collaborations with materials science 

and physics researchers. 

 On the downside, however, the culture of small, atomic collaborations may be a 

hindrance to attacking important areas that require the formation of collaborative teams 

from across many disciplines. However, the DLMF work demonstrated a capacity to 

form a larger team of collaborators than is typical of this division. This ability to form 

larger, effective collaborations may, in the long run, bode well for ACMD’s ability to 

cope with future multidisciplinary problem areas in which the ACMD will be called on to 

participate. 

 In regard to the charge of the Director of NIST to assess the technical merits and 

scientific caliber of the current laboratory programs relative to comparable programs 

worldwide: There are areas in which the accomplishments and capabilities of the ACMD 

are excellent and surpass the capabilities of comparable programs worldwide. The ability 

of staff to engage with physicists and materials scientists to use mathematics and 

computing to provide insight on experiments is among the most successful such efforts in 

the world. The Digital Library of Mathematical Functions is a unique and enduring 

accomplishment. The Quantum Communications Program seems to be among the best in 

the field. 

 In other areas the level of technology in the division work, while still excellent, is 

below the state-of-the-art level. That is, the work is excellent by academic standards in 

terms of the development of novel mathematical methods but is below the state of the art 

when measured against the work of the division’s peers in other mission agencies in 

meeting the goals of providing mathematical modeling and simulation expertise to the 

rest of their organizations. For example, in the area of large-scale partial differential 

equation (PDE) simulation, capabilities are mostly (but not exclusively) two-dimensional. 

In addition, the level of mathematical rigor varies widely across the division. Code 

verification is now viewed as an essential component for any large-scale simulation 

project. Very systematic convergence studies similar to those done in the Parallel 

Hierarchical Adaptive MultiLevel (PHAML) project might be adopted in other areas as a 

method of code verification. 

 As to the charge to assess the alignment between laboratory R&D efforts and 

those services and other mission-critical deliverables for which that laboratory is 

responsible: While the level of technical expertise in the ACMD is certainly sufficient to 

meet upcoming challenges, some significant cultural problems in the program 

composition and scientific culture in the division could inhibit the response of the ACMD 

to the challenges that the division views as critical to meeting its future needs. 

 The challenges facing the ACMD are multiscale and multiphysics, involve 

complex geometries and new applications, and are of necessity multidisciplinary. In 

addition, computational science is facing a major disruptive change in technology. This 

change is the transition to multiple computing cores on a single processor, for which 

many of the methods for parallelization and software engineering developed over the past 

15 years lead to very low performance. Since this is an architectural change on the level 

of a single processor, it will affect scientific computing at all scales. 
18 

 The response of the larger applied mathematics community to the vastly increased 

complexity of these problems has been to form larger teams, sustained over long periods 

of time, in order to span the broad range of intellectual capabilities and challenges 

required to make a successful attack on such problems. This is being done at universities, 

at funding agencies, and at other national laboratory systems. This kind of change has not 

yet taken place in the ACMD, and in fact it runs contrary to the more traditional “small 

science” model in applied mathematics that has served the ACMD well in meeting many 

of its mission requirements. It is certainly possible for the ACMD to form larger teams, 

as evidenced by the DLMF and (to some extent) by quantum communications projects. 

However, if the problems that NIST is investigating or will investigate are multiscale and 

multiphysics, then such an approach will need to become more widely used in the 

ACMD. Such a change may well require the ITL and NIST management to reconsider 

the role of the ACMD in the laboratory as well. Owing to resource constraints, it may be 

necessary to focus on a set of technologies and applications that lead to a high degree of 

synergy, possibly at the cost of reducing the range of stakeholders at NIST. Another 

possible approach to meeting these challenges, particularly in the simulation software 

area, would be to collaborate with the laboratories in other agencies that would bring 

strengths complementary to those of the ACMD (such as its success in collaborating with 

physical scientists) and with whom the division could share the responsibility of software 

development. The ITL and the ACMD should consider strategies to change the scientific 

culture of the ACMD to meet the increased challenges facing the division. In addition to 

broader, multidisciplinary simulations, discrete simulations might also be an area in 

which the ACMD can address important challenges that align with the NIST mission 

(network standard, for example). 

ADVANCED NETWORK TECHNOLOGIES DIVISION 

 The mission of the Advanced Network Technologies Division is to “provide the 

networking industry with the best in test and measurement research.”22 Its goals are “to 

improve the quality of emerging networking specifications and standards,” and “to 

improve the quality, reliability, resilience, robustness, security, and interoperability of 

networked systems.”23 The ANTD assists in the conception and development of national 

priorities in which networking is an essential enabler, and additionally it proactively 

participates in the implementation of those developments.

The ANTD is composed of two groups: the Emerging and Mobile Network 

Technologies and the Internet and Scalable Systems Metrology groups. The ANTD has a 

regular staff of 29, and there are 10 guest researchers. 

 The ANTD focus areas are core networking infrastructure, ad hoc networks, edge 

networks, complex systems, and networks and applications (cloud computing, smart grid, 

and health care). The division’s delivery mechanisms include specifications and 

guidelines, models and prototypes, test and measurement tools, reference 

implementations and demonstrations, journal and conference papers, workshops and 

conferences, and standards activities. 

 22 Donna F. Dodson, ANTD and CSD, “Advanced Network Technologies Division,” presentation 

to the panel, Gaithersburg, Maryland, March 21, 2011, p. 2. 23 Ibid., p. 3. 
19 

High-Level Observations and Recommendations 

 The Advanced Network Technologies Division is strong and healthy, exhibiting 

an awareness of and competence in achieving its mission within the ITL and within NIST 

as a whole. Morale and motivation across the division appear high. The ANTD leverages 

its relatively small size by taking advantage of opportunities for collaboration, both 

internally with other divisions and with external organizations. It understands its mission 

and strives to balance various demands on its attention, including mandates, supporting 

science, standards advocacy, and strategic collaborations. The division demonstrates an 

aptitude for responding to externally imposed agendas, often under tight time lines with 

limited resources, while attempting to maintain a solid scientific foundation. 

 Networking is a key enabling technology for many, if not all, of the high-priority 

activities of the ITL and NIST in general. National innovation and industrial 

competitiveness are fueled by a robust, secure, and accessible national infrastructure. The 

ANTD is and should continue to be at the forefront of the national discussion on vital 

issues, standards, and policies of networking and worldwide communications 

infrastructures. 

 Since the previous NRC review in 2009, the ANTD has made significant progress 

on that panel’s suggestion for expanding the division’s exposure in the greater 

community outside of specific standards organizations. The ANTD has increased the 

quantity and quality of its publications, including papers in conference proceedings and 

journals that the panel noted as being of high quality, and a staff member received a bestpaper award. The ANTD also has widened its external collaborations, in some instances 

with grants and sometimes by seeking out companies, universities, and other research 

organizations with common interests. 

 Despite these observed accomplishments, the ANTD faces several challenges. 

First, it needs to acquire without delay a full-time director to articulate ANTD’s values 

and interests and to define effectively long-term directions for the division and its 

projects. Second, as a key collaborator in many projects, the ANTD finds itself 

understandably drawn into activities that address many national priorities; these activities 

are not only aligned with NIST’s charter but also positively impact national policy and 

the economy. However, as a consequence of so many commitments, the division seems to 

be understaffed, and the researchers’ ability to balance short-term and long-term NIST 

goals can be impaired. Ideally, near-term projects in areas such as smart grid and safety 

networks can be developed into core competencies, aligned with long-term strategic goals 

in these important areas. 

Recommendations 

 Following are the recommendations of the panel with respect to the Advanced 

Network Technologies Division: 

1. The ITL should fill the position of chief of the ANTD with a permanent chief. 

2. ANTD management should address the understaffing issue in the division, 

and in particular it should ensure that there are adequate resources to handle 

both the short- and long-term needs of the division. 
20 

3. ANTD management should create a strategic roadmap for the technical work 

of the division. The roadmap should be useful in managing the division’s 

resources and portfolio of activities. 

4. In addition to its portfolio of important activities, the ANTD should embrace 

the future importance of cloud computing and draw up a plan for ANTD’s 

leading role in a multidivision initiative within the ITL. 

Project-Specific Observations and Recommendations 

Internet Infrastructure Protection 

 In the area of Internet infrastructure protection, the ANTD has an exemplary 

portfolio that represents technical foundations, standards leadership, policy guidance, and 

adoption facilitation. The ANTD group provided an exceptionally lucid and compelling 

presentation for the panel, detailing the motivation, accomplishments, and impacts of its 

outstanding multifaceted work in this area. Particularly in the area of standards—with 

regard both to the definition of standards (e.g., in which group members have co-authored 

key standards documents) and to their adoption (e.g., IPv6 and the .gov deployment of 

the DNSSEC)—the group is at the forefront of the field. The group’s leadership here 

results in part from its sustained and long-term efforts in the area. Developing formal 

methods to prove security properties of the various systems under study might be a 

worthwhile research topic; although this topic had been explored in the past, recent 

developments may justify another look at it as a possible growth area. 

Smart Grids 

The area of smart grids has tremendous impact and importance, and it fits into 

NIST’s mission for supporting infrastructures that increase U.S. competitiveness. This is 

new work for the division, and consequently the current work is somewhat narrowly 

focused and does not take advantage of the breadth of the possible challenges. This 

narrow focus may be because the group is responding to specific external requests. 

Networking researchers can and should play a significant role in helping to define and 

evaluate future smart grid architectures, including and beyond the communications 

aspects of smart grids. The networking field, with more than 40 years of combined 

experience in developing a highly scalable, informationally decentralized network-ofnetworks, may well be able to contribute architectural, measurement, management, 

control, and security techniques to the smart grid. 

Public Safety Communications 

The efforts of the ANTD in public safety communications are aimed at 

developing performance-analysis tools to evaluate deployment scenarios and to evaluate 

new technologies. These efforts are in collaboration with a number of federal agencies, 

including the Department of Homeland Security (DHS) and the National Institute for 

Occupational Safety and Health (NIOSH), as well as the Institute for Telecommunication 

Sciences in Boulder, Colorado. The activities thus far have been a conceptually 

straightforward evaluation of wireless-network deployment scenarios with various 
21 

channel models, with existing radio technologies. The private sector has likely developed 

similar (possibly proprietary) tools. Before embarking on a large-scale or long-term 

activity in this area, an analysis of existing tools (both public and private) should be 

conducted, and partnerships with any promising tool developers should be explored. 

Body Area Networks 

Body Area Networks is a cross-divisional program (involving collaborators from 

the ANTD and the Applied and Computational Mathematics, Statistical Engineering, 

Software and Systems, and Information Access divisions) with a number of outside 

collaborators. The panel was briefed only on the project developing a radio-frequency 

propagation model for implantable and wearable medical sensors. The group is making 

solid progress on the development of the radio propagation model using detailed models 

of the human body and on the demonstration of the use of these models to determine 

channel performance. These models have been included in a modeling tool. Model 

validation for body surface propagation has proven extremely valuable and has provided 

important insights; additional validation and continued (and extended) collaboration with 

external research groups will be valuable. Congratulations are in order for the group’s 

receipt of the Institute of Electrical and Electronics Engineers Personal, Indoor and 

Mobile Radio Communications Conference Best Paper Award. 

Measurement Science for Complex Information Systems 

The area of measurement science for complex information systems is an excellent 

example of core competency development through multidisciplinary, multidivisional 

research (e.g., involving researchers from the ANTD and the Statistical Engineering and 

the Applied and Computational Mathematics divisions). This work addresses an 

increasingly common challenge in networking (and, more generally, the emerging 

interdisciplinary field of network science) in which complex, large-scale systems must be 

evaluated over a high-dimensional parameter space. The interdisciplinary contributions 

here make the whole greater than the sum of the parts. The connection to a specific 

application (cloud computing in this case) and the choosing of model details (such as the 

virtual machine types and hardware platform classes) based on an Amazon Elastic 

Compute Cloud (EC2) data center, are beneficial. The transition from congestion control 

to a topical new area while retaining all previously acquired knowledge is a 

commendable accomplishment. This is a well-chosen area with strategic vision. 

COMPUTER SECURITY DIVISION 

The Computer Security Division is composed of three groups: Cryptographic 

Technology, Systems and Emerging Technologies Security Research, and Security 

Management and Assurance. The CSD’s mission, “[to] provide standards . . . to protect 

our nation’s . . . information systems,”24 is being achieved exceptionally well. Generally 

speaking, the division’s activity can be divided into two categories: (1) activities that 

directly create and communicate mission-critical deliverables, and (2) basic research and 

 24 Donna F. Dodson, ANTD and CSD, “NIST Computer Security Division,” presentation to the 

panel, Gaithersburg, Maryland, March 21, 2011, p. 2. 
22 

development activities that support the identification, improvement, and creation of those 

mission deliverables. 

 Mission-critical deliverables include a broad range of standards (including 

cryptographic standards), technical and management guidelines, validation, testing, and 

measurement tools, and technical reference materials and specifications to support 

agency implementations. As a shining example, the NIST Special Publication 800* series 

is renowned for providing technically sound, unbiased, relevant guidelines that are 

frequently adopted voluntarily in private-sector procurements and practices and often 

mandated by the OMB for use by the federal government. It is generally agreed that 

NIST security guidelines are consistently superior to guidelines from other sources, 

which are often vague, unrealistic, incomplete, or idiosyncratic. 

 The division program includes the following: coordination of large new efforts 

such as the cryptographic hash competition and the Security Content Automation 

Protocol Program; independent good judgment as exercised in the specification of the 

transition to new key lengths; the development of conceptual taxonomies that structure 

research in new fields such as cloud computing; the development of validation of 

products, tools, and techniques through tools and trained outsiders; and outreach to 

agencies, industry, and academia through NIST-managed workshops, external 

conferences, and government committee participation. Given the broad scope of the 

publication series, some publications will need updating soon; the authors are aware of 

this need. The National Vulnerability Database (NVD) is another example of an 

outstanding collaborative effort that is widely used and appreciated across the federal 

government and by U.S. business. Efforts in the important area of authentication, such as 

the study of biometrics risks and participation in the public-private NSTIC digital identity 

initiative, tackle a messy, urgent problem for which no fully satisfactory solution is ready 

yet. 

 Research and development activities explore basic science and engineering that 

will sustain the division’s ability to create high-quality deliverables into the future as new 

challenges emerge. Several of these projects are closely related to the division’s programs 

in standards and technical guidance. For example, the work on combinatorial circuit 

analysis and on side channel and differential power analysis is related to the division’s 

work on cryptographic module validation. Such research work is important in that it 

provides the division with the technical know-how required to evaluate modules or 

components that implement cryptography. 

 The merit and caliber of the division’s mission deliverables are widely 

acknowledged as being world-leading. Historical examples of this leadership include the 

Advanced Encryption Standard (AES) introduced in 2001. The AES has been accepted 

internationally and is an exemplar for conducting open competitions for cryptographic 

standards. The CSD is at present conducting an international competition for a new 

secure hash algorithm, the so-called SHA-3, and all indications are that the quality and 

integrity of this process are at the same high levels. A significant number of other 

security-related standards have been developed or are in process now (block cipher 

modes, digital signatures, and key management, for example), and many influential 

special publications (SPs) have been written (SP 800-108, Recommendation for Key 

Derivation Using Pseudorandom Functions; SP 800-132, Recommendation for PasswordBased Key Derivation, Part 1: Storage Applications; and SP 800-38B, Recommendation 

for Block-Cipher Modes of Operation: The CMAC Mode for Authentication). 
23 

Security tools such as the National Vulnerability Database have become highly 

regarded international resources that are critical to modern security engineering. These 

CSD activities are singular and are setting global standards for quality. Their value and 

success have led others to create derivative products (sometimes not as high quality) that 

may confuse users about the source of the products. Accordingly, NIST and the CSD 

should protect their brand in this space as much as possible. For example, developing 

quantitative analysis techniques and tools that define and estimate metrics from the NVD 

would be one way to add value and to further NIST’s branding in this area. 

 The merit and caliber of the division’s basic R&D activities are comparable to 

those of its peer, mission-oriented government laboratories such as those operated by the 

Department of Energy, Department of Defense (DoD), National Aeronautics and Space 

Administration, and National Institutes of Health. R&D activities of the division that are 

most closely aligned with current and future NIST mission deliverables include work in 

biometrics, cloud computing, virtualization, secure boot processes, test coverage 

efficiency, and mobile platforms. Those activities are of high quality and are comparable 

to applied academic, industrial, and government laboratory efforts. 

 There appears to be solid alignment between the division’s R&D efforts and its 

mission-critical deliverables. When a research project is being proposed and created, it 

will be leading edge to some degree—ahead of the specific standards and guidelines that 

it is intended to support. Such is inevitably the case: research has to run far enough ahead 

of the work on standards and guidelines to provide meaningful technical input. However, 

this constraint means that any given research program may prove to be inapplicable to 

technical standards, either because the research does not succeed or because the 

directions of the market, standards, and technology diverge from the direction of the 

research effort. As a general principle, given constraints on resources and the dynamic 

nature of IT security technology, the division should be mindful of the relevance of its 

research projects to the remainder of its mission and should be willing to sunset projects 

in those cases in which the project has begun to achieve industrial or commercial success 

or the focus of the project has diverged from the mainstream direction of information 

technology or from the division’s work on standards and guidelines. The Role Based 

Access Control Program appears to have achieved a measure of industrial success and is 

perhaps a candidate for handing off to industry. 

 A recommended new initiative for the division would be to undertake a formal 

collaboration with the Software and Systems Division in the area of secure software 

development. Application-level vulnerabilities are a growing focus of hostile attacks in 

both locally hosted and cloud computing environments. It is important that NIST bring 

together its resources in security and software development to contribute in this area and 

to prepare for the development of standards and guidelines that will help to raise the level 

of the nation’s security in this important area. The ITL should consider creating a 

collaborative effort between the CSD and the SSD that would be responsible for the 

creation of standards and guidelines on secure software and enterprise-level system 

development for use by government, industry, and academia. The CSD is also 

encouraged to investigate, as appropriate, NIST’s role in metrics and guidelines for 

privacy, a subject that was not specifically presented to the panel. 

 In sum, this well-managed division has a depth of expertise that contributes to 

task forces defending the United States against advanced cyberthreats. This contribution 

is particularly important given the ongoing national and international attention to the 
24 

cybersecurity challenge. Overall, staff in this division are spread thin, but they seem to be 

juggling priorities well. The morale in the division appears to be quite high, creating a 

positive work environment that has attracted top talent. These two facts—thinly spread 

resources and a history of attracting top talent—suggest that the division should consider 

hiring as a priority. 

INFORMATION ACCESS DIVISION 

Organization and Operation 

 The mission of the Information Access Division is to conduct research of 

performance metrics and guidelines, standards and testbeds, and community-wide 

evaluations in order to support the development of advanced “technologies for intuitive, 

efficient, and effective access, manipulation, analysis and exchange of complex 

information.”25 The division’s projected benefits of its work include the expedited 

development of these technologies and their transition into the commercial marketplace 

and into government applications. 

 The IAD consists of five major groups: Retrieval, Image, Multimodal 

Information, Visualization and Usability, and Digital Media. The IAD also pursues four 

crosscutting program areas: biometrics technology, human language technology, humansystem interaction, and multimedia technology. The division’s staff consists of 66 

permanent employees, 12 contractors, 14 guest researchers, and 7 students—the IAD is 

the largest division in the ITL in terms of staffing. 

 The educational background of the personnel is roughly as follows: one-half has 

master’s degrees, and one-quarter each has doctoral or bachelor’s degrees. The range of 

technical activities covered by these five groups is diverse, yet they can be generally 

described and broadly grouped into types of activities as follows. 

 Biometrics: The use of biometric features for personal identification, 

particularly related to face, fingerprint, iris, voice, and, more recently, 

multimodality, which uses a combination of features for the same purpose; 

 Retrieval (and search): The retrieval of information in the form of image, text, 

video, and three-dimensional objects, in response to user queries; 

 Human language technology: Community-wide technology evaluations that 

assess progress in speech and language for text analysis, machine translation, 

speech analysis, and speaker and language recognition; 

 Visualization and usability: The study of usability by humans related to 

specific implements and systems such as electronic voting machines, health 

information records, and biometric devices; and 

 Multimedia technology: Evaluations, standards, and challenge problems in 

video analysis, digital data preservation, three-dimensional imaging, and 

biomedical imaging. 

 25 Curt Barker and Leslie Collica, ITL and IAD, “Information Access Division (IAD),”

presentation to the panel, Gaithersburg, Maryland, March 21, 2011, p. 5. 
25 

 The biometrics and retrieval activities enjoy a long track record of 

accomplishments. The usability initiative is emerging and showing the potential of 

important impacts. The video analytics, multimedia event detection, and the retail 

surveillance projects exemplify emerging projects of smaller scales. 

 The funding structure of the division is unique among ITL divisions. Sixty 

percent of the division’s budget comes from OA funding (other agencies, such as from 

the DHS, DoD, FBI, and Intelligence Advanced Research Projects Activity), which is 

mostly security-related. Only about a quarter of the budget is classified as IAD base 

funding, and less than 15 percent is from other sources at NIST. 

 The output of the work in the division includes the following: 

1. Standards and guidelines: Defining the best practice along a specified 

dimension or aspect of the relevant technology development; 

2. Data repository: Acquiring, verifying, organizing, and supporting the various 

important standard data sets required for developing and evaluating the 

relevant technologies; 

3. Evaluation and benchmark results: Conducting and reporting impartial 

community-wide evaluation results to help gauge the progress in the relevant 

technology; 

4. Outreach: Organizing workshops (some, e.g., the Text Retrieval Conference 

[TREC],26 have been going on annually for almost 20 years) to present 

findings of the technology evaluations that bring together people from 

academia, industry, and government; hosting students and guest researchers; 

and 

5. Reports, publications, and software: Publishing and archiving relevant results 

and reports; providing benchmark software (e.g., fingerprint image quality and 

matching). 

Division Accomplishments and Observations by the Panel 

 The IAD is responsible for meeting a number of high-value and critical national 

priorities and mandates. Several groups in the division are uniquely qualified to address 

these mandates. Many scientists in the IAD continue to play a major role commendably. 

Examples of these mandates include the USA PATRIOT Act, the Enhanced Border 

Security and Visa Entry Reform Act of 2002, the Help America Vote Act of 2002, and 

the Health Information Technology for Economic and Clinical Health Act of 2009. 

 Within the division’s charter, the groups demonstrate a high level of technical 

competence in conducting long-standing, large-scale technology evaluations in critical 

technologies that are extremely important to the government, business, and academia, not 

only in the United States but worldwide. Examples include open community evaluations, 

such as TREC, and large-scale biometric testing (e.g., fingerprint, face, iris, and speech). 

Given the staff’s expertise and its neutral role, NIST evaluations conducted by the IAD 

are considered fair and beneficial to the sponsors and to technology developers, and they 

foster research in academia. 

 26 TREC is a forum for evaluating technologies for detecting, searching, and retrieving query text 

in a large set of documents. 
26 

 The division appears to take a proactive role in identifying new directions for 

pursuing its mission and is adapting well to meet the national mandates and progress in 

technology. Examples include the progression of the Retrieval Group from the retrieval 

of spoken documents in TREC to video retrieval in TRECVid (TREC for video 

“documents”). The Image Group started out mainly in fingerprint matching and 

evaluation, but it now has broadened its expertise and conducts large-scale evaluations in 

face, iris, and multimodal biometrics. Examples of some other new initiatives include the 

evaluation of computerized tomography (CT) lung image analysis and usability studies 

for biometrics. 

 The IAD demonstrates unique capabilities in establishing biometric data standards 

and large-scale performance evaluation benchmarks. 

Recommendations 

 Following are the recommendations of the panel with respect to the Information 

Access Division: 

1. The Information Access Division supports the development of technologies 

and their transition into the commercial marketplace as well as government 

applications. The division currently relies on substantial and sustained 

amounts of other agency (OA) funding (approximately 60 percent of IAD 

funding). Most of the OA funding is security-related (from the Department of 

Homeland Security, the Department of Defense, the Federal Bureau of 

Investigation, and the Intelligence Advanced Research Projects Activity). The 

reports, standards, and evaluation studies of the IAD are closely followed by 

academia and industry. In light of increasing foreign dominance of the 

biometric industry, IAD’s reliance on OA funding, and IAD’s work in support 

of biometrics technology development, it is important that the IAD and the 

ITL remain mindful of the NIST mission to promote U.S. innovation and 

industrial competitiveness, and so IAD efforts should continue to place 

highest priority on the needs of the nation’s commerce even while pursuing 

activities involving international sponsors. 

2. The IAD should establish a long-term vision and identify some fundamental 

research problems (e.g., relevant measurement science) that it determines 

should be pursued in addition to the ongoing activities. 

3. The Image Group should highlight its work in biometrics and differentiate its 

work from that of other groups in the IAD that also deal with images. 

4. The NIST administration should initiate a review of the interagency 

negotiation process, joined by the individual and relevant external agencies if 

necessary, to reduce the unnecessary administrative impediments that may 

adversely affect the technical progress and jeopardize future contracts. 

5. The ITL should work with the NIST administration, and possibly the policydefining organization, that is, the Department of Commerce, in reviewing the 

institutional approval process related to the Institutional Review Board (IRB) 

to maximize efficiency and minimize unnecessary latency. 

6. The IAD should enhance outreach and external interaction as well as 

personnel professional development. For example, a NIST-wide internship 
27 

program at the graduate level with a tracking mechanism may prove helpful in 

raising the institutional image, the promotion of IAD activities, and the 

recruitment of new talents. 

7. The IAD should review its recruiting policy and practice so as to ensure a 

healthy influx of new and young talent. 

SOFTWARE AND SYSTEMS DIVISION 

The mission of the Software and Systems Division is “to accelerate the 

development and adoption of correct, reliable, testable software leading to increased trust 

and confidence in deployed software, systems, and products.”27 Four research groups 

comprise the SSD: Software Components, Information Systems, Interoperability, and 

Electronic Information. The SSD recently hired a new division chief and two managers. 

The division staff performs high-quality work and is having good impact on a number of 

high-visibility, national priority projects. 

On the question of the degree to which the division’s programs in measurement, 

standards, and services achieve their objectives: The SSD work to date in software testing 

and conformance is strong, although the SSD should look to the future and position the 

division to get measurement, standards, and data sets out ahead of urgent needs. 

Regarding the overall merits and caliber of the program: The SSD efforts in niche areas 

are of unquestioned leadership, and in general the quality of work is comparable to that of 

its peers in any other government or private organization. In addition, inter- and 

intralaboratory collaborations are a key strength of several members of the SSD group. 

On the question of the alignment of division R&D efforts with its mission, the work in 

the SSD is well aligned with the missions of the SSD, the ITL, and NIST; however, 

application-level projects could be better leveraged such that the methods, techniques, 

and tools developed for one project are readily available for reuse in new projects. It is, 

of course, clear that customer needs must be met, but a continuing focus on the potential 

generalizability of the technologies developed within the context of a particular project 

will ensure that the SSD staff is ready to take on new projects more quickly and 

effectively. 

Formal Performance Metrics 

Foundational work can be challenging to measure objectively. One possible 

method for measuring the external impact of foundational work might be through 

applications that are built on the core methods and techniques. For example, the merits 

of software testing could be measured by assessing the improvements in voting systems 

subsequent to their having undergone rigorous NIST-based testing. Measuring 

applications should lead to increased emphasis on customer-oriented metrics. For 

example, Software Assurance Metrics and Tool Evaluation (SAMATE)-related 

improvement in static analysis tools (perhaps as measured at Static Analysis Tool 

Exposition [SATE] events) and overall quality of code (perhaps as measured by vendors 

of software testing tools) could augment more internal, process, or local metrics. 

 27 Ram D. Sriram, SSD, “Software and Systems Division,” presentation to the panel, Gaithersburg, 

Maryland, March 21, 2011, p. 8. 
28 

Strategic Plan 

The new SSD leadership is in the early stages of developing a strategic plan. To 

accelerate the process, the leadership identified an outside expert who will work with it to 

help define goals and objectives. As part of developing the strategic plan, the group 

should re-examine its mission statement and should consider the relative benefits of 

aligning the SSD technical organizational structure with foundational methods rather than 

with specific applications. As projects come and go, the organization will thus be better 

positioned to bring together the appropriate expertise for new projects as they arise. Once 

such a new organizational structure is in place, it should become clear where there are 

gaps in expertise, and additional formally trained individuals in gap areas should be 

hired. 

Application-Level Projects 

Externally mandated and funded projects are subject to sunsetting, and the team 

should be ready to let go of those application areas that are no longer priorities. 

Application-level projects are important opportunities to fulfill SSD’s mission, but it is 

important that the division continue to build the core approaches, methods, and 

technologies particularly so that, as new application-level projects arise, the group is well 

positioned to address them. 

Reference Data Sets 

 A core role of the ITL is to generate and curate reference data sets. Although this 

role requires significant effort, other ITL divisions have been highly successful in such 

efforts. For example, the ITL work on the TREC project has been exemplary, and it 

would serve the SSD well if it were also to take a lead in generating, curating, and 

maintaining reference data sets, including, if possible, those in the health domain. 

SSD Research Groups 

Software Components Group 

The SSD should consider technology developments that have occurred and are 

likely to occur in the first part of the 21st century, and it should reevaluate the activities 

of the Software Components Group. The National Software Reference Library (NSRL) 

for software for mobile devices is a good new effort. Additional energy should be 

applied to improving and speeding up the process of generating and distributing the 

reference library given the new, and now much more common, models of commercial 

software distribution. Examples include auto-updates and application downloads, 

software as a service, and cloud computing. 

The software assurance projects such as SAMATE, SATE, and the SAMATE 

Reference Data Set (SRD) are solid, appropriate SSD projects. More metrics related to 

external impact should be used to evaluate the (likely very positive) impact of these 

projects. In particular, the SSD could quantify impact by comparing old and new tools 

against the same code base, and possibly it should gather statistics from tool vendors on 
29 

industry-wide improvements in software quality. On the SRD in particular, the SRD 

code snippets are too small (100 lines of code [LOC]). The SSD should add larger code 

examples—perhaps some from SATE—into the SRD (e.g., Emacs [~300K LOC C], 

Chrome [~3M LOC], BSD network stack). 

Information Systems Group

The voting work being conducted by the Information Systems Group is moving 

apace. The group should consider the implications of the completion of the voting 

project, given that the Election Assistance Commission may sunset this project. SSD 

staff should aggressively publish on this work so that it, and, more generally, NIST, 

receive appropriate credit for their contributions. 

On bioinformatics and bioimaging, the group is collaborating well with other 

laboratories at NIST. The SSD should capture best practices here, and they should be 

replicated across the ITL. The medical imaging work has the potential for impact by 

advancing the quality and consistency of measurement in clinical use. However, this 

work and other SSD efforts in the biosciences are less well aligned with the SSD mission 

than are some other SSD projects. If the SSD wishes to build strength here, it should 

clarify how this project contributes to its core foundational work and then recruit domain 

leadership in that area. 

Interoperability Group 

Test methods developed by the SSD for meaningful use certification have been 

somewhat helpful to the Electronic Health Record (EHR) vendor community and the 

Office of the National Coordinator for Health Information Technology (HIT). (This 

work allows EHR vendors to appear on the Certified HIT Product List.) As the funding 

from the American Recovery and Reinvestment Act of 2009 (Public Law 111-5) comes 

to an end, work on testing infrastructure for health IT has the potential for use in new 

application areas that the Interoperability Group should consider. The energy devoted by 

the group to “tools for tools” is well placed. It is helpful for NIST to bridge and help 

bring consistency across various interoperability groups (e.g., the Nationwide Health 

Information Network [NwHIN] and Integrating the Healthcare Enterprise [IHE]); it is 

nevertheless important to maintain both the perception and the reality of NIST neutrality, 

particularly in the IHE involvement. 

Electronic Information Group 

The smart grid framework and roadmap work is a contribution to the community. 

The specific work on 1588 time synchronization is interesting. The framework and 

roadmap document for the smart grid released as a draft is likely to have positive impact 

on the smart grid community and future industrial development. 
30 

STATISTICAL ENGINEERING DIVISION 

The Statistical Engineering Division has a 65-year history of consistent and 

fundamental contributions to the central mission of NIST through the development and 

application of statistical methodology for metrology. This methodology supports the 

basic NIST activities of producing measurements and quantifying their associated 

uncertainties. 

The SED advances its mission of supporting research in measurement science, 

technology, and the production of standard reference materials (SRMs) through 

numerous scientific collaborations within NIST and externally. The SED conducts 

statistical research and provides important training and educational activities within and 

beyond NIST. It serves as a unique national and international resource for the metrology 

and standards communities and more broadly in high-profile contexts where an 

acknowledged impartial broker of data analysis and interpretation is needed. 

 The SED is composed of two groups: one at the NIST campus in Gaithersburg, 

Maryland, and the other at the NIST campus in Boulder, Colorado. The Gaithersburg 

group has 20 technical staff members plus a support secretary; the Boulder group has 3 

technical staff members and a secretary. 

Achievement of Stated Objectives 

The SED mission statement is that the SED “seeks to contribute to research in 

information technology, to catalyze scientific and industrial experimentation, and to 

improve communication of research results by working collaboratively with, and 

developing effective statistical methods for, NIST scientists and our partners in 

industry.”28 The division is highly successful in achieving these objectives outside of the 

ITL, where its contributions (inside and outside of NIST) are highly regarded, effective, 

and visible. The management and staff of the division also believe that there are 

additional opportunities for genuinely substantial contributions to existing efforts inside 

the ITL (as personnel limitations allow). 

Inside NIST, the SED provides essential support for basic metrology efforts (for 

example, in the production of standard reference materials). The division’s personnel 

also have deep collaborations with NIST scientists in both traditional and emerging areas 

of science and technology, ranging from neutron physics, to the large-scale measurement 

of greenhouse gas concentrations, to basic signal processing, to the mapping of return 

values of extreme wind gusts. The division’s lead role in the new NIST (Innovations in 

Measurement Science) Shape Metrology Program is evidence of the quality of its work 

and effectiveness in NIST collaborations. 

Outside of NIST, the SED responds on an emergency and ad hoc basis to national 

and international needs, is engaged on an ongoing basis with important U.S. researchers 

and agencies, and contributes materially to international metrology groups. Recent 

examples of important externally focused work of these types include the development of 

an oil budget calculator for the Deepwater Horizon oil spill; work with the Office of Law 

Enforcement Standards on body armor performance; collaborations with medical 

researchers on computer tomography and optical medical imaging; long-term work with 

 28 See http://www.nist.gov/itl/sed/mission.cfm. Accessed July 11, 2011. 
31 

the U.S. Departments of Justice and Homeland Security and the Office of National Drug 

Control Policy; membership on technical committees of the International Organization 

for Standardization (ISO); contribution of basic material to new ISO standards 

development; and invited participation in important international workshops on 

mathematics and metrology. 

The SED makes substantial educational contributions both inside and outside 

NIST and effectively disseminates sound statistical methodology for metrology. Division 

personnel are active in the development and delivery of metrology-related statistics short 

courses from Gaithersburg to South America and Europe. The division maintains and has 

recently updated the online Engineering Statistics Handbook,29and it develops and 

disseminates specialized software supporting best practices in statistical metrology. It 

has an excellent record of refereed publications, both papers done jointly with subjectmatter scientists in fields of collaboration, and publications in the statistical (and 

statistical metrology) literature. 

The most obvious opportunity for the SED to have additional impact is within the 

ITL itself. If the fairly severe need for a larger PhD-level staff can be alleviated, a 

number of emphases in the ITL could be catalyzed by more involvement of the SED. 

These possibilities include work in data management and visualization, work in pattern 

recognition and machine learning, and the laboratory’s work in forensics. The SED must 

be a central contributor to NIST work in forensics to the extent that the Daubert 

requirement30 for the evaluation and statement of potential error rates is to be upheld. The 

situation is not unlike SED’s role in developing quantifications of measurement 

uncertainties. While there is already some division activity in this area, notably in the 

National Ballistics Comparison Project, much more is possible and needed as personnel 

are available. 

Technical Merits and Caliber of Work 

 The SED work is technically excellent and central to the overall mission of NIST. 

The division provides state-of-the-art support for traditional NIST activities such as the 

production of standard reference materials, calibration studies, and assessments of 

interlaboratory studies. It is an innovative partner in the development of complicated 

emerging measurement technologies like high-speed opto-electronic measurement. Its 

external reputation for excellence is easily established through a consideration of its 

solicitation to work on high-profile, pressing national and international issues such as the 

Deepwater Horizon oil spill disaster, the estimation of Colombian coca cultivation, and 

law enforcement body armor reliability; its steady stream of internal NIST and external 

awards (including three Department of Commerce Gold Medals in 2010 and the Youden 

Award from the American Statistical Association in 2009); its record of peer-reviewed 

publications in statistics, metrology, and basic sciences (which are strong in terms of both 

the venues used for dissemination and the number of papers published); and invitations to 

 29 Available at www.itl.nist.gov/div898/handbook. Accessed August 15, 2011. 30 The reference is to the 1993 Daubert v. Merrell Dow Pharmaceuticals, Inc., U.S. Supreme 

Court decision that set current rules and requirements for the admissibility of expert scientific testimony in 

federal courts. Among other things it requires that error rates of methods used to support expert scientific 

testimony be known and stated. 
32 

international service, both on society committees and in the providing of training in 

statistical best practice. 

Alignment of Research with Laboratory Services and Deliverables 

The alignment of the SED research with ITL services and deliverables is 

excellent. The SED projects and participations fully contribute to the ITL roles of 

advancing science and engineering, setting standards and requirements for unique 

scientific instrumentation and experiments, data, and communications. It is the very 

nature of good statistical research that it is driven by the real data-analysis needs of a 

substantive field. The SED research is excellent and thus both inherently and effectively 

collaborative in nature (inside and outside NIST). 

Developments Since the 2007 and 2009 NRC Assessments 

 The SED management continues to be excellent and gaining momentum. It is 

proactive in finding and defining important projects and in aligning the division’s work 

with ITL and NIST priorities. It is proactive in marshaling appropriate funding and 

promoting international linkages. The leadership is technically active and effective, and 

the group morale appears to be high. The number and visibility of projects (including the 

new NIST Innovations in Measurement Science project) are increasing. 

 Two strong hires have been made since the previous NRC review, but two 

compensating retirements have prevented the growth in the number of PhDs strongly 

recommended in both the 200731 and 200932 reviews. There have been no new 

partnerships established with academicians and no division postdoctoral researchers since 

the 2009 review. 

Division Needs 

 Increasing the number of PhDs in the SED is a critical need (identified, as noted 

above, in the 2007 and 2009 NRC reviews). This is particularly true in light of potential 

retirements and the exponential explosion of the complexity of statistical and 

measurement methodologies, and of NIST needs (related, e.g., to forensics work). At 

least one hire per year for the next 5 years is needed in order to cover both imminent 

retirements and any other attrition and to provide for growth of at least a 2 to 3 full-timeequivalent staff in the next 5 years in order to handle the increasing complexity and 

volume of SED involvement in important new initiatives and areas. In addition, an 

increase in visiting faculty and other scientists is needed to help foster long-term external 

collaborations. More than one-time events and/or visits and a much broader scope here 

would allow the division to leverage its trove of excellent problems and opportunities to 

 31 National Research Council, An Assessment of the National Institute of Standards and 

Technology Information Technology Laboratory: Fiscal Year 2007. Washington, D.C.: The National 

Academies Press, 2007. 32 National Research Council, An Assessment of the National Institute of Standards and 

Technology Information Technology Laboratory: Fiscal Year 2009. Washington, D.C.: The National 

Academies Press, 2009. 
33 

increase not only its own output, but the quality and relevance of U.S. academic 

statistical research. 

 For the direct development of the SED staff, increased flexibility, support, and 

opportunity for visits by SED personnel to academic and non-academic institutions to 

establish long-term collaboration would be beneficial. So also would increased funding 

opportunities and support for participation in conferences and other professional 

statistical activities. The latter is important not only for the development of existing staff, 

but also to make the division attractive to the best new potential hires. 

Opportunities for the Future 

 Overall, the situation in the SED is quite healthy. Important and fascinating 

opportunities outstrip the division’s present physical ability to respond. There is much 

important work going on within the ITL that could genuinely benefit from increased SED 

involvement and collaboration. If the small group in Boulder were larger, there would be 

the opportunity to collaborate with scientists at the National Center for Atmospheric 

Research and the National Oceanic and Atmospheric Administration. More could be 

done in the way of participation on editorial boards and the organization of sessions (at 

both statistical and metrological meetings). More could also be done to raise the SED 

profile in the statistics community. The division has access to some of the most 

interesting statistical problems in the world, and it does some of the most interesting 

physical-science-related work in the world. Statisticians broadly should know more 

about opportunities and the work at NIST. 
34 

4 

Summary of Conclusions 

The Information Technology Laboratory of the National Institute of Standards 

and Technology is a well-managed science and engineering facility contributing in 

important ways to the nation’s scientific and technical research and development needs. 

The ITL supports the NIST mission through its own mission “to promote U.S. innovation 

and industrial competitiveness by advancing measurement science, standards, and 

technology through research and development in information technology, mathematics, 

and statistics”33 

To support its mission, the ITL has focused its R&D agenda on eight broad 

program areas: complex systems; cyber and network security; the enabling of scientific 

discovery; identity-management systems; information discovery, use, and sharing; 

pervasive information technologies; trustworthy information systems; and virtual 

measurement systems. Through programs in these areas, the ITL works to support 

important national programs and mandates. Examples among many of these mandates 

include the following: the USA PATRIOT Act and the Enhanced Border Security and 

Visa Entry Reform Act of 2002, the Help America Vote Act of 2002, and the Health 

Information Technology for Economic and Clinical Health (HITECH) Act of 2009. 

Each of the six divisions comprising the ITL is well managed, staffed with highly 

capable researchers, and addressing problems of national and international interests. For 

example, the ITL supports the National Initiative for Cybersecurity Education, working 

to build a framework to promulgate education, training, and awareness of cybersecurity 

issues to every segment of the population. 

The recommendations in this report are designed to address problems that exist in 

the ITL. These problems include the need to increase staffing in order to continue to 

support ITL’s mission; the need to find a new permanent, full-time chief of one division; 

and the need to sort out the vision in the Software and Systems Division. 

In summary, NIST in general and the Information Technology Laboratory in 

particular are exciting places to work. The work is challenging, interesting, and clearly 

of importance to industry and to the commerce of the nation. 

 33 Cita M. Furlani, ITL Director, “The Information Technology Laboratory,” presentation to the 

panel, Gaithersburg, Maryland, March 21, 2011, p. 3.