Visit the National Academies Press online, the authoritative source for all books from the 

National Academy of Sciences, the National Academy of Engineering, the Institute of

Medicine, and the National Research Council: 

• Download hundreds of free books in PDF 

• Read thousands of books online, free 

• Sign up to be notified when new books are published 

• Purchase printed books 

• Purchase PDFs 

• Explore with our innovative research tools 

Thank you for downloading this free PDF. If you have comments, questions or just want 

more information about the books published by the National Academies Press, you may 

contact our customer service department toll-free at 888-624-8373, visit us online, or 

send an email to comments@nap.edu. 

This free book plus thousands more books are available at http://www.nap.edu.

Copyright © National Academy of Sciences. Permission is granted for this material to be 

shared for noncommercial, educational purposes, provided that this notice appears on the 

reproduced materials, the Web address of the online, full authoritative version is retained, 

and copies are not altered. To disseminate otherwise or to republish requires written 

permission from the National Academies Press. 

ISBN: 0-309-14507-4, 32 pages, 8.5 x 11, (2009)

This free PDF was downloaded from:

http://www.nap.edu/catalog/12768.html

An Assessment of the National Institute of 

Standards and Technology Information Technology 

Laboratory: Fiscal Year 2009 

Panel on Information Technology; National Research 

Council 
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html

AN ASSESSMENT OF THE 

NATIONAL INSTITUTE OF STANDARDS 

AND TECHNOLOGY 

INFORMATION TECHNOLOGY 

LABORATORY 

FISCAL YEAR 2009

Panel on Information Technology 

Laboratory Assessments Board 

Division on Engineering and Physical Sciences 

THE NATIONAL ACADEMIES PRESS 

Washington, D.C. 

www.nap.edu
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html

THE NATIONAL ACADEMIES PRESS 500 Fifth Street, N.W. Washington, DC 20001 

NOTICE: The project that is the subject of this report was approved by the Governing Board of 

the National Research Council, whose members are drawn from the councils of the National 

Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The 

members of the panel responsible for the report were chosen for their special competences and 

with regard for appropriate balance. 

This study was supported by Contract No. SB3141-06-Z-0011, TO #6 between the National 

Academy of Sciences and the National Institute of Standards and Technology, an agency of the 

U.S. Department of Commerce. Any opinions, findings, conclusions, or recommendations 

expressed in this publication are those of the authors and do not necessarily reflect the views of 

the agency that provided support for the project. 

International Standard Book Number-13: 978-0-309-14506-0 

International Standard Book Number-10: 0-309-14506-6 

Copies of this report are available from 

Laboratory Assessments Board 

Division on Engineering and Physical Sciences 

National Research Council 

500 Fifth Street, N.W. 

Washington, DC 20001 

Additional copies of this report are available from the National Academies Press, 500 Fifth 

Street, N.W., Lockbox 285, Washington, DC 20055; (800) 624-6242 or (202) 334-3313 (in the 

Washington metropolitan area); Internet, http://www.nap.edu. 

Copyright 2009 by the National Academy of Sciences. All rights reserved. 

Printed in the United States of America 
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html

The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged 

in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the 

general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate 

that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president 

of the National Academy of Sciences. 

The National Academy of Engineering was established in 1964, under the charter of the National Academy of 

Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection 

of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. 

The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, 

encourages education and research, and recognizes the superior achievements of engineers. Dr. Charles M. Vest is 

president of the National Academy of Engineering. 

The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of 

eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. 

The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be 

an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and 

education. Dr. Harvey V. Fineberg is president of the Institute of Medicine. 

The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad 

community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal 

government. Functioning in accordance with general policies determined by the Academy, the Council has become the 

principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in 

providing services to the government, the public, and the scientific and engineering communities. The Council is 

administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Charles M. Vest 

are chair and vice chair, respectively, of the National Research Council. 

 www.national-academies.org
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html

iv 

PANEL ON INFORMATION TECHNOLOGY 

JEFFREY D. ULLMAN, Stanford University, Chair 

GEORGE V. CYBENKO, Dartmouth College 

MARK E. DEAN, IBM Corporation 

WAYNE H. ENRIGHT, University of Toronto 

DEBORAH L. ESTRIN, University of California, Los Angeles 

DAVID R. FERGUSON, Boeing Corporation (retired) 

JAMES D. FOLEY, Georgia Institute of Technology 

ERIC H. GROSSE, Google Inc. 

JAMES M. LANDWEHR, Avaya Laboratories 

PATRICK D. LINCOLN, SRI International Corporation 

STEVEN B. LIPNER, Microsoft Corporation 

ALEXA T. McCRAY, Harvard Medical School 

DAWN MEYERRIECKS, Independent Consultant, Purcellville, Virginia 

DEBASIS MITRA, Bell Laboratories, Lucent Technologies 

HILARIE K. ORMAN, Purple Streak, Inc. 

DAVID W. SCOTT, Rice University 

W. TIMOTHY STRAYER, BBN Technologies 

STEPHEN B. VARDEMAN, Iowa State University 

JAMES L. WAYMAN, San Jose State University 

Staff 

JAMES P. McGEE, Director 

CY BUTNER, Senior Program Officer 

LIZA HAMILTON, Administrative Coordinator 
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html

v 

Acknowledgment of Reviewers 

 This report has been reviewed in draft form by individuals chosen for their diverse 

perspectives and technical expertise, in accordance with procedures approved by the National 

Research Council’s Report Review Committee. The purpose of this independent review is to 

provide candid and critical comments that will assist the institution in making its published 

report as sound as possible and to ensure that the report meets institutional standards for 

objectivity, evidence, and responsiveness to the study charge. The review comments and draft 

manuscript remain confidential to protect the integrity of the deliberative process. We wish to 

thank the following individuals for their review of this report: 

John Bailar III, University of Chicago, 

Suzanne Bakken, Columbia University, 

Steven Bellovin, Columbia University, 

John Hopcroft, Cornell University, 

Craig Partridge, BBN Technologies, 

Richard Pew, BBN Technologies, and 

Stephen A. Vavasis, University of Waterloo. 

 Although the reviewers listed above have provided many constructive comments and 

suggestions, they were not asked to endorse the conclusions or recommendations, nor did they 

see the final draft of the report before its release. The review of this report was overseen by 

Alton D. Slay, Warrenton, Virginia. Appointed by the National Research Council, he was 

responsible for making certain that an independent examination of this report was carried out in 

accordance with institutional procedures and that all review comments were carefully 

considered. Responsibility for the final content of this report rests entirely with the authoring 

panel and the institution. 
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html

vii 

Contents 

SUMMARY 1 

1 CHARGE TO THE PANEL AND THE ASSESSMENT PROCESS 3 

2 GENERAL ASSESSMENT OF THE INFORMATION TECHNOLOGY 

 LABORATORY 5 

Some Important Activities at the Laboratory, 5 

Programs and Projects, 6 

Budget Issues, 8 

Raising the Profile of the Laboratory, 10 

3 ASSESSMENT OF THE LABORATORY DIVISIONS 12 

Mathematical and Computational Sciences Division, 12 

Statistical Engineering Division, 13 

Software and Systems Division, 15 

Computer Security Division, 18 

Advanced Network Technologies Division, 20 

Information Access Division, 21 
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html

1 

Summary 

The Information Technology Laboratory (ITL) of the National Institute of 

Standards and Technology (NIST) has been assessed by a panel of experts appointed by 

the National Research Council (NRC). The panel visited the six divisions of the 

laboratory and reviewed their activities. The scope of the assessment included the 

following criteria: (1) the technical merit of the current laboratory programs relative to 

current state-of-the-art programs worldwide; (2) the adequacy of the laboratory budget, 

facilities, equipment, and human resources, as they affect the quality of the laboratory’s 

technical programs; and (3) the degree to which the laboratory programs in measurement 

science and standards achieve their stated objectives and desired impact. Based on its 

assessment using these criteria, the panel formed the following observations and 

recommendations, among others discussed in the report. 

 Observations: 

1. The technical merit of the programs is often excellent and is generally high. 

Specific examples are highlighted throughout the report. 

2. There are large, temporary changes in budget levels overlaying the normal 

funding cycles, and at the ITL, budgeting is having more influence than it 

should on the progress of technical work. The issues are the benefits and 

potential risks of “soft money” (outside funding), the inability to hire critical 

staff in some areas, and questions of how best to use incoming, short-term 

funds. 

3. Facilities and equipment were found to be adequate, with the exception that 

the Mathematical and Computational Sciences Division needs additional 

space and the Information Access Division has need of a usability laboratory. 

4. There has been rapid development of the matrix structure of the laboratory, 

with programs cutting across the traditional divisions. There are benefits 

already derived from this approach, but also small risks. 

5. Soft money continues to be an important aspect of laboratory operations, and 

it causes staff to worry about the safety of their jobs and causes other 

confusion that should be addressed. The use of soft money has benefits and 

potential risks. As long as potential risks are monitored and avoided, a policy 

of encouraging a search for solid external support for sound, internally vetted 

projects is worthwhile. 

 Recommendations: 

1. ITL staff, perhaps led by the program managers, should look for linkages with 

external organizations such as research universities and laboratories. The 

recent addition of temporary funding associated with the economic recovery 

can help build these connections. 
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html

2 

2. The ITL should make efforts to raise its profile through outreach (connections 

with major research universities and laboratories, hosting faculty, postdoctoral 

researchers, and other short-term visitors; and staff participation in 

professional service) and publication (in highly respected journals and 

conferences). 

3. Program managers who are capable of providing technical leadership and also 

devote effort to promoting the interests of their programs should be regarded 

by the staff as positive contributors, even if they are no longer writing code or 

doing other technical tasks associated with individual projects. 

4. There is a need for additional senior technical leadership. 

—The Software and Systems Division (SSD) needs to hire a strong health 

informatics leader. 

—NIST should appoint a full-time chief for the SSD, which currently has 

an acting chief who divides time between leading the division and 

working in the Office of the ITL Director. 

—The panel found multiple cases of the SSD’s suffering from a lack of 

sufficient focused leadership at a time when the SSD is being asked to 

be the lead in several important efforts, such as health care. 

5. SSD leadership should encourage its staff toward greater innovation and 

redirection in keeping with developments in the broader research and 

scientific community. 

6. Apart from the current chief, there has been no perceptible growth in the 

permanent staff of the Statistical Engineering Division for years. The division 

is short-staffed, and such growth should be pursued with urgency before the 

next review. 

7. The ITL needs a process for sunsetting programs and encouraging bottom-up 

development of new programs. 
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html

3 

1 

Charge to the Panel and the Assessment Process 

 At the request of the National Institute of Standards and Technology (NIST), the 

National Research Council (NRC) has since 1959 annually assembled panels of experts 

from academia, industry, medicine, and other scientific and engineering environments to 

assess the quality and effectiveness of the NIST measurements and standards 

laboratories, of which there are now nine,1 as well as the adequacy of the laboratories’ 

resources. In 2009, NIST requested that five of its laboratories be assessed: the NIST 

Center for Neutron Research; the Center for Nanoscale Science and Technology, the 

Information Technology Laboratory (ITL), the Chemical Science and Technology 

Laboratory, and the Electronics and Electrical Engineering Laboratory. Each of these was 

assessed by a separate panel of experts; the findings of the respective panels are 

summarized in separate reports. This report summarizes the findings of the Panel on 

Information Technology. 

For the fiscal year (FY) 2009 assessment, NIST requested that the panel 

consider the following criteria as part of its assessment: 

1. The technical merit of the current laboratory programs relative to current 

state-of-the-art programs worldwide; 

2. The adequacy of the laboratory budget, facilities, equipment, and human 

resources, as they affect the quality of the laboratory’s technical programs; 

and 

3. The degree to which the laboratory programs in measurement science and 

standards achieve their stated objectives and desired impact. 

The context of this technical assessment is the mission of NIST, which is to 

promote U.S. innovation and industrial competitiveness by advancing measurement 

science, standards, and technology in ways that enhance economic security and 

improve the quality of life. The NIST laboratories conduct research to anticipate future 

metrology and standards needs, to enable new scientific and technological advances, 

and to improve and refine existing measurement methods and services. 

In order to accomplish the assessment, the NRC assembled a panel of 19 

volunteers whose expertise matches that of the work performed by the ITL staff.2

 The 

panel members were also assigned to six subgroups (division review teams), whose 

members’ expertise matched that of the work performed by staff in the six divisions in 

the ITL: Mathematical and Computational Sciences, Statistical Engineering, Software 

 1

 The nine NIST laboratories are the Building and Fire Research Laboratory, the Center for 

Nanoscale Science and Technology, the Chemical Science and Technology Laboratory, the Electronics and 

Electrical Engineering Laboratory, the Information Technology Laboratory, the Manufacturing Engineering 

Laboratory, the Materials Science and Engineering Laboratory, the NIST Center for Neutron Research, and 

the Physics Laboratory. 2

 See http://www.itl.nist.gov/ for more information on ITL programs. Accessed May 1, 2009. 
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html

4 

and Systems, Computer Security, Advanced Network Technologies, and Information 

Access. The panel met at the NIST facilities in Gaithersburg, Maryland, on April 1-3, 

2009. After the full panel met for a session of overview presentations on the ITL and 11 

ITL programs by the ITL management and staff, the panel divided into its six review 

teams, and each (led by a team leader chosen from within the panel) then visited its 

respective ITL division for about a day. During these visits, the review team members 

attended presentations, tours, demonstrations, and interactive sessions with the ITL staff. 

Subsequently, the entire panel assembled for about a day, during which it interacted with 

ITL and NIST management and also met in a closed session to deliberate on its findings 

and to define the contents of this assessment report. For each assessment cycle, the panel 

membership is composed of new members (approximately one third) and returning 

members (approximately two thirds) who participated in previous assessment cycles. 

The approach of the panel to the assessment relied on the experience, technical 

knowledge, and expertise of its members, whose backgrounds were carefully matched to 

the technical areas of ITL activities. The panel reviewed selected examples of the 

technological research covered by the ITL; because of time constraints, it was not 

possible to review the ITL programs and projects exhaustively. The examples reviewed 

by the panel were selected by the ITL. The panel’s goal was to identify and report salient 

examples of accomplishments and opportunities for further improvement with respect to 

the following: the technical merit of the ITL work, its perceived relevance to NIST’s own 

definition of its mission in support of national priorities, and specific elements of the 

ITL’s resource infrastructure that are intended to support the technical work. These 

examples are intended collectively to portray an overall impression of the laboratory, 

while preserving useful suggestions specific to projects and programs that the panel 

examined. The assessment is currently scheduled to be repeated biennially, which will 

allow, over time, exposure to the broad spectrum of ITL activity. While the panel applied 

a largely qualitative rather than a quantitative approach to the assessment, it is possible 

that future assessments will be informed by further consideration of various analytical 

methods that can be applied. 

The rest of this report is organized in two chapters. Chapter 2 discusses issues that 

apply broadly to several or all of the divisions or to the ITL as a whole. Chapter 3 

presents observations specific to each ITL division. The comments in this report are not 

intended to address each program within the ITL exhaustively. Instead, this report 

identifies key issues and salient programs and projects relevant to those issues. Given the 

necessarily nonexhaustive nature of the review process, the omission of any particular 

ITL program or project should not be interpreted as a negative reflection on the omitted 

program or project.
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html

5 

2 

General Assessment of the 

Information Technology Laboratory 

 The total FY 2009 projected available funds for the Information Technology 

Laboratory are $104.6 million, of which $40.0 million (38 percent) is other-agency 

funding, $63.9 million (61 percent) is scientific and technical research services (STRS) 

funding, and $0.7 million (1 percent) is other NIST funding. The total of all ITL staff is 

332 staff members, of whom 37 percent are computer scientists, 21 percent are 

mathematicians/statisticians, and 14 percent each are information technology (IT) 

specialists, engineers/physicists, and administration and support personnel. Additional 

information on the funding for the 11 crosscutting key ITL programs is provided below. 

SOME IMPORTANT ACTIVITIES AT THE LABORATORY 

Many things are working very well at the Information Technology Laboratory. 

The ITL has established 11 key programs that cut across the traditional ITL divisions, 

and the ITL uses a matrix approach to manage them. The panel observed that the 

program structure, discussed below, has been adopted well and rapidly. This section 

discusses some important positive aspects of the overall program. 

Role as the “Honest Broker” 

Many ITL activities can be characterized as being a national or international 

resource. In many cases, ITL staff are the only such resource available. In others, they 

have established a role as the neutral party that is appropriately charged with evenhanded measurement or evaluation of the quality of products. The challenge-problem 

activities mentioned below are important examples. For another example, ITL staff 

performed the empirical studies that allowed proposed standards for fingerprint and iris 

recognition to be confirmed as realistic. 

Challenge Problems 

There is a methodology for advancing technology that is especially appropriate in 

many of the activities for which the ITL is responsible, and it is both effective and 

pervasive in the ITL culture. A number of teams are responsible for issuing challenges 

to the research community in the form of task definitions, test data sets, and associated 

ground truth on which software is to be evaluated. The prototype program was the Text 

Retrieval Conference (TREC), at which the information-retrieval community has been 

given yearly challenges such as finding relevant documents or answering questions. More 

recent programs of this type include Text Retrieval Conference Video Retrieval 

Evaluation (TRECvid), Multiple Biometric Grand Challenge (MBGC; multimodal 

biometrics), and machine translation (MT). A similar program is Secure Hash Algorithm 
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html

6 

3 (SHA3), a design challenge in secure hashing, in which ITL staff and community 

volunteers analyze the work of the participants. 

The international respect given to the ITL “challenge problems” allows the U.S. 

government to leverage a comparatively small investment into major technical progress 

on problems of immediate interest to many governmental agencies. Most of these 

challenge problems appear to be financed by agencies outside the Department of 

Commerce, with the ITL performing required research and providing test development 

and execution. With the ITL setting up carefully defined tests and distributing data sets, 

government agencies can entice competing academic and industrial organizations to 

focus research and development (R&D) monies on narrowly defined problems of 

immediate practical interest. 

In addition to providing a selection of benchmarks for quantifying progress in the 

field, as teams from around the world compete to be the best at the yearly tasks, the 

management of challenge problems presents serious research problems for the ITL staff 

itself, as they address the metrology involved. For example, there have been attempts to 

automate, or partially automate, the process of evaluating the quality of machine 

translation from language to language. 

PROGRAMS AND PROJECTS 

There has been very rapid progress in the Information Technology Laboratory 

with respect to putting programs in place. Following a matrix management approach, the 

divisions retain responsibility for administration and remain the focal point for a 

particular discipline, for example, statistics or security. Projects are intended to address 

particular problems or classes of problems that draw on several disciplines, both across 

the ITL and, in some cases, across NIST. 

The programs and their budgets (as of March 25, 2009, and broken out in terms of 

budget total and distribution by ITL headquarters (HQ) and ITL divisions1

) are as 

follows: 

1. Complex Systems (Total = $2,421,000: HQ = $329,000, MCSD = $1,019,000, 

ANTD = $498,000, IAD = $177,000, SSD = $289,000, SED = $109,000) 

2. Cyber and Network Security (Total = $14,804,000: HQ = $330,000, MCSD = 

$26,000, ANTD = $2,679,000, CSD = $11,769,000) 

3. Enabling Scientific Discovery (Total = $4,737,000: HQ = $327,000, MCSD = 

$2,890,000, CSD = $33,000, SED = $1,487,000) 

4. Identity Management Systems (Total = $9,142,000: HQ = $318,000, ANTD = 

$34,000, CSD = $4,365,000, IAD = $4,424,000) 

5. Information Discovery, Use, and Sharing (Total = $7,788,000: HQ = 

$461,000, MCSD = $939,000, ANTD = $125,000, IAD = $4,136,000, SSD = 

$1,534,000, SED = $593,000) 

 

1

 MCSD, Mathematical and Computational Sciences Division; ANTD, Advanced Network 

Technologies Division; IAD, Information Access Division; SSD, Software and Systems Division; SED, 

Statistical Engineering Division; CSD, Computer Security Division. 
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html

7 

6. Pervasive Information Technology (Total = $1,833,000: HQ = $287,000, 

MCSD = $148,000, ANTD = $500,000, CSD = $90,000, IAD = $184,000, 

SSD = $624,000) 

7. Trustworthy Information Systems (Total = $3,267,000: HQ = $329,000, 

MCSD = $176,000, CSD = $16,000, IAD = $212,000, SSD = $2,534,000) 

8. Virtual Measurement Systems (Total = $1,030,000: HQ = $244,000, MCSD = 

$400,000, SED = $386,000) 

9. Quantum Information (Total = $2,621,000: MCSD = $796,000, ANTD = 

$1,696,000, CSD = $129,000) 

10. Voting (Total = $3,702,000: CSD = $1,103,000, IAD = $979,000, SSD = 

$1,620,000) 

11. Healthcare Information Technology, or Health IT (Total = $1,246,000: HQ = 

$602,000, CSD = $49,000, SSD = $595,000). 

The ITL presented high-level introductions describing the scope of each of the 11 

programs in existence at the time of the panel’s review. However, the panel’s ability to 

drill down to the technical activities of each program was limited to those projects that 

were covered in individual division-based reviews. At future reviews it would be useful 

to hear more detailed descriptions of the technical activities of each program, especially 

those aspects that by their nature cut across several divisions. In addition, the ITL should 

consider having external, independent reviews of programs periodically. 

One deficiency in some of the program presentations to the panel was a lack of 

clear identification of the focus or theme of the program. For some programs the name 

implies the focus (this is particularly true in some of the congressionally mandated 

programs), but other programs did not properly identify a national need being addressed 

and, as a result, appeared to be more of a collection of individual projects without any 

overarching national goal as a driver. It should be a specific responsibility of the 

program manager to ensure that this issue is addressed in all presentations and 

documentation provided to the panel in the future. 

Other programs appeared to be an uncomfortable collection of group activities 

with no explicable goals or metrics. The Pervasive Information Technologies Program 

lives up to its name in the sense of being a set of technologies, and each group is working 

on standards, but there is no discernible effort to have these technologies form something 

larger than its parts. Also, overall there did not seem to be a clear and cohesive roadmap 

for the activities subsumed under the Health Information Technology effort. 

By contrast, the Quantum Information Program seems well suited to matrix 

organization, and the NIST research expertise has salutary effects on a wide swathe of 

other activities, particularly in helping practitioners understand the possible impacts of 

quantum computing on cryptographic security. 

Role of Program Managers 

Program managers appear free to be advocates for their programs and do not seem 

burdened with routine administrative tasks. Their role in practice appears to be evolving 

and differs among programs. Program managers should be, at least to some extent, 
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html

8 

advocates and marketers for the programs that they lead. In many cases it makes sense 

for them to look for external funding or for customers in the other laboratories of NIST 

for the projects associated with their programs. It is important for program managers to 

be recognized technical leaders in their fields; the more important the program, the more 

vital it is to have a strong and widely known advocate in a leadership role. For 

example—repeating the panel’s suggestion made 2 years ago—it would be helpful to 

bring in a strong scientist (expert in health informatics, maybe an M.D.) to lead the 

medical program efforts, especially given the recent thrust to develop electronic medical 

records. 

There are concerns among some of the staff that the creation of programs and 

program managers has increased the amount of “overhead” and reduced the number of 

people actually doing the work. If the program managers are capable of providing 

technical leadership and also devote effort to promoting the interests of program 

participants, then they should be regarded as positive contributors, even if they are no 

longer writing code or doing other technical tasks associated with individual projects. 

 The group of program managers, being mostly younger or midcareer people, will 

eventually provide a valuable cadre of experienced managers within the ITL. 

Sunrise and Sunset of Projects 

Although the programs are too new to have provided much opportunity to deal 

with project evolution, it would be wise to think about the process and philosophy for 

both terminating projects and generating new ones. Projects that support a particular 

standard can risk continuing and consuming ITL resources forever if no external body is 

willing to take ownership of the standard. In general, there should be a process in place 

for doing a critical review of projects periodically to make sure that they are fulfilling a 

need and functioning well. 

Likewise, there should be a process in place for creating new projects. Many of 

these will be mandated by Congress or will arise from a need within another laboratory of 

NIST itself. However, it would be stimulating to encourage bottom-up proposals. This 

approach may interact in important ways with the matter of external (outside agency 

[OA]) funding, discussed below. The philosophy in both the process for terminating 

projects and the process for creating new projects should be to base evaluations and 

decisions on what efforts are most appropriate for the ITL within NIST’s mission. 

BUDGET ISSUES 

There are large, temporary changes in budget levels, overlaying the normal upand-down progression of funding cycles. At the ITL, budgeting is having more influence 

than it should on the progress of the technical work. One issue is the role of “soft 

money”—the resources that come from outside NIST. A second issue is that in some 

places within the ITL it has not been possible to hire critical staff. The third major issue 

is how the arrival of the recently announced temporary funding can be used to benefit the 

technical goals of the laboratory rather than to result in a burden on already-overloaded 

staff. Each of these issues is discussed below. 
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html

9 

Soft Money 

The ITL divisions vary greatly in how much OA money they rely on. The 

Information Access Division (IAD) is the only division that receives the majority of its 

budget from outside NIST. In addition, the Statistical Engineering Division (SED) and 

the Mathematical and Computational Sciences Division (MCSD) receive significant 

amounts of money from other laboratories at NIST. 

Taking money to perform tasks for others could be harmful. It could lead to the 

ITL’s personnel devoting effort to problems that are not worthwhile science just because 

there is money available to support staff. In extreme cases, the role of NIST as an 

impartial broker could be jeopardized. The panel has not, however, seen these problems 

in practice. Some of the externally funded work is among the most interesting and 

impactful in the ITL. 

The existence of staff supported by soft money can lead to a certain paralysis and 

stagnation. For example, some research staff in the IAD expressed a reluctance to invest 

their time until the outside money was in hand (something that does not always happen 

on the most desirable schedule). There may also be an inhibition to seeking outside 

sources for good projects. The ITL leadership has made it clear that well-conceived 

projects with outside funding do not present a risk to the research staff; the laboratory 

will backstop research personnel against unforeseen loss of OA money. However, staff 

are fearful and risk-averse, especially in the current economy. The use of soft money has 

benefits and potential risks. As long as potential risks are monitored and avoided, a 

policy of encouraging a search for solid external support for sound, internally vetted 

projects is worthwhile and likely to lead to the direct funding of new, important, and 

relevant projects. 

Hiring 

There is a similar problem concerning the filling of staff positions: researchers are 

unwilling to expend the effort to search for a new person without a clear mandate to do 

so. With the new crosscutting structure, some staff are unclear about who is responsible 

for conducting a search. And of course busy people are not anxious to devote a lot of 

time to an effort that can come to naught for a variety of reasons. 

 The situation in the Statistical Engineering Division is of significant concern. The 

dynamic division head who was hired about 2.5 years ago has developed good support 

and rapport with his staff and ITL management. However, the problem noted by the 

panel 2 years ago remains: there have been no other new hires at SED for at least 7 years, 

and a number of new and important branches of the field and emerging scientific needs 

are not covered by permanent staff. This issue should be creatively and energetically 

addressed now. 

On a positive note, the ITL was able to deal with one of the problems of 2 years 

ago: the inability to hire scientists in nontraditional fields at appropriate salaries. A 

linguist was hired for work on the machine-translation challenge problems. The process 

of such hires should be further regularized, because there is likely to be a continued need 
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html

10 

for additional personnel in soft sciences for such projects as accessibility of voting 

machines. 

Short-Term Money 

The ITL noted its responsibility for spending a large part of the money that has 

been allocated for economic recovery (through the American Recovery and Reinvestment 

Act of 2009 [Public Law 111-5]), especially in the area of electronic medical records, but 

in many other subdisciplines as well. It would be ideal if the fellowships and short-term 

arrangements could be used to develop longer-term collaborations—ones that could last 

beyond the money and perhaps lead to joint ventures in better times. Yet there is a 

concern, both from the staff and the panel, that the net result will be that the scientists do 

a lot of work functioning as program managers and will get nothing out of it to support 

their own research activities. 

The staff, perhaps led by the program managers, should start looking for those 

external groups, such as at research universities, where there is expertise that could fit 

well with the programs. They should be proactive in making known both the opportunity 

and the needs of their teams. 

RAISING THE PROFILE OF THE LABORATORY 

While the various ITL units differ in their approach to external visibility, 

performance evaluation should stress some of the points made below, which include a 

few principles that could be applied to benefit ITL scientists and improve their ability to 

fulfill their mission. 

Outreach 

The ITL should try to build connections with major research universities and 

research laboratories. The existence of short-term funding support should help initiate 

these connections. Soft money should be used to search for and invite more faculty, 

postdoctoral researchers, and other short-term visitors. 

The ITL should encourage staff to participate in professional service, such as 

service on conference committees (tasks other than program committee work often lead 

to recognition and membership on later program committees) or participation in 

speakers’ bureaus. 

Publication

Many ITL scientists have extensive publication lists; for those who publish 

regularly, it is important to be selective in their venue. Publication is far from the only 

way to impact science. In many areas of computer science (although not mathematics or 

statistics), conferences are more respected than journals are, because top conferences are 

harder to get into than are even the best journals. The top few conferences in a field will 

often have acceptance rates of around 10 percent. The ITL should give more attention to 
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html

11 

the quality of publication venues and to reward staff who can publish their work in the 

places with the greatest visibility and prestige. Journal and conference papers serve not 

only to disseminate project results, but also to advertise the capabilities and successes of 

a program and to provide a permanent, searchable record for future users. 
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html

12 

3 

Assessment of the Laboratory Divisions 

MATHEMATICAL AND COMPUTATIONAL SCIENCES DIVISION 

 

The panel holds essentially the same view expressed in the 2007 ITL assessment 

report:1

 “The Mathematical and Computational Sciences Division (MCSD) has a wellformulated view of the way it contributes to national priorities by advancing science and 

industrial innovation, and individual contributors understand how their work fits with the 

goals of NIST. In particular, the teams understand the importance of simulation-based 

engineering and how to deliver their technology to scientists and practitioners, inside and 

outside NIST, who are the ultimate users.” 

During the 2007 review, staff and management expressed a desire for more 

postdoctoral positions and had plans to actively recruit applicants in order to increase the 

likelihood of attracting excellent postdoctoral candidates. This tactic appears to have 

paid off, as the MCSD is now in line for four postdoc positions. Also, in 2007 the 

division chief was worried about developing an eventual successor and about finding 

program managers within the group. This concern appears to have been sufficiently 

addressed through the new program-management structure. Two technical people have 

stepped up to the program-management tasks. They have developed into highly capable 

program managers who are running excellent programs and learning the ins and outs of 

management and program management. 

Another change is that in 2007 the project to create the Digital Handbook of 

Mathematical Functions seemed to be having trouble bringing the publication of the 

handbook to a conclusion. Now the technical content of the handbook is complete, and a 

publisher for the accompanying book has been selected. Publication of the handbook is 

expected soon. 

 

Particular Programs and Projects That Deserve Mention

 

Two ITL programs, Enabling Scientific Discovery (ESD) and Virtual 

Measurement Systems (VMS), are managed by MCSD leaders but draw on crossdisciplinary teams. They are vehicles for strong, multidisciplinary science and 

mathematics and are supportive of the simulation-based engineering important to national 

goals. Both programs have track records of delivering technology to users: scientists and 

practitioners inside and outside NIST. 

The ESD program has a goal of improving the accuracy, precision, efficiency, and 

complexity in metrology, improvements needed as simulation-based engineering 

continues to grow in importance. An example project is Modeling Grain Boundary 

 1

 National Research Council, An Assessment of the National Institute of Standards and 

Technology Information Technology Laboratory: Fiscal Year 2007. Washington, D.C.: The National 

Academies Press, 2007, p. 11. 
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html

13 

Premelting in Binary Alloys. This theoretical study will shed light on the material 

properties of binary alloys. Projects within the ESD program typically use state-of-theart tools that result in simulations with more accurate physics. 

The VMS program recognizes that failures of some of our major simulation tools 

have been implicated in major engineering failures. For example, a NASTRAN (NAsa 

STRuctural ANalysis) simulation underestimated stress by 47 percent, resulting, at least 

in part, in the collapse of an oil platform in 1991. The VMS program is concerned with 

the validation and verification of such simulations. It is attempting to define a metrology 

infrastructure for virtual measurements that will provide a platform for understanding 

uncertainty in modeled effects, trueness relative to benchmarks, and traceability leading 

to trust. For example, the micromagnetic modeling project is developing an open-source 

simulation tool that will become a measurement standard for validating micromagnetics 

simulations. The MCSD is also performing the first-of-its-kind investigation of rendering 

uncertainties. (Rendering is the generation of an image [a rectangular array of pixels] 

based on a virtual three-dimensional scene, and a virtual scene is a set of geometric 

objects parameterized in three-dimensional coordinates.) Since this investigation is new, 

it is hard to predict the outcome. However, rendering does play an important role in 

virtual measurement, so understanding uncertainties in visualization will certainly be 

important. 

The project on Quantum Information is an example of the MCSD’s looking (far) 

into the future, in this case the future of quantum computing, and beginning to build a 

measurement science framework for this new technology. The Ion Trap Computing 

Benchmarks project is investigating benchmarks that will be needed to measure the 

performance of quantum computing. 

 

Issues for the Future

Management of the MCSD is aware of two potential management problems 

arising in the near term. One is finding and recruiting high-quality personnel in 

traditional areas of applied mathematics and physics for new positions resulting from the 

stimulus package (the American Recovery and Reinvestment Act of 2009) and from 

retiring staff. The second is acquiring needed space commensurate with the current 

facilities and staff expectations for new hires. 

STATISTICAL ENGINEERING DIVISION 

 

The Statistical Engineering Division has a consistent, long-standing mission to 

advance metrology through the appropriate use of statistical methods in the activities of 

producing measurements and expressing their uncertainty; this mission is of central 

importance at NIST. This mission is accomplished through many deep scientific 

collaborations that cut across NIST and other governmental agencies in the 

characterization of materials and processes, for example, with standard reference 

materials (SRMs); through methodological research; and through training and 

educational activities. For another example, the SED’s statistical metrology effort is a 

unique national and international capability and resource. 
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html

14 

Technical Merit of Programs 

The overall caliber of the statistical metrology effort remains very strong. The 

SED’s portfolio of projects includes broad, long-standing collaborations with individual 

scientists and groups that lead to jointly authored publications in the substantive fields. 

Areas of important current work range from metrology for bulletproof vests and infrared 

vision devices used by police and fire officers to neutron physics spectroscopy. 

Important work supported by other government agencies includes projects such as one 

with the Department of Homeland Security for radiation-detection monitors deployed at 

border crossings. 

The SED is involved with several of the recently created ITL programs, including 

Complex Systems, Virtual Measurement Systems, and Enabling Scientific Discovery. 

These are appropriate ITL areas and show good promise for important SED 

contributions. At the same time there are other ITL programs, such as that in multifactor 

experimentation, where the science could benefit from increased substantive 

contributions of SED expertise, but such opportunities to contribute to ITL programs are 

at present not being fully met owing to a lack of resources. 

Overall, the statistical metrology research program is state of the art. Researchers 

publish regularly in international metrology journals and are among the leaders in 

international metrology efforts. However, the group has drastically reduced its 

participation in national and international meetings owing to budget choices that are not 

consistent with the long-term SED mission. 

 

Human Resources and Facilities 

 The panel’s 2007 ITL assessment report concluded that, among the ITL activities, 

the SED was most in need of immediate enhancement of its capabilities. This situation 

has not been successfully addressed. The staff is the same as it was 2 years ago. Apart 

from the current chief, recent permanent hires were made 7 and 8 years ago. The 

dynamic chief, who was hired around 2.5 years ago, has developed good support and 

rapport with his staff and ITL management. The number of ITL and NIST activities is 

even greater now than in 2007, and the need for staff expansion and rejuvenation is 

greater. The ITL should address this issue creatively and energetically now. Further, the 

SED budget has apparently permitted very little external professional activity in the past 

year. This is a serious problem that must be addressed. The SED facilities and 

equipment are adequate. 

Achieving Objectives and Desired Impact 

 The existing SED programs are having their desired impact and helping NIST to 

achieve its overall mission and objectives. As national priorities and needs evolve, so 

must the metrology infrastructure and methodology addressing such needs. One 

important emerging area involves measurement for greenhouse gas management 

programs (such as would be needed for cap-and-trade programs currently under 

consideration by Congress). This area requires new statistical research appropriate for 
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html

15 

the measurement technology, along with the associated modeling and uncertainty 

analyses. 

 Another important emerging area involves spatial measurements for climate 

change and standards for reporting error bars for long-term model forecasts. Research in 

such areas is desirable and shows promise but is currently limited owing to the staffing 

issues raised above. 

The SED’s impact could be increased through raising its external profile within 

the statistical community. Both the SED and this community would benefit from greater 

awareness of the important, challenging problems arising and being successfully 

addressed at NIST. Creative approaches for regularly communicating activities and 

opportunities could also help increase the pool of desirable candidates for short-term, 

visiting and regular positions. The SED should compete for invited paper sessions at 

international meetings. The division should take advantage of its presence in Boulder, 

Colorado, to make strategic alliances with the National Center for Atmospheric Research, 

also located there, if the SED chooses to focus new resources in that arena. 

SOFTWARE AND SYSTEMS DIVISION 

Overview 

 The Software and Systems Division comprises three research groups, Software 

Components, Information Systems, and Interoperability. Efforts addressing several of the 

crosscutting, key ITL programs, most notably the Voting and the Healthcare Information 

Technology programs, are included within the R&D portfolio of the division. The 

division currently has an acting chief, who also has significant responsibilities in the 

Office of the ITL Director. The lack of strong scientific and administrative leadership 

within the SSD and also, in some cases, at the programmatic level is cause for concern. 

This is particularly salient given the impending large influx of American Recovery and 

Reinvestment Act dollars for some of the SSD activities. No clear vision or plan was 

articulated for how these dollars would be most effectively spent. Research staff appear 

to be comfortable with their research projects, although at times this comfort level may in 

fact be at odds with innovation and redirection in keeping with developments in the 

broader research and scientific community. Several research staff are involved in projects 

of great national import, yet they do not seem to be exercising the leadership in those 

national settings that is required for having maximum impact. 

Selected Projects and Programs 

Voting Systems

The SSD efforts in voting and related technologies respond to congressional 

initiatives such as the Help America Vote Act of 2002 (or HAVA; Public Law 107-252). 

HAVA directs NIST to assist the Election Assistance Commission in the development of 

voluntary voting system guidelines, and HAVA directs NIST to chair the Technical 

Guidelines Development Committee. NIST activities in the area are executed by the SSD 
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html

16 

and include a study of the computer security of voting systems and methods to prevent 

fraud and privacy violations. The SSD works to develop voluntary standards but does 

not certify voting machines for use in elections. 

 The SSD efforts provide a baseline assessment capability but do not go far 

enough. Voting is the very core of the nation’s democracy, and vulnerabilities in its 

voting systems pose a direct threat to the basis of its government. The SSD should 

increase the stringency of its voluntary standards, perform and coordinate more active 

evaluations of voting machines and systems, and work with election commissions to 

improve assurance of all voting systems immediately. 

Software Assurance Metrics and Tool Evaluation

The SSD Software Assurance Metrics and Tool Evaluation (SAMATE) program 

focuses on the planned and systematic set of activities which ensure that software 

processes and products conform to requirements, standards, and procedures. The SSD’s 

energy is focused on two specific challenges: achieving trustworthiness (ensuring that no 

exploitable vulnerabilities exist, of either malicious or unintentional origin) and achieving 

predictable execution (providing justifiable confidence that software, when executed, 

functions as intended). The SSD efforts in this area in the past have concerned sourcecode security analysis. More recently, the SSD has developed support for analysis of 

Web application vulnerabilities and binary-code security analysis. 

 In general, the SSD’s efforts have yielded good results, better industry and 

academic measurement, and net improvement in static analyzers. The division’s solid 

technical contributions are made within a principled framework by a skilled and 

dedicated team. The engagement of the SSD with the static analysis community has 

yielded improvements and, it is hoped, will continue to yield improvements in the quality 

of static analysis tools and overall assurance available for large software systems 

developed using those commercial tools. 

 As software services are increasingly delivered through the Internet, Webapplication vulnerability is a problem of increasing importance. It is good that the SSD 

has developed tools and metrics for Web-application vulnerability scanning. The SSD 

should redouble efforts in this area. Further, it should expand features of the Webapplication scanning benchmarks to include cross-site scripting and other modern attack 

types. Also, binary-code analysis is an important area, and the SSD should continue to 

grow its practice in this area. 

Computer Forensics

 The SSD’s participation in this activity is focused on computer forensics 

investigations with an emphasis on providing the discipline and rigor essential to support 

the needs of law enforcement. A number of studies have verified the increasing 

importance of computer forensics as everyday reliance on computing devices and their 
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html

17 

infiltration for unlawful purposes increase exponentially. In addition, the current 

administration has chosen to make heavy initial investments to strengthen national 

information technology assets vital to the public interest. 

 The SSD has made modest but impactful, insightful, and highly leveraged 

investments in this area. Unfortunately, the pace of change in computing devices and 

associated infrastructure, combined with the need for computer forensics support for law 

enforcement in the face of the increasing threat, is entirely unmatched from an SSD 

investment perspective. Increased investment is required simply to match the demands of 

the environment. The following immediate areas of investment concern should be 

addressed: 

1. The explosion in the use of mobile devices, combined with the rate of 

innovation and product release, cannot be addressed with current resource 

assignments. 

2. The National Software Reference Library must expand its scope to encompass 

virtual distribution mechanisms and the nearly constant stream of product and 

service upgrades that are common service elements in all information 

technology offerings. 

Grid Computing Systems

 There are various definitions for “applied distributed computing,” including 

distributed high-performance computing resources (traditional “grids”), horizontally 

scaled transaction systems, clouds supporting distributed storage services, and high-end, 

near-real-time business analytics. The SSD has defined its grid scope as large-scale 

organizations of distributed computing resources providing on-demand services to 

computing and/or data-intensive applications. The technical work being performed is 

sound and based on a robust analytical framework largely ignored by current U.S. 

commercial grid implementations. This reveals inherent weakness in the SSD’s 

engagement strategy with U.S. business activities and the resulting lack of application 

and adoption of the SSD’s work. Engagement with U.S. business and government 

entities currently employing, on a global scale, grids and clouds, is essential. 

Engagement with such formulative organizations as the Open Cloud Consortium might 

be useful. 

Computational Biology

 

The Computational Biology project is a component of a broader, key SSD 

program (Information Discovery, Use, and Sharing). The initial goals of the project are 

modest and narrow in scope—computational biology is a very large field. The project has 

so far focused on cellular and subcellular image processing and analysis. (The 

cellular/subcellular and the organism/population/community levels are a focus given in 

the FY 2009 Administration Research and Development Budget Priorities 
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html

18 

Memorandum.2) The SSD research staff collaborates with other SSD divisions and, 

appropriately, with the NIST Chemical Science and Technology Laboratory. The staff 

also has collaborations with university and other research groups outside NIST. Next 

steps for the project were not explicitly addressed, and it is not clear how or whether the 

project intends to grow beyond its current focus. 

Health Information Technology 

 The Health Information Technology effort is both a set of projects and a key 

program within the SSD. The goal of the program is extremely ambitious: to ensure that 

the technical infrastructure of the U.S. health information technology network is correct, 

complete, and testable. The projects presented to the panel (Medical Devices, Clinical 

Document Validation, Semantic Interoperability, and Testing Infrastructure) were uneven 

in their approach and potential impact. Overall, there does not seem to be a clear and 

cohesive roadmap for the activities subsumed under this effort. 

In 2007 the panel recommended recruiting strong health informatics leadership 

that would be commensurate with the broad and complex agenda of this program. It is 

even more pressing that such leadership be put in place now, given the current emphasis 

on health information technology at the presidential level. Billions of dollars have been 

set aside for health information technology, with explicit expectations for results. (NIST 

will receive $20 million for health information technology.) If the panel’s 

recommendation was not implemented because the SSD was not able to attract a 

scientific leader of the highest caliber to take on the leadership of this very important 

activity, the SSD should consider a variety of other hiring mechanisms. For example, 

using an Intergovernmental Personnel Act or Special Expert mechanism would allow the 

SSD to hire someone at a competitive salary for a short period of time (1 to 3 years). 

Such an individual would, working with the SSD leadership, be able to provide a muchneeded strategic vision and implementation plan for the program. 

COMPUTER SECURITY DIVISION 

Progress at the Computer Security Division since the panel’s 2007 review has 

been impressive. The CSD now exhibits a stronger research program, a stronger and 

better-established division management team, and more focus on programs of national 

importance. The group is conducting state-of-the-art research and technology 

development in an area that is of growing importance to U.S. leadership in commerce. 

The CSD is required by Congress or the Executive Branch to respond to a number 

of program requirements. These include the 60-day study of Cyber Security, the Federal 

 

2 “Memorandum for the Heads of Executive Departments and Agencies,” from John H. 

Marburger III, Director, Office of Science and Technology Policy, and Stephen S. McMillin, Acting 

Director, Office of Management and Budget: Subject: FY 2009 Administration Research and 

Development Budget Priorities, p. 6, August 14, 2007. Available at 

http://www.ostp.gov/galleries/Budget09/FY2009FINALOMB-OSTPRDPriorityMemo.pdf. Accessed 

August 24, 2009. 
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html

19 

Information Security Management Act of 2002 (FISMA; Public Law 107-347), security 

for electronic voting, and the federal identification card initiative (Homeland Security 

Presidential Directive 12, or HSPD-12). In too many cases, these programs come to 

NIST as unfunded mandates that can distract staff and management from planned 

research. The CSD is also required to sustain established standards and guidelines. Both 

of these factors make it important that the division’s full-time staffing continue to grow, 

so that it will be possible for research to continue at an appropriate level. That said, the 

division’s research program is stronger than it was 2 years ago—a result attributable to 

increased funding and stronger leadership. 

 The CSD has initiated a research project aimed at measuring security. This is a 

very difficult research challenge but one that is worth tackling, and the project leader is 

bringing new ideas to the task. 

 Several staff members have been hired in the area of cryptography research, and 

the CSD plans to add more staff. Cryptography is an extremely important area for the 

division, and it is vital that there be a critical mass of expert cryptographers. Building the 

skills of newly hired staff takes a significant amount of time, so hiring now is a 

worthwhile investment for the future. The division could gain a great deal of leverage 

through academic partnerships; the current small core of cryptographers needs 

augmentation in order to create enough of a gravitational field to attract visiting scholars 

and more postdoctoral fellows. 

The CSD is attending to the task of making the guidance for federal agencies 

more actionable and effective. The initiation of FISMA Quick Start Guides and the 

intent to collaborate or integrate efforts with the national security community and the 

Consensus Audit Guidelines are impressive efforts. The division is also taking a fresh 

look at the effectiveness of its cryptography module accreditation program and of 

FISMA; NIST needs this kind of leadership to balance its scientific perspective with 

federal requirements. 

The CSD should increase its efforts in the following areas and should describe its 

progress in these areas during the next panel review (in 2011): 

 Within the new ITL program structure, some activities that are very relevant 

to security are not conducted in the CSD or in the security program. For 

example, software assurance and software tool evaluation—critical to building 

software that is resistant to attack—are encompassed in the Trustworthy 

Information Systems Program, and the presentation of CSD activities to the 

panel did not cover these important topics. Although the separation of work 

in secure network protocols from the CSD is an artifact of the history of the 

ITL, the program structure does not seem to be leading to the expected 

collaboration between network and security experts. More cross-divisional 

interaction would strengthen the admirable work of both divisions. 

 Because there is no closed-form nontrivial solution to the problem of building 

a secure system, effective security depends at a fundamental level on the 

ability to bring the perspective of an attacker to bear. The need for an attacker 

perspective is important in several areas, including the definition of security 

configuration standards such as the Federal Desktop Core Configurations and 
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html

20 

the development of management guidance such as SP800-53 (the FISMA 

security controls guideline). The division is beginning to reflect more of an 

attacker perspective in its work, but it should enhance its focus and investment 

in this area. 

 “Cloud computing” is one of the possible futures of commercial computer 

services, and the division is taking an early look at the security issues of this 

developing technology. 

 In the area of electronic medical records, the CSD could play an important 

role in developing a framework for security and privacy. This is a 

complicated and crucial issue facing health care providers, and current 

initiatives need to face it squarely as they assess technologies and develop 

standards. 

ADVANCED NETWORK TECHNOLOGIES DIVISION 

The Advanced Network Technologies Division is generally healthy and stable, 

and the staff is involved and enthusiastic about its work. The division activities are 

generally aligned with the mission of the laboratory. The division members participate 

broadly in the crosscutting programs, which speaks to the desirability of their skills and 

their willingness to collaborate outside their immediate circles. Also noteworthy is the 

Guest Researchers program. The number of guest researchers, the diversity of their home 

organizations, and their contributions to the division’s work programs are impressive. 

As discussed in this section, there are substantial variations in the projects 

presented to the panel in detail. The projects on Secure Naming (Domain Name Systems 

Security), Secure Routing (Border Gateway Protocol), and Internet Protocol Version 6 

(IPv6) Security and Deployment, all having a common focus on Internet infrastructure 

protection, successfully blend several desirable characteristics. There is understanding of 

the problems at a fundamental level; tools, tests, and testbeds are adequately covered; 

there is engagement at the policy level, with the group responding to requests from 

government and industry for its expertise and also proactively advancing its agenda in 

various forums; and there is engagement in deployments at the national level. The picture 

that emerges is of research projects in areas of national importance that are having 

significant impact. 

The project on Measurement Science for Complex Information Systems leverages 

methods from the mathematical sciences, especially statistics, to achieve state-space 

reduction by several orders of magnitude. Quite rightly these methods have been 

designed to tackle a specific, yet fairly general, application space, which is Transmission 

Control Protocol (TCP)-based congestion control in the Internet. The work is deep and 

broad. Important challenges lie ahead. First, as the title of the project implies, the goal is 

to apply the methodology to complex systems generally, and before that can happen there 

needs to be a distillation of the existing methods. The group should justify the original 

goal by applying the methodology to at least one other major complex system. The 

project output should be offered for peer review at top conferences and journals to a 

greater extent. This is worth doing for several reasons, including raising the group’s 

visibility. 
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html

21 

The Seamless Mobility project focuses on handover modeling and performance 

analysis and is closely coupled to standards activities. More specifically, this work is for 

WiMax (“Wireless Worldwide Interoperability for Microwave Access,” IEEE 802.16 

standard) and related standards on predicting the “link going down” condition and 

minimizing service outage. Although the effort shows skill, the focus on mathematical 

models without the benefit of feedback of measurements from experiments and testbeds 

makes the project rather narrow and promises to yield incremental gains. The extent to 

which elements of the project have been done before remains unclear. 

The Body Area Networks (BAN) project focuses on the emerging area of radiofrequency (RF) propagation from medical implants. The project has contributed a virtual 

reality model for RF propagation in the body and surroundings. The work is fresh, 

focused, and well connected to leading players in the world, including standards bodies 

and vendors. The promise of value from this early effort is large. It is a matter of some 

regret that the participants in the BAN channel model effort, apart from the work at 

NIST, have been from outside the United States. The ITL group should work to correct 

that situation. 

The Public Safety Communications project, while having several drivers, is 

focused on video quality and specifically on error concealment. The work is of high 

quality, but its narrow focus is likely to lead to only incremental gains. Also, there are 

exciting research developments in the area of compressive sampling that have much to 

offer here. The group should broaden the focus and increase the emphasis on new 

research methods. 

The area of Internet measurements and data analysis appears to offer great 

potential for the special and extraordinary skills that exist in the ANTD. While the 

Cooperative Association for Internet Data Analysis is already an important presence, 

there are large, rapidly evolving spaces that are uncharted. The ANTD will benefit from 

exposure here, and the Internet will likewise benefit from the ANTD’s contributions and 

insights. 

INFORMATION ACCESS DIVISION 

If a single theme can be said to characterize the work of the Information Access 

Division, it is that of developing standards, metrics, and evaluations for automated 

understanding of digital media, such as text retrieval, automated recognition of humans 

from images and voice, language recognition, and human activity classification from 

video. Organizationally, the IAD is divided into five groups with significant overlap of 

activities: Multimodal Information; Retrieval; Image; Visualization and Usability; and 

Digital Media. A methodology common to the majority of these groups is that of the 

“challenge problems,” most of which are externally funded. 

The IAD is the second largest division within the ITL, with a total staff of 72. 

But of all ITL divisions, the IAD has the most externally funded work, with 58 percent of 

its total budget ($11.4 million of $19.5 million—estimates as of August 13, 2009) in FY 

2009 coming from outside-agency sources. This is a significantly greater percentage of 

OA work than the FY 2009 ITL average across divisions—28 percent ($25 million of $89 

million)—and represents nearly half of all OA funds received by the ITL. 
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html

22 

Technical Merit of the Programs 

The IAD programs, research, and standards development are generally excellent. 

Specifically, the impressive challenge-problem concept is an excellent way for the 

government to leverage academic and industrial development of solutions to problems of 

long-standing government interest, such as text retrieval, human recognition, and 

language identification from both text and speech. These programs are generally funded 

by other government agencies, with the IAD serving as the scientific and organizational 

resource and the honest broker assuring the participants of the neutrality and integrity of 

the tests. 

Under the challenge-problem concept, the IAD, with collaboration from the 

funding organizations, establishes a technology challenge through a series of workshops, 

free to all comers. Examples of such challenge problems include the Text Retrieval 

Conference and the Speaker and Language Recognition Evaluations. Although details of 

the protocols vary by sponsor, the tests center on problems of interest to the government, 

as precisely defined through the development of a task and test data set. The IAD 

establishes the data requirements to meet the sponsors’ technology development needs, 

then either gathers the data itself or works with a third party (such as the Linguistics Data 

Consortium of the University of Pennsylvania in the case of speaker and language 

identification), to create a data set characterizing the problem of interest. The data are 

partitioned, with one set sent to volunteering participants for development and the other 

set retained for the test. It is common, when possible, to release all of the data to the 

research community after the test. 

The challenge-problem concept has several appealing aspects: academics and 

industry are encouraged to work on problems tailored directly to government needs; 

government can motivate technology improvements without directly funding research; 

government can use the outcomes of the evaluations to establish the feasibility of 

deployments and to direct future research funding; academic and industrial groups can 

assess their own performance against the state of the art on common data sets, using the 

resulting information to commercial advantage; and developmental data sets are made 

available to the community for future research. 

The standards organized and fostered by the IAD, such as the American National 

Standards Institute (ANSI)/NIST-ITL-1/2007 (“Data Format for the Interchange of 

Fingerprint, Facial, and Other Biometric Information”), serve as the de facto standard for 

biometric data interchange both within the U.S. government (such as between local law 

enforcement and the Federal Bureau of Investigation) and within international 

organizations (such as between European governments and the International Criminal 

Police Organization, Interpol). The international acceptance of this standard is an 

indication of the world’s positive opinion of the technical merit of the IAD work. 

Contributions to the development of the ANSI/NIST and other standards (such as the 

International Organization for Standardization/International Electrotechnical Commission 

Joint Technical Committee Subcommittee 37: Biometrics, ISO/IEC JTC1 SC37 biometric 

series) and performance of the research necessary to support implementation of the 

standards have made an invaluable contribution to national and international law 

enforcement and security activities. 
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html

23 

Human Resources and Facilities 

The IAD’s human resources and facilities are adequate, but there is need for a 

usability laboratory to support the increasing data collection requirements of the 

Visualization and Usability group, particularly in the area of voting machine assessment 

and biometrics. It is encouraging that since the panel’s previous visit in 2007, the IAD 

has acquired social scientists in linguistics, psychology, and human factors as direct 

employees or subcontractors to assist with the usability work. The ITL-recognized job 

categories must keep pace with both technology and tasking changes within the IAD, and 

the ITL must institutionally recognize the continued need for social and health scientists 

to support, for example, the visualization and usability and the speech efforts. 

The heavy reliance on OA funding has created pressures within the IAD; it has 

allowed growth and new opportunities for the division, but it has predictably forced 

program managers into marketeering to support continued funding. Although NIST 

internal funding through the congressional budgeting process is not completely stable, 

there are problems brought on within the IAD by even less stable external funding. The 

perceived instability in the external funding has created reservations within the IAD 

management with respect to acquiring the human resources necessary to take on 

additional, high-value projects, such as usability in medical informatics and 

computer/data security. The IAD program managers expressed uncertainty over what 

processes would be followed by ITL management in the event of a retraction in OA 

support. ITL leadership must make clear to the IAD what safety-net processes would be 

in place in the event of an unexpected decline in OA funding. ITL management should 

also make available stable and dependable internal bid and proposal monies from the 

scientific and technical research services (STRS) category of the budget in order to 

support the IAD marketing burden implicit in the acceptance (in FY 2009) of $11.4 

million in OA funding. 

 

Achieving Objectives and Desired Impact 

The existing IAD programs have a strong international impact and are 

contributing to the ITL mission and objectives. The IAD has a strong culture of both 

publication and participation in academic and industrial conferences, workshops, and 

forums. This increases the impact of the division’s work through the process of 

dissemination of results to stakeholders. However, when normalized by the number of 

PhD researchers on the IAD staff, the number of conference and journal papers falls short 

of what would be expected at a major university. Consequently, the IAD should place 

additional emphasis on promoting first-rate publications. 

There is a gap in the program of work that spreads from the IAD to other 

divisions across several programs—that gap being a systematic and research-grounded 

analysis of privacy. The National Research Council has in several past studies noted the 

importance of the consideration of privacy as a fundamental design factor in information 

technology development. Privacy cannot be forced on technologies purely at the level of 
Copyright © National Academy of Sciences. All rights reserved.

An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2009

http://www.nap.edu/catalog/12768.html

24 

applications. The IAD work, particularly that of human recognition and personal 

information retrieval and the supporting data collection efforts, carries significant 

potential privacy impact that needs to be methodologically considered. The IAD must 

develop expertise in privacy theory and practice.